Remote Position
[Remote] Principal Software Engineer, Distributed Systems
🌍 Remote · United States
💼 Full-time
Note: The job is a remote job and is open to candidates in USA. Sigma Automate is hiring a Principal Software Engineer focused on distributed systems and platform reliability. The role owns the architecture and engineering practices that ensure long-running and scheduled workflows execute reliably despite component failures, while improving observability, messaging, database, Kubernetes, and production reliability systems.
Responsibilities
- Design and improve the systems responsible for executing long-running and scheduled infrastructure workflows
- Durable workflow execution
- Idempotent operations
- Automatic retries and exponential backoff
- Worker leases and heartbeats
- Stuck-job detection and recovery
- Workflow timeouts and cancellation
- Failure isolation
- Dead-letter handling
- Checkpointing and workflow resumption
- Concurrency control
- Distributed locking and lease management
- Reconciliation loops
- Graceful degradation during dependency failures
- Evaluate and implement workflow technologies such as Temporal or equivalent systems where appropriate
- Own and improve Sigma's asynchronous execution architecture
- Ensure event-processing failures cannot silently cause workflows to stop progressing
- Help design application patterns that remain reliable under high concurrency
- Partner with application engineers to identify and eliminate patterns that create production contention or reliability risks
- Improve the reliability of Sigma deployments running in containerized environments
- Kubernetes workload architecture
- Pod lifecycle and failure recovery
- Resource limits and capacity planning
- Autoscaling
- Health checks
- Graceful shutdown
- Rolling deployments
- High availability
- Infrastructure-as-code
- Disaster recovery
- Backup and restore validation
- Design systems that tolerate node, pod, process, and dependency failures without losing customer work
- Make the state of Sigma understandable in real time
- Build and improve:
- Metrics
- Distributed tracing
- Structured logging
- Dashboards
- Alerting
- Workflow-level telemetry
- Kafka consumer monitoring
- Database performance monitoring
- Worker health monitoring
- Queue depth and backlog monitoring
- Help establish engineering practices expected of mature enterprise platforms
- Service-level objectives and indicators
- Error budgets
- Capacity planning
- Load testing
- Failure-mode testing
- Chaos testing
- Incident response
- Root-cause analysis
- Production readiness reviews
- Runbooks
- Automated recovery
- Disaster-recovery testing
- Move Sigma from detecting incidents to preventing and automatically recovering from them
Skills
- You have significant experience operating production distributed systems where reliability matters
- Kubernetes
- Kafka or similar event-streaming systems
- PostgreSQL
- Distributed systems
- Asynchronous worker architectures
- Workflow orchestration
- Temporal, Cadence, Step Functions, Conductor, or similar systems
- Python backend systems
- Cloud infrastructure
- Infrastructure-as-code
- Observability platforms
- High-availability architectures
- You understand concepts such as: • At-least-once delivery • Idempotency • Leases and fencing tokens • Distributed locks • Leader election • Retry semantics • Backpressure • Eventual consistency • Transaction boundaries • Failure domains • Reconciliation • Split-brain scenarios • Distributed tracing
- Most importantly, you have personally diagnosed and improved production systems experiencing real distributed-system failures
Company Overview