Site Reliability Engineer

Site Reliability Engineer (Remote) — Mid-Level, Senior, and Engineering Manager SRE · Contract-to-Hire · Remote (U.S., Eastern or Central time) · U.S. Citizen or Permanent Resident ABOUT ARSENAL AI Arsenal AI is a Pittsburgh-based venture studio and AI consulting firm. We build and invest in AI-native companies, and we embed engineering teams directly with our clients to ship production software. ABOUT THE CLIENT Arsenal AI is hiring on behalf of a fast-growing healthcare technology company that is transforming the way health systems coordinate and deliver care through secure, modern technology. Its teams use current cloud-based tools and services to design and build complex applications. Every engineer is a valued contributor who brings their own experience and perspective to the problems its users face. SUMMARY We are hiring a fully remote Site Reliability Engineering (SRE) team for our client: an engineering manager, senior engineers, and mid-level engineers. This new team works across engineering, product operations, and security to make production systems measurable, observable, and recoverable. Its scope covers monitoring and alerting, incident response, availability reporting, and resilience testing. You will help build the team's practices from the start. That means setting the first availability targets and defining how they're measured, making sure the right team gets each alert, and turning incident follow-ups into real fixes. You will see the results in less on-call noise, faster detection, shorter recovery times, and availability numbers that customers and auditors can trust. This role overlaps with DevOps and Platform Engineering. You will work in AWS and Kubernetes, write infrastructure as code, and build automation. The difference is that you own production reliability and the incident response practice first. Strong software development skills are a must, because much of the job is replacing manual, recurring operational work with code. This is a contract-to-hire role. You join as a W2 employee of Arsenal AI working full-time on the client's team, with a path to convert to a full-time role with the client based on performance and business need. THE ROLES SRE Engineering Manager — Leads the new SRE team: hiring and onboarding, the on-call program, the incident response practice, and planning the team's work. Partners with application, product operations, and security leads on who owns alerting, triage, and escalation, and stays hands-on in design reviews and major incidents. Senior SRE — Owns major parts of the reliability practice: SLOs, alerting, incident command, disaster recovery testing, and deployment safety. Provides technical leadership through design reviews and mentoring. Mid-level SRE — Builds and runs monitoring, dashboards, runbooks, and automation with guidance from senior engineers. Joins the on-call rotation and grows into incident command. WHAT YOU'LL DO Own the reliability of monitoring and alerting: make sure alerts fire, reach the right team, and are themselves monitored. Define and maintain SLIs, SLOs, and availability targets, and report uptime and incident metrics (such as MTTD, MTTR, paging volume, and postmortem action-item closure). Build and run the incident response practice: severity definitions, the incident commander role, tabletop exercises, postmortems, and tracking action items to closure. Work with Security on decisions about external and regulatory reporting. Improve alert quality as an ongoing responsibility: broaden error coverage, tune and de-duplicate noisy alerts, and make sure every paging alert has a runbook. Build dashboards and detection for system health and saturation across databases, workflow engines, message queues, and logging infrastructure. Plan, run, time, and document backup/restore, failover, and maintenance-window procedures to meet audited recovery objectives. Improve deployment safety with post-deploy health checks, gates after failed deployments, and automated rollback. Write code that reduces operational work, such as internal tools for secret rotation, platform upgrades, and recurring compliance tasks, in Go, TypeScript, and Pulumi. Turn recurring operational obligations into scheduled, visible practices that keep running during a busy week. Work with application teams, product operations, and Security to agree who owns alerting, triage, and escalation, including the security incident triage tier. Write and maintain runbooks, incident response plans, and disaster recovery documentation. Participate in code reviews, design sessions, all sprint ceremonies (standup, sprint review, retrospective, grooming), and an on-call rotation. Senior and manager roles: provide technical leadership through design reviews, written trade-off analyses, and constructive peer reviews; serve as an escalation point during major incidents; and coach other engineers, including onboarding new people into on-call. REQUIRED QUALIFICATIONS Site Reliability Engineering, production operations, DevOps, or Platform Engineering experience, including on-call rotations (AWS preferred): mid-level 2–5 years; senior 5+ years; engineering manager 5+ years, including time leading or mentoring engineers Bachelor's degree in Computer Science, Software Engineering, Computer Engineering, or a related field (or equivalent practical experience) Strong software development skills, with hands-on experience in Go and/or TypeScript (the client's primary languages), enough to build and maintain operational tooling and automation Experience with scripting languages such as Bash and/or Python Hands-on experience with observability and alerting stacks such as Grafana/LGTM (Mimir, Loki, Tempo), Prometheus/PromQL, or Splunk, including designing alerts people can act on and reducing alert noise Experience with incident management, such as serving as or supporting an incident commander, applying a severity model, and running blameless postmortems whose action items are tracked to completion Experience defining SLIs and SLOs and reporting availability to internal and external stakeholders Hands-on experience with core AWS services (ECS/Fargate, RDS, EBS, CloudWatch, IAM, Lambda, SSM/Secrets Manager) Experience operating Kubernetes workloads, ideally AWS EKS, including stateful services (StatefulSets, persistent volumes) Experience with Infrastructure as Code, ideally Pulumi (Terraform, CDK, or CloudFormation also acceptable) Linux administration and troubleshooting experience Experience with version control, especially Git and GitHub Ability to work fully remote within the continental USA in an Eastern or Central time zone (excluding New York), with a high-speed internet connection Strong verbal and written communication skills. You will write runbooks and incident documentation that hold up under pressure, and work with other teams to agree who owns what. DESIRED QUALIFICATIONS Experience with ServiceNow ITSM/CSM (major incident and outage records, assignment group rules) Experience planning and running backup/restore, disaster recovery, and failover tests against defined RTO/RPO targets Experience with synthetic monitoring, uptime monitoring, and public status pages Experience with anomaly detection or statistical alerting, and capacity/saturation monitoring Experience with deployment safety practices: automated rollback, post-deploy verification, and deployment gates (Jenkins experience a plus) Experience building an on-call program: rotations, shadowing, onboarding, and runbook coverage Experience automating secret rotation and other recurring operational procedures with least-privilege access Experience working in a FedRAMP and/or HIPAA-compliant environment, including contingency planning and incident reporting requirements AWS certifications (SysOps Administrator, DevOps Engineer, Solutions Architect, etc.) Experience working in an Agile development environment ELIGIBILITY REQUIREMENTS This engagement supports a U.S. Department of Veterans Affairs contract. These are federal contract requirements, not internal company policy. Confirm you can meet them before applying. U.S. citizenship or U.S. permanent residency A REAL ID-compliant driver's license and a valid passport, for federal identity proofing Successful completion of a background check and drug screening before your start date