[Remote] Director, AI Automation Engineering

Note: The job is a remote job and is open to candidates in USA. Impiricus is hiring a Director, AI Automation Engineering to own the quality and reliability layer for AI systems across its internal platform and customer-facing products. The role builds automated testing infrastructure, evaluation frameworks, and release gates, while partnering with AI and product teams and developing the automation function.

Responsibilities

• Own the infrastructure that continuously verifies whether autonomous agents are taking the actions we intend, before and after they reach production
• Stage, test, and lifecycle-manage bots and agent workflows before they reach production
• Establish baseline quality metrics and coverage standards for AI initiatives
• Build the infrastructure to run automated research chains and agent evals at scale
• Create automated evaluation frameworks for LLM outputs: accuracy, safety, bias, and regression
• Implement RAG evaluation, prompt regression testing, and golden-set benchmarking
• Define and enforce quality thresholds that AI Engineers and builders must meet before ship
• Partner with AI Researchers and Engineers on eval design for new model and agent patterns
• Build the release-gating infrastructure that governs when an agent is trusted to act in production, including staged rollouts and rollback authority
• Develop AI automation solutions to ensure that agentic coding delivers highly functioning software while maintaining quality across the product portfolio
• Integrate testing into GitHub Actions (or equivalent) for AI model and agent releases
• Maintain existing test suites and keep them current as models, prompts, and tools change
• Work with AI Solutions Architects on eval criteria for workflow automations they build in the field
• Partner with Principal AI Product on launch-readiness and QA for AI projects across the org
• Support product teams (Ascend, DocUpdate, Platform) on AI feature validation without owning their roadmaps
• Document standards, runbooks, and patterns so non-engineers building in Sage can self-serve basic testing

Skills

• 5+ years building, evaluating, or governing production AI/ML systems, with direct experience in agentic AI, LLM-based automation, or autonomous system safety
• Direct experience evaluating LLM outputs: RAG eval, prompt regression, safety and accuracy benchmarking
• Experience building CI/CD pipelines with integrated test gates (GitHub Actions or similar)
• Comfortable working across AI platform infrastructure and product-facing AI features
• Operates independently as a senior IC and communicates quality standards to engineers and non-engineers
• Proficiency in Python and standard testing frameworks (pytest or equivalent)
• Experience with agent testing, bot lifecycle management, or AI agent orchestration platforms is a strong plus
• Familiarity with LLM observability and eval tooling (LangSmith, Braintrust, custom harnesses, etc.) is a strong plus
• Background in healthcare, pharma, or regulated environments where quality gates matter is a strong plus
• Prior experience standing up a QA or automation function from scratch, with appetite to grow into leading a team

Benefits

• Medical, dental, and vision coverage for you and your dependents + on-demand healthcare concierge
• Pre-tax savings options for healthcare and dependent care, with monthly employer contributions to HSA (if enrolled in a high-deductible plan)
• 100% paid short- and long-term disability, plus life and AD&D insurance
• Flexible vacation policy
• Paid parental leave to spend time with your newborn, adopted, or foster child (available after 6 months)
• If you're close to an office, we encourage spending some time in-person to collaborate and connect. If not, remote is always an option.
• 401(k): Save for your future with tax advantages (and company match!)

Company Overview