[Remote] Staff Applied AI Engineer, Product & Agent Performance

Note: The job is a remote job and is open to candidates in USA. Arcadia is a healthcare technology company that builds data and analytics platforms for providers, payers, and life sciences organizations. The Staff Applied AI Engineer owns product-layer decisions and performance for agentic capabilities, including agent behavior, retrieval, context, memory, evaluation, and escalation. The role partners with Product and Engineering to improve the accuracy, safety, transparency, and scalability of AI in healthcare workflows.

Responsibilities

• Design and iterate on agent behavior across real, live workflows, including long-horizon, multi-turn agentic tasks
• Design retrieval and context architecture so the right source data reaches a model in the right structure and agents remain grounded in real data rather than filling gaps with assumptions
• Design memory and state handling across multi-turn and multi-agent flows, determining what is carried forward, summarized, or dropped and why
• Create context and prompt templates that combine few-shot examples, structured formatting, and reasoning scaffolding for consistent agent behavior
• Improve performance through prompting, tool-use strategy, and context construction, validated through direct experimentation rather than guesswork
• Build and run evaluations against real production conditions to measure performance, regressions, failure modes, and edge cases
• Author evaluation rubrics, quality heuristics, and thresholds that weight failures by severity and cost, not just frequency, and monitor those measures against production behavior
• Design and validate escalation paths that route agents to human review based on confidence and uncertainty while preserving safety and consistency under adversarial and edge-case conditions
• Design for cost-aware performance alongside latency, reliability, and accuracy through efficient context construction and tool-call economy
• Evaluate and sign off on model changes by baselining current behavior, running comparative evaluations, and making the go/no-go call before a change reaches a customer
• Maintain product-level AI documentation, including model cards, intended use, limitations, and known failure modes, so customer-facing teams work from actual agent behavior
• Partner closely with Product and product managers to ensure agents are not just capable, but steerable, trustworthy, and ready to scale

Skills

• Bachelors degree in Engineering or related field and/or masters of Data Science or related field
• 4+ years of production software engineering experience, including 3+ years of hands-on ownership of ML, LLM, or agentic systems in production, with direct experience in healthcare, finance, or another regulated industry
• Demonstrated ability to diagnose why an agent failed, correctly attribute the fix to instruction, retrieval, context, or memory design, and weigh failures by severity and cost rather than frequency alone
• Hands-on experience with RAG architecture, production-grounded evaluation frameworks, and fallback or human-in-the-loop logic for automated systems
• Working familiarity with AWS AI/ML services, including Bedrock and SageMaker, sufficient to build and evaluate effectively in Arcadia's environment
• Evidence-led judgment and the credibility to push back on launch decisions, paired with a builder's instinct to run the experiment and move from a production failure to a fix
• Experience applying AI to healthcare data or workflows where safety, transparency, and calibrated uncertainty directly affect care teams or patients
• Experience with long-horizon, multi-turn or multi-agent workflows and product-level AI documentation such as model cards

Benefits

• Flexible, remote-friendly culture
• Employee-driven programs and initiatives for personal and professional development

Company Overview

Company H1B Sponsorship