[Remote] Agentic AI, Senior Software Development Engineer

Note: The job is a remote job and is open to candidates in USA. Zillow is a real estate platform helping people navigate buying, selling, financing, and renting. The Agentic AI team is seeking a backend engineer to design and scale evaluation frameworks, tracing and observability systems, and platform infrastructure for reliable, trustworthy AI agents. The role partners with applied scientists, machine learning engineers, and product leaders to improve AI quality, reliability, and product impact.

Responsibilities

• Design, build, and scale evaluation frameworks for Zillow’s agentic AI experiences
• Build tracing, observability, and quality measurement systems for production AI agents
• Create platform infrastructure that enables domain teams across Zillow to build on top of the Evals platform without bottlenecks
• Partner with applied scientists and machine learning engineers to integrate new AI evaluation capabilities into production systems
• Help evolve how Zillow measures AI quality, reliability, trustworthiness, and product impact
• Stay current with emerging agentic AI paradigms, evaluation techniques, and LLM tooling, and translate them into practical platform innovation
• Support scaling, reliability, performance optimization, incident response, and cost management for the evaluation layer
• Apply first-principles thinking to ambiguous problems and iterate quickly on novel solutions

Skills

• 4+ years of backend engineering experience, with a track record of designing, shipping, and operating scalable production ML services
• Experience building platform infrastructure or developer-facing services consumed by multiple teams; you've thought carefully about APIs, user experience, reliability, and what it means to have internal customers
• Hands-on experience with ML Evals & Observability frameworks (Databricks MLflow, or evolving LLM frameworks preferred - LangSmith, Braintrust, Promptfoo, or similar)
• Experience evaluating third-party AI platforms and making principled build vs. buy decisions for platform infrastructure
• Fluency working across applied science, ML, product, design, and engineering; you translate between disciplines without losing precision
• Comfort with LLMs, agentic systems, and evaluation frameworks; familiarity with orchestration tools (LangChain, LangGraph, MCP, or similar)
• A habit of reaching for AI-assisted development tools in your own workflow; you don't just build for AI, you build with it
• Strong ownership and a growth mindset; you're energized by ambiguity, not slowed by it
• Advanced degree in Computer Science, Machine Learning, or a related field, or equivalent practical experience building products using frontier LLMs, multimodal models, or agent-based systems
• Experience designing and operating evaluation pipelines: offline batch evals, online scoring, cost/latency/signal tradeoffs, and closing the loop from user signals back to training data
• Evolving LLM frameworks preferred - LangSmith, Braintrust, Promptfoo, or similar

Benefits

• This position is also eligible for equity awards based on factors such as experience, performance and location.
• Through Cloud HQ, our strategic approach to distributed work, most employees have the flexibility to work from wherever they’re most productive.

Company Overview

Company H1B Sponsorship