[Remote] AI Quality Engineer

Note The job is a remote job and is open to candidates in USA. Dealstitch AI is a technology company focused on private-market applications and agent systems. The AI Quality Engineer will own automated evaluation pipelines for AI agents, assess performance across reasoning, tools, and retrieval, maintain production datasets and observability, and continuously improve the quality standards for shipped systems.

Responsibilities

Own the evaluation pipelines that gate every change we ship, keeping them automated and running in CI/CD rather than leaning on manual spot-checks Keep agents scored at the step level, not just the final answer, across tool selection, planning and reasoning chains, and retrieval quality, so subtle regressions surface before clients do Extend the eval toolkit as new failure modes appear LLM-as-a-judge, RAG faithfulness and relevance, hallucination detection, prompt and agent regression suites, and red-teaming for safety Maintain the golden datasets that mirror real production inputs and the tracing and observability that watch our agents once they are live Own the quality bar for what ships and raise it over time with the rest of the engineering team Skills Strong software engineering with Python, and a feel for testing non-deterministic systems, where classic assertion-based tests are not enough Experience evaluating LLM and agent systems agent graphs (e.g. LangGraph), eval frameworks (DeepEval, RAGAS, G-Eval, LangSmith, or similar), and LLM-as-a-judge methods Fluency with RAG and retrieval quality, plus observability and tracing for agents in production A bias for measurement you define what "good" means, then build the harness that proves it Bonus data-pipeline testing experience, or exposure to private markets, financial services, or enterprise data Benefits Direct access to decision-makers See the impact of your work in weeks, not quarters Company Overview Only 8% of private market investors have successfully deployed AI — and Gartner finds roughly half of all pilots fail before go-live. It was founded in 2023, and is headquartered in Denver, Colorado, USA, with a workforce of 2-10 employees. Its website is https//www.dealstitch.ai.