AI EVALUATION ENGINEER

Location: Remote Working Hours: Must overlap with Central European Time (CET) About the ProjectWe are building a Generative AI regulatory science co-pilot for regulatory submission teams. The system functions like a virtual Health Authority reviewer, extracting and assessing regulatory risks, mapping submission content to relevant regulatory guidance and precedent, verifying that key messages are supported by clinical data, and anticipating potential Health Authority questions. Because the system directly informs regulatory submission strategy, its outputs must be accurate, evidence-based, and traceable to reliable sources. We are seeking an experienced AI Evaluation Engineer to develop and execute evaluations for the system’s core reasoning engines, working closely with regulatory affairs subject-matter experts (SMEs) and the product engineering team.

Responsibilities

Build gold-standard datasets: Collaborate with SMEs to annotate regulatory and clinical documents, define expected outputs, develop annotation guidelines, manage adjudication, and measure inter-rater agreement. Evaluate information extraction: Measure precision, recall, and completeness of extracted regulatory risks, comparing successive engine versions against established baselines. Evaluate retrieval and ranking: Assess retrieval recall (Recall@K), ranking quality (MRR and nDCG), and question coverage when retrieving regulatory guidance, precedent, and previous Health Authority interactions. Validate grounding and traceability: Develop code-based checks to ensure findings are supported by valid evidence and citations. Identify fabricated guidance, fabricated precedent, and unsupported regulatory claims. Evaluate AI reasoning: Design and validate LLM-as-a-Judge rubrics for regulatory relevance, evidence sufficiency, risk assessment, and explanation quality, calibrated against SME judgment. Evaluate pipeline performance: Build evaluation datasets and baselines for individual stages of the multi-stage engine, including evidence structuring, question coverage, retrieval, ranking, mapping, and grounding. Coordinate end-to-end reviews with SMEs. Report findings and improve reliability: Translate failure analysis into actionable insights for the product team and package evaluations into regression test suites that run in CI/CD. Required Qualifications5+ years of hands-on experience in AI/ML evaluation, natural language processing (NLP), or applied data science, including recent delivery experience with LLM, RAG, or agent-based systems. Demonstrated experience building evaluation frameworks and gold-standard datasets for information extraction and retrieval/ranking systems. Candidates should be prepared to share examples of their work. Strong understanding of evaluation metrics and statistics, including precision, recall, F1, Recall@K, MRR, nDCG, inter-rater reliability (e.g., Cohen’s kappa or Krippendorff’s alpha), and confidence intervals for small evaluation sets. Experience designing and validating LLM-as-a-Judge systems against human expert ratings. Experience managing annotation projects with domain experts, including annotation guidelines, adjudication, and quality control. Strong Python programming skills and experience with evaluation frameworks such as Ragas, DeepEval, promptfoo, Inspect, or equivalent in-house tools. Experience working with LLM APIs, Git, and CI/CD workflows. Ability to work independently, operate with minimal onboarding, and deliver against fixed milestones. Professional fluency in written and spoken English. Preferred QualificationsExperience in regulatory affairs, regulatory intelligence, or regulatory submissions, including FDA/EMA guidance, Common Technical Document (CTD) structure (e.g., Modules 2.5 and 2.7), Health Authority questions, and labeling. Experience evaluating AI systems in regulated (GxP) environments. Familiarity with clinical trial outputs, including Clinical Study Reports (CSRs), tables, listings, and figures (TLFs), statistical outputs, and drug development. Experience evaluating document-comparison, claim-verification, or fact-checking systems. Publications or open-source contributions in AI evaluation, information retrieval, or biomedical NLP. How to ApplyInterested candidates should submit their resume and cover letter to [email protected].