[Remote] Domain Expert, AI Evaluation Task Writer: Healthcare (Contract, Remote)

Note: The job is a remote job and is open to candidates in USA. Patronus AI is hiring a healthcare domain expert to create realistic tasks for evaluating and training AI models. The role involves building fictional healthcare records, solving tasks, writing grading criteria, testing tasks against current models, and reviewing work for accuracy and clarity.

Responsibilities

• Build realistic, fully fictional patient and operational records from a provided scenario: clinical notes and discharge summaries, lab and imaging reports, medication lists and care plans, coding worksheets and provider queries, prior authorization requests and medical necessity criteria, utilization review notes, claims, remittances and denial letters, payer policies and eligibility responses
• Write clear task requests that state what the deliverable is and how uncertainty and justified assumptions should be handled
• Solve each task end to end and document a reference solution with the clinical or operational reasoning behind every step
• Write explicit scoring criteria that let two independent graders reach the same verdict, including acceptable alternative answers and anything that would make a response unsafe
• Run tasks against current models, find where and why they fail, and adjust difficulty
• Check your own and peers' tasks for clinical accuracy, ambiguity, answers exposed in the materials and shortcuts that skip the intended reasoning
• Incorporate reviewer feedback within agreed turnaround times and flag guideline gaps or edge cases you run into

Skills

• 3+ years of hands-on experience in one or more of the four areas above, as a physician (MD/DO), nurse practitioner, physician assistant, registered nurse or pharmacist, or in clinical documentation, medical coding, utilization review, care management, revenue cycle or payer operations. You need to have done the work yourself, not only supervised it
• Current or past licensure or certification in your field (for example MD/DO, NP, PA, RN, PharmD, CPC, CCS, CDIP, CCDS, CCM or CRCR)
• Working knowledge of the guidelines, criteria and documentation standards that govern your area (for example clinical practice guidelines, ICD-10-CM/PCS and CPT coding rules, InterQual or MCG criteria, CMS and payer policies)
• Careful reading, and the habit of citing the chart entry, code set or policy behind every conclusion
• Clear, precise writing in professional English
• Ability to work independently and asynchronously against deadlines
• Board certification or credentials beyond the minimum for your role (for example a clinical specialty board, CPHQ, CRCR or CHFP)
• Experience across settings (inpatient, outpatient, payer, pharmacy, post-acute)
• Experience with the systems used in your area, such as Epic or Oracle Health, Solventum (formerly 3M) 360 Encompass, InterQual or MCG, Availity or a payer portal, and a claims clearinghouse
• Teaching, precepting, board-question writing or clinical case development, or prior experience writing or reviewing AI evaluation tasks
• Familiarity with how AI models fail (fabricated citations, outdated guidelines, missed contraindications, wrong codes, overconfident answers)

Benefits

• Fully remote, asynchronous
• A potential bonus for each accepted task
• The trial task is paid

Company Overview

Company H1B Sponsorship