[Remote] Domain Expert, AI Evaluation Task Writer: Legal (Contract, Remote)

Note: The job is a remote job and is open to candidates in USA. Patronus AI is hiring a legal domain expert to create realistic legal tasks that challenge AI models. The role involves preparing scenario-based documents, solving tasks, defining grading criteria, and reviewing tasks for accuracy and difficulty.

Responsibilities

• Build realistic evidence files from a provided scenario: services agreements and amendments, email threads, pleadings and discovery requests, privilege logs, docket entries, closing checklists and disclosure schedules, signature packages, cap tables, contract playbooks and redlines, request intake records, data maps and retention schedules
• Write clear task requests that state what the deliverable is and how uncertainty and justified assumptions should be handled
• Solve each task end to end and document a reference solution with the reasoning behind every step
• Write explicit scoring criteria that let two independent graders reach the same verdict, including acceptable alternative answers and the limits of what the solver is authorized to decide
• Run tasks against current models, find where and why they fail, and adjust difficulty
• Check your own and peers' tasks for legal accuracy, ambiguity, answers exposed in the materials and shortcuts that skip the intended reasoning
• Incorporate reviewer feedback within agreed turnaround times and flag guideline gaps or edge cases you run into

Skills

• 3+ years of hands-on experience in one or more of the four areas above, as a paralegal, legal assistant, contracts manager or specialist, privacy or data protection analyst, or attorney. You need to have done the work yourself, not only supervised it
• Working knowledge of the rules, procedures and documentation standards that govern your area (for example federal and local court rules and discovery practice, closing mechanics, contract playbooks and CLM workflows, GDPR and CCPA request handling)
• Careful reading, and the habit of citing the clause, message or docket entry behind every conclusion
• Clear, precise writing in professional English
• Ability to work independently and asynchronously against deadlines
• Paralegal certification (for example NALA CP or NFPA RP), a law degree, CIPP/US or CIPP/E, or a contract management credential
• Experience with the systems used in your area, such as Relativity, Everlaw or DISCO, a virtual data room, Ironclad, Icertis, Agiloft or Conga, or OneTrust, TrustArc or Transcend
• Law firm, in-house and legal operations experience across more than one setting
• Prior experience writing or reviewing AI evaluation tasks, exam questions, case studies or training material
• Familiarity with how AI models fail (fabricated citations, misread clauses, miscounted deadlines, overconfident conclusions)

Benefits

• A potential bonus for each accepted task
• The trial task is paid
• Fully remote, asynchronous

Company Overview

Company H1B Sponsorship