Research Scientist, Human Data & Evaluation

Job Description:

• Translate ambiguous client questions into clear research questions, hypotheses, and study designs
• Develop research proposals and methodologies from scratch
• Identify new research directions and help define what Contra Labs should measure, study, and build
• Design human-data collection strategies, evaluation protocols, rubrics, annotation schemes, and experiments
• Develop prompt strategies for eliciting, evaluating, and comparing model behavior
• Determine sampling, evaluator selection, calibration, quality-control, and reliability methods
• Design methodologies across creative and multimodal domains including design, image, video, and agentic workflows
• Analyze qualitative and quantitative evaluation data
• Explain why patterns occur and their implications
• Synthesize findings into recommendations, research narratives, and hypotheses
• Produce client-ready reports, presentations, benchmarks, and research publications
• Independently own research engagements from initial scoping through final delivery
• Collaborate with project leads, creative experts, research engineers, and data/engineering teams
• Maintain research rigor and data quality on fast client timelines
• Support multiple concurrent projects at different stages
• Participate in client scoping and proposal development
• Collaborate with Strategic Project Leads on client delivery

Requirements:

• Based in San Francisco, California
• 3–5+ years of relevant research experience
• Master’s, PhD, or equivalent industry research depth in HCI, Human-Centered AI, ML, design research, or a related field
• Strong qualitative and quantitative research skills
• Experience in human evaluation, experimentation, data collection, UX/HCI, annotation, or model behavior research
• Ability to independently design and execute rigorous research
• Ability to decide what data to collect, why, and how to analyze it
• Ability to turn data and observations into insights, research directions, and actionable recommendations
• Comfortable analyzing data with Python and/or SQL
• Strong written communication and ability to explain methodological tradeoffs and research decisions
• Comfortable operating independently through ambiguity and balancing research rigor with startup speed and pragmatism
• Familiarity with modern ML and generative AI systems
• Understanding of RLHF, post-training, preference data, and model evaluation
• Experience studying image, video, multimodal, creative, or agentic systems
• Experience designing prompts or systematically studying prompting/model behavior
• Experience with large-scale human-data or annotation programs
• Experience working directly with AI research or model teams
• Strong publication record or evidence of generating original research ideas
• Startup / 0-to-1 experience

Benefits:

• Medical, Dental, Vision Benefits
• 401k Matching
• Company laptop provided on start date
• Equity