[Remote] AI Test Engineer

Note: The job is a remote job and is open to candidates in USA. INSPYR Solutions is recruiting for an AI Test Engineer; the hiring client is not identified. The role builds and maintains AI testing frameworks, automates evaluations, validates AI-enabled products, and provides quality insights across a DevEx portfolio.

Responsibilities

• Help design, develop, and maintain AI testing and evaluation frameworks for generative AI, predictive AI, and AI-powered developer experience (DevEx) solutions
• Create and execute automated test suites that validate AI model behavior, output quality, accuracy, reliability, and performance
• Develop automated evaluation pipelines that integrate with CI/CD workflows and support continuous AI quality validation
• Perform functional, regression, performance, and scenario-based testing of AI-enabled applications and services
• Analyze AI-generated outputs to identify defects, inconsistencies, hallucinations, bias, performance issues, and other quality concerns
• Build and maintain test datasets, benchmark scenarios, prompt libraries, and evaluation artifacts used to assess AI systems
• Collaborate with software engineers, AI engineers, and platform teams to improve model quality and testing coverage
• Define, measure, and report AI quality metrics such as accuracy, relevance, latency, consistency, robustness, and user satisfaction
• Support root-cause analysis, troubleshooting, and validation activities for AI-related defects and incidents
• Document test strategies, test results, evaluation methodologies, and recommendations for technical and non-technical stakeholders
• Contribute to AI governance and responsible AI efforts by supporting evaluation activities related to reliability, fairness, transparency, and safety
• Support delivery of strategic DevEx initiatives and AI-enabled platform capabilities across the software development lifecycle

Skills

• US Citizen, GC Holders or Authorized to Work in the U.S
• Bachelor's degree in Computer Science, Information Systems, Software Engineering, Data Science, or a related technical field
• 3+ years of experience in software testing, quality engineering, test automation, or AI/ML testing
• Experience designing and executing automated test strategies for complex software applications
• Strong programming and scripting skills in Python
• Experience building automated test frameworks and test pipelines
• Experience working with APIs, automated integration testing, and CI/CD environments
• Understanding of machine learning, generative AI, large language models (LLMs), or AI-powered applications
• Experience performing data validation, test result analysis, and defect investigation
• Familiarity with modern testing methodologies, quality engineering practices, and software delivery lifecycles
• Knowledge of statistical analysis and evaluation techniques used to validate system performance
• Experience using source control systems and collaborative development practices
• Strong analytical, problem-solving, and troubleshooting skills
• Excellent written and verbal communication skills
• Ability to work independently in a fast-paced, agile environment while collaborating effectively across teams
• Experience testing generative AI applications, LLM-based solutions, or Retrieval-Augmented Generation (RAG) systems
• Experience with AI evaluation frameworks such as DeepEval, Ragas, LangSmith, or similar tools
• Knowledge of LLM observability, monitoring, and traceability practices
• Experience working with Azure AI, GitHub Copilot, OpenAI, Hugging Face, LangChain, or comparable AI platforms
• Familiarity with prompt engineering, benchmark dataset creation, and AI output validation techniques
• Understanding of Responsible AI principles and AI governance frameworks
• Experience with performance testing, scalability testing, and reliability engineering
• Experience supporting developer platforms, DevEx initiatives, or software engineering productivity tools
• Experience working within enterprise or regulated technology environments

Benefits

• Remote work (ET hours)

Company Overview