Senior QA Automation Engineer
About The Company IDT Technologies, a division of IDT Corporation, is a leading player in the telecommunications industry with a rich history dating back to 1990. Headquartered in New Jersey, IDT has established itself as an industry leader in prepaid communication and payment services, serving millions of customers worldwide. The company is publicly traded on the New York Stock Exchange (NYSE) and employs over 1,500 professionals across more than 20 countries. With revenues exceeding $1.5 billion, IDT continues to innovate and expand its global footprint. Its commitment to delivering reliable, cost-effective communication solutions has made it a trusted partner for both consumers and enterprises alike. Net2Phone, a subsidiary of IDT, complements this legacy by providing comprehensive cloud communications solutions. It offers an integrated platform that includes voice, messaging, chat, and collaboration tools, supporting both domestic and international calling to over 23 countries. Net2Phone’s innovative approach enables businesses to streamline their communication channels, enhance customer engagement, and improve operational efficiency. Together, IDT and Net2Phone form a robust ecosystem dedicated to transforming how organizations connect and communicate in a digital-first world. About The Role We are seeking an experienced Senior QA Engineer to take ownership of quality assurance for our cutting-edge AI voice virtual agent. This role is ideal for a professional who is comfortable reasoning about complex language models and reading SIP traces, with a keen interest in testing non-deterministic systems. The successful candidate will be instrumental in ensuring the reliability, accuracy, and robustness of our voice virtual agent, which interacts with users via live phone calls, transcribes speech, interprets intent, reasons over large language models, and generates spoken responses within strict latency constraints. The role involves designing innovative testing strategies for a multifaceted system that combines speech recognition, natural language understanding, telephony, and AI reasoning. The engineer will work closely with cross-functional teams, embed within Agile Scrum teams, and contribute to all phases of the development lifecycle. Responsibilities include building evaluation frameworks for non-deterministic outputs, automating call scenarios, diagnosing failures across the stack, and extending automated testing coverage integrated into CI/CD pipelines. The role offers an exciting opportunity to influence product quality at a foundational level, shaping how conversational AI systems are tested and validated in real-world conditions.
Qualifications
Minimum of 5+ years of experience in quality assurance, with recent hands-on ownership of complex, integrated products. Practical experience testing SIP/VoIP systems, including call flows, SIP signaling, media handling, softphones, or SIP trunking. Ability to troubleshoot using SIP traces or packet captures is essential. Proven creativity in test design, especially for live scenarios involving complex conversational behaviors. Proficiency in Python, capable of independently developing and maintaining automation frameworks. Experience testing non-deterministic or probabilistic systems, with a clear approach to designing assertions for variable outputs. Strong analytical skills to define, justify, and track quality metrics, leveraging data to inform release decisions. Hands-on experience with formal Test Management System (TMS) tools and Jira for issue tracking and project management. Excellent command of English, both spoken and written, with reliable overlap with US Eastern business hours.
Responsibilities
Embed fully within a Scrum team, actively participating in all ceremonies and maintaining associated artifacts to ensure continuous quality improvement. Own the end-to-end test strategy for the AI voice virtual agent, covering language models, speech recognition, and telephony layers. Design and develop test scenarios that capture complex conversational behaviors, including multi-turn dialogues, interruptions, ambiguous inputs, out-of-scope requests, human escalation, and failure recovery mechanisms. Build and maintain evaluation frameworks for non-deterministic outputs, utilizing golden datasets, semantic assertions, and automated regression detection to monitor model and system changes. Conduct adversarial testing to identify hallucinations, grounding errors, prompt injections, and potential data leaks, ensuring system robustness against malicious inputs. Prepare and provision SIP devices, endpoints, and call scenarios to automate comprehensive call origination and media control testing. Diagnose failures across the entire system stack, from LLMs and speech recognition to call routing and media pathways, isolating root causes effectively. Extend automated testing coverage and integrate test suites into CI/CD pipelines, establishing release gates that uphold quality standards. Define, track, and report on core quality metrics such as task completion rate, intent accuracy, transcription quality, response latency, and escalation correctness. Maintain detailed test documentation within the Test Management System, ensuring full traceability to business requirements. Communicate quality risks and issues effectively to stakeholders, supported by data and comprehensive analysis.
Benefits
Remote work flexibility, allowing you to operate from anywhere. Competitive B2B employment terms with attractive compensation packages. Stable employment with long-term growth opportunities within a dynamic organization. Access