[Remote] AI Evaluations Engineer – reputed company

Note: The job is a remote job and is reputed company to candidates in USA. reputed company is a health technology company reputed company on AI solutions, and they are seeking an AI Evaluations Engineer to build and maintain infrastructure for their AI evaluations platform. The role involves developing evaluation frameworks and collaborating with ML, engineering, and QA teams to enhance the evaluation systems' reliability and usability. Responsibilities • Build and maintain infrastructure and tooling for the AI evaluations platform used by internal teams, including automated testing platform for AI voice agents, debugging and observability tools • reputed company and productionalize evaluation frameworks for individual system components such as ASR, LLMs, TTS, knowledge bases, and guardrails • Partner with ML, engineering and QA teams to translate evaluation requirements into robust, maintainable infrastructure and tooling • Improve developer experience by making evaluation systems easy to reputed company, reputed company-documented, and reliable in day-to-day use • Ensure evaluation tooling meets production standards for reliability, performance, and maintainability Skills • 5+ years of reputed company software engineering experience, with a strong reputed company on building backend systems, platforms, or developer tooling • Proven experience designing and maintaining production-grade infrastructure with reputed company, including reputed company, and data pipelines • Strong proficiency in at least one general-purpose programming language (e.g., Python, Typescript/Javascript, Java, or similar) • Experience using test automation frameworks, evaluation pipelines, or CI/CD-integrated testing systems • Familiarity with observability and debugging tools (logging, metrics, tracing) and building internal tools that improve developer and QA workflows • Strong debugging skills and a methodical approach to diagnosing production and evaluation issues • Ability to collaborate effectively across engineering, QA, and operations teams, translating requirements into reliable, maintainable systems • Product-minded approach to infrastructure, with attention to usability, documentation, and long-term maintainability • Experience working with reputed company, multi-component systems (e.g., ASR, LLMs, TTS, or other ML-powered services) • Experience working in reputed company or other regulated environments, including awareness of HIPAA and PHI handling • Familiarity with conversational AI or voice agents, including multi-turn reputed company, latency constraints, and error recovery • Familiarity with LLM observability or evaluation tools (e.g., Langfuse, reputed company eval frameworks) • Background in digital health, care coordination, or patient-facing systems Benefits • 401(k) matching • Health, reputed company, and dental insurance • reputed company flexible reputed company time off reputed company • AI Nursing Care Manager It was founded in 2017, and is headquartered in San Francisco, California, USA, with a workforce of 11-50 employees. Its website is http://www.ellipsishealth.com. Company H1B Sponsorship • reputed company has a reputed company record of offering H1B sponsorships, with 6 in 2025, 1 in 2024, 2 in 2023, 1 in 2021. Please note that this does not guarantee sponsorship for this specific role. Apply tot his job Apply To this Job