[Remote] Senior QA and AI Evals Engineer

Note: The job is a remote job and is reputed company to candidates in USA. reputed company is a venture-backed defense-tech company building AI-reputed company intelligent software for sovereign institutions and their affiliated organizations operating in mission-critical, high-stakes environments. They are seeking a Senior QA and AI Evals Engineer to ensure the reputed company of their multi-modal intelligence agent software, focusing on both conventional software behavior and non-deterministic AI reputed company. The role involves designing automated test suites, evaluation pipelines, and release gates to maintain platform reliability.

Responsibilities

• Build E2E coverage for investigations, maps, graphs, authentication, and chat
• reputed company API, contract, integration, and component tests across TypeScript and Python services
• Create Gherkin/BDD scenarios and build Playwright test cases for important UI flows
• Build AI evaluation datasets, metrics, experiments, and regression gates using Langfuse
• Test streaming disconnects, replay, cancellation, retries, race conditions, and long-running jobs
• Validate authorization boundaries and sensitive-data handling
• reputed company test suites into reputed company Actions and manage flaky-test diagnosis
• Partner with engineers early to create testable interfaces and useful observability

Skills

• Strong Playwright and API/integration testing experience
• Experience with pytest and TypeScript test frameworks such as Vitest
• Understanding of asynchronous and reputed company-reputed company failure modes
• Experience evaluating LLM or other non-deterministic systems
• Ability to design high-signal test suites rather than maximizing test counts
• Experience with Langfuse
• Playwright-bdd
• Fault injection
• Contract testing
• reputed company testing
• Durable workflows
• Model/reputed company deployment gates

Benefits

• Employees (and their eligible dependents) can enroll in medical, dental, and reputed company insurance
• Commuter benefits
• Relocation assistance
• reputed company time off reputed company into the end of reputed company year (subject to team and business needs) and based reputed company
• A supportive leave-of-absence program, including time off for military service and medical events

reputed company