QA Engineer for Generative AI

Title: QA Engineer for Generative AI Reports To: QA manager, who reports to Parker Wightman, Senior Director of Engineering Location: Draper, UT / Remote (USA) About The Role Jump is looking for a US–based QA engineer to help coordinate and run data labeling/annotation campaigns used to improve our AI/ML systems and evaluate/review production system outputs, such as meeting notes, recap emails, and tasks; answers in our Ask Anything feature; our pre-meeting prep product; and our AI agents. This role blends process design and hands‑on testing. You’ll use AI evaluation rubrics prepared by our product managers or data team to improve our products so our customers get accurate transcripts, summaries, and action items every time they interact with Jump. You’ll go deep into AI best practices and limitations. You’ll partner closely with Engineering, Product, and Customer teams to ship quickly and confidently. Familiarity with Jump (as a user, beta tester, or close to advisor workflows) is a big plus. If you already have AI evaluation experience, that’s great—but it’s not required. We’ll teach you our approach; candidates with AI evaluation experience will be compensated accordingly. What You’ll Do • Serve as the embedded QA engineer on two pods (Jump’s cross-functional teams), collaborating with product managers to evaluate AI outputs, run exploratory and regression testing, and unblock engineers and PMs. • Learn and track AI/ML quality signals, including golden datasets, prompt/regression suites, and metrics such as WER, diarization accuracy, action-item precision/recall, summary faithfulness, hallucination rate, and PII handling. • Build dashboards for quality KPIs (defect escape rate, flake rate, regression coverage, MTTD/MTTR, AI eval scores) and drive continuous improvement. • Partner with Product and Engineering to ensure requirements are testable, edge cases are captured, and AI evaluation rubrics are clear and repeatable. • Foster a no-drama, direct-and-kind culture that moves with high-quality velocity. About You • 3+ years in QA or Quality Engineering for SaaS products • Strong exploratory testing skills and clear, concise written communication for reproducing issues • Curiosity and aptitude to learn ML/AI evaluation (prompt testing, golden sets, offline evals, safety/guardrails) • Familiarity with AI prompts, LLMs, and the Jump product (as a user or employee) • You don’t need a traditional STEM background to excel here. You’ll thrive if you • Get excited about spotting patterns • Have a strong grasp of human language and thought processes • You might have a background in • Editing • Technical writing Nice-to-haves: • Comfortable reading software system logs and finding patterns in messy data • Familiarity with fintech or other regulated environments • Experience with BigQuery or other data warehouses • Experience with web API testing • Basic familiarity with query languages, relational databases, and other data storage systems What You’ve Done • Built or scaled a QA function (process, tooling, reporting), or partnered with product managers and engineers to identify and resolve AI-related bugs • Written great documentation, bug reports, or other clear technical writing • Interacted meaningfully with LLMs and AI outputs • Nice-to-haves: • Designed and executed AI evaluation workflows (golden datasets, human-in-the-loop scoring, clear rubrics)—a plus but not required; candidates with this experience may be considered for higher compensation • Created risk-based test plans and lightweight automation that caught regressions early About Jump Jump’s mission is to empower financial advisors and their clients to thrive in the age of AI. Our product is an intelligent meeting assistant that helps advisors with notetaking, task management, follow-ups, and data syncing—built to serve advisors and firms of all sizes, including the largest financial institutions in the world. Jump was launched in 2023 by a team of experienced software entrepreneurs with backgrounds including Harvard, Stanford, Google, Divvy, BILL, and WeWork. We’re recognized as the leading product in our category and are growing quickly, backed by top-tier venture capital firms. Jump’s culture • High Velocity • World Class • Direct + Kind + No Drama We believe in building tight teams of extraordinarily capable people. Come join us to transform the advisor and enterprise experience with state-of-the-art technology. Compensation • Salary: $75k to 90k (DOE) • Equity • Health benefits Apply tot his job Apply To this Job