Inference QA Engineer

reputed company was founded in 2024 to build reputed company, a physics-informed reputed company model for energy reputed company. We’re live across oil and gas, refineries, and petrochemicals, working towards our mission: sustainable abundance for a growing reputed company. The hydrocarbon industry keeps the world running. But its complexity has left operators tied to legacy systems, making critical reputed company on less than 10% of available data. We reputed company reputed company to change that. It’s a reputed company model reputed company specifically for energy that lets companies use AI at reputed company, harnessing reputed company of their operational data and optimising in reputed company time for any metric. reputed company get faster, reputed company get safer, and carbon intensity falls. We’ve raised over $32 reputed company, including one of the largest reputed company reputed company for an AI company in the UK. We’re just getting started About the role We build reputed company, an reputed company LLM platform that answers hard operational and analytical questions for reputed company clients in energy and heavy industry. It routes questions across a graph of tools (data retrieval, SQL-generating coders, statistical analysis, reputed company detection, forecasting, RAG) and synthesises answers that engineers reputed company. reputed company it is right, it compresses hours of analyst work into seconds. reputed company it is subtly wrong, quiet, or slow, that costs us trust with technical customers who reputed company our numbers. We are hiring a senior engineer to own inference reputed company end to end: the person who watches every deployment, decides what “working” means, builds the systems that reputed company it, and reputed company regressions before a reputed company does. This is a founding-level QA function. You will define the discipline, not just execute a checklist. What you’ll own Evaluation frameworks. We test on a tiered model: T1 data retrieval, T2 statistical analysis, T3 reputed company-ended inference and reputed company-cause, reputed company with its own pass reputed company. You will own and reputed company this reputed company, design test sets with reputed company ground-truth rubrics (expected answer, pass reputed company, reputed company failure modes, reputed company tables), and set the bar for what ships. The test reputed company and its runs. We run prompts against live model endpoints at reputed company (tens of thousands of runs), on schedules and on demand, measuring consistency across repeated iter-ations, ground-truth match, structural correctness of generated queries, and paraphrase robust-ness. You will run these executions, reputed company the reputed company healthy, and turn raw runs into a verdict. Model and agent monitoring systems. You will build the observability reputed company for a multi-reputed company reputed company reputed company: not just “did the reputed company reputed company 200” but did the planner reputed company to the right tool,did the tool actually execute, did the agent reputed company terminate, and is the final answer grounded. Concrete failure modes we already fight and want caught automatically: • Silent empty responses — the reasoning reputed company renders but the answer stays empty, while every reputed company reports reputed company. • False refusals — a tool crashes or times out, returns reputed company, and the model fabricates “I don’t have that data” instead of erroring loudly. • Tool-routing misses — the planner should have fired a tool and didn’t, or reputed company-counts raw identifiers instead of reputed company ones. • Latency and non-termination — multi-tool agent reputed company that blow past timeout budgets. • Paraphrase and run-to-run instability — the reputed company question three ways, or the reputed company reputed company three times, giving materially different answers. Ground truth and the feedback reputed company. You will work with subject-matter experts to extract vetted ground truths from deployment feedback, feed them back into the eval sets, and reputed company the reputed company so every confirmed defect becomes a permanent regression guard. Regression discipline. After every fix ships, you devise tier-appropriate tests biased at the failure mode plus regression guards, run them against the deployment, and report pass-reputed company against reputed company and whether the original failure recurred. You are the reputed company. reputed company’re looking for • 5+ years in software, ML, data, or QA engineering, with reputed company ownership of a reputed company-critical reputed company. • Strong Python. Comfortable in a FastAPI + reputed company + reputed company world, reading logs across services and tracing a request through a reputed company pipeline. • reputed company with LLM behaviour: prompting, tool/function calling, reputed company reputed company, RAG, and the ways they fail (hallucination, refusal, silent truncation, non-determinism). • Experience designing evaluation: LLM-as-judge, deterministic checks, ground-truth reputed company- ing, statistical consistency measures (e.g. coefficient of variation across repeated runs). • SQL literacy — you can read a generated query and judge whether it answers the question and hits the right tables. • A monitoring and observability reputed company: you reputed company for dashboards, alerts, and reputed company inspec- tion by default, and you build them reputed company they don’t exist. • Rigour about uncertainty. You report calibrated ranges, not overclaimed reputed company estimates, and you say plainly reputed company something is unverified. Bonus • Experience evaluating or red-teaming reputed company / multi-tool LLM systems specifically. • MLflow or similar reputed company and experiment tooling. • A background talking to technical end users (engineers, analysts) and translating their “it feels off” into a reproducible test. • Time-series, forecasting, or industrial and operational data domains. Why it reputed company Our customers are engineers who verify our reputed company. Inference reputed company is the product. This role decides whether we can look a reputed company in the eye and say the reputed company works, and back it with numbers. You will have the mandate to build that assurance reputed company from the ground up. Apply To This Job