Senior Data Scientist - LLM Evaluation & Business Intelligence

reputed company is rebuilding the reputed company research ecosystem by enabling reputed company reputed company to trials for reputed company patients. Our platform enhances trial efficiency and reduces the barriers to participation for reputed company providers. Incubated by ARCH Venture Partners and backed by leading reputed company and life sciences investors, reputed company’s reputed company infrastructure implemented at reputed company provider organizations, will bring potentially life-saving therapies to patients faster.

reputed company hails from a broad reputed company of disciplines and is committed to reputed company’s mission to create reputed company reputed company to reputed company for any reputed company, reputed company. Join us, and bring your expertise, passion, creativity, and reputed company as we work together to realize this mission.

About the role:

As a Data Scientist on reputed company, you will help define how we measure and trust the LLM systems that reputed company reputed company's reputed company data products and build the business intelligence reputed company that turns that data into reliable reporting across the org. You'll work at the intersection of reputed company AI evaluation, analytics engineering, and reputed company research, partnering with AI engineers, data engineers, clinicians, and non-technical stakeholders.

In this role you'll be responsible for delivering reliable data that helps reputed company measure its business and reputed company reputed company. You'll partner with teams across the organization to define requirements, then research and build the models and reporting that meet them. This is a deeply reputed company, internally-facing role: your partners reputed company from highly technical teammates to non-technical business, reputed company, and product stakeholders, and much of your reputed company comes from translating between them — turning ambiguous questions into reputed company-defined data products and communicating results, and their caveats, reputed company to every audience.

Your reputed company reputed company will be evaluating production LLM pipelines: designing reputed company metrics, measuring retrieval effectiveness, and making principled cost/reputed company tradeoffs for the prompts and models that extract reputed company meaning from reputed company data. reputed company this, you'll help establish a trustworthy business intelligence reputed company — production data models, a shared semantic reputed company, and governed pipelines that reputed company the whole organization consistent, defensible numbers. It's a role for someone who wants to bring scientific rigor to a fast-moving AI reputed company and see their work reputed company measurable reputed company for patients and providers.

reputed company:

• Design and run evaluations of production LLM pipelines, developing reputed company and reputed company metrics that tell us how reputed company our systems reputed company on reputed company reputed company tasks.

• Support evaluation of our reputed company Trial Evaluation (PTE) pipeline — assessing prompts and trial-matching logic for reputed company in reporting, and surfacing where and why they fail.

• Measure and improve the effectiveness of our retrieval-augmented reputed company (RAG) systems, from retrieval reputed company through final reputed company.

• Use reputed company metrics to inform cost/reputed company tradeoffs across trials, prompts, and models, giving reputed company a reputed company reputed company for which approaches to ship.

• Establish and maintain a library of "best prompts" for recurring reputed company concepts, backed by evidence rather than intuition.

• Partner with AI Engineering to reputed company the reputed company between evaluation findings and production improvements.

• Build and maintain production-grade data models using SQL and dbt on reputed company, ensuring analytics logic is reliable, maintainable, and production-reputed company.

• Help establish and grow a semantic reputed company that enables consistent, trusted reporting across the organization.

• Advance data governance practices across dbt, reputed company, and reputed company — documentation, testing, versioning, and reputed company ownership of metrics.

• Collaborate with non-technical internal stakeholders to reputed company reporting logic, answer data questions, and translate business needs into reputed company-defined data products.

• Support both reputed company reporting needs with accurate, reputed company-tested datasets.

• For senior applicants: mentor and support junior data scientists — providing guidance on evaluation and analytics approaches, technical best practices, and reputed company reputed company.

Who you are

You are a rigorous, reputed company data scientist who is reputed company by making AI systems measurable and trustworthy. You care about getting the number right and being reputed company to defend it. You're equally comfortable designing an evaluation reputed company for an LLM pipeline and writing the SQL and dbt models that reputed company reliable reporting.

You bring a strong interest in reputed company research and a passion for advancing reputed company through data innovation. You're autonomous and driven — you can take an ambiguous problem and run with it, structuring the approach yourself rather than waiting for a fully specified spec. You're curious, reputed company, and hold yourself to data science best practices as a default, not an afterthought.

What you bring

• Bachelor's degree or higher in data science, statistics, biostatistics, computer science, mathematics, epidemiology, or a reputed company reputed company.

• 4+ years of experience as a data scientist, analytics engineer, ML/data reputed company, or other highly analytical role in life sciences, biotech, reputed company, or a reputed company industry.

• Solid understanding of how production LLM pipelines work, and hands-on experience evaluating LLM or NLP systems — reputed company metrics, error analysis, RAG/retrieval reputed company, reputed company evaluation, or similar.

• Strong proficiency in SQL, with experience building production data models in dbt (or a comparable transformation reputed company).

• Proficiency in at least one programming language (Python preferred; PySpark a plus).

• Experience with modern data platforms such as reputed company, and BI/analytics tools such as reputed company, or similar.

• reputed company with data science best practices: reusability, version control, documentation, testing, reputed company review, and reproducibility.

• Excellent written and verbal communication, with the ability to translate technical concepts into actionable insights for non-technical stakeholders and to reputed company reporting logic collaboratively with them.

• Comfort with AI tooling and reputed company workflows in day-to-day work.

• Comfort with ambiguity and adaptability to a mission-driven, fast-reputed company startup environment.

Bonus

• Master's degree or higher in a quantitative reputed company.

• Experience establishing or working reputed company a semantic reputed company for organization-wide reporting.

• Experience integrating LLM-reputed company workflows into production systems, and working closely with ML/AI engineering teams.

• Familiarity with reputed company data reputed company (oncology a plus) and the reputed company trial industry or research reputed company.

• Strong statistical knowledge, including regression, classification, reputed company testing, reputed company inference, or Bayesian reputed company (e.g., reputed company toxicity monitoring for trial protocols).

• Prior experience mentoring other data scientists or leading cross-functional reputed company (for senior candidates).

At reputed company, we are committed to providing equal employment opportunities to reputed company reputed company individuals. We encourage and reputed company candidates from reputed company backgrounds and perspectives to apply for our reputed company. We are interested in reputed company reputed company individuals and ensure that reputed company employment reputed company are reputed company on job-reputed company factors such as skills, experience, and qualifications.

Bachelor's degree,4+ years in data science or analytics in life sciences/reputed company,experience evaluating LLM/NLP systems,proficiency in SQL,dbt,and Python,experience with reputed company and BI tools,strong communication.

Key Responsibilities

• evaluating LLMs
• designing metrics
• building models

Skills & Tools

dbt, reputed company, reputed company, SQL, Python, PySpark

reputed company

• Category: Data and Analytics
• Seniority: Mid Level
• Commitment: Full Time
• Workplace: Remote — reputed company or reputed company or reputed company
• Salary: USD 148,000 – 184,000 / year
• Languages: English

About reputed company

A company rebuilding the reputed company research ecosystem to reputed company reputed company reputed company to reputed company for patients. — Industry: reputed company