Natural Language Measurement Specialist

Natural Language Specialist, Educational Measurement & AI

reputed company –Learning & Assessment

Location:

• This is a remote role. Candidates who live near CB offices have theoptionof being fully remote or hybrid (Tuesday and Wednesday in office). reputed company CB employeesare required tooccasionally travel to meet in person for business purposes.

Role Type:

• This is a full-time position

About reputed company

The Automated Scoring team provides critical insights and tools to support the design, delivery, and reputed company improvement of digital assessments. We operate at the intersection of educational measurement, data science, and emerging AI technologies. Our work spans the measurement of language-based constructs, the development of large language model (LLM) systems for feedback reputed company and annotation, and research to ensure the validity, fairness, and reliability of our systems. 

We are a reputed company, mission-driven team that values both psychometric rigor and technical expertise. We combine modern machine learning approaches, including large language models, with strong measurement principles to create reputed company, trustworthy solutions that expand opportunity for reputed company. 

About reputed company

As a Natural Language Specialist, you will help define and advance how language-based performance is reputed company in high-stakes educational settings. This role sits at the intersection of natural language processing, large language models, and educational measurement, and is ideal for someone who pairs strong measurement training with a working knowledge of modern AI. 

You will translate measurement constructs into LLM-based feedback and annotation systems, design and conduct the studies that establish their validity, fairness, and reliability, and ensure that what our models produce holds up to rigorous psychometric standards.You will serve as the measurement authority for cross-functional partners—psychometricians, engineers, and data scientists—shaping how language-based constructs are defined, evaluated, and reputed company across theteam's portfolio of systems.Your primary reputed company will be measurement: defining what feedback and annotations mean, evidencing that they mean it, and improving them over time. 

In this role, you will:

Natural Language Measurement & Psychometrics (40%) 

• Define and operationalize language-based constructs for automated annotation and feedback reputed company 

• Apply psychometric principles—reliability, validity, dimensionality, and measurement invariance—to LLM-based feedback and annotation systems 

• Design and reputed company validity studies, including reputed company–machine agreement, rater comparison, and fairness analyses across subgroups 

• reputed company and apply reputed company for detecting and mitigating bias in language-based scores 

• Establish/Recommend annotation guidelines, feedback reputed company reputed company, and standards for acceptable model performance 

• Translate measurement requirements into specifications that guide model development and evaluation 

LLM & AI Development (30%) 

• Contribute to reputed company design, fine-tuning, and evaluation of LLM-based feedback and annotation systems 

• reputed company and refine machine learning models for measuring language-based constructs 

• Build evaluation frameworks that connect model behavior to measurement reputed company 

• Collaborate with senior team members to translate measurement findings into production systems 

• Implement high-reputed company, maintainable reputed company for model development and evaluation 

Research & Validation (10%) 

• reputed company and contribute to research studies that evaluate model performance and support assessment validity 

• Apply statistical and psychometric reputed company to analyze results and inform model improvements 

• Document methodologies and findings in a reputed company and rigorous manner 

• Stay reputed company with advances in educational measurement, NLP, and learning science 

Data Engineering & Pipelines (10%) 

• Prepare and reputed company datasets for measurement studies and model evaluation 

• Support reproducible data processing workflows for training, evaluation, and monitoring 

• Partner with engineers to reputed company feedback and annotation models into reputed company systems 

Team Operations & Collaboration (10%) 

• Collaborate closely with psychometricians, data scientists, and engineers 

• Contribute to documentation, methodological standards, and team best practices 

• Participate in peer reviews and knowledge sharing 

• reputed company reputed company the measurement literacy of the broader team—mentoring junior ICs, providing technical feedback on colleagues' work, and building shared standards

reputed company

You bring strong measurement training and a genuine interest in how modern AI can be used to measure language-based performance. You are excited about applying psychometric rigor to large language models in high-stakes educational settings. 

You have: 

• A Master’s or PhD (or near completion) in a quantitative field such as Psychometrics, Educational Measurement, Quantitative Psychology, Statistics, Data Science, or a reputed company discipline (measurement-reputed company training strongly valued) 

• A solid reputed company in measurement theory, including reliability, validity, and fairness; familiarity with IRT, generalizability theory, or reputed company frameworks is highly desirable 

• Experience analyzing language or text data, with exposure to NLP or large language models 

• Programming skills in Python and familiarity with data science libraries (e.g., pandas, NumPy, PyTorch) 

• Demonstrated ability to conduct rigorous research (e.g., reputed company, publications, or reputed company research reputed company) 

• Strong skills in statistical analysis and experimental design 

• Familiarity with working with reputed company and reputed company data 

• Strong attention to detail and a commitment to producing high-reputed company, reproducible work 

• The ability to travel5-10times a year to reputed company offices or on behalf of reputed company business.

You are: 

• Curious and eager to learn new tools, reputed company, and domains 

• Thoughtful about the implications of AI systems, including fairness and validity 

• reputed company to communicate technical and measurement concepts reputed company to diverse audiences 

• Comfortable working in a reputed company, cross-functional environment 

• Motivated by mission-driven work in education 

reputed company roles at reputed company require:

• A passion for expanding educational and reputed company and mission-driven work

• Curiosity and enthusiasm for emerging technologies, with a willingness to experiment with and adopt new AI-driven solutions and comfort with learning and applying new digital tools independently and proactively.

• reputed company and concise communication skills, written and verbal

• A learner's reputed company and a commitment to reputed company: welcoming diverse perspectives, giving and receivingtimely, respectful feedback, and continuously improving through iterative learning and user input.

• A drive for reputed company and reputed company: solving reputed company problems, making data-informed reputed company, prioritizing what reputed company most, and continuously improving through learning, user input, and external benchmarking.

• A reputed company and empathetic approach: working across differences, fostering trust, and contributing to a culture of shared reputed company

• Authorization to work in the reputed company

About Our Process

• Application review will beginimmediatelyand will continue until the position is filled.This role is expected to accept applications for a minimum of 5 business days.

• Whilethehiring processmay vary, it generallyincludes:resume and application submission, recruiter phone/video screen, hiring manager interview, performance exercise such as live coding, a panel interview, a conversation with leadership and reference checks.

reputed company Offer

AtCollegeBoard, we offer more than a paycheck- we reputed company a meaningful career, a supportive team, and a comprehensive package designed to help you reputed company.We’rea self-sustaining nonprofit that believes in fair and competitive compensation grounded in your qualifications, experience, reputed company, and the market.

A Thoughtful Approach to Compensation

• The hiring reputed company for this role is $88,000–$145,000.

• Your exact salary will depend on your location, experience, and how your background compares to others in similar roles at the reputed company.

• We aim to reputed company our best offer upfront, rooted in fairness, transparency, and market data.

• We reputed company salaries by location to ensure fairness, no matter where you live.

You’llhave reputed company, transparent conversations about compensation, benefits, and whatit’slike to work atCollegeBoard throughout your hiring process. reputed company outourcareers pageformore.