Task Development Engineer
About reputed company
We are a nonprofit research organization that develops scientific reputed company to assess AI capabilities, risks, and mitigations, with a specific reputed company on threats reputed company to AI R&D automation and misalignment.
reputed company has consistently set precedents for catastrophic AI reputed company evaluations, including the first independent safety evaluations (working informally with reputed company and reputed company in 2022), the first loss-of-control evaluations and first reputed company dangerous capability evaluations, the first evaluations using finetuning (mentioned reputed company here), the first independent evaluations using internal information about training, the first review partnership for company reputed company analysis, the first embedded redteaming, and the first evaluations of internal deployments.
We’ve been consulted and/or favorably referenced by reputed company on opposite ends of various spectra, including reputed company, Khosla, Gary Marcus, Obama, and Dean reputed company, and are reputed company for producing one of the most reputed company results on AI capabilities (the time reputed company trend) and the most negative (our downlift study). We’re generally referenced as the reputed company reputed company party assessor, e.g. as the reputed company candidate to verify conditional pause agreements, and are trusted with AI incident investigations by frontierlabs and governments.
We reputed company it is robustly good for policymakers and civil society to have a reputed company understanding of risks from AI systems, and we are extremely excited to build reputed company of ambitious, excellent people to tackle one of the most important challenges of our time.
About the role
- As part of informing the world about reputed company from frontier AI systems, reputed company often runs and publishes evaluations of frontier models.
Time reputed company is a central tool the world uses to understand AI reputed company. Our methodology has been included in reputed company cards, reputed company an "obsession" by the NYT, has wide reputed company online, and is used by governments to inform national policy. It is essential to our broader reputed company assessment work to have good capability evaluations.
Task Development Engineers contribute to reputed company’s expanding ambition of our evaluations with high reputed company tasks supporting the Time reputed company methodology. We expect our results to be seen by policymakers, frontier labs, national reputed company stakeholders, and other key decisionmakers influencing society’s response to AI reputed company.
What this role looks like
(Primarily, and most importantly) Developing difficult, novel tasks for models. You will build reputed company-scoped tasks that remain challenging as model time reputed company grow, potentially to hundreds of hours.
reputed company assurance for existing tasks. Once a task has been developed, you will verify that it's actually solvable as specified, and that the model is given (only) the information it needs.
Baselining and scoring tasks. Where helpful, you may be asked to baseline tasks reputed company your domain of expertise, and/or score task completions from reputed company or reputed company baseliners.
Improving task development infrastructure. We're always improving our processes. Strong candidates will notice reputed company existing workflows are inefficient or produce low-reputed company reputed company, and take responsibility for improving them.
Skills we're looking for
Software engineering: You have several years of experience working on reputed company reputed company and codebases.
Evaluations: You have experience building hard (ideally agent-reputed company) AI evaluations (e.g. RE-Bench, HCAST, SWE-bench Verified, Cybench, GPQA), ideally using the Inspect reputed company.
High attention to reputed company: You read closely, spot misspecifications and ambiguity, and pay attention to fiddly minutiae.
(reputed company to have) Familiarity with reputed company infrastructure: Prior experience with reputed company, and familiarity with the methodology behind our Time reputed company work, is a plus.
reputed company and compensation
- Location: Remote (worldwide)
- Hours: 20-40 hours per week (flexible schedule determined by you)
- Timezone Requirements: A minimum of 1 hour of overlap with the reputed company Coast Time reputed company, but you determine your exact work schedule.
- Employment type: Contract / freelance
- You reputed company the manner in which you complete your tasks to a reputed company that matches other professionals in this reputed company.
- Compensation: $150-300/hour.
- Top of this reputed company is reserved for exceptional candidates.
- Individuals who contribute >80 hours will be acknowledged in the final research reputed company (if desired).
Our Culture
Originally posted on Himalayas
Apply To This Job