Apply here. Applications close on Jan 3, 2024. This role is for an employee who’d work remotely.
About METR (formerly ARC Evals)
METR does empirical research to determine whether frontier AI models pose a significant threat to humanity. It’s robustly good for civilization to have a clear understanding of what types of danger AI systems pose, and know how high the risk is. You can learn more about our goals from Beth’s talk.
Some highlights of our work so far:
- Establishing autonomous replication evals: Thanks to our work, it’s now taken for granted that autonomous replication (the ability for a model to independently copy itself to different servers, obtain more GPUs, etc) should be tested for. For example, labs pledged to evaluate for this capability as part of the White House commitments.
- Pre-release evaluations: We’ve worked with OpenAI and Anthropic to evaluate their models pre-release, and our research has been widely cited by policymakers, AI labs, and within government.
- Inspiring lab evaluation efforts: Multiple leading labs are building their own internal evaluation teams, inspired by our work.
- Early commitments from labs: Anthropic credited us for their recent Responsible Scaling Policy (RSP), and OpenAI recently committed to releasing a Risk-Informed Development Policy (RDP). These fit under the category of “evals-based governance”, wherein AI labs can commit to things like, “If we hit capability threshold X, we won’t train a larger model until we’ve hit safety threshold Y”.
We’ve been mentioned by the UK government, Obama, and others. We’re sufficiently connected to relevant parties (labs, governments, and academia) that any good work we do or insights we uncover can quickly be leveraged.
Job mission
You'll contribute to some of our most crucial work, helping us test model capabilities by analyzing their performance and generally improving our assessment framework.
As part of your role, you will:
- Review transcripts of language model agents attempting various complex tasks, noting down concrete mistakes and identifying broad classes of failure modes and impressive capabilities.
- Implement and test different tasks in which to assess the agents’ performance in order to identify their strengths and weaknesses.
- Implement and test potential improvements to the language model agents’ scaffolding in order to demonstrate their different capabilities.
- Generally support the research team by conducting experiments on and analyzing the behavior and capabilities of language model agents trying to complete various tasks.
This is a full-time, fully-remote role, although substantial overlap with United States Pacific Time working hours is required. This role is compensated at US$108,000 per year.
About you
To succeed in your role, you will likely need:
- A good working understanding of language model capabilities and of modern ML.
- Diligence and the capacity for routinely reviewing large amounts of data manually. Most of our experimental setups are complex and result in largely freeform data requiring manual review and evaluation.
- To carefully think about what makes an LM agent’s action “good” or “bad” in a variety of nuanced scenarios.
- Strong scripting skills: Ability to design and build a nice system for managing experiments and data in order to quickly answer questions about our data, make figures, and run experiments given an API to a model.
- The ability to learn quickly, especially in a coding context: We test our LM agents in a variety of settings and tasks, most of which are likely to be unfamiliar to any given person.
Application process
We primarily use work tests for our evaluations. If we proceed with your application after review, we typically ask you to do 1–3 of these and 1-2 interviews.
The first trial task is often less than an hour, while later trial tasks can be longer. If we ask you to do a work test that is expected to take 2 or more hours, we will compensate you at a competitive hourly rate for your time.
After the first 1-2 work tests, we typically schedule a short phone or video call to discuss what you're looking for in a role, ask you any questions we have, and provide time for any other questions you have.
The final stage in our hiring process is usually a few-week trial period; we're open to figuring out how to manage this with each candidate.
Our application is short, with just a single required answer. Apply here!
We encourage you to apply even if your background may not seem like the perfect fit! We would rather review a larger pool of applications than risk missing out on a promising candidate for the position.
We are committed to diversity and equal opportunity in all aspects of our hiring process. We do not discriminate on the basis of race, religion, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status. We welcome and encourage all qualified candidates to apply for our open positions.