How to apply to AI safety fellowships (and beyond)

About a year ago, I decided to go all-in on applying to AI safety fellowships, and around the end of 2025 I got into MATS. I think there’s a small art to communicating your skills legibly. When I’ve spoken with others about how I answer application questions, they seem to appreciate my advice. I wrote a MATS 9 Retrospective which was well-received, so consider this to be similar advice, but for applying to jobs or fellowships.

Applying to AI safety fellowships or doing job applications is an adversarial process: The goal of an application process is to measure how well the candidate would do in the position they’re applying for, but this process is noisy. Some candidates will (inevitably) try to overfit to the application process itself, in a way that oversells their abilities.

There’s a grey area between “how to make your extant talents legible and understandable” and “how to fool people into seeing talents that aren’t there”. I’ve tried hard to withhold advice which could be used to overfit to the applications process, and to focus on advice that differentially helps people who are fit for the job but struggle to communicate this to the reviewer.

Many application processes have significant flaws that lead to them being noisier than they should be (including those at companies you think should know better). I’m unsure why this is the case, I suspect the issue is that this process is recreated at ~every company, and every company thinks they’re a special snowflake with special hiring requirements such as “very smart people”. Put less cynically: hiring is a hard problem, people who get good at hiring often get promoted away from hiring, and there are often very few ways for feedback to flow from the applicants to the people doing the hiring.

With my disclaimers out of the way, I’ll split the rest of this into some advice for making your skills legible in general and then some advice specific to AI safety fellowships.

How to apply

Differentiate yourself from the Average Joe

There will be lots of Average Joes applying alongside you, and you want to make sure you’re not mistaken for an Average Joe. Put yourself in the mind of the reviewer: the vast majority of applications will be from Average Joe. The application before yours, and the application after yours, will be from Average Joe. Average Joe is great, very fun to talk to. But Average Joe is not quite cut out for the job. You’ve done cool and interesting things, Average Joe has done mediocre and mildly interesting things. We love Average Joe, but probably wouldn’t hire them.

During the application process and as you’re describing your experience, your projects and accomplishments, it is absolutely critical that whatever you write could not be mistaken for something Average Joe could write.

Let me be crystal clear: it is insufficient to accurately describe the impressive things you’ve done. You must describe it in a way that maximally distances yourself from what an Average Joe could have written.

There are hundreds of Average Joes applying to this position, and only one of you. There is substantial overlap between the best-written applications from Average Joes and the worst-written application from yourself. You’re going up against the best-written application from all Average Joes from across the world, so you need to put a lot of effort into writing things that no Average Joe could write. Every sentence should be evidence that you are not an Average Joe, so that you maximally distinguish yourself from the rest of the applicant pool. I do quite literally mean you should read each sentence you write during an application, and ask “If I didn’t know the person who wrote this, what’s my lowest possible estimate for their skills and abilities?“. You need to ensure the lowest estimate is as high as possible.

Making your skills legible

The book Seeing Like A State gives a slightly new meaning to the word legible that’s extremely useful. You can have the skills needed to do a job, but this is basically irrelevant when considering whether you’ll get the position or not. The only thing that matters is whether you can convince the person reviewing your application that you have these skills. If you cannot legibly communicate to the reviewer that you have the skills needed for the job, they will reject your application.

This is critical: it is insufficient to be cracked. You must be able to communicate to someone and have them understand that you are cracked. If you cannot communicate your skills, you might as well not have them.

It is hard to properly communicate your skills via an online application. This is partly because writing is hard, but it’s also partly because there are so many other people who don’t have your skills but are trying to claim that they do. It’s very hard to separate honesty from deception if all you have are some interview questions.

This is why many people get jobs via recommendations or friends-of-friends: Recommendations are generally terrible at communicating the skill of the person, but the recommender is inherently staking their social capital on saying “this person is honest and won’t try to trick you when you evaluate whether they’re fit for the job”.

In lieu of getting a recommendation from someone, you will need to convince whoever’s reviewing your application that 1. you’re fit for the job and 2. you’re better than the other people who are also applying. This is different from just being fit for the job and being better than the other people who are applying: I’ll assume you actually are fit for the job, so the challenge is in effectively (and honestly!) communicating this to the reviewer.

Concrete ways to be more legible

The requirement for skills to be legible is why certificates, PhDs, degrees, references, and journal publications are often requested as evidence: they’re (usually) hard to fake and communicate your skills in a way that’s standardised and easy to understand.

Contributions to open source projects are okay, but significantly less legible because they require that the reviewer understands the project and understands your niche technical contributions to that project. With modern AI coding, open-source contributions say less about your programming ability than they did in the past.

Put yourself in the reviewer’s shoes

This is a bit tricky to express properly, I fear I’ll explain an idea, but not the correct idea and you’ll come away thinking you know what I’m gesturing at but nonetheless I failed to explain the idea properly.

When you answer a question, it’s helpful to ask yourself: what’s the least qualified Average Joe who could reasonably write that answer? You want to ensure the least qualified Average Joe who could reasonably write your answer is still someone who’d get accepted.

Less adversarially, you should try to imagine the state of the reviewer’s mind as they read your answer¹ and try to make them more likely to accept you. Often applications will ask something like:

What’s the most impressive thing you’ve built or accomplished? (50 words)

And it’s important to realise that you’re answering the question behind the question. Your goal is not to describe the most impressive thing you’ve built, your goal is to describe the most impressive thing you’ve built that can be explained in 50 words and is likely to get you accepted. Often (but not always!) these will be the same. If the most impressive thing you’ve built requires 40 words of background just to explain the context, but the second most impressive thing you’ve built is easy and quick to explain, you should describe the second most impressive thing.

The internet is more meritocratic, so use it

The internet (mostly) doesn’t care where you’re living or who you know. If you’re disadvantaged because you don’t live in the right area or don’t know the right people or didn’t go to the right school, you should put more effort into your online presence than you otherwise would. I don’t think most people appreciate that ~everyone is online. It might be really tricky to get important people to respond to your email, but if you write something interesting that shows up in their feed, they’re quite likely to read it!

You should write publicly if you are talented but lack the credentials which usually make those talents legible. Well-written and informative technical essays tend to go viral on the websites that matter: technical people love to read deep-dives into peculiar topics. If you have skills and a good understanding of niche areas, you can make these skills legible by writing things online (in English) and sharing them on LessWrong/HackerNews. Having a personal website is useful but not critical.

This is especially the case if you’re not in the Bay Area or otherwise in a scene where you can discuss your thoughts in person. Writing online is a way to share your thoughts with people who’d otherwise never give you the time of day.

This extends (somewhat) to Twitter. Having a stronger presence online is a very high leverage way to get the attention of very powerful people². Everyone uses the same internet, and everyone scrolls roughly the same timeline. It’s significantly easier to get your ideas in front of interesting people via Twitter than (for example) by flying to San Francisco and knocking on the front door of the organisation you want to work for. Powerful people are usually open to hearing interesting ideas, but they can’t allow any random person to book a 20 minute meeting to discuss their interesting idea. The internet is an incredibly powerful way of getting good ideas in front of people who matter, and if you do this well you can often use this credibility to unlock other opportunities.

While it’s commonly done, I’d recommend against putting your efforts into building the following of an anonymous Twitter account. It’s very hard to “cash out” that reputation into opportunities or job offers without removing the anonymity. While putting your name and face to your opinions is scary, people (in my experience) are more likely to engage with a named profile precisely because there’s a human who’s staking some small amount of reputation on this opinion.

That being said, it is hard to build a following on Twitter if you don’t have a few friends such that you can all mutually like each other’s tweets (and so launder each other’s credibility). Laundering credibility is incredibly common practice (both online and offline) so it’s useful to find friends who can vouch for you in this way. Twitter does do weird things with country borders and limiting the reach of posts based on where you’re posting from, but I still think the above advice is applicable. For example, I saw a significant increase in my following and reach when I was posting from London or The Bay compared to when I was posting from South Africa (although this is confounded by MATS and gaining followers from friends at MATS, so possibly this is less of an effect than I’m making it out to be).

Ensure that skimming your CV still leaves a good impression

If I look at your CV for 15 seconds, what information do I see? I should see things that are directly relevant to the application, for example language model and research experience. There’s a lot written about CVs and how to make a good one, and I imagine your favourite LLM could help you with the layout and styling, but don’t trust it to write the prose for you. You should try to view your CV with fresh eyes and notice what jumps out at you during the first 15 seconds. If someone had to make a decision knowing nothing else, would they accept or reject you?

If I look at it from 5 metres away, what stands out? I sure hope it’s things like “Intern @ Impressive Company” or “Built Cool Project” and not “The internship was from January to March 2025”. If someone only reads the headings on your CV, do they come away wanting to give you the position or not? Do the headings differentiate you from the Average Joe, or could Average Joe have written exactly the same headings as you have?

CVs tend to have lots of structured data (e.g. 3 items of job experience, where each job has a start date, end date, company name, location, etc). This can cause them to become bloated with redundant information that’s implied by other parts of your CV. You should look carefully at every word on your CV, and ask if it’s pulling its weight. I do mean literally every word. If you studied at the University of , you probably don’t need to also include that you were studying in the location of , . I say you don’t need to include it, because (in general) the location won’t make someone more or less likely to hire you. You do literally need to look at each word and ask if removing the word would make someone less likely to hire you. Every word should make a reviewer more eager to get you on their team ASAP.

Consider how you can improve yourself

Notice the little seed of doubt that appears in your mind when an application implicitly asks for something you don’t have. Google Scholar? PhD? Which Ivy League? Not all of these are solvable, but some of them are, and you should listen to what the applications are telling you they want. If you see lots of applications asking if you’ve done ARENA, then you should try to do ARENA. If you see lots of applications asking “Share things you’ve done related to AI Safety” then you should try to do lots of things in AI safety.

Basically, you should look at each application as both a potential for you to get in, but also a very strong signal for what you need to do next time round to better your chances. If you’re applying to the fellowships over and over again and answering the questions in basically the same way each time, you’ll probably continue to get rejected. Take the questions as a rubric against which you should improve yourself.

A little love letter for typst

Typst is great, it’s like LaTeX but significantly easier to use and faster. I spent some time migrating my CV to Typst and it’s paid off massively. I can strongly recommend it, especially nowadays when doing this is just one prompt away.

Part of the benefit is that it’s programmable, so my CV looks something like:

= Boyd Kane CV
== Experience
#from-yaml("snippets/2026-mats-extension.yaml")
#from-yaml("snippets/2026-mats.yaml")
#from-yaml("snippets/2024-aisct.yaml")
...

And each of those #from-yaml(...) functions reads a YAML file that looks something like:

title: MATS 9 with Alex Turner & Alex Cloud
location: Berkeley, California
from: "Jan 2026"
until: "Apr 2026"
desc: |
  - The project by my coauthor #link("https://joneedssleep.github.io/")[Jo Jiao] & I was accepted as one of 9 MATS #link("https://www.youtube.com/watch?v=4nsCTYRS1H4")[Symposium Spotlight talks] out of a cohort of ~100 fellows, and will be continued during the MATS 9 Extension in London. It has been accepted for publication in NeurIPS 2026 (reviewer scores: 5/5/4 out of 6).
  - LLMs behave differently in evaluations than they do when we're not watching them. It's possible that an LLM would behave misaligned in some very narrow range of scenarios (e.g. when it could exfiltrate its weights _and_ nobody's watching _and_ it has internet access _and_ ...).
  - Alex Turner (ex-Google DeepMind) and Alex Cloud (Anthropic) are my mentors as I work on a project to distinguish between LLMs that would exhibit this behaviour and LLMs that won't.
  - This project uses finetuning as a method of evaluation: by measuring how easily an untrusted LLM is able to learn some misaligned behaviour, we can get an estimate for how likely the unfinetuned LLM is to behave misaligned.

And then when I compile the .typ doc, it gives me a CV listing that looks like this:

This is great. I can easily add/remove items from my CV without messing with the formatting. And as a side-effect, I’ve got an LLM-friendly log of every project, job, internship, talk, or thing of interest I’ve ever done. Which brings me to:

Maintain an LLM-friendly version of your work experience

A few years ago I converted all the job-relevant projects and positions I’ve done into YAML files (as described above). At the time this was just so I could move things over to Typst, but this has been doubly useful in that it allows me to easily describe what I’ve done to LLMs.

There are many ways in which this is valuable, but to highlight one which was critical to me applying to many AI safety fellowships while working full-time. It was pretty tough to get motivation some nights to actually go through yet another application process (especially in the beginning). Nearly every question gave me serious imposter syndrome and I’d often be left with writer’s block trying to find an answer, questioning why I should even bother. Most questions are phrased in a way that made me doubt whether anything I had done was enough:

What are 1–2 reasons why you would be a good fit to do empirical AI research? Please give a concise description of each piece of evidence and explain why it’s relevant. Max 200 words³

How experienced are you with tuning hyperparameters: Write about your hyperparameter tuning experiences. For LLMs, what tricks do usually try to make them work at a target task?³

What do you consider to be your top achievement(s)? (Max. 50 words) Please share 1-3 achievements that you are especially proud of. These could be scholastic, like your academic performance at university or in an international olympiad, and extracurricular, like winning a competitive sporting event.³

In these scenarios, I’d use LLMs to remind me of the interesting things I’ve done that are relevant here. This very consistently reminded me of extremely relevant experience that I have and projects that I’ve done. The prompt was something like:

Here’s my experience and a question for an AI safety fellowship. Do not answer the question for me, but give a bullet list of relevant parts of my background that I should mention in my response.

Making it easy for myself to answer many questions like the above was very important for me being able to do well in the application process.

I’ll repeat this again: Do NOT get LLMs to respond to the questions for you. I gave Opus 5.5 the above question along with my pre-MATS context, and the response it gave failed to make a good case for why I should be accepted. Opus does very well at answering the question as it is posed: it cites 2 projects I had done. However, Opus fails to answer the meta-question that sits behind all application questions: “Why should I accept this application?“.

Keep a text log of your questions/responses

If you end up applying to many positions, you’re going to answer many nearly-identical questions. You should keep a log (mine was just a faq.md file) of each question you got asked and what you answered. This is immensely helpful:

1. When you get through to the next interview round, many interviewers will refer to your answers and you really don’t want to be struggling to remember what you said.
2. Sometimes, different applications will ask ~identical questions⁴ about your background, why you want to work in AI safety, etc. Being able to refer to a previous response and lightly edit it will help you fill out more applications without becoming more tired.

To be clear, do not reuse your answers if they’re irrelevant to the question. The reason you’re applying to one application should not be the same as the reason you’re applying to another application. If you think they’re the same, you need to go dig deeper and figure out why they’re different.

If in doubt, you should probably default to writing the answer de novo. Even for something generic-sounding like “Describe your current career plans and aspirations”, you should phrase your career plans in a way that emphasises your goals as relevant to the position you’re currently applying for.

Do not make up fake career plans just to match with the job. That way leads to sadness.

But you should prioritise the aspects of your existing career plans that the mentor would be most interested in seeing. Do not bend the truth about what you want to do. But if your career plan involves something that’s mostly irrelevant to the current position, don’t go yapping about it for ages and ages.

Applying to MATS and other AI-safety fellowships

These sections are primarily phrased with technical AI safety fellowships in mind, although I think there’s lots of advice that is relevant to other job applications.

The AI Safety fellowship pipeline

Like it or not, there’s a bit of a pipeline to the AI safety fellowships, and some are easier to get into if you’ve done an earlier-stage fellowship first. Getting into a later-stage fellowship without any prior qualifications is hard. Doing earlier stage fellowships is a really good way to make your skills legible, because you get more time and input from progressively more important people who can later recommend you for progressively more interesting and impactful fellowships.

I’m going to be a bit explicit about this pipeline (and risk offending some people who disagree with me) because even if I get the details wrong, I think it’s valuable to communicate that there is a pipeline, and applying to a fellowship without enough legible skills will possibly result in you feeling despondent and low-value when you’re rejected. Skipping the earlier stages of the pipeline is possible and I encourage people to try. But I think it’s important to communicate that your odds of getting into the Anthropic Fellows Program are not the same as your odds of getting into SPAR. Here’s Opus 5.5’s rough ranking, and I approximately agree: https://claude.ai/share/9ecdbf1f-604c-426b-a4ca-da58545ac8cd.

If you feel bitter because you’ve been rejected and you believe the fellowships only accept Ivy League graduates, my advice is to make your skills more legible. Being an Ivy League graduate is a very legible indicator of skill, but so are options such as earlier-stage AI safety fellowships or getting involved with (or starting!) your local AI safety interest group.

Different streams are more or less competitive

I can’t speak for every fellowship, but generally you don’t apply to just The Fellowship, you actually apply to multiple streams within The Fellowship. This is very consequential! When submitting your application, you’re not competing with everyone who applied to The Fellowship, you’re just competing with the people who applied to the stream. So while MATS is extremely competitive, your preferred stream within MATS might not be as competitive. This is especially the case if you’re interested in relatively niche ideas for that fellowship that probably get fewer applicants. If you see a stream that you think fits your interests very well but you think you’re underqualified for the fellowship, consider applying anyway.

Apply to multiple streams and to multiple fellowships

Fellowship applications are noisy, so if your skills are not maximally legible you’ll probably be rejected even though you shouldn’t have. Similarly, someone else will likely be accepted even though they should have been rejected. There’s a fair amount of luck involved, and applying to more fellowships gives you more chances at being lucky.

Luckily, there are several fellowships, and each of them has several streams. Do not spam the applications process with pointless applications. But please do apply to more than one stream or fellowship if multiple options seem to be a good fit. Making your skills more legible increases your chances of being accepted into any single fellowship or stream, but applying to multiple fellowships/streams gives you more chances to get lucky.

It’s a bit tricky to give advice on how many streams you should apply to. Doing the application is usually a lot of work, and doing it well is more on top of that. I can maybe give advice on when to not apply to a stream: if you’re answering the questions or doing the take-home assessment and you just can’t be bothered, you notice yourself struggling to find the motivation, or there are just many other things that seem interesting in that moment, you should consider not finishing the application. Most streams or fellowships will have questions on the application that are somewhat similar to the kind of work you’ll do during the stream/fellowship, so if you’re struggling for motivation to answer the questions, you’ll likely struggle for motivation during the fellowship itself.

The mentors (usually) make the final decision

As said above, for some fellowships, you actually apply to a particular stream and then the mentor(s) of that stream make the acceptance decision. If possible, you should heavily tailor your application to your mentor’s interests. Don’t be fake or deceptive! But if (for example) you did a small project looking at the security implications of hardware side-channels but you normally exclude this from your applications because it’s not usually relevant to AI safety, but the mentor you’re applying to happened to also be interested in side-channels as they relate to ASI, then you absolutely should customise your CV and application to describe this project on hardware side-channels.

Like all things, this is about making your skills legible to the person reviewing your application. Most people don’t understand hardware side-channels so there’s no point in including the project in every application you make. But if a reviewer has prior experience in side-channels, they’re more likely to see the skills required for your particular project, so describing your side-project is a very legible way of showing them how good you are.

In general, if you’ve got experience in some niche field and your potential mentor happens to have previous work in that field, you should consider highlighting this in your application. With this in mind, you should try to read as much as you can (with your eyeballs, not Claude’s!) about the person who’ll be reviewing your application and making the final decision. This includes their blog, online profiles and research papers.

Don’t make the mistake of faking alignment with your mentor’s values. At best this will get you into the fellowship you want but then you’ve got to spend three months doing something you dread. There are better things to do than waste your own time pursuing a goal you don’t care about. Gaining prestige in a field you don’t care about is like climbing the wrong mountain and being surprised that you didn’t get the view you wanted.

If you’re a “risky” option, try start with low-commitment options

If you think you’re skilled but can’t seem to get anyone’s attention, it might be that you’re too high-risk at the moment. Mentors might be looking at your application and thinking “this person will either be great or be atrocious, I can’t risk 3 months of my time on that”. This is a rational decision on their part, and you can change their mind by becoming less risky. It can be hard to know if you’re a risky option, it requires putting yourself into a reviewer’s shoes and looking through your application critically. Becoming more legible can often make you less risky, for example doing another less-prestigious fellowship or working on a related personal project.

But how do you get into a fellowship if this requires having already gotten into a fellowship‽ The answer is that you start with the low-stakes fellowships (online, shorter, no funding) and progress towards higher and higher stakes fellowships (in-person, longer, more funding).

It can be tricky to take a lower-stakes option because it implies that you were less capable than you thought you were. You might have the skills to do great things, but if you cannot convince other people that you have these skills then you will not be given the opportunity to do these great things. The good thing is that there’s always a lower-stakes option that you can do to make your skills more legible, regardless of what you get rejected from. I strongly recommend against applying to the same position over and over again unless you’ve significantly improved yourself and made your skills more legible in between the different applications.

Lower stakes options might look like doing some or all of the ARENA course in your free time on Google Colab GPUs, or publicly writing up your thoughts on a paper you read (bonus points if you disagree with the author’s decisions). I recommend ARENA because it is very highly regarded in the AI safety community but there’s no application process in between you today and the you who’s completed the syllabus. Average Joe has not done ARENA by themselves. I recommend writing your thoughts because many AI safety research questions ask you to read a particular paper and then either suggest follow-up experiments or to critique the paper. If you’ve got public critique of papers or have run follow-up experiments on other experiments you thought might be interesting, this is incredibly valuable to people trying to assess whether you’re a good fit. Average Joe has not bothered to write substantive feedback on popular research papers.

Consider an 80,000 Hours advising call

I applied and got a call, and found this very helpful in getting a third party to highlight my weaknesses and confirm my strengths. The advisor also offered various follow-ups with different industry professionals who would have been extremely useful, had I not been accepted into MATS shortly afterwards.

Fellowships are ~constantly accepting new applications

While they don’t literally have rolling applications (yet) the frequency for the big fellowships is much more than annual. If you want to apply, there’s no need to wait until the applications actually open for you to prepare your CV. You should start now to get your CV and thoughts in order, instead of waiting for the applications to open and then having to grind to get things ready in time. You can pre-emptively look at mentors from previous cohorts (they’ll likely mentor again) to see who you want to apply to. You can also look at research papers from mentors you admire to see what sort of work they do and whether you think you’d enjoy doing that same work.

Your success is roughly proportional to your effort per application

You can definitely always put more effort into manually (not with an LLM) looking through everything about an application. You should heavily study the mentors on the streams you’re excited about, and read their recent papers. If you can be critical while doing so (“why did they choose X hyperparameter?”, “why didn’t you consider Y?”) this is even better, most mentors appreciate thoughtful critique and discussion of their work.

This is definitely a rabbit hole for some mentors, there’s always more you can do and more you can learn about them. I’m not sure when is the right place to stop, but you should be trying to discover if the work they do is something you’d like to do yourself. If you’re not excited by their research, you probably don’t want to apply to work on their stream, even if they’re “famous”.

Look through ~all the mentors’ pages

This is a bit of a nightmare. For the bigger streams there are often dozens of mentors, and there are often a half-dozen fellowships. I’m not going to pretend it isn’t a lot of work, but it is hard to avoid if you want to be selective with your time. I relied on heavily filtering early on (e.g. I had to completely ignore governance and non-US options in order to make the workload manageable) and from there I just spent a lot of time reading their biographies and websites.

Luckily, this is a process you can start in advance, before applications for that fellowship technically open. Even if the “official” mentor pages haven’t opened up yet, many fellowships will have a list of their previous mentors (e.g. https://www.matsprogram.org/mentors) and probably you can get Claude to surface useful information like previous papers and research interests.

If you get through to a later interview stage, it’s probably worth taking a day off of work

I forget when, but at some point I had an enormous number of stream applications due in just a few days’ time window. I was working full time, and most of the applications assume you’re able to answer their questions 24/7. I ended up taking a sick day off of work and just read & wrote applications the whole day. Looking back, I think this was well worth my time to do, although I’m not sure if I’d have felt differently if I hadn’t gotten into MATS. Because I was fresh and well-rested, I managed to get a lot more done than I would otherwise and that particular day at work wasn’t very special or unique.

Rejection feels like shit

It’s really not fun. It hurts, and makes you question whether you should bother doing anything to get better. To the extent that you can, I don’t think you should beat yourself up about getting a rejection letter. But it is hard, and many rejection letters do very little to cushion the blow or suggest productive next steps.

After taking a break and clearing your head, I recommend trying to review all the answers you gave to all the questions. Review your CV, your references, and your online presence. Ask how you can change these things to be closer to a promising AI safety researcher. Often there’s a several-month gap between one round of AI safety applications and the next, so I encourage you to be ambitious with what you can do in those several months.

The goal here is to learn from the application process, and to try to avoid feeling dejected. You have incredibly valuable information now about what does work and what doesn’t. You should do your best to learn from this information such that you can spend the next few months doing ambitious projects that improve your chances for the next cycle. Try to get involved with a local AI safety org (or think about starting one). Make your work more legible.

You should probably self-study the ARENA curriculum

I knew ARENA was good, but if you’ve got some time to work through the content on the ARENA schedule, it’s seen as very high signal and many applications explicitly ask if you’ve done ARENA. They didn’t seem to strongly discriminate between doing it on your own online or in-person. If I hadn’t gotten into MATS, working through the ARENA coursework on my own would have been my top priority.

A long list of application questions

Here are all the application questions I answered, slightly deduplicated, anonymised, and sorted arbitrarily. If you’re wanting to apply for an AI safety fellowship but there are none available, I strongly recommend going through this list and either answering each question, or considering what you can do to make your future answer to each question the best it possibly can be.

Show all the questions

• Are there any previous projects or experiences you’d like to highlight as especially relevant? For e.g. you can highlight if you’ve previously participated in fellowships such as PIBBSS, MATS, ERA, Pivotal, Talos, or similar programs and include the dates in which you did so.
• Are you interested in working in a team of four talented, conscientious people (plus us as mentors)? We think people who answer “yes” to this question are probably a better fit for our stream, but it is not a strict requirement.
• Could you give us a sense of what your longer-term career paths might be? This will help set the agenda for our call, so think of at least two different career paths you could see yourself pursuing and list some pros and cons for each.
• Describe a time when you faced a non-technical challenge collaborating with other people on a complex project. How did you address it? (Suggested length: ~250 words) Feel free to share other information that would help us understand what it’s like to be your teammate.
• Describe a time where you had to demonstrate high agency as described here? (100 - 200 words) This refers to a time where you took initiative to solve a problem, create something new, or drive a project forward despite obstacles or uncertainty. This could be academic, professional, or personal.
• Describe your current career plans and aspirations
• Explain your strongest disagreement with other alignment thinkers. Consider only your own inside-view understanding, and don’t defer to others’ expertise. (Suggested length ~250 words, but it’s okay to write more)
• For [Multi-Agent safety, AI for Facilitating Human Cooperation, Gradual Disempowerment], describe the research you’d like to undertake during the fellowship and why you think it is important?
• Give an example of an empirical AI safety research project you’d be interested in working on, and why? Please mention any relevant experience if applicable. Max 300 words
• Have you completed Chapter 0 of the ARENA curriculum, or, if you join our stream, would you commit to completing it before the start of the program (Jan 5)?
• Have you done surveys/literature reviews? Or show some evidence on how fast you can read papers, and what kind of useful takeaways you usually obtain from papers. Put “N/A” if this question is not applicable.
• How experienced are you with tuning hyperparameters: Write about your hyperparameter tuning experiences. For LLMs, what tricks do usually try to make them work at a target task?
• How long do you configure the deep learning environment: Describe one of your past experience for the most complicated server that you’ve set up the deep learning environment on. What did you do, and how long does the entire configuration take? (Basically configuration here means everything before you can start to run the program.)
• How will [Fellowship] help you to achieve your long-term career aspirations? What are you hoping to get out of the program? (max. 800 characters)
• If you’re open to working in a team, briefly reflect on how you might make this go well. What would a great collaboration look like to you? In your answer, you might draw on prior experience in teams, but you don’t have to. (max 250 words).
• Imagine it’s 2027 and you completed [Fellowship] a year ago. What specific role or project are you now working on that represents your highest-impact contribution to AI safety or governance? (100-200 words)
• In 1-2 sentences per project, please provide a short summary of the project and what you contributed to it.
• Is there anything in particular you’re looking to get out of our potential collaboration? It’s okay to leave this blank. In that case, we’ll assume your answer is “I want to grow as an ML researcher and make contributions to AI safety.”
• Key factors of AI progress in the next ~5 years? (max. 1500 characters)
• Link 5 papers or blogposts relevant to AI control, oversight or model evaluations that you’re excited about (they don’t have to be your own). We may ask you about these in interviews.
• Link to a personal blog post
• Link to code samples from past projects (e.g. GitHub links or uploaded zip files), ideally a substantial project and (if possible) a machine learning project (these could be the same or different projects).
• Links to things you’ve done or your experience
• Mitigating Gradual Disempowerment. Develop concepts and tools that will allow us to develop mitigations to preserve human agency and ensure that our institutions serve us. (500 - 1000 words, 3 hour proctored time limit)
• Other notable skills or experience
• Please add a link to 1–2 writing samples, code repositories, or similar that are illustrative of your knowledge or skills relevant to the fellowship, and then briefly give any important context on this piece of work (e.g. whether other people helped produce it).
• Please describe a hard technical or research problem you solved. What was the bottleneck and how did you overcome it? What impact did solving the problem have? (200 words)
• Please link impressive AI projects that you’ve built.
• Please provide 1-2 code samples from past projects (e.g. GitHub links or files).
• Please provide links to any of your projects or posts related to AI safety.
• Please share any academic, professional, or personal accomplishments you are proud of. (max 200 words)
• Please share any other information about your background in ML that you would like us to know.
• Please talk briefly about an area of technical work right now you’re most interested in or excited about, and why. (~3 sentences)
• Propose a follow-up experiment to section 3 of this paper and explain the relevance of the experiment to AI safety efforts. (Suggested length: ~250 words for experiment, 1-4 sentences for relevance.)
• Provide two reference contacts. These contacts should ideally relate to your past AI, AI safety, research, engineering, and/or other relevant experience.
• Question: What do you consider to be your top achievement(s)? (Max. 50 words) Please share 1-3 achievements that you are especially proud of. These could be scholastic, like your academic performance at university or in an international olympiad, and extracurricular, like winning a competitive sporting event.
• Read the following report by Model Evaluation & Threat Research (METR): Measuring AI Ability to Complete Long Tasks. For additional technical details, you can see the full-length paper here. In section 7.2.1 of the full paper, the authors discuss “systematic differences between our tasks and real tasks”, including automatic scoring, no interaction with other agents, lax resource constraints, unpunishing, and static environments. Which one of these systematic differences do you expect to have the largest impact on the generalizability of the study’s results? Explain your reasoning. (1600 characters)
• Take a look at the research projects for [Mentor]‘s stream that I’m interested in (Reasoning Model Interpretability, Red-Teaming and Elicitation, and Model Organisms). What are 1-3 pieces of evidence that you’d be able to do good research in this stream? They don’t have to be standard credentials! Please describe them and why they’re relevant. Aim for conciseness (e.g. a single page is likely sufficient). Feel free to write up a high level plan for a research project you’d want to work on as evidence that you are able to generate and/or flesh out interesting directions that we could work on together.
• Tell us about a recent achievement where you displayed proactivity, grit, and self-motivation
• Tell us about any interest or involvement you’ve had in effective altruism.
• Tell us about your most significant technical accomplishment.
• What (if any) relevant experience do you have for working in the track you have selected? (750 characters)
• What NLP/general AI tasks do you usually read papers about, e.g., AI safety, LLM interpretability, LLM for coding, computational social science, etc. (and how many papers for each task, as a rough estimation)?
• What are 1–2 reasons why you would be a good fit to do empirical AI research? Please give a concise description of each piece of evidence and explain why it’s relevant. Max 200 words
• What are some (possible) long-term career goals for you (e.g., a bit like “where do you see yourself in your 40s”)?
• What are your career aspirations? (100 - 250 words) Please outline 2-3 potential career paths you’re considering and explain how this fellowship fits into your trajectory.
• What are your long-term career aspirations? (max. 800 characters) We ask this to better understand your interests and motivations. We know that career plans often evolve and might be broad at this point, so feel free to express uncertainty.
• What are your long-term career goals
• What attracts you to [Company]‘s mission of protecting humanity from AI risks?
• What concerns you the most about AI development?
• What deep learning models have you coded before (e.g., adaptations of open-weight LLMs, VLMs, etc)? If applicable, support your answer with your GitHub repos
• What do you hope to gain from participating in [Fellowship]? (200 words)
• What do you think is the most promising way to gain helpful information about the probability that advanced AI’s are scheming right now? (<300 words)
• What do you want to work on? Please give a 3-5 paragraph pitch for your research idea that fits this stream.
• What have been your preferred sources for learning about AI Safety topics? Please be specific (e.g., name particular researchers, papers, podcasts, courses, books rather than general categories).
• What is a result in AI Safety, Governance, and/or ML that you have found interesting/surprised you? Did it make you change your mind about something? (max. 1000 characters)
• What is an important opinion or idea about AI that you wish more people understood? Please explain the opinion or idea in 200 words or fewer. Separately, in 150 words or less, explain why you wish more people understood it and why that might be important.
• What is one potential project or research question you would be interested to work on as part of [Fellowship]? (2000 characters) You don’t need a fully scoped research idea at this stage, and nothing you write here is a commitment. If you’re accepted to the programme, your project can evolve substantially (and may differ entirely) after you meet your research manager and mentors. However, it is a useful exercise to brainstorm what you could actually work on, and it helps us understand your research interests. Please make an effort to highlight key sub-questions and the theory of change (how would the research ultimately lead to reducing large-scale catastrophic risks from advanced AI?).
• What is your aim for this research collaboration/internship? Answer using bullet points the following:
• What is your favourite piece of research/theory? Why?
• What is your motivation for applying to [the Empirical Research] workstream? Max 150 words
• What is your motivation to apply to this programme? What do you hope to get out of it?
• What knowledge areas are you familiar with, or do you like to spend your time thinking about? If applicable, share a link of your notes etc.
• What previous research experience do you have? Please feel free to provide references to your work. (Max. 100 words)
• What previous research experience do you have? Prior research experience in fields like ML/CS/math/philosophy may play a role in your ability to do alignment research. That said, if you have done any research in any field that you are proud of, we would also like to hear about that. Feel free to refer to your LinkedIn/resume.
• What’s the most awesome thing you’ve ever done?
• What’s your experience with AI safety? If you have any projects or posts, please provide links when able. If you have no experience with AI safety research in particular, please feel free to leave this section blank.
• What’s your motivation for applying to this stream and project? (max 100 words)
• Why do you want to be part of [Fellowship]? (150 - 300 words) Here you can mention (but are not required to) your personal motivation for AI safety work, why [subfield] research specifically interests you, and how participating now fits your longer-term goals for contributing to the field.
• Why do you want to work at [Company]?
• Why do you want to work in AI Safety?
• Why do you want to work on reducing risks from advanced AI through this fellowship? (1200 characters)
• [50-100 words, max 300] What are 1-3 pieces of evidence that you’d be able to do good research in these streams? (These don’t have to be standard credentials!) Please concisely describe them and why they’re relevant
• [Your Previous Papers & Coding Skills] If applicable, list your past projects/papers, where each line includes (1) link to the paper (published and unpublished writeup), (2) conference and year (if applicable), and (3) part of the code that you programmed (with GitHub links, and notes about which parts you implemented)
• [≤ 150 words] Which topic(s) would you be excited to work on as part of these streams and why? Feel free to select one of the example project ideas provided in the stream descriptions or propose your own.
• [≤ 200 words] Describe a research project that you have done, what went well and some takeaways for improvement.

Conclusion

I hope this helps some people make their skills more legible and increases the talent pool that goes towards AI safety. I think communicating your skills is a hard problem, and often there are little to no mechanisms for feedback when you do poorly.

I do wish this process weren’t so adversarial, to the extent where I have to recommend meta-gaming and reasoning about the grader reviewer. I do think there’s a better way to structure applications that removes a lot of these inherent problems, but I’m not going to pretend like everyone will “just” get better at job applications.

Footnotes

1. The connection between this and LLM meta-gaming is left as a ponderance for the reader. If you are reading this as someone who reviews applications, I’d like to also raise the connection between attempts to clamp down on meta-gaming and LLMs just becoming better at hiding their meta-gaming. Applying for high-status fellowships is inherently a competitive, adversarial environment and if you require applicants to play it you should not be surprised when they succumb to Moloch. ↩

2. Hello there, powerful person who is reading this. ↩

3. This is a real question by the way, completely verbatim. ↩ ↩² ↩³

4. You could say they match byte-for-byte. ↩