About the AI Safety Institute 

The AI Safety Institute is the first state-backed organisation focused on advancing AI safety for the public interest. We launched at the Bletchley Park AI Safety Summit in 2023 because we believe taking responsible action on this extraordinary technology requires a capable and empowered group of technical experts within government.  

We have ambitious goals and need to move fast.  

1. Develop and conduct evaluations on advanced AI systems. We will characterise safety-relevant capabilities, understand the safety and security of systems, and assess their societal impacts. 
2. Develop novel tools for AI governance. We will create practical frameworks and novel methods to evaluate the safety and societal impacts of advanced AI systems, and anticipate how future technical safety research will feed into AI governance. 
3. Facilitate information exchange. We will establish clear information-sharing channels between the Institute and other national and international actors. These include stakeholders such as policymakers and international partners. 

Our staff includes senior alumni from OpenAI, Google DeepMind, start-ups and the UK government, and ML professors from leading universities. We are now calling on the worlds top technical talent to join us. This is a truly unique opportunity to help shape AI safety at an international level. 

As more powerful models are expected to hit the market over the course of 2024, AISIs mission to push for safe and responsible development and deployment of AI is more important than ever. 

 What we value: 

• Diverse Perspectives: We believe that a range of experiences and backgrounds is essential to our success. We welcome individuals from underrepresented groups to join us in this crucial mission. 
• Collaborative Spirit: We thrive on teamwork and open collaboration, valuing every contribution, big or small. 
• Innovation and Impact: We are dedicated to making a real-world difference in the field of frontier AI safety and capability, and we encourage innovative thinking and bold ideas. 
• Our Inclusive Environment: We are building an inclusive culture to make the Department a brilliant place to work where our people feel valued, have a voice and can be their authentic selves. We value difference and diversity, not only because we believe it is the right thing to do, but because it will help us be more innovative and make better decisions. 

ABOUT THE SAFEGUARDS TEAM

As AI systems get more powerful and integrated into society, the safeguards that prevent a bad actor from misusing a system will become more important. The AI Safety Institutes Safeguard Analysis Team assesses the susceptibility of advanced AI systems to attack, including by testing system safeguards. 

The Safeguard Analysis Team is different to other AISI research teams it evaluates attacks that unlock unsafe model capabilities, as opposed to evaluating unsafe capabilities themselves. The team takes a holistic view of security, meaning both ML-based and traditional security attacks and protections are in-scope for the Teams work.

The Teams priorities include research-oriented responsibilities like assessing the threats to frontier systems and developing novel attacks and engineering-oriented ones, such as building infrastructure for running evaluations.  

ABOUT THE ROLE 

The Safeguard Analysis Team is hiring specialists with skillsets that lean in the direction of either or both of Research Engineer and Research Scientist. We're hiring for all of our sub-teams:

1. Emerging Threats research how current and future advances in the functionality of frontier systems changes AISIs assessment of their security against misuse.
2. Information Security Protocols analyse the information security protocols used to protect frontier systems, evaluating them for flaws and assessing how the choice of protocol affects the systems risk profile.  
3. Controls and Mitigations develop recommendations for measures that control and mitigate the risks from frontier systems being misused, considering in-scope a range of sociotechnical interventions by various actors. 
4. Evaluations & Red Teaming build and run AISIs suite of evaluations to measure how easily the protective measures of frontier AI systems can be defeated.

Were hiring at various seniority levels within the team, including in Senior Research Engineer / Research Scientist positions. Calibration on final title, seniority and pay will take place as part of the recruitment process.

As a Research Scientist/Research Engineer in the Safeguard Analysis Team, you will be part of the team that defines AISIs approach to measuring and improving frontier AI security.  You will contribute to a research portfolio that makes frontier AI systems more secure, with responsibilities that could include: 

• Designing or executing AISIs safeguard evaluations against frontier AI systems and analysing their outcomes.
• Contributing to AISIs strategy for measuring and improving frontier AI security, including developing AISIs threat models.
• Developing novel attacks against AI systems and exploring their mitigations.
• Understanding the space of AI controls and mitigations and investigating which defences may be best placed to reduce risk.
• Building tooling to allow more rapid and reliable evaluation exercises.
• Undertaking research projects to make progress on relevant open questions, like predicting future model vulnerabilities.
• Working closely with the Safeguard Analysis Workstream Lead, Research Engineers and other Research Scientists.
• Collaborating with external experts, contractors, partner organisations and policy makers to coordinate and build on external research.

A strong applicant for this role will have conducted technical research in machine learning, computer system security, or information security.  

We expect experts at the intersection of these areas to be rare, so we encourage you to apply no matter what research expertise youre excited to bring to the Institute.  

We look for some of the following skills, experience and attitudes: 

• Experience in industry, relevant open-source collectives, or academia in a field related to machine learning, AI, AI security, or computer security.
• Strong Python skills.
• Strong written and verbal communication skills.
• Experience with large language models, potentially related to prompt engineering, tooling, or fine tuning.
• Excellent knowledge of training, fine-tuning, scaffolding, prompting, deploying, and/or evaluating current cutting-edge machine learning systems such as large language models.
• Record of building software systems to meet research requirements and have led or been a significant contributor to relevant software projects, demonstrating cross-functional collaboration skills.
• Knowledge of statistics.
• Ability to work in a self-directed way with high agency, thriving in a constantly changing environment and a steadily growing team, while figuring out the best and most efficient ways to solve a particular problem.
• Bring your own voice and experience but also an eagerness to support your colleagues together with a willingness to do whatever is necessary for the teams success and find new ways of getting things done within government.
• Have a sense of mission, urgency, and responsibility for success, demonstrating problem-solving abilities and preparedness to acquire any missing knowledge necessary to get the job done.
• Improving scientific standards and rigour and/or technical standards across a team, through mentorship & feedback.
• [For engineers]: At least 6 years of experience writing production quality code and designing, shipping, and maintaining complex tech products. Given the changing nature of the field, its most of all important to us to build a team with strong problem-solving skills and a preparedness to acquire any missing knowledge necessary to get the job done. 

CORE REQUIREMENTS

• You should be able to spend at least 4 days per week working with us
• You should be able to join us for at least 12 months
• You should be able work from our office in London (Whitehall) for parts of the week, but we provide flexibility for remote work

The Department for Science, Innovation and Technology offers a competitive mix of benefits including:

• A culture of flexible working, such as job sharing, homeworking and compressed hours.
• Automatic enrolment into the Civil Service Pension Scheme, with an average employer contribution of 27%.
• A minimum of 25 days of paid annual leave, increasing by 1 day per year up to a maximum of 30.
• An extensive range of learning & professional development opportunities, which all staff are actively encouraged to pursue.
• Access to a range of retail, travel and lifestyle employee discounts.
• The Department operates a discretionary hybrid working policy, which provides for a combination of working hours from your place of work and from your home in the UK. The current expectation for staff is to attend the office or non-home based location for 40-60% of the time over the accounting period.

Job contact :

• Name : [email protected]
• Email : [email protected]

Recruitment team

• Email : [email protected]