Sunishchal Dev is an AI Evaluations Research Scientist at RAND CAST (Center on AI, Security, and Technology) where he works as an ML engineering lead. He has worked on building benchmarks for LLMs & Biodesign Tools, evaluating biosecurity implications for open weight LLMs, leading science of evaluations research (LLM autograders and human baselining), and doing frontier model risk assessment. Dev comes from a decade long career doing data science for climate change solutions and management consulting for enterprise AI implementations.

Jeffrey Lee is an AI biosecurity research scientist at the RAND Center on AI, Security, and Technology where he is a biology subject matter expert. He is a molecular biologist with a background in global health, synthetic biology, and science policy. His work primarily focuses on designing evaluations that assess AI systems to better understand the potential risks from maligned use of highly-capable AI models. Lee was a former Technology and Security Policy Fellow at RAND.

Model Chaining in AIxBio Evaluations

AIxBio evaluations are often implemented to assess how a single model performs. However, in a real-world setting, actors may choose to use multiple LLMs (or LLM-agents) to obtain an answer or design outcome, perhaps chaining together several LLMs and feeding outputs from one into another. In this project, we will examine how model chaining impacts performance on biology, and time-permitting, biosecurity relevant benchmarks. Mentees will also have the opportunity to design their own evaluation to use for this project.

Activity Scanner Tool for Risk Assessment of Logs (ASTRAL)

Current model log scanners are geared more toward isolated features and are customized for specific uses. In this project, we will build an activity scanner tool for risk assessment of logs (ASTRAL) aimed at automatically extracting key information such as the nature of the threat, a possible description of the actor, assign a risk level, and other elements of risk assessment. Mentees will then generate different multi-turn conversations with an LLM around dual-use biology research using different personas and objectives. They will then deploy ASTRAL to characterize the logs and provide a preliminary actor profile and risk assessment that can be compared against the initial personas and objectives.

Model Chaining in AIxBio Evaluations

Mentees will be a good fit if:

• They are critical thinkers

• They have experience in biology, biosecurity, or are in an adjacent field and highly interested 

• They are independent workers and self-motivated, but can also be a team player

• They have experience in AIxBio evals - either designing or implementing (especially with the Inspect platform)

Activity Scanner Tool for Risk Assessment of Logs (ASTRAL)

Mentees will be a good fit if:

• They are critical thinkers

• They have experience in threat modeling or a security-mindset

• They have experience in biology, biosecurity, or are in an adjacent field, and are highly interested

• They are independent workers and self-motivated, but can also be a team player

• They have experience building LLM evaluation tools or scanners (autograders/LLM judges)