Jeffrey Lee
AI Biosecurity Evaluations Research Scientist, RAND
-
Jeffrey Lee is an AI biosecurity research scientist at the RAND Center on AI, Security, and Technology where he is a biology subject matter expert. He is a molecular biologist with a background in global health, synthetic biology, and science policy. His work primarily focuses on designing evaluations that assess AI systems to better understand the potential risks from maligned use of highly-capable AI models. Lee was a former Technology and Security Policy Fellow at RAND.
-
Model Chaining in AIxBio Evaluations
AIxBio evaluations are often implemented to assess how a single model performs. However, in a real-world setting, actors may choose to use multiple LLMs (or LLM-agents) to obtain an answer or design outcome, perhaps chaining together several LLMs and feeding outputs from one into another. In this project, we will examine how model chaining impacts performance on biology, and time-permitting, biosecurity relevant benchmarks. Mentees will also have the opportunity to design their own evaluation to use for this project.
Activity Scanner Tool for Risk Assessment of Logs (ASTRAL)
Current model log scanners are geared more toward isolated features and are customized for specific uses. In this project, we will build an activity scanner tool for risk assessment of logs (ASTRAL) aimed at automatically extracting key information such as the nature of the threat, a possible description of the actor, assign a risk level, and other elements of risk assessment. Mentees will then generate different multi-turn conversations with an LLM around dual-use biology research using different personas and objectives. They will then deploy ASTRAL to characterize the logs and provide a preliminary actor profile and risk assessment that can be compared against the initial personas and objectives.
-
Model Chaining in AIxBio Evaluations
Mentees will be a good fit if:
They are critical thinkers
They have experience in biology, biosecurity, or are in an adjacent field and highly interested
They are independent workers and self-motivated, but can also be a team player
They have experience in AIxBio evals - either designing or implementing (especially with the Inspect platform)
Activity Scanner Tool for Risk Assessment of Logs (ASTRAL)
Mentees will be a good fit if:
They are critical thinkers
They have experience in threat modeling or a security-mindset
They have experience in biology, biosecurity, or are in an adjacent field, and are highly interested
They are independent workers and self-motivated, but can also be a team player
They have experience building tools or scanners