Evaluating AI-assisted social engineering

This project aims to evaluate AI-assisted social engineering and create novel defense mechanisms such as personalized spam filters and agentic scam alerts. We seek to scale up our AI-phishing research to populate our evaluation benchmark (https://scambench.com/) with new and more diverse data. We are also launching economic cost analyses of how AI changes the incentive structure for attackers and defenders, and creating tools and support programs specifically aimed at vulnerable population groups. For more information, read our paper on evaluating AI-powered scams.

This project can recruit both technical and governance fellows. Below are some representative tasks one can expect during the fellowship period.

Technical Candidate (ML Engineer)

• Expand our AI agents to include voice and SMS-based phishing simulations.

• Improve the OSINT functionality of our scam-training tool to accommodate diverse user demographics (e.g., students and seniors).

• Review and refine our current AI tool’s persuasion techniques, and conduct experiments on how to measure and mitigate different persuasion attack types..

• Create a blue team AI agent that tests the effectiveness of our scam-generating (red team) agents.

• Investigate how social-engineering techniques integrate into the broader chain of automated cyberattacks, including technical intrusion methods such as those documented in Anthropic’s recent AI-espionage case: https://www.anthropic.com/news/disrupting-AI-espionage

Governance and Policy Candidate

• Create a plan for how to better align ScamBench with AISI’s guidelines for evaluation benchmarks (https://www.gov.uk/government/publications/ai-safety-institute-approach-to-evaluations/ai-safety-institute-approach-to-evaluation).

• Map relevant compliance, legal, and ethical requirements for scaling and working with ScamBench (e.g., data protection, privacy, and consent frameworks).

• Map out and implement strategies to recruit participants for ScamBench to scale its data and reach.

• Review and improve our metrics for different types of persuasion techniques and scams (such as the grandchild-in-distress, EZ Toll, and Mastercard verification scams).

• Develop and implement a plan to ensure comprehensive demographic representation.

Technical Candidate (ML Engineer) – Qualifications

Required

• Background in computer science.

• Familiarity with AI development, equal to having completed Arena (https://github.com/callummcdougall/ARENA_3.0).

Bonus

• Web development experience.

• Interest in cybersecurity, social engineering, and human-computer interaction.

• Experience with Docker, Kubernetes, and API integrations.

Governance and Policy Candidate – Qualifications

Required

• Prior experience in community engagement or nonprofit outreach, and building strategic partnerships.

• Ability to conduct structured interviews, including calling or meeting with seniors, caregivers, or other target groups to gather qualitative data.

• Experience in gathering user feedback, such as designing and conducting short interviews, surveys, or feedback sessions with users (including seniors and caregivers) to assess the effectiveness of our AI-phishing tool and ScamBench, identify usability issues, and collect insights to improve training materials and benchmarks

• Comfort in learning the basics about legal compliance for data management. For example, ensuring we maintain compliance with data protection and privacy requirements across different US states, streamlining verification processes, ensuring ethical consent collection, secure data storage, and anonymization practices, and staying up to date on relevant laws and ethical standards for handling sensitive user information.

Bonus

• Experience from working with vulnerable population groups.

• Organizational and project management skills, such as scheduling participant sessions, keeping track of datasets, and coordinating research tasks.