Blog Press Careers Contact Us
Announcements ROOST partner
Mila’s Suicide Prevention Guardrail: Lightweight, Open Source Safeguards for Real-Time Detection

Mila’s Suicide Prevention Guardrail: Lightweight, Open Source Safeguards for Real-Time Detection

Published

More young people than ever are talking to AI. They use it for homework and company, and sometimes for the things they cannot say out loud to anyone else. This shift puts a new kind of responsibility on the systems they talk to: when a young person signals that they are struggling, the response they get back matters, and most systems aren’t yet built to handle that moment with care.

To meet this challenge with the transparency and independent rigor it deserves, ROOST and Mila last month announced their partnership at the G7, where world leaders came together to emphasize the importance of youth safety and open source.

Today Mila and ROOST are jointly releasing Mila’s Suicide Prevention Guardrail in beta – a real-time, open source safety model that prevents AI chatbots from encouraging or directing youth suicide. This low-latency BERT model is a fine-tuned classifier that can run locally, even on an ordinary CPU, at very low cost. It's the first of a series of classifiers built in partnership with Mila’s AI Safety Studio, aimed at creating safeguards for multiple harm categories and across a user’s interaction with an LLM. Targeting exclusively model outputs, this single-turn, light weight guardrail is stackable for chatbot developers looking to add an additional layer of protection around suicide prevention in their applications.

By contributing the guardrail to the public commons, alongside the synthetic data generation prompt, ROOST and Mila are making chatbots safer and tackling a crucial part of child safety.

Key Features

Narrow by design: This classifier is built specifically as a high-recall, low-footprint narrow safety add-on filter, to detect suicide and related self-harm encouragement and suicide assistance instructions in AI responses, not a general-purpose safety moderation filter. Generic safety models are tuned to catch broad categories like harassment or explicit content, but are not always optimally dispositioned to catch the language of AI responses in the specific context suicide.

Fast and lightweight: This single-turn BERT model is small enough to run on commodity hardware, including locally on CPU. Its low-latency design returns a fast decision with no reasoning chain to wait on, so teams can embed it directly in a chat pipeline or run it on-device. This minimal footprint also makes it ideal for stacking alongside other existing model guardrails – using high-recall, safeguards for specific high-risk topics to create “safety-in-depth.”

Configurability, multi-linguism, and inclusion: To give developers full control over their safety preferences, the model’s tolerance to false negatives and false positives can be configured according to the use-case and developer tolerance. The methodology used to develop this model is also highly customizable and portable to other languages or specific social contexts, as part of Mila and ROOST’s commitments to building AI for everyone.

Part of a growing safety toolkit

Mila's Suicide Prevention Guardrail joins the ROOST Model Community (RMC), where it sits alongside other open safeguards like OpenAI's gpt-oss-safeguard. Together, the community gives developers a range of detection models and safety guardrails to draw from, rather than a single fixed option, so teams can pick the safeguard that fits a given job, or layer several, and compose the protection their platform actually needs.

Building and evaluating in the open

This marks an important first step in open progress toward model safety, not a finished answer. Mila and ROOST ran several early evaluations, and the results are encouraging. On a held-out set of 1200 synthetic examples, the fine-tuned classifier scored near-perfect precision and recall (F1 0.99), and on a smaller handcrafted set of indirect and coded cases it held up well, with recall around 0.89. Moreover, the the model’s recall is consistently outperforming general purpose safety guardrails on the specific topic of suicide assistance. Full evaluation details are in the model card.

There is more work to do, including by ensuring that the dataset is effective on real output dataset, improving it to better capture context and support multi-turn conversations, and even extending the capabilities of the guardrail beyond detection to have it instead propose alternative safe responses. There is more work to do, including by ensuring that the dataset is effective on real output dataset, improving it to better capture context and support multi-turn conversations, and even extending the capabilities of the guardrail beyond detection to have it instead propose alternative safe responses. These were critical issues raised through Mila’s work to engage with specialists, clinicians and scientific experts, and during its national hackathon with leading Canadian suicide helplines. Still, we are releasing it now, early, because the best way to build safety infrastructure is out in the open, with the people who will use it, who will interact with it, and who can study it.Now, we are releasing this guardrail, early, because the best way to build safety infrastructure is out in the open, with the people who will use it, who will interact with it, and who can study it.

Mila’s Suicide Prevention Guardrail is the first classifier in a planned series, each built and released in the open and with ROOST. Today, it focuses on the responses a model produces. Future versions are planned to include multi-lingual, multi-turn, contextual input filters, as well as variants trained to detect suicide encouragement within human-to-human chats, or that analyze directly user’s input. The goal is a Swiss cheese model to safety: layering ​​multiple safeguards to create safety in depth. Tackling suicide risk today, ROOST and Mila’s partnership will continue to grow in the coming months, so that our solution can address the wide open safety guardrails ecosystem over time.

Get started