Blog Press Careers Contact Us
ROOST partner Announcements
Welcoming Roblox’s Safety Models to the ROOST Community

Welcoming Roblox’s Safety Models to the ROOST Community

Published

Roblox’s open source safety tools are joining the ROOST Model Community (RMC): Sentinel, a detection solution for identifying egregious risk in large-scale chat conversations; PII Classifier, an AI model for detection of attempts to share or request personally identifiable information (PII); and a voice safety classifier, a model for detecting policy violations in voice, covering 30 languages. Roblox is also releasing a new benchmark dataset – the Roblox PII Classifier Benchmark – alongside these models so that more organizations can test, build, and release privacy classifiers for their users and communities.

Online abuse and harm do not arrive in one format or as one issue. They come in different modalities like text and audio, and can show up as a single violating message or a dangerous pattern only discoverable by investigators across weeks of conversations. Those who protect people online face distinct challenges in detection at different points in a safety operation. Roblox has been open sourcing their internal technology used to combat these challenges, and bringing them into the RMC is the next stage of sharing their safety innovations with the wider field.

A detection suite to help keep users safe online

PII Classifier: real-time filtering

The PII Classifier classifies whether text is asking for or giving personally identifiable information (PII). In many scenarios, sharing PII can be a clue in safety investigations. Strangers might try to get minors to share their personal information, or bad actors may provide it in doxxing attacks. Roblox’s PII classifier applies two labels: privacy_asking_for_pii and privacy_giving_pii to catch both ends of the spectrum. Privacy risks can be hard to detect because the cases developers may most want to catch may contain no PII at all. The statement: "What school do you go to" for example carries no personal data, and yet is exactly the kind of signal that a safety team wants to be made aware of. By detecting the ask as well as when sensitive information is shared, this model can be useful against attempts to move a conversation off-platform and out of safety teams’ visibility.

The PII Classifier is fine-tuned on XLM-RoBERTa-Large for multilingual support and built to catch creative spelling, character substitution, and implicit references.

Roblox is also releasing the Roblox PII Classifier Benchmark, a synthetic set of multiuser chats built around the evasion techniques a classifier like this should be able to capture. While open models are becoming easier to publish than before, the data used to evaluate and fine-tune them is not. By publishing this dataset and its benchmark, Roblox is enabling the entire field of privacy and safety developers – giving them the tools they need to compare homegrown or commercial PII classifiers with open source options, and to even build their own classifiers and evaluations on top of it.

Sentinel: patterns over time

Some harms, like child grooming, happen through multiple interactions over time. In these instances, no single message may be enough to act on, but a pattern of borderline signals across many messages is. Sentinel accumulates weak per-message signals across a rolling window of a user's recent communications and surfaces the pattern for investigation by a safety team. It compares ongoing communication against two indexes, one built from benign interactions and one built from communications removed for child endangerment violations; teams seed the indexes with examples from their own platform.

Because Sentinel aggregates scores across a user's recent messages rather than judging each in isolation, making grooming, one of the hardest detection problems, tractable.

Through collaboration with ROOST, Roblox improved the effectiveness of Sentinel when used outside of Roblox’s environment. Following feedback from the ROOST community, version 2.0 of Sentinel is dockerized for easier deployment and adds an evaluation harness, so teams can tune against their own labelled data, and increases explainability, so a reviewer can see what surfaced a case. This is exactly the promise of the open source safety movement: a model or library released into the open can reach new communities, be studied, validated, and extended to bring more teams advanced safety capabilities.

A voice safety classifier for audio moderation

Roblox’s voice safety classifier scores audio segments against Roblox's policy categories across 30 languages: asking for PII, discriminatory language, harassment, sexual content, illegal and regulated content, dating and romantic content, profanity, and disruptive audio. It is a transformer distilled from a larger teacher model, trained on over 300,000 hours of Roblox’s internal voice chat datasets, using both machine and human-labeled data. The model is a 0.3B-parameter model published as F32 safetensors under an Apache 2.0 license, and expects 16kHz mono audio in segments up to 15 seconds.

Audio remains an underserved modality in open safety tooling. Most open safety models cannot process audio at all, which means teams running voice products have been building this layer from scratch. By releasing this model, Roblox is giving an invaluable building block to the broader safety community – allowing platforms that routinely see voice content to more easily adopt or develop classifiers to protect their users.

Stacking Open Safety Models

Each tool accomplishes a different goal, but used together they can create cohesive safety coverage. Beyond the specific case of Roblox's toolkit, this broader question of when and how to “stack” safety models has become a recurring conversation topic within the RMC. For example, organizations can currently pair content safety reasoning models like gpt-oss-safeguard with faster, cheaper models like CoPE-B-A4B and Shieldstral or specialized models like Mila’s Suicide Prevention Safeguard. This enables a multi-tiered approach that lets organizations optimize based on their priorities, such as speed and cost, and the unique factors of their community, violation volume and complex-case frequency.

By expanding the community of open, accessible, and modular safety models, the RMC aims to help teams access the detection tools they need to keep their communities safe. The RMC provides a forum in which organizations can evaluate how multiple safety models perform in combination, and in which model creators and AI labs can learn how to openly release and collaborate on safety models. Community members share what they are actually running: where coverage overlaps, where the gaps are, what it costs to run models in sequence. Model creators meanwhile hear from teams whose use cases are not their own: safety practitioners at companies large and small, with different communities and contexts, who then share and validate best practices. By sharing these insights, the safety community grows stronger and develops together.

Try the Models Out

Get Involved

ROOST's mission is to make online safety tools open, shared, and auditable. The RMC is where that happens: safety practitioners, AI researchers, and model creators building detection tooling in the open.

ROOST is committed to expanding access to open safety infrastructure. The addition of Roblox's tools to the ROOST Model Community enables organizations to address text, voice, and conversation-pattern detection requirements without developing capabilities independently. We look forward to continued collaboration with Roblox and with all partners contributing to this work.