Announcements ROOST partner coop
More Ways to Protect Children with Coop and a Milestone for Transparency

More Ways to Protect Children with Coop and a Milestone for Transparency

Published

Today, OpenAI announced a new classifier for detecting child sexual abuse material (CSAM), and to ensure that any organization, no matter its size and safety budget, could benefit from this innovation, OpenAI chose to integrate with Coop at launch. Coop is the world’s first trust and safety tool recognized as a Digital Public Good. Built to identify previously unseen material, the classifier joins the perceptual hash matching and Google's Content Safety API integrations that Coop already supports, giving any organization another powerful layer of detection within the review and reporting workflows they already rely on.

OpenAI is launching with the Coop integration and making the classifier available at no cost to organizations that apply and are approved. As with other free tools for detecting novel CSAM, approval requires that an organization have a way to review flagged content and report it to the National Center for Missing & Exploited Children (NCMEC), given the sensitivity of the material involved. Coop is the only free tool that provides both, which means organizations of any size can detect, review, and report CSAM without paying for any part of the process.

Bringing CSAM detection to every organization

Any organization that allows image or video uploads, or provides a way to generate or edit images or videos with AI, will, sooner or later, have to contend with CSAM. Detection has traditionally relied on two complementary approaches: hash matching, which compares uploads against digital fingerprints of material that has already been identified, and classifiers, which estimate the likelihood that an image contains abuse material even if it has never been seen before. Classifiers have become increasingly important as new abuse material continues to be created, driven by harms such as sextortion and sadistic online exploitation, as well as a wave of synthetic CSAM made with generative AI. Newly created material, by definition, won't match any existing hash.

Yet access to these technologies has long been uneven. Hash matching has historically been difficult to implement, with scarce documentation and opaque processes to get access, while classifiers for novel CSAM have often been prohibitively expensive and difficult to access. Even once an organization secures access, a detection signal is only the beginning: using it responsibly also requires a place for trained reviewers to assess flagged content, decide what happens next, records of each decision, and a reliable pipeline for reporting to NCMEC. For the many developers without a full-time safety team, building that infrastructure has been out of reach.

Coop was designed to close that gap by giving teams the infrastructure that surrounds detection: review queues, automated enforcement rules, analytics, appeals, and built-in NCMEC reporting, all free and self-hosted. New detection tools plug into those same workflows, so teams don't need to rebuild their systems each time one becomes available. That is why we worked with OpenAI to connect Coop with the new classifier, so that everyone has access to the tools they need to benefit from the latest innovations in child safety, not only those with the resources to build around them.

The consequences of this gap are already visible in reporting. A small group of companies accounts for the overwhelming majority of reports to child safety hotlines, while thousands of registered providers under-report or don't report at all, and tightening regulation without improving the tooling risks a reality in which only incumbents can comply. We believe—and are frequently proven right—that there is a much wider coalition of the willing: organizations eager to do their part to keep children safe online and to proactively detect and report this material, but who simply lack the necessary tools. As more organizations gain access to better tools, reporting numbers may rise even if the underlying volume does not. Figures from NCMEC, the Internet Watch Foundation, and other hotlines reflect detection and reporting rates rather than the true volume of abuse.

A step toward transparency in the use of AI for child safety

Alongside the classifier, OpenAI is publishing a model card that describes how the classifier was built with assistance from NCMEC, how it was evaluated, and where its limitations lie. We see this as a meaningful milestone, and one that we have long advocated for.

Earlier this year, ROOST's Camille François and Vinay Rao, together with Margaret Mitchell and Yacine Jernite of Hugging Face and J. Nathan Matias of Cornell, wrote in Tech Policy Press about why model cards are an urgent necessity for child safety. Child safety classifiers are widely deployed, yet they are not documented in a standardized way, leaving safety teams without the information they need to set thresholds, plan for human review, or understand where a tool may fall short. In that piece, we described a "tragedy of first movers," in which the difficulty of evaluation and the legal constraints around this material discourage anyone from publishing first. Coop utilizes UX design to nudge all model builders towards greater transparency by creating a model card default for any third-party integrations included, even if it meant one would stare at empty fields.

OpenAI CSAM Image Classifier Model Card in Coop

OpenAI's model card goes a long way toward moving past that impasse as the first ever model card for a CSAM classifier. It documents the classifier's performance on both photographic and AI-generated material, reports results across age, race, and sex, and is candid about where performance is weaker, including for images of the youngest children and for some demographic groups. By naming those gaps openly, it helps set a roadmap for the field and shows where work is still needed. It also notes what has not yet been tested, such as video, and explains why precision measured on a held-out evaluation set can look very different on real-world traffic. This is the kind of disclosure organizations need to set thresholds, plan for human review capacity, and understand how a tool is likely to behave on their own platforms.

OpenAI's model card also calls for shared, independent benchmarking of CSAM detection systems, a need no single company can meet on its own. It is the question at the heart of ROOST's new research into the accuracy of CSAM detection tools, led by our inaugural Scientific Fellow, Dr. Rebecca Portnoff, and we look forward to working with OpenAI and others across the field on it. OpenAI's model card brings the field closer to tools that we can actually understand and that safety teams can rely on to find abuse material quickly and help protect children. We hope this example encourages other providers of child safety tools to follow.

How the classifier works in Coop

The classifier is built on the same infrastructure as OpenAI's Moderation API, which Coop already supports through its OpenAI integration. Teams already using that integration will find the CSAM classifier alongside the sixteen OpenAI moderation signals available in Coop today. Once organizations are approved by OpenAI, they can use the same API key or set a new one specific to the CSAM Classifier to use with Coop.

Coop integrations, including the OpenAI CSAM Image Classifier

The classifier currently returns two scores, each from 0 to 1: one for suspected CSAM and one for suspected AI-generated CSAM. Within Coop, these work like any other signal, so teams can set thresholds and combine them with hash matching and Google's Content Safety API to route flagged images to review queues and confirmed material into Coop's NCMEC reporting pipeline.

Coop routing rules, configuring a rule using the OpenAI CSAM Image Classifier

Getting started

Organizations interested in using the classifier with Coop can get started in four steps:

  1. Apply to OpenAI through OpenAI's application form. OpenAI reviews each applicant, including whether NCMEC reporting is in place, whether they do manual review, and what their current stack looks like. This process happens directly with OpenAI, outside of Coop and ROOST.

  2. Deploy Coop using the deployment guide and configure NCMEC reporting, which can be done while your application is under review.

  3. Add your API key once approved. In the OpenAI integration settings in the Coop dashboard, enter the API key associated with the Org ID approved for the CSAM classifier, in the new field next to the existing OpenAI API key. Until then, the classifier's signals will appear in the dashboard but remain disabled, with a note pointing to the approval process.

  4. Benchmark and route. Following the model card's guidance, review what the classifier flags on your own traffic to choose thresholds, then use automated enforcement rules to route flagged images to review queues and confirmed material to the NCMEC reporting pipeline.

Get Involved

ROOST is committed to ensuring that the most capable child safety tools, and the information needed to use them well, are available to every organization. We're grateful to OpenAI for their partnership on this integration and for their leadership on transparency, and we look forward to continuing to work with the community to make free, open child safety infrastructure stronger.