Protecting Children Online Takes More Than Safeguards: They Must Be Open

Many conversations about child safety treat online safeguards as a checklist: content filters, reporting mechanisms, and age-appropriate design. What rarely gets discussed is how those safeguards are actually built, or what technology sits underneath them. At ROOST, we think that is where a commitment to safety either becomes real or quietly disappears.

In November 2025, ROOST President Dr. Camille François joined a high-level meeting in Rome on child dignity in the age of artificial intelligence. The gathering brought together technologists, faith leaders, researchers, and child protection advocates around a shared declaration on protecting children online. That declaration set out six pillars, including protection and safety, ethical and transparent technology, and global governance. As part of this, our team shared the following proceedings with Rome's leaders, stating that those commitments cannot be met unless we change how safety technology itself gets built.

As that argument has only gotten more urgent, we're publishing the paper we shared with Rome’s leaders here in full. Because the safeguard that exists only inside a single company is not the same as a safeguard anyone can study, modify, and use. Our paper makes three points:

1. Regulation is outpacing the toolbox. Governments increasingly require platforms to detect not just known child sexual abuse material, but new and AI-generated material and grooming behavior. Hash matching works for known CSAM and is widely deployed. For everything else, the ecosystem is remarkably thin.

2. That gap entrenches the largest platforms. A small group of companies account for the overwhelming majority of reports to child safety hotlines. Thousands of registered providers under-report or don't report at all. Tightening the law without fixing the tooling produces stricter rules in theory, but creates a reality where only incumbents can comply, through systems whose blind spots nobody outside can measure.

3. The four freedoms apply to safety, too. The free software tradition names the freedoms to use, study, modify, and share. Applied to child protection this means smaller platforms and NGOs can deploy real defenses; auditors and researchers can test for failures and bias across languages; communities can adapt tools to their own context; and innovation can travel freely.

ROOST’s projects show what’s possible when safety tools are openly designed, built, and distributed. Coop, our content review and enforcement console comes with integrated NCMEC reporting and free child safety detection tools. Coop is open source and free for any organization. This enabled small teams to do NCMEC reporting for the first time.

We believe everyone deserves online safety, and that means every organization needs access to the technologies that help protect users, particularly the most vulnerable.

Read our full proceedings below.

*****

Abstract

In Rome, our declaration on “Child Dignity in the Artificial Intelligence Era” rightly emphasised the need for robust safeguards to protect children online. Yet whether any such safeguards are effective, accountable, and globally accessible depends not only on what they do, but on how they are built and shared.

This text argues that, in 2025, it is no longer sufficient simply to demand safeguards in the abstract. To honour the moral commitments expressed in our six pillars—especially protection and safety, ethical and transparent technology, and global governance—we should now insist that core safety tools be developed and distributed as open-source, shared infrastructure.

Open-source safeguards allow independent scrutiny of how systems behave; they enable families and survivors to understand harm and seek redress; and they give smaller platforms, NGOs, and public-interest projects the means to protect children without licensing or reinventing proprietary systems. Meanwhile, regulation around the world is rapidly, and rightly, moving toward stronger obligations to detect child sexual abuse material (CSAM) and predatory behaviours online. The toolbox for detecting new, previously unseen CSAM (including AI-generated material) and grooming remains brittle, expensive, and opaque. If we do not change course, we risk a state of affairs in which only a handful of large platforms can comply—behind closed safeguards whose biases and blind spots we cannot measure or correct. Recent developments in open-source safety tooling, however, provide a clear blueprint for a different path.

In many policy discussions, “safeguards” are treated as a checklist. Practitioners name content filters, reporting mechanisms, age-appropriate design, and move on. Because of time constraints, our discussions during the High Level Meeting did not dwell on how these safeguards are actually engineered. But that technical architecture is where our moral commitments either become real, or quietly disappear.

A safeguard that exists only as a proprietary system inside a single company is not the same as a safeguard that exists as shared, auditable infrastructure that others can inspect, improve, and deploy. When we say that every child has a right to dignity online, we are also making a claim about the distribution of technical power. Protection should not depend on whether a child happens to use a service owned by one of a few firms that can afford sophisticated, internal safety tooling.

From my perspective as a technologist, it is time to lift our expectations. It is no longer enough to ask companies and institutions to “do better” on safety. We must ask specifically how safeguards are built and implemented, and how they may be shared for auditability and public benefit. Concretely, that means recognising that there is a moral imperative to treat core child-protection safeguards as public goods and to develop and release them as open-source tools.

The free software tradition outlines four basic freedoms, which are here relevant in the context of safety safeguards: the freedom to use a system for any purpose; to study how it works; to modify it; and to share the original or modified system with others. For child safety, these map directly onto the obligations we claim to uphold.

The freedom to use safety tools means that smaller platforms, NGOs, schools, and public-interest projects can actually deploy robust defences rather than being priced out or locked out. The freedom to study them is what allows independent auditors, regulators, and researchers to see how they behave in practice: to test their blind spots, to measure bias across languages and communities, to understand when they fail. The freedom to modify them allows different communities and jurisdictions to adapt safeguards to their specific needs, instead of accepting a one-size-fits-all system designed elsewhere. And the freedom to share improvements is how we move from isolated efforts to a genuine safety commons.

Without these freedoms, “safeguards” easily become a black box. When abuse spreads unchecked, when recommender systems keep surfacing self-harm content, or when chatbots coax minors toward dangerous behaviour, families are told that something went wrong inside a system they cannot see. Open-source safeguards do not guarantee justice, but in many cases make it possible to retrace what happened, to understand design choices, demand change, and collaborate towards visible improvements for all. They are also essential for global equity: today, the most advanced tools live inside a few well-resourced companies in North America and Europe, while smaller and open services are held to the same expectations with none of the same infrastructure.

At the same time, laws are justifiably hardening around the protection of children online. Many countries around the world have recently passed or are considering legal frameworks that would require providers not only to detect and report known child sexual abuse material, but also new CSAM and grooming, and to deploy increasingly proactive technology to do so. The problem is that the tools they are supposed to use often do not exist at scale, certainly not in a way that respects the freedoms above. For known CSAM, hash-matching technology exists and is widely used by the largest firms. For new CSAM, especially AI-generated imagery, the ecosystem is extremely sparse: essentially a handful of proprietary or gated classifiers, expensive to run and largely opaque. Small and mid-size organizations often face a harsh choice: deploy a black-box service they cannot audit, or do almost nothing.

The consequences are already visible. A small group of large platforms account for the overwhelming majority of reports to child safety hotlines, while thousands of registered providers either under-report or do not report at all. In practice, the ability to detect and report abuse at scale is concentrated in a few companies. If we continue to tighten legal obligations without addressing this tooling gap, we will get stricter laws on paper, but a reality in which only incumbents can comply—through closed, unaccountable systems whose biases and blind spots we cannot even see. Children on smaller or emerging platforms will remain less protected.

There is, however, another path. In recent years, international initiatives and industry efforts have begun to treat safety technology itself as something that should be shared. Under The Christchurch Call, governments and companies have committed to developing shared open source tools to help platforms of all sizes tackle harmful content. At the Munich Security Conference, major technology companies have recognised that the tools used to counter deceptive AI content may need to be open-sourced to be effective. And several large platforms have started publishing parts of their own trust-and-safety stacks under open licences.

When OpenAI released one of its moderation models through Robust Open Online Safety Tools (ROOST) as an open-weight system, OpenAI’s Head of Safety Systems put the point simply: “safety tools should be available for all to study, modify and reuse,” and should help build “an open ecosystem of safety technologies adapted to the AI era.” It is the right standard to hold technology platforms to if they are to take child dignity seriously.

Conclusion

In Rome, we heard heart-breaking accounts of the new ways AI systems can harm children: AI-generated sexual abuse material, recommender systems that trap minors in cycles of self-harm content, conversational agents that gain their trust and then steer them toward dangerous behaviour. The question is not whether we can build stronger safeguards. Technically, we can and must. The question is what kind of architecture we choose.

One path is through ever-stricter legal obligations that can hardly be met by a sparse, proprietary, and un-auditable ecosystem of proprietary safety tools. That path entrenches the largest platforms, leaves smaller services struggling or non-compliant, and relies on safeguards whose performance and bias we cannot study, modify, or share.

The other path leads to a world where organizations, big and small, contribute to protecting children through a shared, open commons of safety tools. The emerging movement around open-source infrastructure for online safety—under initiatives like The Christchurch Call and ROOST, and in decisions by companies to open their safety models—shows that this path is already open to us.

We should now say this clearly: in the AI era, it is not enough to demand safeguards and leave the details of implementation to others. We must insist that the key safeguards protecting children are built and shared to guarantee everyone’s freedoms to use, study, modify, and share them. Only then can we claim to be building a safety infrastructure that protects every child, not just those on a few large, opaque platforms.

Professor Camille François, Columbia University School of International and Public Affairs (SIPA)

Co-founder, President, ROOST