Being proactive about predictive systems

Log in to listen to this article

Digital platforms increasingly rely on predictive systems to shape how users, including minors, experience online services, recommending content, ranking information, and personalizing interactions based on observable user behaviour. 

Many of these systems are developed in-house and operate continuously as part of a product’s core architecture, influencing how users engage with a service in real time. 

“Opaque algorithms direct [children] toward extreme material, while manipulative designs influence their behaviour…,” Michael O’Flaherty, the Council of Europe’s Commissioner for Human Rights, has warned.

This observation underscores the real-world consequences of algorithmic design choices, particularly for younger users who may be more vulnerable to engagement-driven features. These concerns are no longer purely theoretical, as evidenced by recent U.S. court rulings, which have brought renewed scrutiny of how these kinds of design choices may translate into legal and regulatory accountability for platforms.

European regulators are responding to these risks by increasing protections for minors in digital environments. For example, Article 28 of the European Union’s Digital Services Act (DSA) requires providers of online platforms accessible to minors to implement appropriate and proportionate measures to ensure a high level of privacy, safety, and security. 

More substantively, Article 28(2) requires measures to reduce exposure to age-inappropriate content, such as gambling or pornography, and prohibits platforms from presenting targeted advertisements based on profiling and personal data when they’re aware with reasonable certainty that the user is a minor.

Perhaps most notably, these obligations apply to all online service providers operating in the European Union, reflecting the legislation’s broad reach, even as specific requirements are scaled in proportion to a company’s role, size, and impact. In this way, safeguarding minors’ rights while using online services becomes an industry-wide responsibility for all online platforms across the E.U. 

Article 28(3) of the DSA makes explicit that online service providers are not expected to collect additional personal data solely to determine whether a user is a minor. Instead, compliance focuses on implementing reasonable measures to identify and protect young users, with regulators able to issue guidance to support effective implementation. 

By embedding these protections into platform design and operations, Article 28 reflects a shift from reactive content moderation towards proactive safeguards that anticipate and mitigate risks to minors online.

Alongside the DSA, the European Union’s Artificial Intelligence Act establishes a risk-based framework for AI systems that influence behaviour or shape decision-making. Together, they highlight an emerging and increased tension with children’s privacy efforts. 

Although AI systems are rarely designed specifically for children, they operate across entire user populations and adjust recommendations based on behavioural signals. Systems built to optimize engagement can influence the experiences of young users even when they are not the intended target. This creates a complicated regulatory and operational environment for platforms and governance teams.

While the European Union has adopted a comprehensive, risk-based framework to address these concerns, regulatory approaches in other jurisdictions are evolving along different lines. In the United States, the approach to youth online privacy and safety is rapidly evolving, but remains fragmented and legally complex. Much of the regulatory momentum is occurring at the state level. For instance, California and New York are targeting “addictive feeds” and the impact of algorithmic design on minors, alongside emerging frameworks similar to age-appropriate design codes that focus on product and service design. 

In Canada, federal privacy reform stalled when Bill C-27—the latest attempt to modernize federal privacy law—died on the order paper in January 2025. However, children’s online safety and privacy have since re-emerged as policy priorities, with the Office of the Privacy Commissioner of Canada advancing a Children’s Privacy Code and the Canadian federal government announcing Bill C-34 in June. 

Bill C-34 would impose new child-safety-by-design obligations on social media platforms and certain AI chatbot services, as well as duties to identify and mitigate online harms, implement age-appropriate safeguards, enhance transparency, and address harmful content and AI-generated interactions. 

While the design requirements will be determined later by regulations, these developments reflect a broader shift toward examining not only the collection and use of children’s data, but also how digital products are designed and experienced by young users.

At the same time, there is growing attention to personalization, excessive use, and so-called addictive design, although these concepts remain difficult to define and regulate consistently. During the OPC’s consultations on the Children’s Privacy Code, addictive design features, such as infinite scroll and autoplay, were identified as examples of harmful practices for children. There was strong support among respondents for prohibiting deceptive design practices that manipulate behaviour or undermine privacy-protective choices.  

In the U.S., efforts are further shaped by constitutional constraints, particularly free speech protections, as well as ongoing litigation that draws analogies to other harm-based industries but faces challenges around proving causation. Tensions also persist between parental control and minors’ autonomy, especially in politically sensitive areas, contributing to a more contested regulatory environment across states. While the U.S. increasingly borrows ideas from international frameworks such as the EU’s Digital Services Act, its approach continues to be more decentralized, litigation-driven, and shaped by competing policy priorities. 

For privacy and governance teams inside technology companies, these developments raise practical questions. Compliance can no longer focus solely on data collection or processing. Teams must also consider how predictive systems operate within a product, how they shape user behaviour, and how regulatory frameworks apply when children are among the platform’s users. Companies need to make design choices that balance engagement, usability, and protection, while also documenting risk assessments and governance decisions for regulators, an often incredibly complex if not impossible task. 

Taken together, these developments highlight an opportunity for privacy and governance teams to proactively embed child safety into product design from the outset, using structured risk assessments and documentation to support responsible innovation aligned with global regulatory expectations. 

* The opinions expressed in this article are the authors’ own, and do not represent the position or opinions of their employers or the Canadian Bar Association.