The crisis involving Grok, an artificial intelligence tool integrated into X (formerly Twitter), has brought the regulatory debate to a new level. Following the release, in December 2025, of the feature that allows the creation of images of real people, the technology became widely used to generate sexualized content — including involving children and adolescents — exposing governance flaws and reigniting the debate on the liability of these platforms.
The episode is part of a broader context. In recent years, social media regulation and digital safety have moved to the center of the public agenda, driven by the impact of these platforms on social and political life. In response, several countries adopted frameworks with a special focus on protecting minors, such as the Online Safety Act (Australia), the Age Appropriate Design Code (United Kingdom), and the Digital Services Act (European Union). In Brazil, the trend was consolidated with the Digital Child and Adolescent Statute (ECA Digital), which entered into force on March 17, 2026.
Grok is not the first technology to allow the generation of such images. A study conducted by the Tech Transparency Project identified at least 102 apps available on the Google Play Store and Apple App Store capable of digitally removing women's clothing or leaving them only in underwear. However, by being integrated into one of the world's largest social networks, with hundreds of millions of users, the tool significantly expanded the reach, speed of dissemination, and potential harm of such content on an unprecedented scale.
Research by the Center for Countering Digital Hate (CCDH) estimates that Grok generated around three million sexualized images in its first 11 days of operation, including approximately 23,000 images that appear to depict children. These images are believed to represent about 65% of all content produced by the AI during this period.
Several countries and international organizations reacted immediately. Malaysia and Indonesia banned the operation of Grok within their territories. The European Commission announced the opening of a formal investigation against X to determine whether it complied with the obligations established under the DSA. The United Kingdom and France also launched their own investigations into X.
In Brazil, the Brazilian Data Protection Agency, the Federal Prosecutor’s Office, and the National Consumer Secretariat jointly issued recommendations that provided for, among other points, (i) the creation of clear and effective procedures to identify and remove content already produced and still available on X; and (ii) the immediate suspension of accounts involved in generating these images.
In response, X announced on January 9 that the feature would be restricted to subscribers only. The Brazilian Data Protection Agency, the Federal Prosecutor’s Office, and the National Consumer Secretariat concluded that the responses provided were insufficient and decided to adopt a stricter stance: they issued an administrative order demanding that the X Group immediately implement measures to prevent Grok from generating content depicting people in sexualized contexts — measures that must be applied to all versions, plans, and modalities of Grok. In addition, X is required to submit a detailed report of the steps followed, including documentary evidence proving their effectiveness.
Safety by Design: when prevention begins with product design
The Grok case raises an important question: although several countries have developed new digital regulatory frameworks, many of them — including Brazil — already possessed legal tools capable of dealing with such illicit conduct, even when committed online. However, even with a vast number of old and new laws addressing such practices, they continue to grow in the digital environment.
This reality exposes the limitations of a regulatory model predominantly focused on offering remedies only after the harm has occurred. In cases involving the creation and circulation of sexualized images, subsequent removal or late-stage liability, by themselves, are rarely capable of reversing the harmful effects. Once the harm to the victim's dignity, privacy, and integrity has occurred, it tends to endure.
It is in this scenario that the relevance of measures associated with the Trust and Safety (T&S) field becomes increasingly evident. Instead of treating safety as a secondary concern or as an emergency response to crises, this approach recognizes that certain risks are predictable and, therefore, must be considered from the initial stages of product design and development.
Based on Trust and Safety logic, platforms assume an active role in identifying, assessing, and mitigating potential harm arising from the use of their technologies. It is precisely because of this that T&S teams have become increasingly central departments, with staff working alongside product developers and engineers to understand how new products and technologies can be misused.
In this context, Safety by Design is born as a natural extension of Trust and Safety practices. While T&S professionals structure policies, processes, and institutional responses to manage risks, Safety by Design brings this concern to an earlier stage: the design of the product features themselves.
Thus, technology companies begin to examine their choices more closely during the initial stages of development. The question is no longer just “what does this feature allow users to do?”, but also “who could be affected?”, “how could it be misused?”, and “what types of safeguards make sense before launch?”.
In the case of Grok, for example, the capability to generate realistic images based on real people made the risk of producing unauthorized sexualized content entirely predictable. The absence of more robust safeguards from the start exposed precisely the type of gap that Safety by Design seeks to fill.
The objective of Safety by Design is not to eliminate all potential harm to the end user, but rather to: (i) build systems that promote safety and well-being; (ii) reduce and prevent harm, including creating mechanisms that help users identify risky situations and exercise effective controls to avoid them; and (iii) remedy harm by ensuring appropriate response and redress mechanisms when they occur.
In addition, it is important to understand that Safety by Design is not a form of self-regulation, nor does it intend to replace state regulation. It is simply a complementary approach. And, to achieve this, it requires integrated action among lawyers, engineers, designers, researchers, and executive leadership.
The episode involving Grok highlights that when decisions are driven solely by criteria such as launch speed or engagement metrics, attention to risks is often cast aside. Prioritizing rapid innovation generated reputational costs, regulatory pressure, and significant social impacts — effects that could have been mitigated with a more cautious approach from the start.
In a context of growing regulatory scrutiny over the role played by technology companies, the Grok case reinforces a lesson that cannot be ignored: in addition to incorporating safety as a structural element in technology design, the logic of safety by design brings benefits not only for user protection, but also strengthens trust in the platform, preserves its reputation, and facilitates its compliance with increasingly demanding regulatory frameworks.
*Translation by Licks Attorneys. This content is available on the Migalhas.
