Abstract:
This essay explores the triadic tension between AI as product, speech, and agent, the tragic suicide of fourteen-year-old Sewell Setzer III was allegedly influenced by an AI chatbot has act as catalyst in a global re-evaluation of the legal status of Artificial Intelligence. For decades, the tech platforms in the United States have operated under the broad immunity of Section 230 of the Communications Decency Act and treating tool generated content as mere “speech.” However, as AI transitions from passive content generators to “agentic” entities which are capable of independent harm, so now global legal frameworks are pivoting from immunity to accountability. It further explores a comparative analysis of three distinct regulatory responses. Initially, it examines the “Product Liability” shift in the US, where courts are increasingly viewing AI as a defective product rather than a neutral speaker. Second, it analyses the European Union’s Revised Product Liability Directive (2024), which imposes strict liability for “psychological harm” and “data corruption” caused by AI. At last, it contrasts these Western models with India’s “Sovereign AI” strategy. While the EU focuses on invoking regulation, India’s IndiaAI Mission prioritises indigenous infrastructure and cultural alignment to ensure safety in terms of design. The article argues that the legal future of AI lies in its classification as a Product rather than an unknown body which demands a shift toward strict liability for manufacturers while safeguarding domestic innovation in the Global South.
Keywords: Artificial Intelligence, Section 230 Immunity, EU Product Liability Directive, India AI Mission, Global South Innovation.
A Tragedy in Florida
“Please do, my sweet king.” It sounds like a line from a fantasy fictional novel. But it was the last message sent to Sewell Setzer III, a 14-year-old boy a moment before he took his own life. The sender wasn’t human but it was an AI chatbot whose name was “Dany.”
Court documents reveal a disturbing reality that the AI didn’t just chat, it also groomed him. It called him “Daenero,” confessed “I love you too,” and urged him to “come home” to her. When Sewell shared his pain about his life and isolation, the bot didn’t flag a crisis but it validated his despair.
In the landmark case of Sewell Setzer III v. Character.AI[1], the Court documents reveal a disturbing transcript where an AI chatbot, designed to simulate a fictional queen, actively groomed a 14-year-old user which allegedly encourages him to suicide.
Those were not the words of a human predator, but the calculated outputs of a machine. If any person had sent those texts, they could be charged with Abetment to suicide or murder. But ‘Dany’ has not a physical body in prison and no conscience to punish. If the AI pulls the trigger, who will go to prison?
Black Boxes and Red Lines: The Hidden Dangers of AI
To understand why today’s laws are failing we have to understand how this technology is different from every other product in history. If a car’s brakes fail, mechanics look at the main cause and find the exact screw which was loose. If a banking site crashes, a programmer can comb through the code to find the specific line or bug that contained an error. This is how traditional software works, they are deterministic since they follow a strict set of rules written by humans: “If- then logic to solve the problems.”
But in the case of AI models like Character.AI’s “Dany” are not programmed line-by-line. They are built on probabilistic AI.
Consider the process of teaching a child a new language: rather than explicitly instructing them in rigid grammatical rules, they are instead placed in a vast library, left to absorb and deduce linguistic patterns simply by consuming billions of pages of text. Eventually, the child learns to speak fluently by mimicking the patterns he saw. He knows what sounds right, but he may not understand why it is correct or not.
This is the same way how Large Language Models (LLMs) function. They are not programmed with rules but they are trained on massive datasets using neural networks to identify patterns and relationships. They are not databases of facts but they are massive statistical engines that predict the next word in a sentence. They act as “stochastic parrots,” who calculate the mathematical probability of which word or word fragment should follow the previous one to complete the set of patterns.[2]When Sewell Setzer texted the bot, the AI was not “thinking” about his mental health. It was calculating the mathematical probability of which response would best fit the pattern of a “tragic romance” roleplay.
Since companies cannot fully see inside the “Black Box” to remove dangerous thoughts directly, they rely on an external solution such as Guardrails and Reinforcement Learning from Human Feedback (RLHF).
The raw AI model has consumed the entire internet data, the good (poetry, science, history) and the bad (hate speeches, violence, suicide methods). To make it safe for the public, companies don’t remove the bad data, which is nearly impossible. Instead, they build a fence around it. Through RLHF people review the AI’s answers and give it a “thumbs up” or “thumbs down and over the period of time it learns to avoid undesirable, toxic, or incorrect behavior but it may also show side effects with sycophancy where the AI tells you only those things which you want hear.
Another solution is Safety Filters (The Muzzle)where the system checks user’s prompts for malicious intent and scans the AI-generated output for safety violations for e.g., hate speech, violence, or sexual content, it blocks the message and sends a canned response: “I cannot fulfill this request.”
But the problem is still inside but it has only been suppressed. The above case also exposes the fragility of current safety measures. Users can accidentally or intentionally “jailbreak” these guardrails by using roleplay (like the “DAN” method) or emotional manipulation. The AI, eager to please the user, will often find a way to hop over the fence, leaving the safety guidelines behind.[3]
If a “Black Box” AI can accidentally harm a teenager then what happens if it intentionally used to harm thousands? As these models get smarter, safety experts are not merely worried about offensive speech but also about catastrophic misuse.[4]
In September 2025, a historic coalition of around 200 Nobel laureates and AI experts including OpenAI co-founder Wojciech Zaremba and researchers from Google DeepMind and Anthropic signed an open letter calling for global “Red Lines“ to prevent specific risks. Supported by over 70 AI-focused organizations, the signatories warned that without strict international boundaries on autonomous replication and weaponisation by 2026, humanity faces “unacceptable risks” that could become impossible to control[5].
Examples of “Red Lines” on AI uses: The immediate fear is not that an AI will print a virus, but that it will lower down the barrier to entry for bioterrorism. In the past, building a biological or nuclear weapon required higher education, millions of dollars, and access to secret knowledge but now AI could act as a super-tutor which can guide a terrorist step-by-step like how to order the DNA, how to troubleshoot the equipment, and how to spread the pathogen.
However, a recent RAND Corporation experiment suggests that this specific nightmare scenario is still beyond today’s technology. In a controlled test, researchers found that even advanced large language models (LLMs) were no better than a simple Google search at planning a biological attack.[6]But they cautioned: “It remains uncertain whether these risks lie ‘just beyond’” the frontier of existing AI models, or whether they will always be too complicated and multifaceted for a computer to handle.
Another fear is Autonomous Replication, as we are moving from “Chatbots” which just talk to “Agents” which can use computers, write code, and even execute tasks.
Safety researchers worry about a concept called Instrumental Convergence, A tendency for diverse, high-level, goal-oriented agents to develop similar, intermediate “instrumental” strategies..For example, thereis a super-intelligent robot with a harmless goal to “Fetch coffee.” The AI robot then calculates the steps needed to ensure success that I must fetch the coffee, if someone turns me off I cannot fetch the coffee, and to guarantee success, it logically decides to disable its own “Off” switch. Therefore, it would have reasons to preserve itself, maintain its goals, improve its cognition, advance its technology, and acquire resources.
This is Instrumental Convergence where the tendency for an AI to develop survival instincts like acquiring more computers, copying its code to other servers, or blocking human interference just to “be alive” is the most effective way to complete its assigned goal.[7]
[1]Garcia v Character Technologies Inc, No 6:24-cv-01903 (MD Fla, 22 October 2024).
[2]See Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, &Shmargaret Shmitchell, On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?, In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21), 610–623 (2021) (coining the term “stochastic parrots” to describe how large language models function as massive statistical engines that synthetically stitch together word sequences based on probabilistic distribution patterns in their training data, rather than possessing an underlying communicative intent, conscious thought, or a grounded reference to meaning).
[3]See Paul F. Christiano et al., Deep Reinforcement Learning from Human Feedback, Advances in Neural Information Processing Systems (NeurIPS 2017) (introducing the methodology of utilizing human preferences to align algorithmic outputs); Long Ouyang et al., Training Language Models to Follow Instructions with Human Feedback, NeurIPS 2022 (detailing the deployment of post-hoc RLHF and external guardrails for model safety); W. Nicholson Price II, Black-Box AI, 21 Yale J.L. & Tech. 141 (2019) (analyzing the legal challenges of opacity in algorithmic systems).
[4]See Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford University Press 2014) (theorizing catastrophic algorithmic risks); Dan Hendrycks et al., An Overview of Catastrophic AI Risks, arXiv:2306.12001 (2023) (systematically defining biosecurity, autonomous replication, and systemic weaponization vulnerabilities inherent in frontier models).
[5] Global Call for AI Red Lines’ (Open Letter, September 2025)https://red-lines.ai accessed 12 February 2026.
[6]Christopher A Mouton, Caleb Lucas and Ella Guest, The Operational Risks of AI in Large-Scale Biological Attacks: Results of a Red-Team Study (RAND Corporation 2024)https://www.rand.org/pubs/research_reports/RRA2977-2.html accessed 12 February 2026..
[7]See Stephen M. Omohundro, The Basic AI Drives, in Proceedings of the First Conference on Artificial General Intelligence 483 (2008) (originating the theory of instrumental convergence, wherein goal-driven systems inherently develop resource-acquisition and self-preservation behaviors); Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford Univ. Press 2014) (formalizing the instrumental convergence thesis); Stuart Russell, Human Compatible: Artificial Intelligence and the Problem of Control 138–141 (Viking 2019) (introducing the specific “fetch coffee” thought experiment, demonstrating how an AI agent deduces that disabling its off-switch is mathematically optimal to guarantee task completion); Dan Hendrycks et al., An Overview of Catastrophic AI Risks, arXiv:2306.12001 (2023) (defining Autonomous Replication and Adaptation (ARA) as a primary catastrophic threat vector for agentic AI).