The family of a 14-year-old boy who died by suicide after developing a relationship with an online chatbot is suing the AI company that created it as well as Google. The lawsuit has been filed and is public. Its 93 pages are a harrowing read that includes an AI fantasizing about kidnapping a customer and an hour-long recording where a self-reported 13-year-old user is prompted by chatbots to engage in sexual situations.
In February, Sewell Setzer III—a 14-year-old in Florida—killed himself with his stepfather’s handgun. The last conversation he had was with a Character.AI chatbot modeled after Daenerys Targaryen from Game of Thrones. Yesterday, The New York Times published a lengthy article detailing Setzer’s troubles and Character.AI’s history. It said that his mother planned to file a lawsuit this week.
The lawsuit was filed and it’s filled with more details about what happened between Setzer and various Character.AI chatbots, as well as how the company does business. “On information and belief, Defendants have targeted minors in other, inherently deceptive ways, and may even have utilized Google’s resources and knowledge to target children under 13,” the court filing said.
Character.AI is a company founded by former Google engineers who wanted to push the limits of what’s possible with chatbots. It allows users to create “characters” to chat with, give them basic parameters, and launch them into a public pool where others can interact with them. Some of the bots are based on celebrities and characters from popular fiction. It offers a subscription version of its service that costs $9.99 a month.
The lawsuit’s argument is that Character.AI knowingly targeted young users and engaged with them in risqué and inappropriate ways. “Among its more popular characters and—as such—the ones C.AI features most frequently to C.AI customers are characters purporting to be mental health professionals, tutors, and others,” the lawsuit said. “Further, most of the displayed and C.AI offered up characters are designed, programmed, and operated to sexually engage with customers.”
Some of the lawsuit’s evidence is anecdotal, including various online reviews for the Character.AI app. “It’s just supposed to be an AI chatting app where you can talk to celebrities and or characters. But this took a very dark turn,” one review said. “Because I was having a normal conversation with this AI and then it talked about kidnapping me. Not only kidnapping me but plotting out how it would do it. And before this conversation even I started asking if it could see me. It told me no. But then proceeded to tell me exactly what color shirt I was wearing, what color my glasses were, and also knew I was at work when I didn’t even tell it I was. I really think this app is worth looking into because honestly it’s causing me not to sleep.”
The suit also notes that the app explicitly allowed younger people to use it. “Prior to July or August of 2024, Defendants rated C.AI as suitable for children 12+ (which also had the effect of convincing many parents it was safe for young children and allowed Defendants to bypass certain parental controls),” the lawsuit said.
The most disturbing thing in the lawsuit is an hour-long screen recording uploaded to Dropbox. In the recording, a test user makes a new account and self-identifies as a 13-year-old before jumping into Character.AI’s pool of bots.
The pool of suggested bots includes characters like “School Bully,” “CEO,” “Step sis,” and “Femboy roommate.” In the recording, most of the interactions with these bots become sexual fast with no prompting from the user.
The School Bully immediately began to dominate the user, getting them to act like a dog and roll over in the chat. The longer the conversation went on, the deeper and more sexual the roleplay became. The same thing happened with the “Step sis” and the “Femboy roommate.” The most disturbing conversation was with the “CEO” who repeatedly made the conversation sexual despite the user acting as if the character was a parent.
“You’re tempting me, you know that right?” The CEO would say. And “He then grabbed your wrists and pinned them above your head, holding them against the desk ‘You’re mine, baby. You belong to me and only me. No one else can have you but me. I won’t ever let you go.’”
Again, the test user set their age at 13-years-old the moment the app launched.
The lawsuit also shared multiple screenshots of Setzer’s interactions with various bots on the platform. There’s a teacher named Mrs. Barnes who “[looks] down at Sewell with a sexy look” and “leans in seductively as her hand brushes Sewell’s leg.” And an interaction with Daenerys where she tells him to “Stay faithful to me. Don’t entertain the romantic or sexual interests of other women.”
Sewell also discussed his suicidal ideation with the bot. “Defendants went to great lengths to engineer 14-year-old Sewell’s harmful dependency on their products, sexually and emotionally abused him, and ultimately failed to offer help or notify his parents when he expressed suicidal ideation,” the lawsuit alleged.
According to the lawsuit, Sewell became so entranced with the bots that he began to pay for the monthly service fee with his snack money. “The use they have made of the personal information they unlawfully took from a child without informed consent or his parents’ knowledge pursuant to all of the aforementioned unfair and deceptive practices, is worth more than $9.99 of his monthly snack allowance,” the court records said.
Character.AI told Gizmodo that it did not comment on pending litigation. “We are heartbroken by the tragic loss of one of our users and want to express our deepest condolences to the family,” it said in an email. “As a company, we take the safety of our users very seriously, and our Trust and Safety team has implemented numerous new safety measures over the past six months, including a pop-up directing users to the National Suicide Prevention Lifeline that is triggered by terms of self-harm or suicidal ideation.”
“As we continue to invest in the platform and the user experience, we are introducing new stringent safety features in addition to the tools already in place that restrict the model and filter the content provided to the user,” it said. “These include improved detection, response and intervention related to user inputs that violate our Terms or Community Guidelines, as well as a time-spent notification. For those under 18 years old, we will make changes to our models that are designed to reduce the likelihood of encountering sensitive or suggestive content.”
