We talk to chatbots as if those conversations are private. But are they?
Most people are confused. A survey commissioned by DuckDuckGo and published in August asked 1,944 American adults about chatbot privacy, and more than half (53%) did not know whether their conversations were being used to train the bots. (They are.)
Only a quarter of respondents knew that chatbot conversations can be subpoenaed. (They can.)
Among self-described AI enthusiasts, 56% said they had confided to a chatbot something they had never told a close friend, parent, colleague, doctor, or therapist.
That’s especially true on Character.AI, a chatbot service where users can chat and form “relationships” with customizable characters, including ones they create and share with the community. On Character.AI, some users chat about thoughts of suicide and self-harm, and they surrender private details like mental health disclosures, ages, private chats and voice calls, deep fears, and relationship struggles.
Character.AI was all fun and games until earlier this month, when the company filed for bankruptcy and opened a court-supervised sale of its assets.
Nobody knows who will end up with the company, or how much of the data on its millions of account holders will be sold as well.
The most valuable asset for sale by the company may be its LLM, which has been substantially trained by users as they build and interact with characters.
Legally, a company can sell personal data about its users if its privacy policy permits the transfer. Character.AI’s privacy policy is spectacularly vague. It says the company may disclose “certain information” in the company’s sale. Nobody knows what that means.
The US Trustee Office, an arm of the Justice Department, wants the judge to appoint a consumer privacy ombudsman to advise the court on user privacy. The judge could restrict the sale of company “records,” including user data.
But here’s the part most people don’t understand: The Bankruptcy Code has no provision specifically addressing trained models. The legal status of LLMs and their training data has yet to be settled in law. That means it’s entirely possible that the data users fed into Character.AI will be sold to whoever buys it.
Character.AI is just one handy example. The larger risk is all the other AI companies out there.
More surprising facts about LLM privacy
The core question is this: Can personal or proprietary information shared in a chat end up inside an LLM, where anyone might coax it out?
The answer is a resounding “uh, maybe.”
First of all, it depends on the AI chatbot service. The companies vary wildly in how they handle chat data, and the differences boil down to four questions: whether training on chats is on by default, whether human reviewers read them, how long they are kept, and whether they feed advertising.
Major chatbot companies generally try to reassure users about privacy. They publish privacy controls, but their own policies admit that consumer chats may be used for training by default, read by human reviewers, or both, and the fine print shifts from company to company.
Tellingly, some of them offer enterprise or business accounts that handle chat sessions privately. For example, they don’t use chat sessions in training data. That business accounts get extra privacy features sounds to me like an admission that individual accounts aren’t private.
Some companies, such as OpenAI, let users opt out of having their chats tossed into the training data. But OpenAI’s help documentation says that turning off training does not delete the chats themselves.
There are three main ways chatbot user data can be exposed. The first is bugs. In March 2023, a bug lurking in an open-source library called “redis-py” let some ChatGPT users see other users’ chat titles and, possibly, the first message of new conversations. More than 1% of ChatGPT Plus subscribers active in a nine-hour window may have had their payment details exposed to other users: names, email addresses, payment addresses, card types, the last four digits of card numbers and expiration dates.
The second is sharing. Some chatbots offer sharing features, and those features can publish chats to the world, whether on purpose or by accident. Once these sessions are posted, search engines are likely to gobble them up and make them searchable. In 2025, Forbes reported that Google’s results showed more than 370,000 Grok conversations after users clicked “share,” and shared ChatGPT chats turned up in search results until OpenAI killed the option.
The third is the absorption of chat sessions into the training data.
Let’s take one simple example: an email address. People often paste in emails so that a chatbot will write the reply. These emails often contain names, email addresses and other information in the header, the signature, or both.
The question I often ponder is this: If emails addressed to one person get pasted into chats hundreds of times, can a stranger ask the chatbot for that person’s address and actually get it back?
In a 2022 study, GPT-Neo models given only a person’s name recovered the right email address no more than 1.24% of the time, but with the domain supplied, the rate climbed to 16.55%, and to 37.06% after five examples. Other factors, including how often an address appeared in the training data and whether surrounding email text came along with it, pushed the rate higher.
Deleting chat records isn’t a remedy, either. Several state attorneys general said deleting chat records may not undo their influence on trained models. And Common Sense Media’s Robbie Torney claimed that exporting or deleting your data “may not mean that that data is gone from model training.”
Note that people not only enter personal information about themselves during AI chats, but about their co-workers, friends, and relatives.
The main thing standing between your secrets and the world is company practice. We depend on the AI companies to keep our information private.
So what happens when those companies sell their assets?
When AI bubbles burst
Online chatter about the “AI bubble” bursting is pretty much constant.
In China, the AI bubble has already burst, according to Reuters. Over the past few months, Chinese AI stocks have been pummeled.
Reuters reported that Shanghai’s STAR50 index of tech stocks fell 40% from its July high in under four months, erasing about $480 billion in market value, and that the broader CSI300 is down about 20% from its June highs.
It’s not a total collapse, but it does put pressure on US stock prices and also provides an incentive to transfer ownership through sales or mergers (though this hasn’t happened in China yet).
Still, the future is unknowable. Which AI companies will go bust as Character.AI did, get sold off, or get hacked?
As for the chatbots themselves, their inner workings are also unknowable.
That’s why we should all listen to the most paranoid amongst us, and never put sensitive information into chatbots. We just can’t trust AI to keep our secrets.
