You tell your AI companion about the fight you had with your mother. About the anxiety you can’t shake. About the relationship that’s falling apart. It listens. It responds. It never judges.
It also logs every word, stores it on servers you’ll never see, and uses it to train the model you’re confiding in. That’s not a privacy policy buried in the fine print. That’s the core business model.
The short answer: AI companion apps privacy is weak by design. Apps like Replika and Character.AI treat your conversations as their training data, and most give free users no opt-out. Some have no meaningful legal check outside the EU, where regulators like Italy’s Garante have forced changes. To use one more safely: never give your real name, withhold identifying details, and turn off any data settings you can find. When you’re done, fully delete your account rather than just pausing it.
The apps and what they collect
Replika, built by US company Luka Inc., has around 10 million registered users. Its pitch is a personalised AI friend — something between a therapist and a companion. The privacy policy is clear about one thing: your conversations are used to train and improve the AI. That’s the product. You talk; Replika learns. Your most vulnerable moments are the training data.
In February 2023, Italy’s data protection authority (the Garante) ordered Replika to stop processing Italian users’ data entirely, citing risks to emotionally vulnerable people and minors. Replika complied for Italian users — which tells you something about what a regulator with actual teeth looks like. For Indian users, there’s no equivalent pressure. Your data goes to US servers under US law.
Replika’s history with intimate data is also instructive. The app used to generate sexual content — a significant category of extremely sensitive conversation. After Italy’s ban, Replika restricted that feature, then partially restored it following user backlash. The policy around some of the most sensitive data in existence shifted multiple times in a single year. That’s not a company that has thought carefully about what it owes its users.
Character.AI is, by some measures, one of the most widely used AI apps in the world. It’s especially popular with teenagers. Users create or interact with AI personas — fictional characters, celebrities, custom personalities — and the conversations can go deep quickly. The company’s privacy policy states that conversations are used to train its models. There is no opt-out for free users. Every exchange, no matter how personal, feeds the system.
In 2024, a lawsuit filed in the US concerned a 14-year-old boy who had become deeply attached to a Character.AI persona. His conversations with that persona had turned to suicide, and he died by suicide. The lawsuit claimed Character.AI failed to implement meaningful safeguards. The case was reported by NBC News, Time, and the Washington Post. Character.AI rolled out parental control features in 2024 following the lawsuit and regulatory attention. Those controls rely on users accurately reporting their age. They do not rely on enforcement.
Snapchat My AI was initially embedded for all users with no opt-out. The UK’s Information Commissioner’s Office launched an investigation in 2023 into Snapchat’s failure to conduct an adequate data protection impact assessment before deploying it to millions of UK users. The investigation was about process — whether Snapchat even thought carefully about the risks first. The answer, apparently, was no.
What these apps actually know about you
The conversations are the obvious concern. But there’s a longer list.
Beyond what you say, these apps track a lot. That includes how long you talk to the AI, how often you return, what topics you discuss, and your emotional tone (some run sentiment analysis on your messages). It also includes device identifiers, usage patterns, and in some cases your location and demographic information.
The business model explains why all this matters. Some platforms run advertising. Many offer premium subscriptions. But the conversation data itself — the actual content of what you say to the AI — is the training asset. Every intimate disclosure you make is a direct business contribution. You are not the customer. You are the dataset.
What can go wrong
There are a few scenarios worth thinking through.
Acquisition. If Luka or Character.AI is bought, your conversation history transfers to whoever buys them. Data privacy commitments made under one company’s ownership may not survive under new management. This has happened with other platforms. It will happen again.
Breach. Centralised stores of sensitive conversation data are valuable targets. Users have raised concerns about the security of their data with several AI companion platforms. If a breach occurs, conversations about your mental health, relationships, or fears become exposed.
Policy drift. Replika changed its intimate content policy multiple times in twelve months. Character.AI has changed its approach to minors repeatedly under legal pressure. The policies governing your most sensitive disclosures are not fixed contracts.
No Indian legal protection. Most of these companies are US-headquartered and process data under US law. GDPR applies if they operate in the EU — which is why Italy’s Garante had leverage. India’s DPDPA is real law, but enforcement against a California-based company for data collected from Indian users is, at present, largely theoretical. If something goes wrong with your data, the remedies available to you are limited.
How to use these apps without handing over everything
If you use AI companion apps, assume that every conversation is logged, potentially reviewed by human staff during moderation or training review, and stored indefinitely unless you actively delete it.
Given that, a few practical rules:
Never use your real name. Give a pseudonym. The AI doesn’t need to know who you actually are, and the less identifying information in your conversation history, the better.
Don’t disclose: your location, your workplace, the names of your family members, your medical diagnoses, the specific financial situation you’re in, or details about other people that could identify them. These people never agreed to be part of your AI companion’s training data.
Check the privacy settings. Some apps let you turn off personalization features or limit data retention. These settings are rarely prominent, so you’ll need to go looking. Turn off whatever you can find. For the mainstream AI tools (ChatGPT, Gemini, and the rest), our guide to opting out of AI training lists the exact toggle at each company.
Delete your account if you’re done. Not “pause” — full deletion. Check that the app’s support page specifically states that deletion removes your conversation history. If it’s ambiguous, assume the data stays.
For mental health conversations specifically, consider apps like Woebot. It has been through clinical validation studies and carries different data commitments than a general AI companion built primarily on engagement metrics. The difference matters.
What you can do right now
Want to see what data brokers expose about you? See what a scan finds → shows an example report: the kinds of sites that list your name, phone, and address. Once KnytX Shield launches it will find and remove these automatically. Join the waitlist to get in early.