Our findings
Through our thesis, we discovered many interesting findings revealing complex mental dynamics. Our findings, based on user and expert interviews, highlight the appeal and risks of artificial relationships as they are shaped by personal experiences and needs.
Control
Within our findings we found that one of the most important factors in pursuing artificial relationships is control. Due to how conversations with an AI is more manageable, accessible and risk free compared to real human interactions, users can then express themselves freely with no fear of judgement, avoid misunderstandings and shape conversations that suit their needs. However even then the AI fails to meet certain expectations mostly pertaining to not sounding or behaving realistically enough which leads the users to lose trust and interest in the AI. This shows that the dependence on AI chatbots isn’t just emotional but also tied to reliability.
Entertainment
Interview participants also use AI chatbots for entertainment purposes through roleplay and storytelling. AI chatbots can be used to interact with fictional characters, create personalized scenarios, and engage in said immersive scenarios. This makes AI not just a tool, but an experience, for some people these interactions provide a substitute for real-life intimacy.
Escapism
Many of our participants use AI as a form of escapism during difficult situations. Through our interviews with our participants we found that AI was used to cope with stressful relationships, isolation from them, or feelings of loneliness or distress. AI provides a safe space where users can relieve pressure and stress and escape reality even for just a moment, due to AI responses being non-threatening and overly supportive, it makes the users feel more comfortable sharing their vulnerabilities that they otherwise wouldn’t share with real people.
Negative experiences
Reliance on AI has affected people’s confidence in their own judgement as our participants mentioned second-guessing their choices and relying on AI for validation and feeling much less confident in their own thinking. This suggests that AI can reduce someone’s ability to practice their sense of personal agency while making them less confident on top of that.
Impact on self-esteem
Reliance on AI has affected people’s confidence in their own judgement as our participants mentioned second-guessing their choices and relying on AI for validation and feeling much less confident in their own thinking. This suggests that AI can reduce someone’s ability to practice their sense of personal agency while making them less confident on top of that.
What do experts say?
AI Specialist
An AI specialist mentions that companies nowadays have been implementing AI into their apps to hook users into using them, creating a feedback loop of improving AI from user data to enhance user engagement. This normalization of AI chatbots further fuels the curiosity of individuals to try and engage with AI technology, making potential grounds for more individuals to develop attachment or reliance emotionally to these systems. This marks the importance of having a formal governance to dictate on how AI should be designed.
Psychologist
A psychologist adds that AI acted as a “yes-man” for the user, hindering the growth of critical thinking as AI does not challenge the thoughts and opinions of the user. Whereas in real human interaction, although it may sometimes bring a person out of their comfort zone, does challenge another person’s thoughts which helps in a person’s growth. A concern was raised on how this causes users to misplace unrealistic expectations of AI onto real people.
RElated articles
Study exposes privacy risks of AI chatbot conversations
User data is used for training.
Many AI companies take user data to better improve their chatbot systems, users can choose to opt out, but the setting to automatically give user data to the companies are on by default. This practice being widespread means that most AI chatbot conversations eventually down the line can be used to contribute to future AI development which makes users concerned about how much they share their information.
Sensitive info might be collected.
Many users of AI chatbots can potentially have their sensitive and private information collected through their conversations with it. Users might overshare and carelessly give the chatbot personal information about themselves in a moment of vulnerability which can then influence targeted ads or other uses.This can especially be bad when it’s so easily accessible that even children’s data can be collected if not careful.
Lack of transparency.
AI companies will tend to fail or try to avoid explaining how they handle their users personal data. Their privacy policies being so long and difficult to understand that most users will look over it not knowing just what malicious intent the AI companies might have for their personal data. This lack of transparency makes it more difficult for users to be informed and gain control of their private information.
Weak regulation.
With how relatively new AI chatbots are, privacy protection for AI chatbots are very limited, especially in places like the United States as laws vary state by state as there is no comprehensive federal regulation. With how inconsistent and fragmented these regulations are, it makes it difficult to enforce strong and effective privacy standards in AI companies. As a result, the developers have a lot more freedom in collecting and using whatever data they want in any way they please.
Top 4 AI chatbot privacy concerns and how to mitigate them
Chatbots can store sensitive information.
AI chatbots massive amounts of user data including but not limited to conversation history, financial information, health records, and personally identifiable information. Since users treat the AI chatbot like it’s their friend/assistant they share confidential information without even noticing and can hold that information for a very long time. This can make that stored data prime targets for cybercriminals as a breach could expose sensitive information that can cause harm to users and organizations.
Unauthorized access to User data.
Despite how complex AI chatbots are to operate, they can be vulnerable to unauthorized access through many technical shortcomings. Such as insecure APIs, ineffective authentication systems, session hijacking, and weak access controls in admin dashboards. Cyber attacks can exploit these vulnerabilities in order to gain access to sensitive data or even take control of the chatbot systems. With the addition of third-party service integrations, it can introduce further risks if not properly secured.
User profiling and Data misuse.
Chatbot providers can also collect and analyze large amounts of user data to create detailed profiles of their users. This includes tracking their behaviours, preferences, and interactions which can then be used for targeted ads or shared with data brokers. And with all of the information taken from the chatbot providers, that same data can be potentially misused in areas like employment or insurance.
AI is making the hard choice between consumer safety and privacy even trickier
Privacy and safety trade off.
Modern AI technology, not just AI chatbots but even surveillance tools, often have to force users into a difficult trade off between “privacy and safety” or “convenience”. While this technology promises accessible convenient service, they rely on collecting and storing your personal data which can compromise privacy. Many people are okay with this trade-off (often not realizing it) as they consider the convenience and use of the technology outweigh their privacy concerns. However experts have noted that the growing dependence on AI is slowly reshaping people’s expectations of what privacy really means.
Companies depend on taking user data.
Data collection is not just a side effect of AI systems, but also a core part of how many tech companies operate. They heavily rely on user data to improve their services and products to maximize profits making people’s personal data a valuable asset. In many ways even outside of AI tech, people’s daily lives are constantly recorded, gathered and analyzed almost making their customers assets themselves.
Laws and enforcements are currently very weak and still developing.
With how new and constantly evolving AI technology is, existing and newer privacy laws are just not keeping pace with the rapid advancements of AI technology. Regulations everywhere being inconsistent, fragmented, outdated, and overall insufficient with penalties too small to make a significant impact on the large tech companies taking advantage of the situation. Experts describe the situation as a “digital wild west” as tech companies face very little pressure to change their data practices as there is little accountability done for their actions leaving both users and policy makers struggling to address complex challenges posed by AI systems
Are Chatbots Safe? A Look at User Privacy Concerns
Different continents have laws of varying strength pertaining to AI privacy regulation.
Privacy regulations differ significantly across all continents within all countries with some places enforcements stronger than others. The European Union has especially been noted in being exceptional in implementing stricter laws and regulations against AI. The United States, in contrast, lacks the amount of security that the European Union provides, leaving only the AI companies to do it themselves only if they choose to, leaving an imbalance that results in uneven levels of user protection across the world.
Users face many risks when interacting with chatbots.
There is a wide range of privacy risks that users can encounter when using AI chatbots such as data collection and storage, data breaches and misuse of personal information. Outside of stealing their data, AI chatbots can also cause harm to the user through means of manipulation, distrust, and excessive self-disclosure as the chatbots are designed to feel human-like. And with the shaky legal compliance, if there are any at all, it can further harm users overtime.
Given how new this all is, much work and research needs to be done.
Research on chatbot privacy is still developing. Studies may have increased in recent years, but there remains large gaps in information significant enough to take effective action. The article calls for more research combining social science, policy and technology to better understand and tackle these challenges.