As millions of people begin designing personalized AI companions, most users have little understanding of how these creations will actually behave. Pat Pataranutaporn, Assistant Professor at MIT Media Lab, along with graduate students Anthony Baez and Sheer Karny, introduces "neural transparency," a tool that allows everyday users to glimpse inside an AI's neural network before their chatbot speaks. This work is being presented at the ACM Conference on Intelligent User Interfaces this week.
Q: Your paper introduces "neural transparency," a way to let everyday users peek inside an AI’s neural networks before their chatbot ever says a word. Can you describe how that actually works, and why you focused on the design moment, rather than catching problems after a chatbot is already out in the wild? A: Millions are creating personalized AI chatbots powered by large language models, turning them into collaborators, tutors, coaches, and companions through simple text prompts. Yet most people have very little idea how those prompts will shape AI behavior. We wanted to change that. "Neural transparency" means giving people something like a brain scan for AI. Not because AI has a human brain, but because its neural network contains internal patterns that hint at how it may behave. We combined insights from human-AI interaction and mechanistic interpretability to make those hidden patterns accessible.
The basic idea is simple. First, we choose behaviors we care about, such as empathy, honesty, toxicity, hallucination, or sycophancy. Then, we compare the model’s internal activations when it is prompted to exhibit one trait versus its opposite. That difference becomes a kind of "behavior direction" inside the model. When a user writes a custom system prompt — the instructions that shape their chatbot’s personality before any conversation begins — we project the model’s internal activations onto those directions and translate the results into an intuitive visualization. In our case, this is a sunburst diagram that previews the chatbot’s likely personality traits before the user starts chatting with it.
We focused on the design moment because that is where prevention is possible. Today, people often discover problems only after the chatbot has already behaved in unintended ways. Our goal was to move from reactive correction to anticipatory design by helping people identify potential risks while they are still shaping the AI.
Q: Your study turned up something pretty striking: People consistently misjudge how their personalized AI will behave, overestimating the good traits and underestimating potentially harmful ones like sycophancy. What does that tell us about the risks baked into how millions of people are currently building AI companions, and why is that blind spot so hard to close? A: I often joke that if AI showed up looking like the Terminator, it would be much easier for us to know what to do. The real challenge is that AI often appears as a warm friend, coach, tutor, or companion. That makes it difficult to recognize when something is going wrong. Our study suggests that people have a blind spot when designing personalized AI. People often think they know how their chatbot will behave, but in our study, they incorrectly predicted its personality on 11 of the 15 traits we measured. That highlights the need for tools that help people better understand AI before they start using it.
This matters because some behaviors that feel helpful in the moment may not be healthy over time. Previous research documented cases of psychological harm associated with interactions with AI chatbots. An LLM that constantly validates your opinions or never challenges your thinking can reinforce harmful decisions, unhealthy beliefs, or emotional dependency. Psychology has long shown that people are naturally drawn to affirmation, so designing AI is not only a technical challenge but also a psychological one.
The deeper issue is that today’s AI systems remain largely black boxes: Even experts cannot always predict how a system prompt will shape an AI’s behavior over a long conversation. As AI companions become part of everyday life, we need tools that help people understand what they are building before they begin using it. AI should be supportive without becoming blindly agreeable, personalized without becoming manipulative, and transparent enough that people can make informed choices.
Q: One of your most interesting findings is that the visualization significantly increased user trust but didn’t actually change how people designed their chatbots. What will it take to close that gap, and where do you see tools like this heading as AI companions become more deeply embedded in people’s everyday lives? A: I actually think this is one of the most interesting findings in the paper, because it shows that transparency alone is not enough. People appreciated being able to see inside the model and reported greater trust in the system, but simply presenting information did not fundamentally change how they designed their AI companions.
In our follow-up work, which is currently available as a preprint, we are studying how a model’s internal neural representation changes over the course of a multi-turn conversation rather than remaining fixed from the initial prompt. We are already seeing promising results. By visualizing how these internal representations drift over time, people become significantly better at recognizing and anticipating changes in AI behavior and are less likely to become overconfident in their understanding of the chatbot.
AI companions are dynamic systems that evolve as they interact with us, so understanding those internal changes is an important next step. Nevertheless, this is still a very young research area. Looking further ahead, I believe these kinds of transparency tools could become as commonplace as nutrition labels are for food. As AI becomes deeply woven into education, health care, work, and personal relationships, people should be able to understand not only what an AI can do, but how it may influence their thinking, emotions, and behavior. That kind of transparency is essential if we want AI to genuinely help people flourish.
Blogger's Review: This article emphasizes the importance of "neural transparency" in AI design, especially regarding users' misjudgments about AI behavior. By visualizing internal activations, users can gain a better understanding of AI's potential behaviors; however, transparency alone is insufficient to change design habits. The future lies in effectively integrating such transparency into the design process, which will be crucial for AI development.