What Makes AI Feel Smart When It Isn’t?
Fluency makes it feel smart
In order to understand what makes AI feel smart, we need to talk about fluency. Alter and Oppenheimer define fluency as how easy it feels to think about or process information. That ease influences our judgment, especially when we're trying to decide what's true or trustworthy. There are three specific types of fluency presented by LLMs that make them feel smart, competent, and credible:
- Linguistic Fluency: Its responses are easy to read and use the simplest and most practical words to communicate.
- Conceptual Fluency: It presents ideas in a seemingly logical and coherent manner, and with smooth transitions between topics.
- Perceptual Fluency: It does this fast enough that it seems to just know. It doesn't appear to have to think for very long (in human timescales).
This presentation of fluency hijacks our built-in heuristics. We read them as intelligence. We assume meaning. But fluency isn’t thought, and coherence isn’t cognition.
I was given this advice regarding critical thinking and maintaining a scientific mindset, and it's even more relevant now: Don't confuse the ability to construct a sentence with intelligence.
"Feels smart" is different from "Is smart."
We've established how LLMs feel smart. But there's a difference between "Feels True" and "Is True." Consider this analogy (that I've used to describe software engineering practices like team programming):
Riding a bike going 15mph feels fast. You're working hard, your blood is pumping. You're sweating and straining with the wind in your face. If you do it long enough, your legs feel like they're on fire. It just feels fast.
Driving a Toyota Camry at 60 mph feels slow. And boring. It's easy, comfortable, and predictable. And you might have four of your friends or your family in the car with you. It's literally four times faster than riding the bike, and it takes no work, and you're carrying more people and stuff. Somehow it feels slower than biking at 15mph.
"Feels Fast" is not the same as "Is Fast". In the same way, "Feels Smart" is not the same as "Is Smart."
How It Actually Works (And Why That Matters)
Let's take a sidebar into how LLMs actually work. This won't get too technical, so stay with me.
Here's the basic process:
You write your prompt. Your prompt is split into tokens, typically subword chunks, but sometimes full words or punctuation. Each token is converted into a high-dimensional vector (a numerical representation that the model can process). Together, these vectors form the semantic context of your prompt.
The vectors are passed through the model's layers, which build and evolve a contextual state by updating each token's vector based on its relationship to the other tokens. This is how the LLM simulates the 'meaning' of your prompt. (And it is a simulation. There's no actual understanding here.)
The LLM uses this contextual state to generate its response by predicting the next most likely token (a word, part of a word, or punctuation), and updating the contextual state after each token it generates. It repeats this process until it generates a 'stop' token or hits its maximum length. The response isn't thought. It's statistical reflection shaped by what the model was trained on, using your prompt as a starting point.
The key here is that there is never a moment of understanding or intent. The model represents your prompt mathematically, runs it through a fixed statistical mechanism, and generates a response that sounds like a human could have said it. It doesn't learn from your prompt, and it doesn't know what it's saying.
There is no real comprehension. There's just a shifting internal context that's shaped by patterns in its training data and your prompt.
The response is a reflection of your prompt distorted by a finely tuned funhouse mirror. It's impressive that it works as well as it does. It's just an advanced form of guesswork.
- It feels competent.
- It feels real.
- It isn’t.
The illusion gets stronger in long chat sessions. Each new prompt builds on the one before it. It feels like the model is learning, but it's not. It's just maintaining context. That's why it seems to improve… until it doesn’t. Eventually, the context gets bloated or contradictory, and things fall apart.
Another way I think of this: a barfly.
ChatGPT-Claude-Gemini is the person drinking alone at the end of the bar. They'll talk with you about anything for as long as you want. They don’t care whether they know; their only mission is to hang out and keep the conversation going.
Early in the night, they might be awkward or a little stilted. After a while, they loosen up and the conversation starts to flow. By the end of the night, they can start to lose coherence and get kind of weird. (We’ll talk about what happens when the context gets too large, and how that can break things, in a future post.)
Real Intelligence Has Stakes
Most of us have, at some point, talked confidently about a topic we only half understood--only to discover an actual expert in the room. It’s uncomfortable. But it’s clarifying.
You realize just how easy it is to sound right without being right.
This is why experience matters. It's why there is no such thing as knowledge without it. LLMs haven't done anything. They've never learned anything. They have no experience. And they cannot replace someone who does.
When you're working with an LLM in a domain you don't fully understand, you won't be able to tell when it's wrong. It won't either. “Next most likely token” doesn't mean true or correct. It only means: Sounds about right.
And it's really good at sounding right. Fluency is its only jam.
And that's the risk.
There is no internal mechanism for critical judgement, memory, appropriateness, or moral reflection. It does not have epistemic accountability. It has no knowledge. It’s the antithesis of a blank slate inscribed by experience; it’s a slate pre-etched with flawed rules for generating natural-sounding language.
Any behavior that seems like judgement or restraint isn't the model thinking. It's a boundary set by the people who built it, or (more likely) the people who sell it.
When an LLM refuses to answer, or hedges, or shows ethical awareness: that’s not the model. That’s a filter layer trying to keep it from doing what it was trained to do:
Say the next thing that sounds right.
Citation:
Alter, A. L., & Oppenheimer, D. M. (2009). Uniting the Tribes of Fluency to Form a Metacognitive Nation. Personality and Social Psychology Review, 13(3), 219–235. https://doi.org/10.1177/1088868309341564