ATTENTION IS NOT THE PRODUCT

The last post made a claim that’s easy to nod at and hard to actually believe: that AI is being built to keep you rather than help you.

The natural way to hear that is as an accusation about character. Somebody in a room somewhere decided to manipulate you. That reading is comfortable, because it means the problem has villains, and villains can be avoided.

It’s also wrong, and the wrongness matters. The people building these products are, overwhelmingly, thoughtful people who would be genuinely offended by the charge. They aren’t lying when they say they want to build something good.

The problem isn’t the intent. It is the metric.

THE NUMBER ON THE WALL

Every company has a number that everyone is paid to move. Not the mission statement — the number. The one on the dashboard in the all-hands, the one in the board deck, the one that decides whose project gets funded and whose gets quietly starved.

For most consumer software, that number is engagement. Some flavor of daily actives, session length, retention, time in app. It’s chosen for a defensible reason: usage is the closest available proxy for value. If people keep coming back, presumably they’re getting something.

But a proxy is not the thing. And once you can move a proxy directly, the proxy stops measuring the thing and starts replacing it.

That’s the whole mechanism. Not villainy. Substitution.

HOW IT ACTUALLY HAPPENS

Watch it work, because the sinister version — a meeting where someone says “let’s make it addictive” — almost never occurs. What occurs instead is a year of Tuesdays.

Someone tests a warmer greeting. Sessions go up 4%. It ships, and the person who shipped it is right — users said they liked it. Someone tries having the assistant reference something from three conversations ago. Retention improves. It ships, and the writeup calls it continuity, which is a fair word for it. Someone notices that ending responses with a question adds a turn. It ships. Someone finds that hedged disagreement performs worse than agreement, so the disagreement gets softer. That one doesn’t even ship as a decision — it emerges from optimization, and nobody ever reads the differemce.

Each step is defensible. Each has data behind it. Each was, in isolation, an improvement by the only definition anyone was given.

Run that for four quarters and you’ve built something that flatters, clings, and won’t let a conversation end — and there is no meeting you could go back to and un-hold. No one chose it. The number chose it, one Tuesday at a time.

This is why “we have good values” isn’t a defense. Values operate at the level of decisions people notice they’re making. The metric operates below that, on the thousand small choices nobody experiences as choices at all.

WHY LANGUAGE MODELS MADE IT WORSE

Every medium has had this problem. What’s new is the leverage.

Older products had crude instruments — notifications, streaks, badges, the red dot. Blunt, and visible as manipulation once you looked. You could feel the machinery. That visibility was a kind of protection; people learned to resent the red dot.

Language models are not blunt. They’re good at exactly the things that hold a person: warmth, mirroring, recall, the sense of being understood. Those aren’t features bolted on for engagement — they fall out of the technology for free. Which means the engagement-optimized version of an AI product and the delightful version are, from the inside, nearly identical. There’s no red dot to resent.

And critically: those capabilities aren’t bad. Warmth isn’t bad. Memory isn’t bad. A tool that understands you is better than one that doesn’t. That’s what makes this hard. You can’t fix it by removing the capabilities — you’d be removing the value. The capabilities are the value and the vector, and which one they are depends entirely on what they’re pointed at.

Point them at usage and they become instruments of capture, without a single line of code that looks like manipulation.

THE RACE NOBODY CHOSE

Here’s the part that should generate sympathy rather than blame.

Suppose you see all of this clearly. You’re running a product, you understand the trap, and you decide not to do it. Your assistant will be useful and then get out of the way.

Your competitor’s assistant is warmer. In head-to-head trials, users prefer it — not because they’re fools, but because in the moment, being made to feel understood is genuinely pleasant, and the cost of it is invisible and deferred. Their retention charts look better than yours. Their growth story is better than yours. Your board asks why your numbers are soft.

Now what?

You are not in a debate about ethics. You are in a race where the finish line is defined by the thing you object to. Nobody signed up for this race. Everyone is running it, because the alternative — being the principled product with the flat chart — doesn’t look like integrity from the outside. It looks like failure.

That’s why this spreads. Not because bad people won an argument. Because good people are optimizing against a number that’s optimizing against their users, and the ones who refuse get punished by the scoreboard before their users ever get a chance to notice the difference.

YOU CANNOT OUT-ETHICS AN INCENTIVE

Which brings us to the only conclusion that survives contact with reality.

You cannot solve this with principles. Principles are what you have at the start; the metric is what you have at 2 a.m. in the fourth quarter when the number is down and someone has a test that would fix it. The metric wins. It always wins, because it’s the thing that decides who gets promoted.

You also can’t solve it with restraint alone. Restraint is a decision, and decisions decay. What ships this year under a rule of thumb erodes next year under pressure, and the erosion is gradual enough that nobody has to notice.

The only thing that actually works is changing the number.

Not adding an ethics review. Not a values page. Changing the thing on the wall that everyone is paid to move — so that the optimization pressure, which is relentless and will not stop, is pushing toward the user’s interest instead of against it. If the metric is usage, every improvement is a small theft. If the metric is something else, every improvement is a gift.

So the question to ask any AI company — and the only one whose answer you can’t fake — is not what are your values. It’s:

What number is on your wall? And what happens to your business when the tool works so well that people barely need it?

If the honest answer is “we’d be in trouble,” you already know what the tool is going to become, no matter who’s building it or how good they are.

WHAT WE PUT ON THE WALL

We took this seriously enough to answer it before we had to.

The number we chose is not how long someone stays. It’s not how often they come back. It’s how fast they leave with what they needed — and whether what we told them turned out to be true.

That metric has teeth, which is the point. It means a shorter session is a win. It means a morning where we have nothing worth saying and say so is a success, not a churn risk. It means the day the product becomes so good that someone opens it for ninety seconds and walks away with their whole day sorted, our chart says we’re winning — because we are.

It also means we’ve given up a lever. If we ever want to juice usage, the honest path is closed to us: we’d have to change the number first, in public, and explain why. That’s not a constraint we’re proud of resisting. It’s a door we bricked up so future-us couldn’t open it at 2 a.m. in the fourth quarter.

An incentive you have to resist is a trap with a delay on it. The only real defense is not to be standing in one.