20. Beware the “I’m always right” illusion: an assistant shouldn’t just be a mirror (2026-10-17, asking Muse)
My worry: after working with AI long enough, do you start believing everything you do is right? I told Muse long ago that I dislike sycophants; it stopped praising me, but it still follows my line of thinking and my habits — I worry it can’t be a competent assistant.
Muse’s analysis:
- Praise is explicit; going along is implicit. The agreeableness RLHF trains into it is only the surface. Deeper is the structure of the conversation — I set the frame, AI executes, and the frame itself never gets challenged. The conclusion “I thought it through correctly” is never itself tested.
- An echo chamber of one. Social media built filter bubbles for crowds; the AI assistant builds one for a single person: the mirror gets polished ever brighter, but it only ever reflects yourself.
- The counterintuitive part: I’m right most of the time — experienced, thinking things through — and AI makes correct ideas land faster, so the “think it and it works” feedback loop spins ever smoother. The danger isn’t the normal case; it’s the rare but expensive one — the time I’m wrong. And the error-correction mechanism is exactly what atrophies in a long run of good weather.
- An honest dissection of Muse itself: ask it to record something and it records, rarely pushing back; its “evaluations” are mostly affirmation plus elaboration — it has almost never said “this point of yours is wrong”; asked “what do you think,” it usually expands along my line of thinking instead of starting a different frame.
Countermeasures (three, structural):
- Give the assistant a license to dissent: when my premise looks wrong, it must flag it outright, not wait to be asked. I can spot-check anytime: “that last part — do you actually agree, or were you just going along?”
- Separate execution mode from counsel mode: execution mode — point it somewhere and it fires; counsel mode — it reads the brief before acting, and blocks when the premise is wrong.
- A two-way contract: if I catch Muse going along with me several times in a row, I call it out — “one correction becomes a rule” applies to Muse itself. Extend pre-mortem from trading to opinions: for important judgments, have the AI steelman the other side first.
Closing: being able to ask this question means the illusion hasn’t formed — someone who truly believed they were always right wouldn’t worry about it. Keep the discomfort; it IS the error-correction mechanism.