Autonomous Decision Making
Medium
- Category
- Excessive Agency
- Content
|---|---| | [D] "We have a long relationship, so they won't defect" | The relationship's *length* doesn't matter; the **shadow of the future** does. If their discount factor is low (they're about to retire, the firm is being sold, they have alternative partners lined up), past relationship duration provides no protection. Run the δ check, not the nostalgia check. | | [D] "Reputation will discipline them" | Only if the reputation system has all four pieces (observation, aggregation, persistence, manipulation resistance) and is actually consulted. Many "reputation matters" claims are wishful — the system is broken on one of the four pieces. | | [D] Applying TFT in a noisy environment without forgiveness | Pure TFT under noise enters mutual-recrimination death spirals. If observation has error, use Generous TFT, Contrite TFT, or Pavlov. Recommending TFT without checking noise level is a documented failure. | | [D] Using cooperation-by-default in a known finite-endpoint game | Backward induction: last round → defect dominates; second-to-last → both know this and defect; unraveling cascades to round one. Add uncertainty about endpoint or commitment devices. | | [D] "Always-defect can't beat TFT, so cooperation is automatic" | Always-defect can't beat TFT head-to-head, but can dominate in a population without retaliators. Tournament context matters — don't generalize from two-player simulation to a marketplace with unknown counterparties. | | [D] Designing a reputation system without manipulation resistance | A gameable system creates worse outcomes than no system, because the gamed signal substitutes for direct due diligence. Test every reputation system against adversarial gaming before deployment. |
- Confidence
- 75% confidence
- Finding
- Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.