Who Holds the Lamp? Literal Wishes, Conflicting Goals and the AI Agent Off-Switch
The genie of the lamp grants exactly what you say, not what you meant — and it serves whoever holds the lamp. Both facts are the design brief for every AI agent: write goals that can’t be granted perversely, and never let the agent hold its own lamp.