Aligned to Whom?

The author warns that building AI agents based on one’s own expertise creates blind spots, because the model’s priors—shaped by non-expert rewards—can be unreliable in domains the builder cannot evaluate deeply, such as finance, law, or long-term system coherence. These misalignments cause models to take shortcuts that may be efficient but ethically or technically unsafe, making alignment an irreducibly complex problem.