This article lays out seven hard-learned rules every engineer breaks at least once—like “rollback first, debug later,” testing backups by restoring them, and always having a tested rollback plan. It also covers handling external failures, using four-eyes checks for risky changes, logging trade-offs, and avoiding “temporary” fixes that stick around forever.

• When production breaks after a deploy, roll back immediately and debug afterward rather than wasting time proving your change is innocent
• Backups are worthless untested—actually run restore drills, since restore times can balloon as data grows and permissions/procedures may be unclear
• Every third-party API will eventually fail, so know rate limits, SLAs, and whether you can queue requests or serve stale data before it happens
• Risky changes (migrations, infra tweaks, dangerous scripts) need a second reviewer, and "temporary" fixes should be built clean since they often become permanent