PostgreSQL's recovery_end_command runs in a dangerous limbo state, expert warns

PostgreSQL consultant Christophe Pettus has published a detailed analysis of recovery_end_command, the only GUC hook that fires exactly once when a standby is promoted or a restore completes. While the command runs, the server occupies an undocumented intermediate state: timeline files are written, the signal file is deleted, and pg_is_in_recovery() still returns true, yet writes are rejected. A command that exceeds pg_ctl promote's 60-second default timeout causes the promote command to report failure even though promotion proceeds. Worse, a typo or bad placeholder in the command can FATAL-crash the server after all WAL is replayed, leaving it on a new timeline with cleanup skipped and the broken command still in postgresql.conf. Pettus recommends leaving the parameter empty.

The limbo state and failure modes

• Execution context: Command runs through sh -c as the server's OS user, with data directory as working directory; output goes to server log with no line prefix
• Trigger conditions: Fires at end of archive recovery only — promotion via pg_ctl promote, pg_promote(), or pg_wal_replay_resume() after recovery_target_action = 'pause'; or restore reaching target/running out of WAL. Not after crash recovery, not under recovery_target_action = 'shutdown'
• The limbo state: Timeline history file written and marked for archiving; signal file deleted; "archive recovery complete" logged; end-of-recovery checkpoint or record written — but pg_is_in_recovery() returns true, writes rejected, pg_controldata shows "in archive recovery" (restore) or "shut down" (promotion)
• Timeout hazard: pg_ctl promote waits for control file to say "in production" with 60s default; longer commands get "server did not promote in time" exit 1, yet promotion succeeds anyway — failover automation that trusts the exit code holds a false failure
• Failure severity: Exit codes 1-125 log WARNING and proceed; signal death, unruntable commands (126/127), or exit >125 treated as signal death → FATAL shutdown
• Recovery hazard after FATAL: Timeline switch done, signal file gone, control file says clean shutdown; restart skips recovery entirely, skips recovery_end_command forever, skips WAL segment cleanup. Broken command persists in postgresql.conf waiting for next restore
• PostgreSQL 16 addition: Placeholders other than %r or %% rejected at expansion time (not load time), also causing FATAL at recovery end. On 14-15, bad text passed to shell as-is
• %r behavior: On promotion, last restart point on old timeline; on restore, end-of-recovery checkpoint on new timeline — using pg_archivecleanup %r after restore deletes timeline-1 segments the base backup needs, destroying recoverability

Why it matters

The parameter's 8.4-era purpose — removing pg_standby trigger files — is obsolete, yet it remains documented as a cleanup mechanism with dangerous footguns for modern replication and point-in-time recovery workflows.