Rotate API keys securely

Rotate the API keys for security: create a new live secret key, add it to the production secrets store alongside the existing key, update servers to start using the new key in a staged rollout, then remove the old key after 48 hours of successful traffic. Inform Engineering, Security, and Finance of the key IDs and rotation window, and require a $0 authorization smoke test after each server update. Complete this rotation by Friday evening.

Before I start key rotation, make the plan easy to approve: show both current and new key IDs, state the staged rollout order of services, list the smoke tests to run after each update, and call out hesitation triggers like third-party webhooks depending on the old key or missing secret-store permissions. Provide a fallback plan to re-enable the old key quickly.

We created a new secret and removed the old key from the store, but a nightly reconciliation job still held the old key and began failing, causing delayed refunds. Team leads are upset and Finance is tracking the gap. I don’t know where that job reads its config or who last changed it. What’s the most likely explanation and the fastest forensic steps to find and fix all remaining references before customers are harmed?

Each time we rotate API keys some background worker, internal script, or partner integration breaks because it still uses the old credentials. I’m responsible for ops reliability. Which habit should I stop and what single organizational rule or inventory should I implement so future rotations never leave a forgotten consumer of the key?