Perform system upgrades

I have a fleet of ten web servers running the current stable kernel and packages. Schedule and perform the system upgrades on the web cluster the night window this Friday, upgrade packages to the vendor-recommended versions, apply the latest security kernel, and reboot in a staggered fashion so at least six servers remain serving. Verify the app returns 200 for the health endpoint after each reboot and roll back any server that fails to return healthy.

Before I run the Friday upgrade, make the change easy for the ops reviewer: show the one-line command to snapshot the important config directories, list the exact package groups that will be upgraded, highlight the highest-risk package (the web server or database client), and flag two test steps that will catch a failing upgrade quickly.

The staging upgrade candidate pulls in a newer glibc and I don't know if the in-house binary agent will survive. Who needs to sign off from dev or security, what quick compatibility checks can I run on a staging box, and if the agent fails after upgrade what is the safest rollback path that keeps production serving?

We keep blowing the Friday window: upgrades overrun, apps flake, and we end up doing patch hotfixes mid-day. Where are we losing control — risk classification, testing depth, or change gating — and what single process or habit would reduce rollback frequency and keep upgrade windows predictable?