Skip to content
WorakornKoonawootrittriron
Back to blog
Network security

Change windows: write and test the rollback before you start

A before, during and after checklist for firewall changes: written go/no-go checkpoints, config snapshots and a tested rollback, mapped to NIST SP 800-128.

Intermediate9 min read

On this page
  1. Why change management counts as a security control
  2. Before the window
  3. During the window
  4. Vendor features that help
  5. Rollback as a first-class plan
  6. After the window
  7. The checklist

A firewall rule is rarely hard to write. What goes wrong is usually everything around it: the step nobody wrote down because "we always do that", the check that was skipped because the window was running late, the rollback that turned out to need a file nobody had saved. Google's SRE team puts a number on the general pattern from its own experience: in their words, roughly 70% of outages are due to changes in a live system.

These are my field notes on running a change window, cleaned up into a checklist you can copy. They are generic on purpose: no vendor is assumed except where I show one product's features as an example. The checklist is split into before, during and after, with one idea running through all three: the rollback is a plan in its own right, written and tested before the change starts, not something you work out at the moment you need it.

Why change management counts as a security control

It is tempting to file change management under "process" and security under "technology". NIST doesn't. SP 800-128, Guide for Security-Focused Configuration Management of Information Systems (August 2011, updated October 2019) describes four phases: planning, identifying and implementing configurations, controlling configuration changes, and monitoring. In the change control phase, the goal is that changes are "formally identified, proposed, reviewed, analyzed for security impact, tested, and approved prior to implementation", and the tools it lists for limiting unauthorised change include access controls, process automation and change windows (SP 800-128, section 2.2.3).

Rollback is in there too. Among the things a configuration change control procedure should cover, SP 800-128 lists "requirements for rollback of changes in the event that problems occur", and its sample change request template has a field for the work plan "including change implementation date, deliverables, and backout plan" (SP 800-128).

The control catalogue says the same thing in control language. In NIST SP 800-53 Rev. 5, the Configuration Management (CM) family includes (full text, PDF):

Control What it asks for, in short
CM-2(3) Retention of previous configurations Keep a number of earlier baseline versions "to support rollback"
CM-3 Configuration change control Decide which changes are controlled, review and approve them with security and privacy impact in mind, document decisions, keep records
CM-3(2) Testing, validation and documentation of changes Test, validate and document changes before finalising them
CM-4(2) Verification of controls After a change, check the affected controls work as intended

Read as a group, those four controls already describe a change window: save the old state, agree the change, test it, check it afterwards. The rest of this post is about doing that at 1 a.m. with tired people.

The rollback lane is drawn before the change starts. At the checkpoint the cursor either carries on to verification or takes the lane back to the snapshot.

Before the window

Most of the work in a calm change window happens days before it opens.

1. Write the change as steps, including the "habit" steps

The steps that cause trouble are the ones experienced people do without thinking: disabling and re-enabling a NAT rule, adding a static route, clearing ARP on a neighbour after a move. If a step is only in someone's head, it is missing from the rollback too. Put every one of them in the plan as a numbered pre-task, task or post-task.

2. Save the current state where you can reach it

Take a backup of the running configuration and keep a copy off the device. SP 800-53's CM-2(3) asks for exactly this, and it is the one thing a rollback can't do without. If you are migrating between platforms, also compare the old and new configurations side by side with objects and groups expanded, so missing objects show up on paper rather than in production.

3. Capture "before" evidence

Run the checks you will run afterwards, now, and save the output: a ping from each interface to its known neighbours, the routing table, session counts, VPN tunnel status, a login to the key applications. "It worked before" is only useful if you can show it.

4. Write down what success looks like

Pick a short list of checks with a clear pass or fail. Google's guidance on canary releases suggests choosing only the top few metrics, "perhaps no more than a dozen", and starting from the indicators that show real problems in the service (SRE Workbook, Canarying Releases). The same discipline works for a firewall change: if you can't say in advance which result means "stop", nobody will agree on it at 2 a.m.

5. Write the rollback trigger and the times

The plan should name, in writing:

  • a start time;
  • a checkpoint time by which the change must pass its checks;
  • a deadline by which the rollback itself must be finished, leaving room before business traffic returns;
  • the trigger: the specific results that mean "roll back", for example any success check failing at the checkpoint.

Time-box the decision, not just the work. The question at the checkpoint is not "can we fix this?" but "did it pass?". If the answer is no, the plan already says what happens next.

6. Name one person who calls go or no-go

The SRE book's chapter on incidents makes the case for a single incident commander who holds the overall state, and it describes how one well-meant but uncoordinated change made an outage worse (Managing Incidents). A change window is a planned version of the same situation. One person calls each checkpoint; everyone else reports facts to that person.

During the window

7. Small steps with a check after each

Apply the change in the order written, and run the relevant check after each step rather than all checks at the end. When something breaks, you then know which step did it. Google's release engineering chapter makes the same argument at a larger scale: frequent releases mean "fewer changes between versions", which makes testing and troubleshooting easier (Release Engineering).

8. Make physical steps reversible too

If cables move, label old and new ports before you touch them, so putting them back is a step anyone on site can follow.

9. "Passed in test, failing in production" means stop

This is the hardest rule to keep and the most useful. If the success checks fail at the checkpoint, start the rollback. Debugging in production past the checkpoint eats the time the rollback needs, and the plan you wrote calmly last week is a better judge than the team at the end of a long night.

10. Communicate small and factual first

Update the people in the change room with what you see. Hold wider announcements until the cause is clear, so you don't have to correct them later.

Vendor features that help

Most firewall platforms keep configuration history and let you restore it. Palo Alto Networks' PAN-OS is a clear example because it separates two configurations: the candidate configuration, where your pending edits live, and the running configuration, which the firewall enforces. A commit is what activates pending changes (Commit, Validate, and Preview; Revert Firewall Configuration Changes).

That split gives you several tools for a change window:

  • Before you commit: Preview Changes compares your candidate configuration with the running one, and Validate Changes reports the errors and warnings a commit would raise without touching the running configuration (same page).
  • Snapshots: the firewall creates a configuration version each time you commit, and you can also save a named snapshot and export it to an external host (Save and Export Firewall Configurations).
  • Reverting: you can revert the candidate to the running configuration to throw away uncommitted edits, or load an earlier configuration version or a named snapshot. Loading replaces the candidate configuration, so the restored settings take effect only after you commit them (Revert Firewall Configuration Changes).
  • Afterwards: Config Audit compares two configuration versions and shows who committed what. Note one limit from the documentation: load and revert operations are not captured in the audit (Perform a Config Audit), so write your rollback down in the change record.
Take the snapshot first. When the new rule fails its checks, it leaves and the saved configuration goes back.

Some platforms go a step further and roll back on their own. On Juniper's Junos OS, commit confirmed activates a change provisionally; if nobody confirms it within 10 minutes by default, the device "automatically rolls back to the previous configuration" (Juniper documentation). That protects you from the classic mistake of locking yourself out of a remote device. Whatever your platform offers, check it in a lab before you rely on it in a window.

Rollback as a first-class plan

A rollback plan deserves the same care as the change plan. In practice that means:

  1. It has its own steps, in order, written as carefully as the forward steps, including the "habit" steps in reverse.
  2. It names the file or version to restore, and where that file is.
  3. It has been tried, at least in a lab or on a non-production pair. A restore you have never run is a hope, not a plan.
  4. It fits in the time left. If the rollback takes 40 minutes, the checkpoint must be more than 40 minutes before the deadline.
  5. It has its own checks: the same "before" evidence from step 3, run again.

After the window

Verify, then keep watching

Run the success checks again and compare them with the "before" evidence. This is CM-4(2) in practice: confirm that the controls you changed are "operating as intended" (SP 800-53 Rev. 5). Then keep an eye on logs and monitoring for a while; some problems only show up when the morning traffic arrives.

Write it up, briefly and without blame

Update the change record and the baseline documents. If you rolled back, a short write-up helps the next attempt. The SRE book describes a postmortem as "a written record of an incident, its impact, the actions taken to mitigate or resolve it", plus root causes and follow-up actions, and says a blameless one focuses on causes without blaming individuals (Postmortem Culture). If the vendor is involved, ask them to reproduce the problem in their lab and agree on the date of the next attempt in the same conversation.

The checklist

Before

  • Every step written down, including habit steps
  • Running configuration backed up and copied off the device
  • Old and new configs compared (for migrations)
  • "Before" evidence captured
  • Success checks with clear pass/fail
  • Start time, checkpoint, deadline and rollback trigger written
  • One go/no-go caller named
  • Rollback steps written and tried in a lab

During

  • One step at a time, check after each
  • Cables labelled before moving
  • Checkpoint called on time; fail means roll back
  • Small, factual updates

After

  • Checks re-run and compared with "before"
  • Monitoring watched into business hours
  • Change record and baseline updated
  • Short blameless write-up if anything went wrong

None of this is clever, and that is the point. A change window goes well when the hard decisions were made in daylight, by people who wrote them down.

Sources

Keep reading

Related posts

Back to blog