Inside a firewall HA failover: heartbeats, session sync, failback

A firewall HA pair is supposed to be invisible. Nobody thinks about it until the day the active unit loses power, a monitored link drops, or someone starts an upgrade, and then everyone wants to know why a VoIP call dropped, why a decrypted session reset, or why the core switch kept sending frames to a dead box for a while.

These are my field notes on what actually happens during a failover. They're vendor-neutral where I can manage it, with Palo Alto Networks (PAN-OS), Fortinet (FortiGate, FGCP) and Cisco Secure Firewall Threat Defense (FTD) terms side by side, and every vendor behaviour links to that vendor's own documentation. Defaults change between releases, so treat the numbers here as a starting point and check the docs for the version you actually run.

                      ┌──────────────────┐                      
                      │  UPSTREAM SWITCH │                      
                      └──────────────────┘                      
                        //■          \\\                        
                    //■/                \\\\                    
                //■/                        \\\\                
             /■/                                \\\             
    ┌────────────────┐                    ┌────────────────┐    
    │      FW-A      │                    │      FW-B      │    
    │     ACTIVE     │•············•······│     PASSIVE    │    
    │     ▪  ·  ·    │    HA heartbeat    │     ·  ▪  ▪    │    
    └────────────────┘                    └────────────────┘    
             \■\\         steady state         ////             
                 \■\\\\                  ■/////                 
                      ┌──────────────────┐                      
                      │        LAN       │                      
                      └──────────────────┘                      
                                                                

Note 1: most designs pick active/passive, and the vendors explain why

All three vendors can run a pair where one unit forwards traffic and the other waits. Two of them also offer active/active, but read the fine print before choosing it:

• Palo Alto supports both modes. Its own comparison says active/passive gives a simpler design and makes routing and traffic flow easier to troubleshoot, and that active/active does not load-balance on its own: you need ECMP, multiple ISPs or load balancers to spread traffic, and the surviving firewall must still be able to carry peak load with content inspection turned on (PAN-OS: HA Modes). VM-Series on AWS and Azure supports active/passive only (PAN-OS: High Availability).
• FortiGate FGCP clusters can run active-passive or active-active, and all members must be the same model with the same firmware and hardware configuration (FortiOS 7.6: FGCP).
• Cisco FTD managed by the Management Center documents active/standby failover for its HA pairs, where one unit passes traffic and the other does not (FMC 7.4: High Availability for FTD).

The point I keep coming back to: active/active only helps if each unit can carry the whole load alone. If it can't, a failure turns into an overload instead of a failover.

Note 2: two kinds of link, two different jobs

Every HA design has a link that answers "is my peer alive and what is it?" and a link that answers "what is my peer doing right now?". The vendors name them differently:

Some details worth knowing:

• PAN-OS HA1 carries hellos, heartbeats, HA state, routing and User-ID sync, and configuration. Heartbeats are ICMP. HA2 synchronizes sessions, forwarding tables, IPsec security associations and ARP tables, flowing one way from active to passive, over Ethernet (EtherType 0x7261 by default), IP or UDP (PAN-OS: HA Links and Backup Links).
• FGCP heartbeats are Layer 2 frames with their own EtherTypes: 0x8890 for heartbeat, 0x8892 for session sync on a dedicated sync interface and 0x8893 for configuration sync. Fortinet recommends connecting two-member clusters back-to-back and keeping heartbeat traffic off user networks, because it contains cluster configuration details (FortiOS 7.6: HA heartbeat interface). Configuration sync runs over the heartbeat link on TCP/703 and UDP/703 (FGCP).
• FTD's failover link carries unit state, hellos, link status, MAC address exchange and configuration replication. The state link is optional but required for stateful failover; Cisco recommends under 10 ms of latency and no more than 250 ms (FMC 7.4 HA).

Split brain is the failure everyone fears: both units decide they're active and answer for the same IP and MAC addresses. Fortinet describes exactly this and offers a backup heartbeat interface to prevent it (HA heartbeat interface). Cisco notes that if the FTD failover link is down at startup, both units become active (FMC 7.4 HA). Redundant control links are cheap insurance.

Note 3: how long "dead" takes to decide

Detection time is a trade-off between speed and false alarms. The defaults are not the same across vendors, which surprises people who move between them:

• PAN-OS: heartbeat every 1,000 ms by default, and three consecutive losses trigger failover (PAN-OS: Failover). The passive unit then waits the promotion hold time before taking over: 2,000 ms in the "Recommended" timer profile, 500 ms in "Aggressive" (PAN-OS: HA Timers).
• FortiGate: heartbeats every 2 × 100 ms, and 20 lost heartbeats mark a member dead, so the default is about 4 seconds. Lower intervals, or the 10 ms unit on supported models, get it below a second (HA heartbeat interface; Failover protection).
• FTD: after three missed hellos on the failover link, a unit sends test messages on its other interfaces, and the standby takes over only if none of them gets a reply; the default detection time for a failed active unit is 15 seconds, configurable from 800 ms to 45 seconds (FMC 7.4 HA).

Faster isn't automatically better. A busy control plane that misses a few hellos can trigger a failover that causes more disruption than the "problem" it reacted to. Cisco, for example, mentions data-plane heartbeats as a way to avoid false failovers when the control plane is congested (same page).

Note 4: session sync decides what users notice

A failover that keeps the session table is "stateful": existing connections continue on the new active unit. But not everything is in that table.

 ACTIVE ●                               STANDBY ●               
 session table            HA data link  session table           
 ───────────────────────  ────════────► ─────────────────────── 
                                                                
                                                                
                                                                
                                                                
                                                                
                                                                
                                                                
                                                                
 ───────────────────────                ─────────────────────── 

What survives, roughly, per vendor:

• Palo Alto syncs sessions, forwarding tables, IPsec SAs and ARP tables over HA2 (HA Links). But "after a failover, firewalls do not support HA sync for decrypted SSL sessions" (PAN-OS: Reference: HA Synchronization).
• FortiGate: session pickup is not enabled by default (Session failover). With session-pickup on, TCP sessions sync; UDP and ICMP need session-pickup-connectionless as well, and Fortinet points out that QUIC (HTTP/3) needs it too (Session pickup). Sessions inspected by proxy-based profiles don't fail over; flow-based ones do, but are no longer inspected afterwards (Pass-through sessions). Management sessions to the firewall itself must restart, while IPsec SAs are synchronized (Terminated sessions).
• Cisco FTD replicates NAT translations, TCP and UDP connections, Snort connection states, ARP tables, IPsec SAs and routing tables, among others. It does not replicate DHCP leases, multicast routing, embryonic and short-lived connections, and decrypted TLS connections, which are reset (FMC 7.4 HA).

The pattern across all three: plain forwarded flows usually survive, and anything where the firewall holds heavy per-session state (decryption, proxies) usually doesn't. If decrypted traffic matters to a business, that's worth saying out loud before a planned failover, not after.

Note 5: the network has to agree the firewall moved

Switches forward frames by MAC address. If the new active unit answered with a different MAC, every neighbour would need to update its ARP cache. So HA pairs move the active unit's IP and MAC addresses (usually a virtual MAC) along with the active role, and the new active unit sends gratuitous ARP so switches relearn which port that MAC lives on.

This is the same idea as VRRP, the standard first-hop redundancy protocol. The current VRRPv3 spec, RFC 9568 (2024, replacing RFC 5798), defines a virtual router MAC of 00-00-5E-00-01-{VRID} for IPv4 (section 7.3), has a backup that takes over broadcast a gratuitous ARP (section 6.4.2), and has a backup declare the active router down after (3 × Active_Adver_Interval) + Skew_Time, where Active_Adver_Interval is the interval the active router advertises; the default advertisement interval is one second (section 6.1). RFC 5798 used the same values under the older "Master" names.

Firewall specifics:

• FortiGate: after failover the new primary's interfaces have the same virtual MAC and IP addresses as the failed primary. It sends 5 gratuitous ARPs by default, 8 seconds apart (arps, arps-interval), and link-failed-signal can briefly shut interfaces to force switches to react if GARP isn't enough. Each cluster in the same broadcast domain should use a unique group ID, or virtual MACs collide (Cluster virtual MAC addresses).
• Palo Alto: the page that describes this is about floating IP addresses in Layer 3 active/active HA. There, when a new active firewall takes over, it sends gratuitous ARPs from each connected interface to tell Layer 2 switches where the virtual MAC now is (Floating IP Address and Virtual MAC Address).
• Cisco FTD: the new active unit takes over the active IP and MAC addresses. If the secondary boots first, it becomes active with its own MACs because it doesn't know the primary's, so Cisco recommends configuring virtual MACs on both units. FTD also doesn't send gratuitous ARPs for static NAT addresses when the MAC changes (FMC 7.4 HA).

My check here is simple: before a test, know which switch ports both units connect to, and watch the MAC address table on the upstream switch during the failover. If the MAC doesn't move, the firewall isn't the problem.

Note 6: monitoring, preemption and flapping

A heartbeat only proves the peer is alive. Link monitoring and path monitoring prove it can still do its job:

• PAN-OS link groups and path groups let you decide whether any or all members must fail. Path monitoring pings every 200 ms by default and marks a destination unreachable after 10 failures (PAN-OS: Failover).
• FortiGate fails over on monitored interfaces, power loss, and optionally SSD failure or sustained memory use (Failover protection).
• FTD monitors named physical interfaces by default and fails over when failed interfaces reach the policy threshold (by default, a single failed interface causes failover), when more than 50% of Snort instances are down, or when disk use passes 90%. A failed standby or a failed failover link does not cause failover (FMC 7.4 HA).

Preemption decides whether the "preferred" unit takes the active role back when it recovers. Every failback is another failover, so it deserves a deliberate choice:

• PAN-OS: lower priority number wins; preemption is disabled by default and must be enabled on both firewalls (Device Priority and Preemption). The Flap Max timer (default 3) suspends a unit that keeps bouncing, counting a flap when a preemption-enabled unit leaves active within 20 minutes or a link or path doesn't stay up for 10 minutes (HA Timers).
• FortiGate: higher priority wins (0–255, default 128), but with override disabled (the default), uptime is compared before priority, with a 300-second margin. A monitored interface failure resets a unit's uptime. Fortinet's docs nickname the two orders MUPS and MPUS: monitored interfaces, uptime, priority, serial number (HA primary unit selection criteria).
• Cisco FTD active/standby: a recovered former active unit comes back as standby rather than reclaiming the role (FMC 7.4 HA).

Note 7: pre-change checks and a safe manual failover

Before I'd touch a pair in a change window, this is the list I'd want to be able to answer "yes" to:

1. Both units healthy and in sync. Same model, same PAN-OS version, up-to-date application, URL and threat databases, and matching licences: Palo Alto lists these as prerequisites (Set Up Active/Passive HA). Both units report their configuration as in sync.
2. Session sync is actually flowing. Palo Alto's upgrade guide confirms this by checking that the HA2 hardware counters increase (show high-availability interface ha2) (Upgrade an HA Firewall Pair).
3. You know what will reset. Decrypted sessions, proxy-inspected sessions, connectionless traffic if it isn't synced (Note 4).
4. A backout plan that doesn't depend on the thing you're testing: console access, and the commands to make the original unit active again.

Then use the vendor's own controlled failover, not a pulled cable:

• PAN-OS: Device > High Availability > Operational Commands > Suspend local device, confirm the peer turns active, then Make local device functional (Verify Failover or Suspend HA).
• FortiGate: execute ha failover set 1, and execute ha failover unset 1 to clear it. Fortinet says it's for testing and maintenance, inside a maintenance window only (Force HA failover).
• Cisco FTD: Switch Active Peer on the Devices > Device Management page of the Management Center, after refreshing the pair's status so the Management Center matches the units (FMC 7.4 HA).

During the test, I watch three things: the HA state on both units, the upstream switch MAC table, and one long-lived test session (an SSH session through the firewall is enough) to see whether it survives.

Note 8: where upgrades of an HA pair go wrong

Upgrades are when HA gets exercised hardest, because each unit reboots on purpose.

• Preemption fighting you. Palo Alto's procedure starts by disabling preemption on one peer, then suspends and upgrades one unit at a time, and re-enables preemption at the end. It also warns that peers two or more feature releases apart put the older one into a suspended state with "Peer version too old" (Upgrade an HA Firewall Pair).
• Assuming the order. FortiGate's uninterrupted upgrade upgrades the subordinates first, picks a new primary from them, then upgrades the former primary; the cluster may end up with a different primary. It also warns that CLI changes that need a reboot can reboot all members at once (Upgrading FortiGates in an HA cluster).
• Forgetting roles swapped. A Cisco TAC walkthrough describes FMC upgrading the standby first, then failing over before finishing the active, and recommends switching back so the primary is active again (Cisco: Upgrade FTD HA managed by FMC). It was written on older releases, so check the guide for your version.

None of this is exotic. HA failovers mostly surprise people when the defaults (detection time, session pickup, preemption) were never chosen on purpose. Read them once, write them down next to the pair, and the few seconds of a failover become boring, which is the goal.

Sources

• Palo Alto Networks: High Availability (PAN-OS 11.1)
• Palo Alto Networks: HA Modes
• Palo Alto Networks: HA Links and Backup Links
• Palo Alto Networks: Failover
• Palo Alto Networks: HA Timers
• Palo Alto Networks: Device Priority and Preemption
• Palo Alto Networks: Floating IP Address and Virtual MAC Address
• Palo Alto Networks: Reference: HA Synchronization (PAN-OS 11.1)
• Palo Alto Networks: Set Up Active/Passive HA
• Palo Alto Networks: Verify Failover or Suspend HA (PAN-OS)
• Palo Alto Networks: Upgrade an HA Firewall Pair (PAN-OS 11.1)
• Fortinet: FGCP (FortiOS 7.6)
• Fortinet: HA heartbeat interface
• Fortinet: Failover protection
• Fortinet: Session failover
• Fortinet: Session pickup
• Fortinet: Pass-through sessions
• Fortinet: Terminated sessions
• Fortinet: Cluster virtual MAC addresses
• Fortinet: HA primary unit selection criteria
• Fortinet: Force HA failover for testing and demonstrations
• Fortinet: Upgrading FortiGates in an HA cluster
• Cisco: High Availability for Firewall Threat Defense (Management Center 7.4)
• Cisco TAC: Upgrade FTD HA Managed by FMC
• RFC 9568: Virtual Router Redundancy Protocol (VRRP) Version 3
• RFC 5798: VRRP Version 3 (obsoleted by RFC 9568)