PAN-OS packet flow: six stages and up to three policy checks

When a rule "should match" but the traffic log says otherwise, the fastest way forward is to know the order in which a PAN-OS firewall makes its decisions. Palo Alto Networks documents this order in its knowledge base article Packet Flow Sequence in PAN-OS. These are my notes on it, grouped into six stages, followed by what the order means when you troubleshoot.

1. Ingress

The firewall parses the packet at layers 2, 3 and 4 and drops malformed packets. At layer 2 it works out the ingress logical interface from the port, the 802.1Q tag and the destination MAC address; if nothing matches, the packet is dropped and a counter goes up. At layer 3 it checks the IP header for problems such as truncation, bad checksums and patterns like the "Land attack" or "Ping of death". At layer 4 it checks TCP and UDP headers for truncation, bad checksums, zero ports and invalid flag combinations.

Two more things happen here. Traffic arriving in an IPsec or SSL-VPN tunnel is decapsulated, given the tunnel interface as its new ingress interface and parsed again. IP fragments are reassembled and fed back to the parser, and overlapping fragments or fragmentation errors cause a drop.

2. Session lookup and setup (the slowpath)

The firewall looks the flow up in its session table using a 6-tuple key: source IP, destination IP, source port, destination port, protocol and the security zone (taken from the ingress interface). For traffic without ports, other fields stand in; ICMP, for example, uses its identifier and sequence number. If there is no session yet, the packet takes the slowpath, where the expensive decisions are made once:

• zone protection, chosen by the ingress zone;
• TCP state: a first TCP packet that isn't a SYN is dropped by default, and SYN floods are handled with SYN cookies or Random Early Drop (RED, the default);
• forwarding: the egress interface and zone depend on the interface mode; in layer 3 mode the forwarding table picks them, and in tap mode the packet is dropped;
• NAT policy, which applies only in layer 3 and virtual wire mode; destination NAT triggers a second route lookup, and a failed source NAT allocation drops the packet;
• User-ID, to map the source IP to a user and group;
• DoS protection policy, which the article says has been looked up before the security policy since PAN-OS 7.0.2 and 6.1.7;
• security policy, with the application still "any". A matching deny rule, or no matching rule at all, drops the packet.

If the traffic is allowed, a session is taken from the free pool (this can fail when limits are reached) and moves from INIT to OPENING to ACTIVE once both directions of the flow are in the lookup table.

"No matching rule" is rarely truly empty. The rulebase ends with two predefined rules that allow all intrazone traffic and deny all interzone traffic, and rules are compared in sequence, with the first match applied.

3. Fastpath

Later packets of the same session skip the slowpath. Packets of a session in a discard state, for example after a policy change to deny or a threat detection, are dropped. For active sessions the firewall refreshes timeouts, starts the half-close or time-wait timers when it sees TCP FIN or RST, applies NAT translation and reassembles TCP. Packets with a payload go on to application identification and content inspection.

4. App-ID

App-ID first checks for an application-override rule. If one matches, the application is set and content inspection is skipped for that session; the Application Override Policy page says such rules bypass layer 7 processing and threat inspection. Otherwise App-ID identifies the application from its signatures, and falls back to decoders and heuristics when signatures are not conclusive.

Once the application is known, the security policy is checked again, this time with the real application. If logging is enabled at session start, the firewall writes a traffic log each time the App-ID changes during the session. Allowed traffic is also given a QoS class here, and for SSL and SSH the decryption policy is looked up.

5. Content inspection

Protocol decoders inspect the flow and look for tunnelling applications. If the application changes, for example when an application is found running inside another one, the security policy is checked yet again. Security profiles (antivirus, vulnerability protection, URL filtering and others) are applied from the matching rule, and a detection with a block action drops the traffic. If SSL forward proxy or SSH decryption is in use, the packet is re-encrypted before it moves on.

6. Forwarding and egress

The firewall applies QoS shaping, fragments the packet if the egress MTU needs it and the packet allows it, encrypts it with IPsec or SSL-VPN if the egress interface is a tunnel, and sends it out of the physical interface.

Why this order matters

• The NAT policy is evaluated before the security policy, but the security policy matches the pre-NAT addresses with the post-NAT zone (NAT Policy Overview). The translation itself is applied on egress. Writing a rule with the translated address, or with the pre-NAT zone, is a common reason a rule doesn't match.
• The first security policy lookup uses the application "any". A rule that allows only ssl can still let the TCP handshake through, and the session is judged again once App-ID knows the application.
• Policy can be checked up to three times in one session: at setup, after App-ID, and if the application changes during content inspection.
• An application override is a trade: you choose the application yourself, and in return the session gets no layer 7 inspection or threat prevention.
• An "allow" in the rulebase still depends on the earlier stages. A packet dropped by zone protection, the TCP SYN check or DoS protection never reaches the security policy at all, so it won't appear as a policy deny.

Matching the stage to the tool

Knowing which stage made a decision tells you which log, counter or capture to read first. A few built-in tools line up well with the stages:

• Policy questions: the test security-policy-match command checks which rule would match the values you give it. The Security Policy Match test in the web interface asks for the From and To zones too, and can list every potential match up to the first allow rule. Give it the pre-NAT addresses and the post-NAT zone, the same view the rulebase uses.
• Drops before the policy: global counters, filtered to one source and destination, show what the dataplane did with the packets. The knowledge base article How to check global counters for a specific source and destination IP address uses show counter global filter packet-filter yes delta yes, and adds severity drop to show only drops.
• Where a packet disappears: a custom packet capture has four stages: receive (the packet arrives on the dataplane), firewall (it matches a session or creates one), transmit (it leaves the dataplane) and drop (processing hit an error and dropped it). A packet seen at receive but never at firewall stopped before a session existed, which points back to stage 1 or the slowpath.

Read in this order, the packet flow stops being a diagram to memorise and becomes a checklist: find the stage, then read the evidence that stage leaves behind.