A first hop redundancy protocol answers one question: which of two routers owns the address the hosts use as their gateway. It answers it very well, and it answers it entirely on the basis of whether the two routers can hear each other on the host segment. Whether the router that wins can actually forward anything is a separate question that the protocol never asks.
That gap is the blackhole. A router whose uplink has failed remains perfectly capable of exchanging messages with its peer on the user segment, so it stays active, continues to answer for the gateway address, and receives every packet the segment sends outward — which it then discards, because it has nowhere to send them. The standby router sits beside it, healthy and idle, and the protocol sees nothing wrong.
This article covers exactly what the blackhole is and why the protocol cannot see it, what should be tracked and why the interface is usually the wrong thing, how the priority arithmetic works and the one setting that makes it do nothing, why a reload produces an outage and which delay covers which scenario, and how to verify the whole arrangement without waiting for a real failure. It is written for the lab rather than for the written exam, and sits alongside the rest of the CCIE Enterprise Infrastructure lab certification track.
The protocol decides ownership from what it hears on the user segment, so a router with a dead uplink remains a perfectly valid winner — and only a tracked object that reflects real forwarding capability changes that.What Exactly Is the Blackhole?
Why can the protocol not see it?
It was never designed to. The protocol elects an owner for a virtual address by exchanging messages on the segment where that address lives, and both routers can exchange those messages regardless of what is happening on their other interfaces. A router whose uplink has failed continues to participate normally, continues to win, and continues to answer for the gateway. Hosts have no way to know and no alternative to try — they have one gateway address and it is being answered.
A Deeper Dive into the Failure
Why hosts cannot route around it
The whole point of the design is that hosts have a single gateway address and never think about redundancy. That simplicity is what makes it work on every device without configuration, and it is exactly what removes any possibility of the host detecting the problem. A host sends to the gateway address, something answers, and the packet is gone.
So the entire burden of detecting an unusable active router falls on the routers. Nothing else in the path has the information or the capability.
The version that actually happens
A physically failed uplink is the textbook case and the rarer one. The common case is an uplink whose link stays up while the path beyond it fails: a routing adjacency that dropped, a device two hops away that lost its own uplink, a firewall in the path that stopped forwarding, or a provider circuit that is up and not carrying traffic.
In every one of those the local interface is up. Interface tracking sees a healthy interface and does nothing. The router stays active and the blackhole persists until somebody notices, which on a user segment means until the calls start.
What makes it hard to diagnose
Everything looks correct. The redundancy protocol shows one active and one standby, as designed. Both routers are up. The interface counters on the active router show packets arriving and none leaving, which is the actual evidence, and it is not where anyone looks first because the redundancy status says the design is working.
The instinct is to check the redundancy protocol, find it healthy, and move on to the hosts. That sequence costs most of the diagnosis time and it is entirely reasonable given what the status output says.
! The redundancy protocol says everything is fine
R1# show standby brief
P indicates configured to preempt.
|
Interface Grp Pri P State Active Standby Virtual IP
Vl10 1 110 P Active local 10.10.10.3 10.10.10.1
!
! The evidence is here instead: arriving and not leaving
R1# show interfaces Vlan10 | include packets input|packets output
R1# show ip route 0.0.0.0
% Network not in table
The asymmetric variant
Where stateful devices sit above the routers, a partial failure can produce traffic that leaves through one router and returns through the other. The redundancy protocol is working correctly, both routers forward, and the stateful device discards flows it has seen only half of.
This is not strictly a blackhole and it is diagnosed the same way: by looking at what forwards rather than at what is active. It is worth mentioning because the fix is the same — make the active role depend on something that reflects the real path.
Why the standby does not help
A standby router is waiting to be told it should take over, and the only thing that tells it is the active router going silent or announcing a lower priority. A blackholing active router does neither. It is present, it is healthy by every measure the protocol uses, and it keeps announcing the priority it was configured with.
Making it announce a lower priority when it cannot forward is the entire subject of the rest of this article, and it is the only mechanism available.
| Failure | Local interface | Protocol sees | Detected by |
|---|---|---|---|
| Uplink cable pulled | Down | Nothing wrong | Interface tracking |
| Routing adjacency lost | Up | Nothing wrong | Route tracking |
| Failure two hops away | Up | Nothing wrong | Reachability probing |
| Firewall stopped forwarding | Up | Nothing wrong | Reachability probing |
| Router reloaded, still converging | Up | Nothing wrong | Preempt delay |
What Should Be Tracked?
What is the right object?
Something that reflects the ability to forward, which usually means a route rather than an interface. Tracking the uplink's line protocol detects a cable failure and nothing else. Tracking the presence of a default route in the routing table detects a lost adjacency as well. Tracking reachability of an address beyond the uplink, by probing it, detects a failure anywhere in the path. Each step up the list catches more and costs a little more to configure and maintain.
A Deeper Dive into Tracked Objects
Interface tracking, and its narrow usefulness
It detects exactly one thing: the local interface going down. That is a real failure and it is the one least likely to go unnoticed, because a down interface generates a log entry, an alert and usually a call from whoever pulled the cable.
It is worth configuring because it is free, and it should never be the only tracking in place. A design whose entire failure detection is interface tracking has protected itself against the failure mode it would have found anyway.
Route tracking, which is the practical default
Tracking whether a specific prefix is present in the routing table catches everything that removes the route: the interface failing, the adjacency dropping, the far end withdrawing the advertisement, an administrative change upstream. That is a large proportion of real failures and it costs one object.
The prefix to track is usually the default route or a summary of the data centre, because those are what the segment actually needs. Tracking the directly connected uplink subnet achieves nothing beyond interface tracking, since the connected route exists whenever the interface is up.
! Track the route the segment actually depends on
R1(config)# track 10 ip route 0.0.0.0 0.0.0.0 reachability
R1(config-track)# delay down 5 up 30
!
! Tie it to the group
R1(config)# interface Vlan10
R1(config-if)# standby version 2
R1(config-if)# standby 1 ip 10.10.10.1
R1(config-if)# standby 1 priority 110
R1(config-if)# standby 1 preempt delay minimum 60 reload 180
R1(config-if)# standby 1 track 10 decrement 30
Reachability probing, for the path beyond
A route can be present and useless. A default route pointing at a next hop that is itself blackholing, or a path through a firewall that has stopped forwarding, leaves the routing table unchanged and the traffic undeliverable.
Probing a specific address beyond the uplink — something in the data centre that the segment genuinely needs — is the only detection that covers this. It costs a small amount of traffic and it catches the failure that everything else misses.
! Probe something the segment actually needs to reach
R1(config)# ip sla 10
R1(config-ip-sla)# icmp-echo 10.100.0.10 source-interface GigabitEthernet0/1
R1(config-ip-sla-echo)# frequency 5
R1(config-ip-sla-echo)# threshold 500
R1(config-ip-sla-echo)# timeout 1000
R1(config)# ip sla schedule 10 life forever start-time now
!
R1(config)# track 20 ip sla 10 reachability
R1(config-track)# delay down 10 up 30
Combining objects
Several tracked objects can be combined into one, either by a logical rule or by a weighted threshold. That is how a design expresses "fail over if the uplink is down, or if the default route has gone, or if the data centre is unreachable" as a single object referenced by the group.
The weighted form is useful where failures are partial: two uplinks, each contributing weight, with a threshold that triggers when both are gone but not when one is. Expressed as a logical rule that would need care; expressed as weights it is arithmetic.
! Any one of these failing is enough
R1(config)# track 100 list boolean or
R1(config-track)# object 1 not
R1(config-track)# object 10 not
R1(config-track)# object 20 not
!
! Or: fail over only when enough capacity is gone
R1(config)# track 200 list threshold weight
R1(config-track)# object 1 weight 50
R1(config-track)# object 2 weight 50
R1(config-track)# threshold weight up 60 down 40
!
R1# show track brief
Damping, so a flap does not become an outage
An object that changes state repeatedly makes the gateway move repeatedly, and every move interrupts traffic. A delay before acting on a change absorbs a flap without acting on it.
The two directions deserve different values. Going down should be fast, because the blackhole is already happening. Coming back up should be slow, because a link that has just returned is the one most likely to fail again, and there is no urgency — the standby router is forwarding perfectly.
show standby brief shows no preempt indicator against the standby router's entry. Fix: configure preemption on the standby router — tracking without it changes a number and nothing else.How Does the Priority Arithmetic Work?
What has to be true?
Three things together. The decrement must be large enough that the active router's reduced priority falls strictly below the standby's. The standby must be configured to preempt, or it will observe the lower priority and take no action. And the standby must itself be healthy, which means it needs its own tracking or it may take over into the same failure. Any one of the three missing produces a design that looks complete and does nothing.
A Deeper Dive into the Arithmetic
Choosing the numbers
A gap between the two routers' priorities, and a decrement larger than that gap. Ten and twenty is a common pairing and leaves room: a ten-point gap crossed by a twenty-point decrement puts the failed router clearly below, with no ambiguity about ties.
Equal priorities after a decrement do not trigger a takeover, because the tie is broken in favour of the current holder. A decrement exactly equal to the gap therefore achieves nothing, and it is an easy mistake to make when both numbers are chosen independently.
Preemption, which is not the default everywhere
Without preemption, a router that becomes eligible does not act. This is deliberate — preemption causes a gateway move whenever the preferred router returns, which is an interruption — and it means that a tracking design must enable it explicitly on the router expected to take over.
The usual arrangement is preemption on both, so that the preferred router reclaims the role when it recovers. That is the right default for a design where one router is genuinely preferred, and it is what makes the delays in the next section necessary.
Tracking on both routers
A standby router with no tracking of its own can take over when the active fails and then blackhole in exactly the same way, because nothing is checking whether it can forward either. On a pair where both have uplinks, both need the same tracking.
The decrements do not have to match, and usually should not: the preferred router needs a decrement that takes it below the other, while the second router needs one that takes it below the first. Working both directions out on paper, once, avoids a design that fails over correctly in one direction only.
! Both routers, symmetric tracking, different starting priorities
R1(config)# interface Vlan10
R1(config-if)# standby 1 priority 110
R1(config-if)# standby 1 preempt delay minimum 60 reload 180
R1(config-if)# standby 1 track 100 decrement 30
!
R2(config)# interface Vlan10
R2(config-if)# standby 1 priority 100
R2(config-if)# standby 1 preempt delay minimum 60 reload 180
R2(config-if)# standby 1 track 100 decrement 30
!
! R1 fails: 110-30 = 80, below R2's 100. R2 preempts.
! R2 fails while active: 100-30 = 70, below R1's 110. R1 preempts.
Several groups on one segment
Where VLANs are split so that half are active on one router and half on the other, each group has its own priorities and each needs its own tracking. Using one tracked object referenced by every group is correct and efficient — the object is about the router's ability to forward, which is the same for all of them.
What must not happen is tracking configured on some groups and not others, which produces a partial failover where half the VLANs move and half blackhole. On a switch with forty groups this is easy to arrive at and hard to spot.
! One object, every group, no exceptions
R1(config)# interface Vlan10
R1(config-if)# standby 1 track 100 decrement 30
R1(config)# interface Vlan20
R1(config-if)# standby 2 track 100 decrement 30
R1(config)# interface Vlan30
R1(config-if)# standby 3 track 100 decrement 30
!
! Verify no group was missed
R1# show standby brief
R1# show running-config | include standby [0-9]+ track
R1# show standby | include Group|Track|Priority
Version, and why it matters
The later protocol version supports sub-second timers, a larger group number range and the address family needed for IPv6. Sub-second timers are the reason to care: with default timers a failover takes several seconds, which is noticeable on a call and invisible on a file transfer.
The version must match on both routers. A mismatch produces two routers that each believe they are active, which is a far worse outcome than the slow failover the change was meant to improve. Reading this once is not the same as being able to do it under time pressure, which is what repetition against realistic CCIE lab practice scenarios is for.
! Sub-second detection, same version on both
R1(config)# interface Vlan10
R1(config-if)# standby version 2
R1(config-if)# standby 1 timers msec 250 msec 750
!
! Or hand detection to a dedicated mechanism
R1(config-if)# standby bfd
R1(config)# standby bfd all-interfaces
!
R1# show standby | include version|Hellotime|Holdtime|BFD
Why Does a Reload Cause an Outage?
What is the sequence?
A router restarts. Its interfaces come up within seconds and the redundancy protocol forms immediately, because it needs only the local segment. The routing protocol takes considerably longer — adjacencies, database exchange, computation, installation. In between, the router has the highest priority, preempts, becomes active, and has no routes. Everything the segment sends is discarded until the routing protocol finishes, which can be tens of seconds.
A Deeper Dive into Timing
Why the redundancy protocol wins the race
It has almost nothing to do. Two routers on a segment exchange messages, compare priorities and agree. There is no database, no computation and no dependency on anything beyond the segment, so it completes as soon as the interface is up.
A routing protocol has to find neighbours, exchange topology, run an algorithm and install results, and on a large network that is a long time by comparison. The gap between the two is the outage, and it exists on every reload of a router configured to preempt.
The reload delay
A delay applied specifically to the first preemption after a restart holds the router back until the routing protocol has had time to converge. It applies once and then stops applying, which is exactly right: the concern is specific to a restart.
The value should exceed the worst observed convergence time for this network, with margin. Measuring it rather than guessing is a fifteen-minute exercise during a maintenance window and it produces a number that is defensible.
The minimum delay
A delay applied to every preemption, including one triggered by a tracked object recovering. Its purpose is different: to absorb a flapping condition so that a link bouncing does not bounce the gateway with it.
Both delays should be configured, and they are not alternatives. The minimum covers flapping and the reload covers restarts, and a design with only one of them is exposed to whichever scenario the other covers.
The synchronisation delay
Where another subsystem must be ready before the router should take over — a stateful feature that needs its state populated, a platform redundancy mechanism — a third delay waits for that subsystem to signal readiness rather than for a fixed time.
It is the most precise of the three because it waits for the actual condition rather than for an estimate of it. It is also the least commonly needed, and where nothing signals readiness it should not be configured, because the delay then runs to its maximum every time.
! All three, with the reload value from measured convergence
R1(config)# interface Vlan10
R1(config-if)# standby 1 preempt delay minimum 60 reload 180 sync 30
!
! What the device is actually waiting for, right now
R1# show standby Vlan10 1 | include preempt|delay|State
Preemption enabled, delay min 60 secs, reload 180 secs, sync 30 secs
State is Standby
Measuring convergence rather than guessing
Reload the router during a window with the standby forwarding, and record the interval between the interfaces coming up and the routing table being complete. That number, plus a generous margin, is the reload delay.
Doing it once per platform and per major software version is sufficient, and it should be repeated after a significant change to the routing design, because convergence time is a property of the network rather than of the device.
! Measure it, once, during a window
R1# show logging | include LINEPROTO.*Vlan10|OSPF.*FULL|HSRP.*Active
!
! Then confirm the delay exceeds what you measured
R1# show standby Vlan10 1 | include reload
!
! And that the routing table is genuinely complete before it fires
R1# show ip route summary | include Total
R1# show ip ospf neighbor | count FULL
The alternative to preemption
Disabling preemption entirely removes the problem and introduces another: after a failover, the role stays where it landed. On a pair where both routers are equivalent that is fine and arguably better, because it removes an interruption.
Where the routers are not equivalent — different uplink capacity, different platform, a defined primary — preemption is needed and the delays are the price. The decision should follow from whether the two routers are genuinely interchangeable.
Interaction with the switching layer
On a switched access layer, a port that has just come up may not forward immediately while the loop-prevention protocol decides what to do with it. Where the router's segment interface is behind such a port, the redundancy protocol may not hear its peer for a period and both routers may briefly believe they are active.
Configuring the switch ports appropriately so that they forward immediately, where that is safe, resolves it. It is worth checking because the symptom — a brief dual-active condition after a change — looks like a redundancy protocol problem and is not.
| Scenario | Covered by | Typical value | Consequence if missing |
|---|---|---|---|
| Router restarts | Reload delay | Measured convergence + margin | Outage longer than the reboot |
| Tracked object flaps | Minimum delay | Tens of seconds | Gateway bounces with the link |
| Subsystem not ready | Sync delay | Its readiness signal | Takes over before state exists |
| Routers are equivalent | No preemption at all | — | An avoidable interruption |
How Do You Verify It Without Waiting for a Failure?
What can be tested?
All of it. The tracked object can be forced down administratively and the priority change observed. The failover can be triggered and timed with a continuous flow running. The reload behaviour can be tested by actually reloading during a window and watching the state change against the routing protocol's convergence. And the arithmetic can be checked on paper in both directions. None of this requires waiting for a real failure, and all of it is skipped more often than not.
A Deeper Dive into Verification
Forcing a tracked object down
Shutting the tracked interface, or removing the tracked route, or making the probed address unreachable, causes the object to go down exactly as a real failure would. The priority change and the resulting takeover can then be observed directly.
This is the single most valuable test and it takes two minutes. It establishes that the object is being tracked, that the decrement is sufficient, and that the other router preempts — the three things that must all be true.
! Force it down and watch all three conditions at once
R1# show track 100
Track 100
List boolean or
Boolean OR is Up
!
R1(config)# interface GigabitEthernet0/1
R1(config-if)# shutdown
!
R1# show track 100 | include Boolean|Change
R1# show standby brief
Interface Grp Pri P State Active Standby Virtual IP
Vl10 1 80 P Standby 10.10.10.3 local 10.10.10.1
Timing the failover
A continuous flow from a host on the segment to something beyond the routers, with the loss counted during a forced failover, gives the real interruption length. That number is what the design delivers, and it is frequently different from what the timer values suggest.
It is also the number to quote when somebody asks how long a failover takes, rather than an arithmetic estimate from the hold time. The two differ because of switching, address resolution updates and the time the new active router takes to be believed.
Testing the reload path specifically
This is the test people skip because it requires reloading a production router. It is also the one that catches the longest outage in this article, and there is no way to establish the correct reload delay without it.
Doing it once, during commissioning, with the standby forwarding and the logs timestamped, produces both the measurement and the confidence. Repeating it after a major software change is worthwhile because convergence behaviour changes with software.
Checking the arithmetic both ways
Write down each router's configured priority, its decrement, and the resulting value. Confirm that each router's decremented priority is strictly below the other's configured priority. Confirm that both have preemption enabled. That is four numbers and two flags and it catches the most common configuration error in this technology.
It is worth doing as a script across the estate rather than per device, because the error is silent and a pair that was correct when built can be broken by a priority change made for an unrelated reason.
! Scriptable, per router
show standby brief
show running-config | include standby [0-9]+ (priority|preempt|track)
show track brief
!
! The four numbers that must be right
R1# show standby Vlan10 1 | include Priority|Preemption|Track
Priority 110 (configured 110)
Track object 100 state Up decrement 30
Preemption enabled, delay min 60 secs, reload 180 secs
What to monitor in production
State changes, which should be rare and each of which should correspond to a known event. The tracked object's state, which is the early warning — an object that goes down and comes back repeatedly indicates a problem that has not yet caused a failover but will. And the count of groups with tracking configured against the total number of groups, which catches the partial-coverage failure.
The second of those is the most useful because it is predictive. A flapping tracked object is a fault that has been absorbed by the damping delay and is invisible in the redundancy protocol's own state.
What to document
Which router is preferred for which groups and why. The measured convergence time that produced the reload delay, and when it was measured. What is being tracked and what it is meant to detect. And the measured failover time from the last test.
The convergence measurement is the one that decays. A reload delay set from a measurement taken before the network doubled in size is a number with no current basis, and the date is what makes that visible.
Blueprint framing
The CCIE Enterprise Infrastructure v1.1 blueprint covers first hop redundancy protocols within its infrastructure services domain, including object tracking. What is examined is usually the interaction — why tracking without preemption does nothing, and why a preempt delay is needed after a reload — rather than the base configuration.
| Test | Method | Establishes | Effort |
|---|---|---|---|
| Object is tracked | Shut the tracked interface | Priority drops as intended | Two minutes |
| Takeover occurs | Same test, watch the peer | Preemption is enabled | Same two minutes |
| Failover duration | Continuous flow, count loss | The real interruption | Ten minutes |
| Reload behaviour | Actually reload it | The correct reload delay | A maintenance window |
| Both directions work | Repeat from the other router | Symmetric arithmetic | Double the above |
Conclusion
The redundancy protocol decides which router owns the gateway address using only what it hears on the user segment, and a router with a dead uplink participates in that conversation perfectly well. So the blackhole is structural rather than accidental: the protocol's winner and the network's usable forwarder are two different questions, and nothing connects them until you connect them.
Tracking is how that connection is made, and what is tracked decides what is detected. An interface catches the cable that was pulled, which is the failure that would have been noticed anyway. A route catches the adjacency that dropped while the link stayed up, which is the one that produces a silent blackhole. A probed address beyond the uplink catches a failure two hops away that leaves the local routing table entirely unchanged. Each step up that list is the difference between a design that looks protected and one that is.
Three things then have to hold together: a decrement large enough to cross the priority gap strictly, preemption configured on the router expected to act, and the same tracked object on every group rather than most of them. And the scenario that produces the longest outage is not a failure at all — it is a planned reload, where the redundancy protocol forms in seconds, the routing protocol takes far longer, and the router becomes the gateway with no routes. A reload delay set from a measured convergence time is what closes that, and measuring it takes one window. More CCIE Enterprise Infrastructure material — labs, protocol breakdowns and study guides — is collected on the SPOTO CCIE site.
External Links
- RFC 2281 — Cisco Hot Standby Router Protocol (HSRP)
- RFC 5798 — Virtual Router Redundancy Protocol Version 3
- RFC 5880 — Bidirectional Forwarding Detection
- RFC 5881 — BFD for IPv4 and IPv6 (Single Hop)
- RFC 826 — An Ethernet Address Resolution Protocol
- Cisco Learning Network — CCIE Enterprise Infrastructure
Reference Notes
- RFC 2281 describes HSRP, in which routers on a segment elect an active router for a virtual IP address based on priority exchanged in hello messages on that segment.
- RFC 2281 states that a router with a higher priority does not take over from an active router unless it is configured to preempt.
- RFC 2281 describes the hello and hold timers that determine how quickly the loss of an active router is detected.
- RFC 2281 notes that the protocol operates on a single segment and makes no assessment of a router's ability to forward traffic beyond it.
- RFC 5798 specifies VRRP version 3, which provides equivalent virtual router election for IPv4 and IPv6 and likewise elects on the basis of priority exchanged on the local segment.
- RFC 5880 specifies bidirectional forwarding detection, a protocol-independent mechanism for rapid detection of forwarding path failures between adjacent systems.
- RFC 5881 specifies the use of BFD for single-hop IPv4 and IPv6 paths, which is the arrangement used to accelerate detection between redundancy peers on a segment.
- RFC 826 specifies ARP, the mechanism by which hosts resolve the virtual gateway address and which must be updated when the active router changes.
- Cisco documentation describes enhanced object tracking, including tracking of interface line protocol, route reachability and IP SLA operation results, and the combination of objects using boolean and weighted threshold lists.
- Cisco documentation describes the tracking delay, which postpones acting on a state change in order to damp a flapping condition, with independent values for the up and down transitions.
- Cisco documentation describes the preempt delay options, including a minimum delay applying to every preemption and a reload delay applying to the first preemption after a device restart.
- The CCIE Enterprise Infrastructure v1.1 unified exam topics include first hop redundancy protocols within the infrastructure services domain.