Tracking a probe rather than an interface is what turns a redundancy design from one that detects cables into one that detects failures. The router sends something, waits for an answer, and the presence or absence of that answer decides which router owns the gateway. It is a simple idea and everything useful about it is in the details of what is sent, to where, how often, and what counts as a failure.
Those details are where deployments go wrong, and mostly in one direction: the defaults are conservative. A probe left at its default settings sends once a minute and waits five seconds for an answer, which means a failure takes the better part of a minute to be noticed. A mechanism added specifically to detect failures faster than the routing protocol ends up detecting them no faster at all, and nothing about the configuration suggests anything is wrong.
This article covers what the probe should actually prove, which operation type and which target to choose, how the timers relate to each other and what the defaults cost, the difference between the two ways of reading a probe's result, and the probe designs that cause the failovers they were meant to prevent. It is written for the lab rather than for the written exam, and sits alongside the rest of the CCIE Enterprise Infrastructure lab certification track.
The probe, the tracked object, the priority decrement and the gateway move are four separately configurable stages, and the total detection time is the sum of the delays each one contributes.What Should the Probe Prove?
What is the question being asked?
Whether this router can still deliver traffic to where the segment needs to send it. That is a statement about a path rather than about a link, which is the whole reason for probing instead of watching an interface. The probe should therefore be aimed at something on the far side of everything that could fail: past the uplink, past the next router, past any firewall, at something in the destination the users actually care about reaching.
A Deeper Dive into the Question
Probing the next hop proves very little
A probe to the directly connected router on the other end of the uplink succeeds whenever the link works, which is what the interface already told you. It adds a small amount of coverage — the far router's own interface being administratively down, for instance — and it does not detect any of the failures that produce a silent blackhole.
It is a common starting point because it is the safest target, and it is a design that has taken on the complexity of probing without gaining the benefit.
Probing past the failure domain
The target should sit beyond every device whose failure would make this router useless. In a typical branch that means something in the data centre. In a campus distribution layer it means something on the far side of the core.
Choosing it involves a conversation about what the segment is for. A floor whose users work entirely against one application has an obvious target; a floor whose users need general connectivity needs something representative rather than something specific.
The dependency you are creating
Whatever is probed becomes load-bearing. If a single host is the target and that host is rebooted for patching, the probe fails, the priority drops and the gateway moves — for every user on the segment, because one server was restarted.
That is the central trade in this design and it has to be managed deliberately. Either the target is something with genuine high availability — a virtual address on a load balancer, a router loopback in the data centre — or several targets are probed and the design requires more than one to fail.
Sourcing it correctly
A probe sourced from the wrong interface can succeed over a path that is not the one being tested. On a router with two uplinks, a probe that leaves by whichever one routing selects tells you that a path works, which is not the question when the point is to detect that this path has failed.
Specifying the source interface pins it. On a design where the router has genuinely only one path this changes nothing and costs nothing; on a design where it has two it is the difference between a meaningful probe and a misleading one.
! Probe past everything that could fail, from the interface
! being tested, and schedule it - the line people forget.
R1(config)# ip sla 10
R1(config-ip-sla)# icmp-echo 10.100.0.10 source-interface GigabitEthernet0/1
R1(config-ip-sla-echo)# frequency 3
R1(config-ip-sla-echo)# timeout 1000
R1(config-ip-sla-echo)# threshold 500
R1(config-ip-sla-echo)# request-data-size 64
!
R1(config)# ip sla schedule 10 life forever start-time now
!
R1(config)# track 10 ip sla 10 reachability
R1(config-track)# delay down 3 up 30
show ip sla summary shows the operation with no recorded return code or an inactive state. Fix: schedule the operation, and include the schedule line in whatever template creates these — it is the single most forgotten line in this technology.Marking the probe
A network that treats traffic classes differently will treat the probe according to how it is marked. A probe marked as ordinary traffic succeeds while a voice class is being discarded, and a probe marked as voice fails during congestion that the rest of the traffic tolerates.
Marking it to match the traffic whose loss would matter most is the reasonable default. On a segment carrying voice, marking the probe accordingly means it detects the conditions that affect calls rather than the conditions that affect bulk transfer.
! Mark the probe so it experiences what the traffic experiences
R1(config)# ip sla 10
R1(config-ip-sla)# icmp-echo 10.100.0.10 source-interface GigabitEthernet0/1
R1(config-ip-sla-echo)# tos 184
R1(config-ip-sla-echo)# frequency 3
!
! Inside a separate routing table, name it explicitly
R1(config)# ip sla 15
R1(config-ip-sla)# vrf INET
R1(config-ip-sla)# icmp-echo 203.0.113.10 source-interface GigabitEthernet0/2
R1(config-ip-sla-echo)# frequency 3
R1(config)# ip sla schedule 15 life forever start-time now
Probing inside a separate routing table
Where the path being tested lives in its own routing table, the probe must be told which one to use. Omitting that produces a probe that is evaluated against the wrong table and either fails permanently or succeeds over an unrelated path.
It is a single keyword and it is easy to omit on a design where the separation was introduced after the probe was written.
| Target | Detects | Misses | Creates a dependency on |
|---|---|---|---|
| Next hop router | Link failure | Almost everything else | Nothing new |
| A router loopback beyond it | Path and routing failure | Application-level failure | That router |
| A single application host | Nearly everything | — | That one host |
| A highly available address | Nearly everything | — | Its availability mechanism |
| Several targets, majority rule | Nearly everything, robustly | — | Nothing single |
Which Operation Type, and to What Target?
What are the choices?
An echo probe, which proves the address answers. A connection probe to a service port, which proves something is listening. An application-level probe, which proves the application responds. And timing-oriented probes that measure delay and variation rather than simply presence. For redundancy tracking the first is usually right and the second is worth considering where the segment exists to reach one specific service.
A Deeper Dive into Operation Types
The echo probe
Simplest, cheapest, and adequate for the question being asked. It proves that packets reach the target and answers return, which is exactly what "can this router still deliver traffic" means.
Its weakness is that the answer may come from something other than the intended target — an intermediate device configured to answer on behalf of others, or a firewall that responds to keep probes quiet. That is uncommon and worth confirming once when the target is chosen.
Connection probes
Opening a connection to a service port proves more: not just that the address is reachable but that something is accepting connections there. On a segment whose purpose is to reach a particular service, that is a more meaningful test.
It also creates a tighter dependency. Planned maintenance on that service now moves the gateway for everyone on the segment, which is a poor outcome. Where a connection probe is used, the target should be a service with its own redundancy rather than a single instance.
! Prove something is listening, not just that a host answers
R1(config)# ip sla 20
R1(config-ip-sla)# tcp-connect 10.100.0.20 443 source-interface GigabitEthernet0/1
R1(config-ip-sla-tcp)# frequency 5
R1(config-ip-sla-tcp)# timeout 2000
R1(config)# ip sla schedule 20 life forever start-time now
!
! And a name resolution probe, where that is the dependency
R1(config)# ip sla 30
R1(config-ip-sla)# dns portal.example.com name-server 10.100.0.53
R1(config-ip-sla-dns)# frequency 10
R1(config)# ip sla schedule 30 life forever start-time now
Probes that measure rather than test
Some operation types measure delay, variation and loss rather than simply succeeding or failing. They are valuable for monitoring and they are a poor basis for a gateway decision, because they are sensitive to conditions that do not justify moving every user on a segment.
Where they are used for tracking, the threshold has to be set from measured normal behaviour, and the damping has to be generous. The simpler test is usually the better choice for a failover decision, with the measuring probes running alongside for visibility.
Where a responder is needed
Probes that measure timing accurately need cooperation from the far end, because the far end has to timestamp what it received. Probes that simply test reachability do not, because the answer either arrives or it does not.
For redundancy tracking, no cooperation is needed, which is a practical advantage: the target can be anything that answers, including a device belonging to someone else.
Several probes, combined
Three probes to three targets, with a rule requiring two of them to fail before the object goes down, removes the single-target dependency entirely. One target being patched changes nothing; the path genuinely failing takes all three down.
The cost is three probes instead of one and a slightly more complex configuration. On a design whose failover affects a whole floor, that cost is trivially justified.
! Three targets, and a rule that tolerates one failing
R1(config)# ip sla 11
R1(config-ip-sla)# icmp-echo 10.100.0.10 source-interface GigabitEthernet0/1
R1(config-ip-sla-echo)# frequency 3
R1(config)# ip sla 12
R1(config-ip-sla)# icmp-echo 10.100.0.11 source-interface GigabitEthernet0/1
R1(config-ip-sla-echo)# frequency 3
R1(config)# ip sla 13
R1(config-ip-sla)# icmp-echo 10.100.0.12 source-interface GigabitEthernet0/1
R1(config-ip-sla-echo)# frequency 3
!
R1(config)# ip sla schedule 11 life forever start-time now
R1(config)# ip sla schedule 12 life forever start-time now
R1(config)# ip sla schedule 13 life forever start-time now
!
R1(config)# track 11 ip sla 11 reachability
R1(config)# track 12 ip sla 12 reachability
R1(config)# track 13 ip sla 13 reachability
!
! Two of three must be up
R1(config)# track 100 list threshold percentage
R1(config-track)# object 11
R1(config-track)# object 12
R1(config-track)# object 13
R1(config-track)# threshold percentage up 60 down 40
R1(config-track)# delay down 5 up 60
Choosing the targets so they fail independently
Three probes to three addresses in the same rack, behind the same switch, fail together for reasons unrelated to the path being tested. The value of combining probes comes entirely from the targets being independent of one another.
Three addresses in three different parts of the destination network is the right shape. If the intent is to detect that the path has failed, the targets should share the path and nothing else.
How Do the Timers Relate, and What Do the Defaults Cost?
What is the relationship?
Three values with a required ordering. The frequency is how often the probe runs. The timeout is how long it waits for an answer before declaring that attempt failed, and it must not exceed the frequency. The threshold is a slower-than-expected marker that affects statistics and one of the two tracking forms, and it should be below the timeout. Left at their defaults these are a minute, five seconds and five seconds — which makes detection take about a minute.
A Deeper Dive into Timing
What the default frequency costs
A probe running once a minute detects a failure somewhere between zero and sixty seconds after it happens, averaging thirty. Add the timeout and any damping delay and the detection time is comparable to a routing protocol's default hold time.
Since the usual reason to add this mechanism is to detect something the routing protocol cannot see, that is not necessarily fatal — a minute of blackhole is better than an indefinite one. But it is far from what the design is capable of, and the improvement costs one line.
Choosing a frequency
A few seconds is appropriate for a gateway decision. Every two to five seconds gives detection within single-digit seconds and generates a negligible amount of traffic for a single probe.
The constraint is scale rather than the individual probe. A router with a handful of probes can run them every second without noticing; a hub with hundreds of tracked branches should think about the aggregate. For an access-layer gateway pair, the aggregate is not a consideration.
The timeout, and why it should be short
Waiting five seconds for an answer that normally arrives in two milliseconds wastes almost the entire interval. A timeout of half a second to one second is generous by three orders of magnitude relative to the normal response and still cuts the detection time substantially.
It must remain above the worst legitimate response time, which on a congested WAN can be considerably higher than the average. Measuring the actual distribution, rather than assuming, is a five-minute exercise with the statistics the probe already collects.
! What the probe is actually experiencing
R1# show ip sla statistics 10
IPSLA operation id: 10
Latest RTT: 2 milliseconds
Latest operation start time: 09:41:22 CST Tue Sep 15 2026
Latest operation return code: OK
Number of successes: 18405
Number of failures: 2
Operation time to live: Forever
!
! And the distribution, which is what sets the timeout
R1# show ip sla statistics aggregated 10 | include RTT|Max|Min
The damping delays, which are separate
The tracked object can delay acting on a change. Downward, that delay adds directly to the detection time and should usually be small or zero — the blackhole is already happening.
Upward, it should be generous. A path that has just recovered is the one most likely to fail again, and there is no cost to waiting: the standby router is forwarding perfectly well. Thirty to sixty seconds upward is a reasonable default and it prevents a flapping circuit from bouncing the gateway repeatedly.
Adding it up
Detection time is the probe interval, plus the timeout for the failing attempt, plus the downward damping delay, plus the redundancy protocol's own transition. With defaults that is roughly a minute; with sensible values it is a few seconds. The arithmetic is worth doing explicitly because each contribution is configured in a different place and none of them is obvious from the others.
! Detection time, added up
! frequency 3 s -> up to 3 s before the failing probe runs
! + timeout 1 s -> 1 s waiting for the answer
! + track delay down 3 s -> 3 s of damping
! + HSRP transition -> under 1 s with tuned timers
! = about 8 s worst case
!
! With defaults: 60 + 5 + 0 + 10 = about 75 s
R1# show ip sla configuration 10 | include Frequency|Timeout|Threshold
R1# show track 10 | include delay|seconds
show ip sla configuration shows the default frequency and timeout. Fix: set both explicitly, and include them in whatever template generates these operations so no probe is ever created with defaults.Scheduling, and the state it produces
An operation that has not been scheduled does not run. It exists in the configuration, produces no results, and the object tracking it is down. The router therefore sits permanently at its decremented priority with nothing to indicate why.
This is worth a specific verification step after creating any probe: confirm that it has run recently and returned a result, rather than confirming that it is configured. The two look identical in the running configuration. Reading this once is not the same as being able to do it under time pressure, which is what repetition against realistic CCIE lab practice scenarios is for.
Reachability or State — Which Form?
What is the difference?
Reachability asks whether an answer arrived. State asks whether the answer arrived within the configured threshold. A slow but successful response leaves reachability up and can take state down. That makes state useful for detecting a degraded path and dangerous as a gateway trigger, because a brief period of congestion becomes a failover. Reachability is the right default and state is a deliberate choice for a specific reason.
A Deeper Dive into the Two Forms
What reachability covers
The probe either got an answer or it did not. A path that is slow, congested or lossy but still working leaves the object up, which is correct for a gateway decision: the router can still deliver traffic and moving every user to the other router would not obviously improve anything.
It is the form to use unless there is a specific argument for the other, and the specific argument is rarer than it sounds.
What state adds, and what it risks
State incorporates the threshold, so a response slower than the threshold is treated as a failure. On a path where a slow response genuinely means the path is unusable — a voice segment, for instance — that is meaningful.
The risk is that congestion is normal on many paths and a threshold set without measurement will be crossed regularly. Each crossing moves the gateway, which interrupts every session on the segment, in response to a condition that was going to clear on its own.
Setting a threshold that is not guesswork
If state is used, the threshold must come from measurement. Run the probe with reachability tracking for a week, read the aggregated statistics, and set the threshold well above the observed maximum rather than above the average.
A threshold set from the average is guaranteed to be crossed. A threshold set from the maximum plus a margin is crossed only when something genuinely different is happening, which is the intent.
! State form: only with a measured threshold and heavy damping
R1(config)# ip sla 10
R1(config-ip-sla)# icmp-echo 10.100.0.10 source-interface GigabitEthernet0/1
R1(config-ip-sla-echo)# frequency 3
R1(config-ip-sla-echo)# timeout 1000
R1(config-ip-sla-echo)# threshold 300
!
R1(config)# track 10 ip sla 10 state
R1(config-track)# delay down 15 up 60
!
! Read the measured distribution before choosing 300
R1# show ip sla statistics aggregated 10
The combination that works well
Two objects on one probe: reachability driving the gateway decision, and state driving an alert. The gateway moves only on a genuine failure, and somebody is told when the path degrades without having to wait for it to fail.
That separates the two concerns properly. A degraded path is an operational problem to investigate; it is not automatically a reason to interrupt every session on a floor.
! One probe, two objects, two purposes
R1(config)# track 10 ip sla 10 reachability
R1(config-track)# delay down 3 up 30
!
R1(config)# track 11 ip sla 10 state
R1(config-track)# delay down 30 up 60
!
! Only the first one touches the gateway
R1(config)# interface Vlan10
R1(config-if)# standby 1 track 10 decrement 30
!
! The second drives a notification instead
R1(config)# event manager applet PATH-DEGRADED
R1(config-applet)# event track 11 state down
R1(config-applet)# action 1.0 syslog msg "Path to data centre degraded"
Return codes, and reading them
The probe records why an attempt failed — no answer, an error response, a name that would not resolve. That distinction is useful during diagnosis and is not visible from the tracked object, which reduces everything to up or down.
Reading the return code when investigating tells you whether the path is silent or actively rejecting, which are different problems with different causes.
! Why the last attempt failed, which the object does not show
R1# show ip sla statistics 10 | include return code
Latest operation return code: Timeout
!
! Timeout = nothing came back
! No Connection = something refused
! OK = it worked, check the threshold instead
!
R1# show ip sla statistics 10 | include successes|failures
Number of successes: 18405
Number of failures: 2
What to watch in production
The success and failure counts, whose ratio is the path's real reliability. The tracked object's change count, which should be very low and whose growth means the design is flapping. And the recorded response times, which drift upward before a path fails outright often enough to be a useful early signal.
The second of those is the one to alert on. A tracked object changing state several times a day is causing gateway moves that users are noticing and that nobody has connected to the tracking configuration.
| Form | Up when | Use for | Risk |
|---|---|---|---|
| Reachability | Any answer arrived | The gateway decision | Ignores a degraded path |
| State | Answer within the threshold | Alerting, or a delay-critical path | Congestion becomes a failover |
| Both, separate objects | — | Gateway on one, alert on the other | None |
Which Probe Designs Cause Their Own Failovers?
What should be avoided?
Four patterns. A single target whose own maintenance triggers a failover. A threshold-sensitive form on a path where congestion is normal. Upward damping set to zero, so a flapping circuit bounces the gateway with it. And probes on both routers targeting the same thing, so that a failure of the target takes both routers down and the gateway lands wherever the arithmetic happens to put it. All four look correct in configuration review.
A Deeper Dive into Self-Inflicted Failovers
The target that gets patched
A single host as the probe target means that host's availability is the segment's availability. Server teams reboot servers, and they are not told that one of theirs is the trigger for a network failover affecting a floor.
Two answers. Target something designed to be always available — a virtual address, a router loopback — or use several targets with a majority rule. Either removes the dependency. What does not work is asking the server team to tell you before rebooting.
Both routers probing the same target
If both routers probe the same address and that address becomes unreachable for its own reasons, both objects go down, both priorities drop by the same amount, and the relative ordering is unchanged. The gateway does not move, which is arguably correct, and both routers are now at a decremented priority for a reason unrelated to either of them.
The subsequent behaviour is what matters: a real failure on one router now has to overcome a priority that is already reduced, and the arithmetic that was worked out for the normal case no longer holds. Checking the arithmetic for the case where both objects are down is a step worth taking.
Flapping, and upward damping
A circuit that comes and goes takes the object with it, and with no upward delay the gateway returns as soon as the object recovers — then leaves again on the next failure. Each move interrupts sessions, and a circuit flapping every few minutes produces a segment that is unusable in a way that looks nothing like a circuit problem.
A generous upward delay converts that into a single move followed by a stable period. It costs nothing, because the standby router is forwarding correctly throughout.
show track shows a high change count and a recent change time. Fix: set a generous upward delay on the tracked object so that a recovered path must stay recovered before the gateway returns.Probes that succeed when they should not
An intermediate device configured to answer on behalf of an address, or a firewall that responds to keep monitoring quiet, produces a probe that succeeds while the real target is unreachable. The design then has tracking that never detects anything.
Confirming once, at deployment, that the answer genuinely comes from the intended target is a two-minute check and it is the only way to find this. Comparing the recorded response time against what a probe to the real target should take is a useful indicator: an answer that comes back far faster than the distance allows is coming from somewhere closer.
A verification procedure
Five steps. Confirm the operation is scheduled and returning results. Confirm the response time is consistent with the real distance to the target. Break the path deliberately and time the detection. Confirm the gateway moved and the peer took over. And restore the path and confirm the upward delay holds the gateway in place for the configured period.
The last of those is the step that is never done, and it is the one that verifies the damping actually works rather than being configured.
! Five steps, in order
R1# show ip sla summary
R1# show ip sla statistics 10 | include return code|RTT
!
! Break it and time the detection
R1(config)# interface GigabitEthernet0/1
R1(config-if)# shutdown
R1# show track 10
R1# show standby brief
!
! Restore, and confirm the up delay holds
R1(config-if)# no shutdown
R1# show track 10 | include delay|seconds remaining
What to document
What is being probed and why that target was chosen. The measured response time distribution that produced the timeout and any threshold. The measured detection time from the last test. And who owns the target, so that a change to it is understood as a change to the network's failover behaviour.
That last item is the one that prevents the most common self-inflicted failover, because it puts the dependency on the record where somebody outside the network team can see it.
Blueprint framing
The CCIE Enterprise Infrastructure v1.1 blueprint covers network assurance and first hop redundancy within its infrastructure services domain, and probe-driven tracking sits between them. What is examined is generally the chain — probe, object, priority, takeover — and the distinction between the two tracking forms, rather than the full operation syntax.
| Design | Failure it causes | Fix |
|---|---|---|
| Single host as target | Patching moves the gateway | Highly available target, or majority rule |
| State form on a busy path | Congestion moves the gateway | Reachability for the decision |
| No upward delay | A flap bounces the gateway | Thirty to sixty seconds up |
| Both routers, same target | Both decrement together | Check the arithmetic for that case |
| Probe answered by something else | Nothing is ever detected | Verify the response time |
| Operation never scheduled | Object permanently down | The schedule line |
Conclusion
Probing rather than watching an interface is what makes a redundancy design detect failures rather than cables. The probe should be aimed past everything whose failure would make this router useless, sourced from the interface being tested, and marked to experience the network the way the traffic that matters does. Whatever it targets becomes load-bearing for the whole segment, which makes target selection a decision about that system's availability rather than a networking detail.
The defaults are the single biggest thing to change. A probe left alone runs once a minute and waits five seconds, so a mechanism added to beat the routing protocol's detection time ends up matching it. Setting the frequency to a few seconds and the timeout to well under a second brings detection into single-digit seconds, and the arithmetic is worth doing explicitly because the four contributions are configured in three different places. And the operation has to be scheduled — an unscheduled probe produces no result, and an object tracking no result is permanently down.
Between the two ways of reading the result, reachability belongs on the gateway decision and state belongs on an alert. State treats a slow answer as a failure, which on a path where congestion is normal converts routine busy periods into gateway moves that interrupt every session on the segment. Run both objects off one probe, let reachability decide and let state tell somebody, and set a generous upward delay so that a flapping circuit produces one move rather than twenty. More CCIE Enterprise Infrastructure material — labs, protocol breakdowns and study guides — is collected on the SPOTO CCIE site.
External Links
- RFC 2281 — Cisco Hot Standby Router Protocol (HSRP)
- RFC 792 — Internet Control Message Protocol
- RFC 2925 — Definitions of Managed Objects for Remote Ping and Traceroute
- RFC 2681 — A Round-trip Delay Metric for IPPM
- RFC 3393 — IP Packet Delay Variation Metric
- Cisco Learning Network — CCIE Enterprise Infrastructure
Reference Notes
- RFC 792 specifies ICMP, including the echo request and reply used by the simplest form of reachability probe.
- RFC 792 notes that ICMP messages may be generated by intermediate systems, which is why an echo reply does not by itself prove that the intended target answered.
- RFC 2925 defines managed objects for remote ping and traceroute operations, including the operation frequency, timeout and the recording of results, which is the model these operations follow.
- RFC 2681 defines a round-trip delay metric, distinguishing a measurement that completed from one that did not and establishing the notion of a waiting time after which a measurement is considered lost.
- RFC 3393 defines IP packet delay variation, the metric measured by jitter-oriented operations.
- RFC 2281 describes HSRP, in which a router's priority determines ownership of the virtual address and a higher-priority router takes over only if configured to preempt.
- Cisco documentation describes IP SLA operations, including the icmp-echo, tcp-connect, dns and udp-jitter operation types and their applicability.
- Cisco documentation states that an IP SLA operation must be scheduled before it runs, and that an unscheduled operation produces no results.
- Cisco documentation describes the relationship between the operation frequency, timeout and threshold, requiring the timeout not to exceed the frequency and the threshold not to exceed the timeout.
- Cisco documentation distinguishes tracking an IP SLA operation's reachability, which reflects whether the operation succeeded, from tracking its state, which additionally reflects whether the result was within the configured threshold.
- Cisco documentation describes tracked object lists using boolean and threshold rules, allowing several objects to be combined so that a defined subset must fail before the combined object goes down.
- The CCIE Enterprise Infrastructure v1.1 unified exam topics include network assurance and first hop redundancy protocols within the infrastructure services domain.