A tracked object is a boolean with a name. Something decides whether it is up or down, and other features watch it and act. The value of the arrangement is that the deciding and the acting are separate: one probe can drive a static route, a policy decision, an address translation adjustment and a notification, and none of those features needs to know how the decision was made.
Most descriptions of this stop at the first use — a static route that is withdrawn when a probe fails — which is the easy half. The route moves in a second or two and the circuit fails over correctly. What breaks is everything that built state against the old path and did not notice it changed, and on a branch router with address translation that is every session on the site.
This article covers what a tracked object can actually drive, how a tracked static route behaves and what the distance ordering does, what has to happen alongside the route change, how to track conditions the router cannot probe for itself, and the tracking designs that flap or fail without anybody noticing. It is written for the lab rather than for the written exam, and sits alongside the rest of the CCIE Enterprise Infrastructure lab certification track.
The tracked object is a named boolean that does not know what watches it, which is what allows one probe to drive a route, a policy, a redundancy group and an arbitrary action at the same time.What Can a Tracked Object Drive?
What are the consumers?
A static route, which is present while the object is up and withdrawn when it goes down. A policy routing next hop, which is used only while the object is up. A redundancy group's priority, which is decremented. An applet, which can run anything at all when the object changes. And other objects, since objects can be combined. One probe therefore drives a complete failover across several mechanisms without any of them being aware of the others.
A Deeper Dive into the Consumers
The static route, which is the common case
A route configured with a tracked object is installed while the object is up and removed while it is down. Without tracking, a static route pointing at a next hop remains in the table as long as the outgoing interface is up, which on any modern circuit means it remains after the path beyond it has failed.
That is the entire reason this exists. A circuit that is up and not carrying traffic keeps a static route pointing into it, and every packet sent that way disappears.
Policy routing
A policy directing traffic to a next hop can be made conditional, so that the next hop is used only while an object confirming its usability is up. When it is down, the policy falls through to whatever comes next — another next hop, or the ordinary routing table.
This is how a design sending certain traffic over one circuit and the rest over another survives one of them failing. Without the condition, the policy keeps directing traffic into a dead path regardless of what the routing table has learned.
! Policy routing that steps aside when the path is gone
R1(config)# route-map GUEST-OUT permit 10
R1(config-route-map)# match ip address GUEST-SUBNET
R1(config-route-map)# set ip next-hop verify-availability 203.0.113.1 10 track 10
R1(config-route-map)# set ip next-hop verify-availability 198.51.100.1 20 track 20
!
R1(config)# interface Vlan30
R1(config-if)# ip policy route-map GUEST-OUT
!
R1# show route-map GUEST-OUT
An applet, which can do anything
An applet triggered by an object's change can run arbitrary commands. That is how the things routing cannot express get done: clearing state, sending a notification, adjusting something on another device, writing a log entry with context.
It is also the mechanism that makes the failover complete rather than partial, because the state that has to be cleared at the moment a path changes is not something any routing feature will clear on its own.
Combining objects
Objects can be combined by a logical rule or by weights, producing an object that is up when a defined combination of its members is. That is how "the circuit is usable" is expressed as one thing when it depends on three separate facts.
Keeping the combination shallow matters for the same reason it matters anywhere: an object depending on objects depending on objects is not something anyone can evaluate while diagnosing a fault at three in the morning.
! One object expressing "the primary circuit is usable"
R1(config)# track 1 interface GigabitEthernet0/1 line-protocol
R1(config)# track 10 ip sla 10 reachability
R1(config-track)# delay down 3 up 30
!
R1(config)# track 100 list boolean and
R1(config-track)# object 1
R1(config-track)# object 10
!
R1# show track brief
Track Type Instance State Last Change
1 Interface GigabitEthernet0/1 Up 02:14:08
10 IP SLA 10 reachability Up 02:13:41
100 List boolean and Up 02:13:41
What an object is not
It is not a measurement and it carries no detail. Up or down, and nothing else. Every consumer sees the same single bit, which is what makes the separation work and what means a consumer cannot react differently to different kinds of failure.
Where different responses are needed for different conditions, that requires different objects rather than one object with more information in it. That is usually the right structure anyway.
| Consumer | Effect when the object goes down | Recovers |
|---|---|---|
| Static route | Route withdrawn | Automatically |
| Policy next hop | That next hop skipped | Automatically |
| Redundancy priority | Decremented | Automatically |
| Applet | Whatever you wrote | Only if you wrote it |
| Another object | Re-evaluated | Automatically |
How Does a Tracked Static Route Behave?
What is the arrangement?
Two routes to the same destination with different administrative distances. The lower-distance one is tracked and is used while its object is up. The higher-distance one is configured permanently and is ignored while the first is present. When the object goes down the first route is removed, the second becomes the best available, and forwarding moves. Nothing switches anything in — the second route was always there and simply became the best one.
A Deeper Dive into the Route Pair
Why the distances matter
Both routes exist in the configuration at all times. Only the better one is installed, and the distance is what decides which that is. Making the backup's distance higher is what keeps it out of the table while the primary is present.
Setting them equal produces both routes in the table simultaneously and traffic balanced across them, which is a different design with different consequences and is usually not what was intended when the words "backup circuit" were used.
! The pair. The second is always configured, never used first.
R1(config)# ip route 0.0.0.0 0.0.0.0 203.0.113.1 track 100
R1(config)# ip route 0.0.0.0 0.0.0.0 198.51.100.1 10
!
! Which one is installed right now
R1# show ip route 0.0.0.0
Gateway of last resort is 203.0.113.1 to network 0.0.0.0
S* 0.0.0.0/0 [1/0] via 203.0.113.1
!
! And what the tracking believes
R1# show ip route track-table
ip route 0.0.0.0 0.0.0.0 203.0.113.1 track 100 state is [up]
Tracking the backup as well
The backup circuit can fail too, and a backup route pointing into a dead circuit is worse than no backup at all — it takes over from a working primary if the primary's object goes down for an unrelated reason, and delivers traffic nowhere.
Tracking both, with each object probing through its own circuit, means that whichever route is installed is one with a confirmed path. It is twice the configuration and it removes the case where the failover lands somewhere no better than where it came from.
! Track both circuits, each through its own path
R1(config)# ip sla 10
R1(config-ip-sla)# icmp-echo 10.100.0.10 source-interface GigabitEthernet0/1
R1(config-ip-sla-echo)# frequency 3
R1(config)# ip sla schedule 10 life forever start-time now
!
R1(config)# ip sla 20
R1(config-ip-sla)# icmp-echo 10.100.0.10 source-interface GigabitEthernet0/2
R1(config-ip-sla-echo)# frequency 3
R1(config)# ip sla schedule 20 life forever start-time now
!
R1(config)# track 10 ip sla 10 reachability
R1(config)# track 20 ip sla 20 reachability
!
R1(config)# ip route 0.0.0.0 0.0.0.0 203.0.113.1 track 10
R1(config)# ip route 0.0.0.0 0.0.0.0 198.51.100.1 10 track 20
The recursion question
A static route to a next hop requires the router to know how to reach that next hop. Where the next hop is directly connected, that resolves through the connected route and the route is valid as long as the interface is up.
Where the next hop is not directly connected, the route resolves through something else, and that something else may resolve through the route itself. Specifying the outgoing interface alongside the next hop removes the ambiguity and is the safe form on a point-to-point circuit.
What the distance means for a routing protocol
Where a routing protocol also provides a default, the distances interact. A static route with a distance lower than the protocol's will always win, and a backup static with a distance higher than the protocol's will never be used while the protocol has a route.
Working out the intended precedence on paper — every source of the same destination, with its distance — takes two minutes and prevents a design where the backup circuit can never be selected because something else always provides a better route.
Timing, added up
The detection time is the object's, which is the probe interval plus the timeout plus any damping. The route change itself is immediate once the object changes. So the failover time is entirely the object's detection time, with nothing added by the routing.
That makes the arithmetic simple and the tuning obvious: everything worth adjusting is in the probe and the object, and the routing contributes nothing to the delay.
The return path
Moving the outbound route does nothing about how traffic comes back. Where the two circuits have different addresses, the return path depends on what the far end knows, which for internet traffic is decided by the source address of outgoing packets and is therefore handled by translation.
Where the two circuits reach a private network rather than the internet, the far end has its own routing decision to make and it has to be arranged. A failover that moves outbound traffic and not inbound produces a path that works in one direction, which is worse than a clean failure.
What Has to Happen Alongside the Route Change?
What else holds state?
The translation table, which is the important one. Its entries record a global address that belongs to the circuit that has just failed, so return traffic for every existing conversation arrives at a circuit that is down — or does not arrive at all. The routing has moved and the translations have not. Also affected: tunnels sourced from the failed interface, and anything caching a path. Only the first needs action, and it needs it immediately.
A Deeper Dive into the Other State
Why the translation table is the problem
Each entry pairs an inside address with a global address drawn from the circuit that was active when the entry was created. After the failover, outgoing packets for those conversations are still translated to the old global address, and they leave by the new circuit with a source address that does not belong to it.
What happens next depends on the provider. Many discard traffic whose source does not belong to the circuit, which means those conversations are dead and will not recover until their entries expire — which with default timeouts is hours.
Clearing it at the moment of change
An applet triggered by the object's change clears the table. Existing conversations are dropped, which is the correct outcome: they were broken anyway, and dropping them lets clients reconnect immediately through the new circuit rather than waiting for entries to expire.
This is the single most valuable thing an applet does in this design, and its absence is why many circuit failovers appear to work in testing and leave users unable to do anything afterwards.
! The applet that makes the failover complete
R1(config)# event manager applet CIRCUIT-FAILOVER
R1(config-applet)# event track 100 state any
R1(config-applet)# action 1.0 syslog msg "Track 100 changed - clearing translations"
R1(config-applet)# action 2.0 cli command "enable"
R1(config-applet)# action 3.0 cli command "clear ip nat translation *"
R1(config-applet)# action 4.0 syslog msg "Translations cleared"
!
R1# show event manager policy registered
R1# show event manager history events | include CIRCUIT-FAILOVER
Tunnels sourced from the failed interface
A tunnel whose source is the failed interface goes down with it, which is correct. A tunnel whose source is a loopback survives if the loopback remains reachable by the other circuit, which is the arrangement to prefer.
Sourcing tunnels from a loopback and ensuring that loopback is reachable over both circuits is the design that makes overlay connectivity survive a circuit change. It requires the far end to be able to reach the loopback over either path, which on an internet-based design means the loopback address must be reachable through both providers — which usually it is not.
So in practice an internet-based tunnel is sourced per circuit and one of them fails with its circuit. That is fine and it should be a deliberate design with a second tunnel over the second circuit rather than an assumption that one tunnel will survive.
What does not need action
Existing application sessions. They were broken by the path change and clients reconnect. Attempting to preserve them is a considerably larger design involving shared state, and for a branch circuit failover it is not worth it.
Cached name resolution answers are similar: occasionally a cached answer points at something that was only reachable over the old path, and the cache expires shortly. It is rarely worth acting on.
Notifying somebody
A circuit failover is an event somebody should know about, and it is easy to miss because the network keeps working. An applet that logs and notifies turns it from an invisible event into a ticket.
Without it, the common outcome is a site running on its backup circuit for weeks, at reduced capacity and cost, with nobody aware that the primary failed.
! Make the event visible, both directions
R1(config)# event manager applet CIRCUIT-DOWN
R1(config-applet)# event track 100 state down
R1(config-applet)# action 1.0 syslog priority critical msg "PRIMARY CIRCUIT DOWN - on backup"
!
R1(config)# event manager applet CIRCUIT-UP
R1(config-applet)# event track 100 state up
R1(config-applet)# action 1.0 syslog priority warnings msg "Primary circuit restored"
!
R1# show logging | include CIRCUIT
Testing the whole sequence
Break the path, and check four things: the object changed, the route moved, the translations were cleared, and a client can reach the internet. The fourth is the one that matters and the third is the one that makes it possible.
Testing only the first two is what produces a failover that passes its test and fails in production, because the state that was not cleared is only a problem when there is state, and a test run on a quiet circuit has none. Reading this once is not the same as being able to do it under time pressure, which is what repetition against realistic CCIE lab practice scenarios is for.
| State | After a failover | Action needed |
|---|---|---|
| Translation table | Points at the dead circuit | Clear it from an applet |
| Tunnel sourced from the interface | Down with the circuit | A second tunnel on the backup |
| Application sessions | Broken | None — they reconnect |
| Resolver cache | Occasionally stale | None, usually |
| Nobody's awareness | Nobody knows | A notification from the applet |
How Do You Track Something the Router Cannot Probe?
What is the mechanism?
An object with no source of its own, whose state is set by something external. An applet, a script, or a management system decides whatever it needs to decide and sets the object accordingly. Every consumer then watches that object exactly as it would watch a probe-driven one. This is how conditions the router cannot measure — an application's health, a business schedule, a decision made elsewhere — become inputs to routing.
A Deeper Dive into External Objects
What it is useful for
Anything the router cannot determine. Whether an application at the other end is actually serving, as reported by a monitoring system. Whether a maintenance window is in progress. Whether a decision has been made to move traffic for a reason that has nothing to do with the network.
It turns those into the same kind of input as a probe result, which means the routing configuration does not have to change to accommodate a new kind of condition.
! An object with no source of its own
R1(config)# track 200 stub-object
R1(config-track)# delay down 0 up 0
!
! Something external sets it
R1# test track 200 state up
R1# test track 200 state down
!
! And a route watches it exactly as it would a probe
R1(config)# ip route 10.100.0.0 255.255.0.0 203.0.113.1 track 200
The applet that decides
An applet running on a timer can evaluate anything expressible as commands and set the object. That includes things no probe covers: a counter crossing a value, a log message appearing, an interface's error rate, the result of a command parsed for a keyword.
It is genuinely powerful and it is also code running on a router with no test environment, which argues for keeping it short and for having it fail in the safe direction if it cannot determine an answer.
! An applet that evaluates something no probe covers
R1(config)# event manager applet CHECK-ERRORS
R1(config-applet)# event timer watchdog time 60
R1(config-applet)# action 1.0 cli command "enable"
R1(config-applet)# action 2.0 cli command "show interfaces Gi0/1 | include input errors"
R1(config-applet)# action 3.0 regexp "([0-9]+) input errors" "$_cli_result" match errs
R1(config-applet)# action 4.0 if $errs gt 1000
R1(config-applet)# action 4.1 cli command "test track 200 state down"
R1(config-applet)# action 4.2 else
R1(config-applet)# action 4.3 cli command "test track 200 state up"
R1(config-applet)# action 4.4 end
The failure mode that is specific to this
An object with no source of its own retains whatever value it was last given, indefinitely. If the thing that sets it stops running — an applet removed, a script whose credentials expired, a management system decommissioned — the object stays at its last value and every consumer keeps acting on it.
Nothing reports this. The object looks healthy, the route is present, and the fact being represented has not been checked for months. It is the quietest failure in this article.
Making staleness visible
Have the setter write the object on every cycle rather than only when the value changes, and monitor the last change time. An object updated every minute whose last change was three days ago is either genuinely stable or abandoned, and the two are distinguishable only if the setter records something on each run.
A log line per cycle is heavy; a counter or a separate timestamp object is lighter. Either is better than an object whose staleness is invisible.
Where an applet is the wrong tool
Anything a probe already covers. A probe is purpose-built, its failure modes are understood, and it does not stop working because somebody edited a configuration. An applet reimplementing reachability testing is more code with more ways to be wrong.
The applet belongs where nothing else can express the condition, and every use of it should be justifiable by naming what a probe could not have determined.
Safe failure direction
An applet that cannot determine an answer — a command that returned unexpected output, a parse that failed — must decide what to do. Leaving the object unchanged is usually right for a condition that is normally stable, and setting it to the safe value is right where an unknown condition should be treated as a failure.
Deciding this explicitly, in the applet, is part of writing it. An applet that silently does nothing on an unexpected result is the one that produces the stale object described above.
Which Designs Flap or Fail Silently?
What should be avoided?
Four patterns. No upward damping, so a flapping circuit moves traffic back and forth and each move clears the translation table and drops everything. A backup that is not tracked, so a failover lands on a path that is also dead. An externally set object nobody refreshes. And a design tested by shutting an interface, which exercises the one failure mode that would have been detected anyway and none of the ones that matter.
A Deeper Dive into the Failures
Flapping, which is worse here than elsewhere
Every change of the object moves the route and, if the applet is doing its job, clears every translation. A circuit that flaps every few minutes therefore drops every session on the site every few minutes, which is considerably worse than the circuit being down.
Generous upward damping is the answer and it costs nothing, because the backup circuit is carrying traffic perfectly well during the wait. A minute or more before returning to a recovered primary is entirely reasonable.
! Fast down, slow up. The backup is fine while you wait.
R1(config)# track 10 ip sla 10 reachability
R1(config-track)# delay down 3 up 90
!
R1(config)# track 100 list boolean and
R1(config-track)# object 1
R1(config-track)# object 10
R1(config-track)# delay down 0 up 90
!
! How often has it actually changed?
R1# show track 100 | include Change|changes
The untracked backup
A backup route with no object behind it is installed whenever the primary is withdrawn, regardless of whether the backup circuit works. A failover to a dead backup is a complete outage arrived at deliberately.
Tracking both, each through its own circuit, means the installed route always has a confirmed path. Where neither is up, the outcome is the same total outage, and at least the objects say why.
Testing the wrong failure
Shutting the interface is the easiest test and the least informative. It exercises the case that interface tracking alone would have caught, and it does not exercise the case the probe exists for — a circuit that is up and not delivering.
A better test is to make the probe target unreachable while leaving the interface up, which is exactly the failure this design was built for. On a lab that is straightforward; in production it can be arranged with a filter applied upstream for a few minutes.
The test that should be run
Traffic flowing, then the path broken without touching the interface, then four checks: the object changed within the expected time, the route moved, the translations cleared, and a client can still reach the internet. Then restore the path and confirm the upward damping holds the route in place for its configured period.
That whole sequence takes ten minutes, exercises every part of the design, and is the difference between a failover that is configured and one that is known to work.
! The four checks, during a real test
R1# show track 100
R1# show ip route 0.0.0.0
R1# show ip nat translations | count .
R1# show logging | include CIRCUIT|Track
!
! And on restore, confirm the up delay holds
R1# show track 100 | include delay|remaining
What to monitor
The change count on every object, which should be very low and whose growth means something is flapping. The last change time on every externally set object, which detects abandonment. And which route is currently installed for the default, which detects a site sitting on its backup without anybody knowing.
The third is the one that saves money. A branch running on its backup circuit for a month is paying for capacity it is not using and running at capacity it did not plan for.
What to document
What each object represents in words, because an object number tells the next engineer nothing. What the applet does and why. The measured failover time from the last real test. And who owns the probe target, since it is a dependency on something outside the network.
The first of those matters most in an incident. A configuration full of numbered objects with no description is one nobody can reason about quickly, and a one-line description on each is free.
Blueprint framing
The CCIE Enterprise Infrastructure v1.1 blueprint covers network assurance and infrastructure services, and tracked objects appear as the mechanism connecting measurement to action. What is examined is generally the chain and the behaviour of a tracked static route within the distance ordering, rather than applet syntax.
| Design | Failure | Fix |
|---|---|---|
| No upward damping | Every flap drops every session | A minute or more before returning |
| Untracked backup | Failover onto a dead path | Track both, each through its circuit |
| Externally set, never refreshed | Stale forever, silently | Alert on the last change time |
| Tested by interface shutdown | The real failure is never exercised | Break the path, not the interface |
| No notification | A month on the backup circuit | An applet that logs both directions |
| Numbered objects, no descriptions | Nobody can reason about it | One line of description each |
Conclusion
A tracked object is a named boolean that does not know what watches it, and that separation is what makes one probe able to drive a route, a policy, a redundancy group and an arbitrary action at once. The route half is straightforward: two routes with different distances, the better one tracked, and the backup always configured and simply becoming best when the first is withdrawn. The failover time is entirely the object's detection time, because the routing adds nothing.
The half that breaks is the state. Translation entries created while the primary was active still map to that circuit's address, so after the route moves, packets leave by the backup with a source that does not belong to it and are discarded upstream. The routing has failed over and the translations have not. An applet clearing them at the moment the object changes is what makes the failover complete, and its absence is why so many of these pass their test and fail in production — a test on a quiet circuit has no state to go wrong.
And the two things most worth getting right are unglamorous. Generous upward damping, because every change drops every session and a flapping circuit with no damping is worse than one that is simply down. And a test that breaks the path rather than the interface, because shutting an interface exercises the one failure that interface tracking would have caught anyway, and none of the ones this design was built for. More CCIE Enterprise Infrastructure material — labs, protocol breakdowns and study guides — is collected on the SPOTO CCIE site.
External Links
- RFC 792 — Internet Control Message Protocol
- RFC 2681 — A Round-trip Delay Metric for IPPM
- RFC 2925 — Managed Objects for Remote Ping and Traceroute
- RFC 5880 — Bidirectional Forwarding Detection
- RFC 3022 — Traditional IP Network Address Translator
- RFC 4116 — Accountability and Multihoming
- Cisco Learning Network — CCIE Enterprise Infrastructure
Reference Notes
- RFC 792 specifies ICMP, including the echo request and reply used by reachability probes that drive tracked objects.
- RFC 2681 defines a round-trip delay metric, including the treatment of a measurement that does not complete within a waiting time.
- RFC 2925 defines managed objects for remote ping operations, including operation frequency, timeout and the recording of results.
- RFC 5880 specifies bidirectional forwarding detection, a lightweight mechanism for rapid detection of forwarding path failures between adjacent systems.
- RFC 3022 describes address translation state, which is held per conversation and associates an inside address with a specific global address for the life of the entry.
- RFC 3022 notes that returning traffic is matched against existing translation state, which is why entries created against one external address are unusable after that address becomes invalid.
- RFC 4116 discusses multihoming, including the dependence of return traffic on the source address used by outbound packets.
- Cisco documentation describes enhanced object tracking, including interface, route, IP SLA and list object types, and the stub object whose state is set externally.
- Cisco documentation describes tracking a static route, in which the route is installed only while the associated object is up.
- Cisco documentation describes the tracking delay, applied independently to the up and down transitions in order to damp a flapping condition.
- Cisco documentation describes the embedded event manager, including track state events and the actions available to an applet.
- The CCIE Enterprise Infrastructure v1.1 unified exam topics include network assurance within the infrastructure services domain.