Border Gateway Protocol (BGP) instability is one of the most critical issues a network engineer can face. When BGP flaps, it consumes router CPU cycles, invalidates forwarding tables, and introduces severe packet loss and routing black holes across autonomous systems.
Diagnosing BGP flapping on Cisco IOS, IOS-XE, and NX-OS devices requires recognizing a fundamental distinction: BGP route flapping versus BGP neighbor (session) flapping. Route flapping occurs when specific prefixes are repeatedly announced and withdrawn while the underlying peer session remains up. Neighbor flapping occurs when the TCP port 179 peering session itself repeatedly transitions between the Established and down states (Idle, Active, or Connect).
This guide walks through how to verify flapping prefixes using route flap statistics, analyze syslog notifications and reset codes, troubleshoot session drops step by step, configure flap dampening, automate proactive monitoring, and collect diagnostic data according to Cisco Technical Assistance Center (TAC) standards.
Table of Contents
Checking Route Flap History and Metrics with show ip bgp flap-statistics
When individual prefixes flap without tearing down the entire BGP peering session, Cisco routers track churn through BGP flap metrics. The primary tool for viewing this behavior is show ip bgp flap-statistics.
Step 1: Confirm BGP Dampening Is Active
Cisco routers only collect flap statistics if BGP Route Flap Dampening is enabled. If dampening is disabled, the command will return an empty table or a notification stating that dampening is inactive. Verify your base configuration before investigating:
Router# show ip bgp dampening parameters
If dampening is active, proceed to verify the flap history across your routing information base (RIB).
Step 2: Inspect Global Flap Statistics
Run show ip bgp flap-statistics (or show bgp ipv4 unicast flap-statistics in modern IOS-XE/NX-OS) to view all prefixes that have accumulated penalty points:
Router# show ip bgp flap-statistics
BGP table version is 14205, local router ID is 192.168.10.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
Network From Flaps Duration Reuse Path
*d 10.100.1.0/24 192.168.1.2 4 00:08:12 00:22:45 65001 65002 ?
*h 172.16.50.0/24 192.168.1.2 2 00:03:10 65001 65003 i
Step 3: Interpret Output Columns and Status Flags
Each column in the output reveals how severely a prefix is misbehaving:
- Status Codes (
*,d,h,s): *(Valid): The route is valid and can be considered for best-path calculation.d(Damped): The route has exceeded the suppress threshold and is currently withheld from the routing table.h(History): The route flapped, was withdrawn, and is not currently reachable, but the router retains penalty history.s(Suppressed): The route is suppressed due to dampening policy.- From: The IP address of the peer advertising the unstable prefix.
- Flaps: The total count of flap events recorded since the history was created or cleared.
- Duration: The elapsed time since the first flap event occurred.
- Reuse: The remaining time before the route’s penalty drops below the reuse threshold, allowing it to become active again.
Step 4: Run Targeted Filters on Specific Prefixes or Autonomous Systems
On production routers carrying full Internet tables, running an unfiltered flap command can overwhelm the CLI buffer. Filter by specific prefix, subnet mask, regular expression, or AS-path filter list:
! Filter on a specific subnet
Router# show ip bgp flap-statistics 10.100.1.0 255.255.255.0
! Filter on paths passing through AS 65002
Router# show ip bgp flap-statistics regexp _65002_
! Inspect full penalty details for a single route
Router# show ip bgp 10.100.1.0/24
In the detailed show ip bgp <prefix> view, locate the lines showing Flap-count, penalty, and dampening status to see the exact penalty points assigned.
Step 5: Clear Flap History After Stabilization
Once the upstream provider or peering link stabilizes, manually reset accumulated penalties and flap counts:
Router# clear ip bgp flap-statistics 10.100.1.0 255.255.255.0
Takeaway: The
show ip bgp flap-statisticscommand allows you to view historical prefix churn, penalty counters, and reuse timers, but it requires BGP dampening to be enabled on the router.
Decoding Cisco BGP Syslog Messages and Peer Reset Reasons
When the issue is neighbor flapping rather than route flapping, Cisco routers generate specific syslog events and record termination codes in the peer state table.
What does %BGP-5-ADJCHANGE: neighbor ... Down mean?
This message is logged whenever a BGP adjacency drops. The suffix of the log entry provides the immediate trigger:
%BGP-5-ADJCHANGE: neighbor 192.168.1.2 Down Hold time expired
%BGP-5-ADJCHANGE: neighbor 192.168.1.2 Down Interface flap
%BGP-5-ADJCHANGE: neighbor 192.168.1.2 Down Peer closed the session
Hold time expired: The local router did not receive any BGP KEEPALIVE or UPDATE packets within the agreed hold time window (default 180 seconds).Interface flap: The physical interface or subinterface over which the peer is reached went down, triggering an immediate teardown.Peer closed the session: The remote peer cleanly closed the underlying TCP connection via a TCP FIN or RST.
What is the difference between BGP Notification sent and BGP Notification received?
Notification sent: The local Cisco router detected a protocol violation or condition (such as an expired hold timer, malformed attribute, or administrative shutdown) and sent an RFC 4271 NOTIFICATION message to the peer before closing the session.Notification received: The remote peer detected an issue and sent an error code to the local router.
The syslog entry typically includes the error code and subcode:
%BGP-3-NOTIFICATION: sent to neighbor 192.168.1.2 4/0 (hold time expired) 0 bytes
%BGP-3-NOTIFICATION: received from neighbor 192.168.1.2 2/2 (bad peer AS) 2 bytes
What do the standard RFC 4271 BGP Notification Error Codes mean?
BGP notifications use a Code/Subcode format:
| Error Code | Name | Common Triggers |
|---|---|---|
| 1 | Message Header Error | Connection synchronized improperly, invalid marker, or incorrect message length. |
| 2 | OPEN Message Error | Subcode 2 indicates Bad Peer AS (mismatched AS numbers); Subcode 1 indicates Unsupported Version; Subcode 3 indicates Bad BGP Identifier. |
| 3 | UPDATE Message Error | Malformed attributes, missing well-known mandatory attributes, or invalid network layer reachability information (NLRI). |
| 4 | Hold Timer Expired | Keepalives were lost in transit due to MTU issues, congestion, or high CPU utilization. |
| 5 | Finite State Machine Error | An unexpected packet was received while in a specific state (e.g., receiving an OPEN while already Established). |
| 6 | Cease | Subcode 1 indicates Maximum Number of Prefixes Reached; Subcode 2 indicates Administrative Shutdown (neighbor shutdown); Subcode 3 indicates Peer De-configured. |
How do you find the historical peer reset reason on a Cisco router?
If the syslog buffer has rolled over, you can retrieve the last drop reason directly from the peer’s data structure:
Router# show ip bgp neighbors 192.168.1.2 | include Last reset
Last reset 00:04:12, due to Peer closed the session
Last reset reason: BGP Notification received, hold time expired
You can also check the counter history:
Router# show ip bgp neighbors 192.168.1.2 | section Notification
This shows the timestamp, code, and raw hex bytes of the last sent and received notification messages.
Takeaway: Syslog messages containing
%BGP-5-ADJCHANGEand%BGP-3-NOTIFICATIONspecify whether the local or remote router triggered the drop, whileshow ip bgp neighbors <IP>retains the precise reset reason even after the event has passed.
Step-by-Step Troubleshooting for BGP Neighbor Flapping
If BGP neighbor sessions drop and re-establish periodically, follow this structured diagnostic workflow to isolate the root cause.
+-------------------------------------------------------------------+
| BGP Neighbor Flapping Diagnostic Flow |
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| Step 1: Check Physical / Link Layer (Errors, Drops, Carrier Flaps)|
+-------------------------------------------------------------------+
| Link is Clean
v
+-------------------------------------------------------------------+
| Step 2: Validate MTU & PMTUD (DF-bit Ping with Large Packets) |
+-------------------------------------------------------------------+
| MTU Passes (1500+ bytes)
v
+-------------------------------------------------------------------+
| Step 3: Inspect CPU & Control Plane Policing (CoPP Drops) |
+-------------------------------------------------------------------+
| Control Plane Uncongested
v
+-------------------------------------------------------------------+
| Step 4: Verify Timers & Fast Failure Detection (BFD Stability) |
+-------------------------------------------------------------------+
| Timers Sane / BFD Stable
v
+-------------------------------------------------------------------+
| Step 5: Check Routing Recursion & Next-Hop Flapping |
+-------------------------------------------------------------------+
Step 1: Check Physical and Data Link Layer Health
An unstable BGP session is frequently caused by intermittent physical connectivity. Check the underlying interface:
Router# show interfaces GigabitEthernet0/0/0 | include line protocol|input errors|CRC|drops
GigabitEthernet0/0/0 is up, line protocol is up
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
142 carrier transitions
- Carrier transitions: Indicates link flaps at Layer 1/2.
- CRC / Input errors: Indicates faulty cabling, optics, or duplex/speed mismatches causing TCP retransmissions and eventual session teardown.
Step 2: Test for MTU Mismatches and Path MTU Discovery (PMTUD) Failures
A classic signature of an MTU issue is a BGP session that reaches Established, stays up for 90 to 180 seconds, and then collapses.
During session establishment, small 64-byte KEEPALIVE packets negotiate the link. However, when the routers exchange their routing tables, BGP sends full-sized UPDATE packets (up to 1500 bytes or 4096 bytes if jumbo frames are configured). If an intermediate device drops packets larger than its MTU, the UPDATE packet never arrives, keepalives are blocked behind it, and the hold timer expires.
Verify MTU reachability using sweeping ping tests with the Don’t Fragment (DF) bit set:
Router# ping 192.168.1.2 size 1500 df-bit
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 192.168.1.2, timeout is 2 seconds:
Packet sent with the DF bit set
.....
Success rate is 0 percent (0/5)
If the ping fails at 1500 bytes but succeeds at 1400 bytes, an MTU mismatch or dropped ICMP Type 3 Code 4 (Fragmentation Needed) packet is breaking Path MTU Discovery.
Fix: Correct the interface MTU across the path, or enforce PMTUD and maximum segment sizing on the BGP peer:
Router(config)# router bgp 65000
Router(config-router)# neighbor 192.168.1.2 path-mtu-discovery
Step 3: Check for High CPU Load and Control Plane Policing (CoPP) Drops
If the router’s control plane is overwhelmed by process-switched traffic or route churn, the BGP process (BGP Router) may not get scheduled in time to process incoming keepalive packets before the hold timer expires.
Check CPU utilization:
Router# show processes cpu sorted | exclude 0.00%
CPU utilization for five seconds: 98%/12%; one minute: 95%; five minutes: 91%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
184 129482100 4829102 26812 48.20% 45.10% 42.15% 0 BGP Router
Check whether Control Plane Policing is dropping TCP port 179 packets under heavy traffic:
Router# show policy-map control-plane | section class-default|bgp
Look for incrementing “drop” counters in classes matching BGP or routing protocol traffic.
Step 4: Verify Timers and Bidirectional Forwarding Detection (BFD)
If keepalive and hold timers are tuned too aggressively (e.g., 1-second keepalive, 3-second hold time) over a jittery WAN link, transient delay will drop the session. Similarly, if BFD is configured with sub-second timers over a congested link, BFD will pull down the BGP session prematurely.
Inspect active timers:
Router# show ip bgp neighbors 192.168.1.2 | include Hold time
Hold time is 180, keepalive interval is 60 seconds
If BFD is active, verify its stability:
Router# show bfd neighbors details
Look for high flap counters or mismatched detection multipliers.
Step 5: Check for Routing Recursion and Next-Hop Loops
For multi-hop eBGP or iBGP sessions peered to loopback addresses, the router resolves the peer IP via an Interior Gateway Protocol (IGP) like OSPF or IS-IS. If the route to the peer’s loopback address is accidentally redistributed into BGP, a recursive routing loop occurs:
1. BGP establishes via the IGP route.
2. BGP learns the peer route, which has a better administrative distance or replaces the forwarding path.
3. The next-hop becomes recursive or invalid.
4. The BGP session drops, restoring the IGP route.
5. The cycle repeats continuously.
Verify how the peer IP is routed:
Router# show ip route 192.168.1.2
Ensure the route to the peer loopback is strictly learned via an IGP or static route, never via BGP.
Takeaway: Neighbor flapping is almost always caused by physical link errors, MTU/PMTUD black holes during large UPDATE bursts, control plane starvation, or recursive routing loops affecting the peer’s IP address.
Configuring BGP Route Flap Dampening in Cisco IOS
When external peers flap routes rapidly, configuring BGP Route Flap Dampening (RFD) prevents those updates from destabilizing your internal network. Dampening assigns penalty points (1000 points per route withdrawal, 500 per attribute change). If the penalty exceeds the suppress limit, the route is dampened. Once the flapping stops and the penalty decays below the reuse limit, the route is reinstated.
Use the following production templates to configure dampening globally or selectively via route maps.
Global BGP Dampening Configuration Template
Apply dampening across an entire BGP address family using default or custom decay timers.
router bgp <LOCAL_AS>
address-family ipv4 unicast
! Syntax: bgp dampening [<half-life> <reuse> <suppress> <max-suppress-time>]
! Default: half-life=15m, reuse=750, suppress=2000, max-suppress=60m
bgp dampening 15 750 2000 60
exit-address-family
Selective Dampening Configuration Template (Recommended)
Applying aggressive dampening to critical routes (such as internal prefixes or major internet transit routes) can create extended outages. Use a route-map to apply differentiated dampening parameters based on prefix length.
! Step 1: Define prefixes to match (e.g., golden routes vs small prefixes)
ip prefix-list CRITICAL_PREFIXES permit 10.0.0.0/8 le 16
ip prefix-list VOLATILE_PREFIXES permit 0.0.0.0/0 ge 24
! Step 2: Define route-map with customized dampening metrics
route-map BGP_DAMPENING_POLICY permit 10
match ip address prefix-list CRITICAL_PREFIXES
! Mild dampening for large/critical blocks: half-life 10m, reuse 1000, suppress 3000, max 30m
set dampening 10 1000 3000 30
!
route-map BGP_DAMPENING_POLICY permit 20
match ip address prefix-list VOLATILE_PREFIXES
! Aggressive dampening for small subnets: half-life 20m, reuse 500, suppress 1500, max 60m
set dampening 20 500 1500 60
!
route-map BGP_DAMPENING_POLICY permit 30
! Default action: fallback to standard dampening
set dampening 15 750 2000 60
! Step 3: Apply the route-map under the BGP process
router bgp <LOCAL_AS>
address-family ipv4 unicast
bgp dampening route-map BGP_DAMPENING_POLICY
exit-address-family
Verification and Operational Inspection
! Display active dampening configuration parameters
Router# show ip bgp dampening parameters
! Display only paths that are currently suppressed/dampened
Router# show ip bgp dampened-paths
Takeaway: Use selective dampening via route-maps rather than global dampening to avoid penalizing well-aggregated, business-critical prefixes while suppressing volatile customer or edge subnets.
Setting Up Proactive Monitoring: SNMP Traps, Syslog Alerts, and EEM
Rather than waiting for users to report connectivity loss, configure automated alerting to detect BGP state changes and flapping routes as they happen.
Step 1: Enable BGP SNMP Traps
Configure the router to send standard SNMP traps whenever a peer transitions between finite state machine (FSM) states:
! Enable generation of traps when BGP neighbor drops or establishes
Router(config)# snmp-server enable traps bgp state-changes
! Direct traps to your Network Management System (NMS)
Router(config)# snmp-server host 10.200.1.50 version 2c COMMUNITY_STRING bgp
Key SNMP Object Identifiers (OIDs) monitored by network surveillance platforms include:
* bgpPeerStateChange (1.3.6.1.2.1.15.7.1 or 1.3.6.1.2.1.15.0.1): Standard RFC 1657 trap sent on any state change.
* cbgpPeerStateChangeBackwardTransition (1.3.6.1.4.1.9.9.187.0.1): Cisco-specific trap generated whenever a session transitions backward (e.g., from Established to Idle).
* bgpPeerLastError (1.3.6.1.2.1.15.3.1.14): Contains the 2-byte error code and subcode of the last notification.
Step 2: Configure Centralized Syslog Alerting
Ensure all neighbor change events (%BGP-5-ADJCHANGE) are forwarded to a SIEM or centralized logging collector:
Router(config)# logging host 10.200.1.60
Router(config)# logging trap notifications
Router(config)# logging source-interface Loopback0
Router(config)# service timestamps log datetime msec show-timezone
Configure your SIEM or log management tool to alert on the string %BGP-5-ADJCHANGE: neighbor .* Down.
Step 3: Automate Diagnostic Capture with Cisco EEM
When a flap occurs intermittently, the state has often recovered by the time an engineer logs in. Use Cisco Embedded Event Manager (EEM) to capture state data at the exact moment of the flap:
Router(config)# event manager applet BGP_NEIGHBOR_FLAP_MONITOR
Router(config-applet)# event syslog pattern "%BGP-5-ADJCHANGE: neighbor .* Down"
Router(config-applet)# action 1.0 syslog msg "EEM: BGP flap detected. Capturing diagnostics."
Router(config-applet)# action 2.0 cli command "enable"
Router(config-applet)# action 3.0 cli command "show ip bgp summary | redirect bootflash:bgp_diag_sum.txt"
Router(config-applet)# action 4.0 cli command "show processes cpu sorted | redirect bootflash:bgp_diag_cpu.txt"
Router(config-applet)# action 5.0 cli command "show policy-map control-plane | redirect bootflash:bgp_diag_copp.txt"
Step 4: Stream Real-Time Telemetry (IOS-XE)
For modern network architectures, configure Model-Driven Telemetry (MDT) to stream BGP state changes in real time via gRPC or NETCONF using the Cisco-IOS-XE-bgp-oper YANG model. This provides microsecond-level visibility into peer state changes without the polling overhead of SNMP.
Takeaway: Combine SNMP backward-transition traps for immediate paging with an EEM diagnostic-capture script to ensure you have control plane, CPU, and peer data preserved at the moment a session drops.
Cisco TAC Technical Reference and Diagnostic Data Collection
When escalating an unresolved BGP flapping case to the Cisco Technical Assistance Center (TAC), adhering to official diagnostic methodology accelerates root-cause identification.
Architectural Scoping and Definitions
- BGP Session Flapping: Governed by RFC 4271. Refers to failure of the transport connection (TCP port 179) or expiration of the FSM keepalive mechanism.
- BGP Route Flap Dampening: Governed by RFC 2439. Defines the mathematical decay model used to penalize fluctuating NLRI without dropping the transport session.
Mandatory TAC Diagnostic Command Matrix
Collect and attach the following outputs before opening a TAC service request:
| Diagnostic Scope | Command Syntax | Purpose |
|---|---|---|
| Global BGP State | show ip bgp summary | Identifies neighbor up/down duration and state/prefix counts. |
| Peer Deep Dive | show ip bgp neighbors <PEER_IP> | Details last reset reason, notification history, and negotiated timers. |
| Route Churn | show ip bgp flap-statistics | Verifies prefix flap history and accumulated penalties. |
| Control Plane | show processes cpu sorted | Checks for BGP Router process starvation or high interrupt CPU load. |
| Buffer History | show logging | Preserves timestamped notification codes and FSM transitions. |
| Aggregated Data | show tech-support bgp | Generates a comprehensive snapshot of memory, configuration, and protocol data. |
Production Safety Guidelines: Debugging Restrictions
- Never run unconstrained debug commands: Avoid running
debug ip bgpordebug ip bgp updateson production routers. On a router carrying Internet routes, this will overwhelm the CPU and cause cascading outages. - Apply conditional debugging: If debugging is requested by TAC, always isolate the debug scope to the specific flapping neighbor:
ios
Router# debug ip bgp conditional group
Router# debug ip bgp neighbors 192.168.1.2 updates - Verify control-plane limits: Run debugs with logging redirected to the internal buffer (
logging buffered), never to the console terminal (no logging console).
Takeaway: Cisco TAC requires targeted neighbor statistics, logging buffers, and control plane load metrics to troubleshoot session instability; never run unconstrained global BGP debugs on production equipment.
Conclusion
Systematically isolating BGP flapping on Cisco routers requires separating route-level churn from session-level drops. Use show ip bgp flap-statistics and route flap dampening to view and control flapping prefixes across the routing table. When the peering session itself is bouncing, rely on syslog %BGP-5-ADJCHANGE messages, notification error codes, and targeted MTU and control-plane checks to identify the root cause. Backed by automated SNMP traps and EEM data collection, this structured approach ensures BGP instability is detected, analyzed, and resolved before it impacts business traffic.











Comments