In-depth BGP troubleshooting

In the TechTarget article Border Gateway Protocol (BGP) troubleshooting: Advanced approach
I'm describing how to troubleshoot BGP connectivity in a customer network with a single BGP-speaking router connected to upstream ISPs.

You can practice the skills described in that article on NIL's remote labs. Cisco partners and employees can get them free of charge on Partner E-learning connection: just click this link after logging into PEC. Everyone else can buy the same labs as part of IP Routing bundle from NIL Data Communications.

The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.

DHCP conflict logging: the true story

The on-line configuration help for the ip dhcp conflict logging configuration command (logging: Record address conflicts in a log file) is one of the more misleading texts I've found in Cisco IOS (and the CCO documentation is not much better). Here's how it actually works ...

If you have configured ip dhcp ping parameters (highly recommended), the router will ping the IP address it intends to allocate to a client before replying to the DHCP request. If the router receives ICMP Echo Reply message (response to ping), the address is obviously in use. If the DHCP conflict logging is enabled (default), the router will log the conflict with a syslog message (not in a separate log file) and put the address on the list of conflicts. The addresses on that list (displayed with show ip dhcp conflict) are not used in the future (similar to the addresses configured with the ip dhcp excluded-addresses command). To reuse a conflicting address, the network operator has to remove it from the list with the clear ip dhcp conflict address (or * for all addresses) command.

The DHCP conflict logging makes sense if the router uses persistent DHCP bindings (called DHCP database agents in Cisco IOS), otherwise any addresses allocated prior to a router reload would be reported as conflicts after the bindings are lost. If you don't use DHCP agents, it's thus best to turn off conflict logging with the no ip dhcp conflict logging configuration command. Even without conflict logging, there's no DHCP functionality loss and no chance of duplicate address allocation, as the router would still check whether an IP address is active before allocating it (and later on, it would be willing to re-check the conflicting IP address).

If you don't use DHCP database agents and you don't disable conflict logging (default setup), you'll have to clear the conflicts manually after a reload and you might potentially exhaust the DHCP pool because of a large number of blocked conflicting addresses.

Note: This article is part of You've asked for it series.

Track interface IP routing detects incorrect interface state

The track number interface name ip routing command is supposed to track an interface readiness to forward IP packets. In reality, it only tracks the interface line protocol status plus the IPCP status in case of PPP interfaces (as well as the actual presence of an IP address on the interface). If you configure IP Event Dampening (with the dampening) command, the interface might be suppressed (unavailable for IP routing), but the track object will report it as available (tested on IOS release 12.4(6)T). This could result in suboptimal HSRP/GLBP decisions if you use track objects to influence HSRP/GLBP priority or actual loss of data if you use such a track object to control policy-based routing.For example, with the following configuration ...

interface Serial0/1/0
dampening 15 500 2000 60 restart 500
!
track 1 interface Serial0/1/0 ip routing

... the interface might be suppressed while the track object would report it's up:

router#show interfaces dampening
Serial0/1/0
Flaps Penalty Supp ReuseTm HalfL ReuseV SuppV ...
0 3235 TRUE 42 15 500 2000 ...
router#show track 1
Track 1
Interface Serial0/1/0 ip routing
IP routing is Up
27 changes, last change 00:03:37
Tracked by:
1207165940

MPLS LDP autoconfiguration

Most MPLS books (mine included) and courses tell you that you have to manually enable MPLS on each interface where you want to run it with the mpls ip interface configuration command. However, this task was significantly simplified in IOS release 12.3(14)T with the introduction of MPLS LDP autoconfiguration. If you use OSPF as the routing protocol in your network, you can use the mpls autoconfig ldp [area number] router configuration command to enable LDP on all interfaces running OSPF (optionally limited to an OSPF area).

As the careful readers of my MPLS books know, it's dangerous to run LDP with your customers; the moment you run LDP with them (Carrier's carrier model is an exception), they can insert any labeled packet into your network, bypassing inbound access lists and sending traffic where it's not supposed to go (even into another VPN). It's therefore vital that you consider security implications before deploying MPLS LDP autoconfiguration.

Using this feature on P routers is absolutely safe, as they have no customer links. You have to be more careful on the PE-routers, more so if you run routing protocols with your customers. The safest configuration method would be to configure LDP autoconfiguration inside a single OSPF area, but even then a configuration error (placing PE-CE interface in a wrong area) could open your network to MPLS-based attacks.

Insert responses to command prompts in Tclsh

I have been aware of the typeahead Tcl command for months, but somehow I never got it to work. It works perfectly in IOS release 12.4(15)T (it might have something to do with other fixes to Tclsh), so to clear interface counters (as Michal would like to do), this is what you can do:

typeahead "y"
exec "clear counter dialer 0";

Warning: if the input is not consumed by the executed commands, it stays in the typeahead buffer; quite dangerous if you have a sequence of commands, as the wrong command could be acknowledged.

An in-depth version of this article is available in the CT3 wiki.

You can find more Tclsh-related information in the Tclsh on Cisco IOS tutorial. Sample Tclsh scripts are available in the Tclsh script library. If you need expert help in planning, developing or deploying Tclsh scripts in your network, contact the author.

This article is part of You've asked for it series.

Skip the “show ip route” legend

Are you as upset as I am with the constant display of the legend in front of the routes displayed with the show ip route command? Two output filters can help you.The easier one is show ip route parameters ¦ begin Gateway (as there is always a line starting with Gateway of last resort ...) before the actual IP routes:

a1#show ip route 172.16.0.0 longer ¦ begin Gateway
Gateway of last resort is not set

     172.16.0.0 255.255.0.0 is variably subnetted, 4 subnets, 2 masks
O       172.16.0.21 255.255.255.255
           [110/51] via 172.18.1.2, 00:04:56, Serial0/0/0.100
O       172.16.0.12 255.255.255.255
           [110/65] via 172.18.1.6, 00:04:56, Serial0/1/0
C       172.16.0.11 255.255.255.255 is directly connected, Loopback0
O       172.16.1.4 255.255.255.252
           [110/113] via 172.18.1.6, 00:04:56, Serial0/1/0

A slightly more complex one matches the first line that has a digit after the leading white space.

a1#show ip route 172.16.0.0 longer ¦ begin ^ +[0-9]+
     172.16.0.0 255.255.0.0 is variably subnetted, 4 subnets, 2 masks
O       172.16.0.21 255.255.255.255
           [110/51] via 172.18.1.2, 00:08:55, Serial0/0/0.100
O       172.16.0.12 255.255.255.255
           [110/65] via 172.18.1.6, 00:08:55, Serial0/1/0
C       172.16.0.11 255.255.255.255 is directly connected, Loopback0
O       172.16.1.4 255.255.255.252
           [110/113] via 172.18.1.6, 00:08:55, Serial0/1/0

If only IOS would have more decent regular expressions, like \s and \d ...

MPLS Traffic Engineering essentials

The MPLS traffic engineering essentials article I wrote for TechTarget describes the fundamentals of MPLS Traffic Engineering, how it compares to traditional traffic engineering available in Frame Relay and ATM networks and what are the distinctive benefits of using MPLS versus a legacy WAN technology.

The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.

Install a static route when an IP address is NOT reachable

In my February IP corner article, Small Site Multihoming, I've described how to install a static route based on reachability of a remote IP address (as measured with the IP SLA feature of Cisco IOS) and one of my readers recently asked an interesting question: “How do you install a static route when an IP address is not reachable?”

Without going into the design reasons that prompted the question, you can actually track when IP SLA measurement fails with an obscure configuration syntax of the track objects.In my example, the route to 1.0.0.0/8 would be inserted in the IP routing table when the ping to 172.16.0.22 fails:

!
! Define and start the IP SLA probe
!
ip sla 53
 icmp-echo 172.16.0.22
 timeout 500
 frequency 3
ip sla schedule 53 life forever start-time now
!
! Define an object that tracks the SLA probe
!
track 13 rtr 53 reachability
!
! Define another object that is the negation of the previous object
!
track 14 list boolean and
 object 13 not
!
! Insert a static route if the second object is UP (thus the 
! IP SLA probe failed)
!
ip route 1.0.0.0 255.0.0.0 Null0 track 14

Note: This article is part of You've asked for it series.

Warm reload does not change the config register

Contrary to what the regular reload does, the warm reload does not change the configuration register value (obviously that's done by ROMMON, which is not involved in the warm reload process). If you just did a password recovery and changed the configuration register back to a normal value, you'd thus be unpleasantly surprised when the NVRAM would be ignored (yet again) after a warm reload (I stumbled across this as I was trying a new IOS release with the reload warm file URL command).

DHCP-based static routes

If you have configured your router as a DHCP client, you can use the default router option received in a DHCP reply as the next-hop for a static route. For example:

ip route 10.0.0.0 255.0.0.0 dhcp

You could use this functionality in scenarios where your core network uses DHCP (for example, in metropolitan networks using layer-2 Ethernet transport from an ISP), but your router needs a different default route.

You can also use this feature to change the administrative distance of the DHCP-based default route (or you could use the ip dhcp-client default-router distance value configuration command that one of the readers described in a comment to a previous DHCP-related post).

Any other good ideas where this might come handy? Post them as comments ...

Reload a router from Tcl script

In his comment, Michal has asked about the ability to execute IOS commands with prompts from Tcl shell. I haven't found a generic solution yet, but you can reload a router from a Tcl script. First you have to define an EEM applet that reloads the router and can be triggered from command-line interface:

event manager applet forceReload
 event none
 action 1.0 reload

Now you can use the exec "event manager run forceReload" Tcl command in your Tcl script to run the applet (and reload the router).

Notes:

• To execute file management commands from Tcl shell, you have to disable prompts with file prompt quiet configuration command;
• This article is part of You've asked for it series.

Basic BGP troubleshooting

In the TechTarget article Border Gateway Protocol (BGP) troubleshooting: Simple approach I'm describing how to troubleshoot BGP connectivity in a customer network with a single BGP-speaking router connected to upstream ISPs.

You can practice the skills described in that article on NIL's remote labs. Cisco partners and employees can get them free of charge on Partner E-learning connection: just click this link after logging into PEC. Everyone else can buy the same labs as part of IP Routing bundle from NIL Data Communications.

The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.

Import DHCP options from an upstream DHCP server

If your router gets its IP address from an upstream DHCP server, it can automatically import the other DHCP options (DNS server, WINS server, domain prefix etc.) into its DHCP pools. For example, if you use a router to connect to a cable or MAN Ethernet ISP (see the following figure), you can use the DHCP option import to minimize your router configuration (and make it fail safe from any changes in the ISP network).

To configure the DHCP option import, use the import all DHCP pool configuration command. You cannot select which options you want to import, but you can override them with other DHCP pool configuration commands.

The import only happens when a DHCP reply is received. To force an immediate import, use the renew dhcp interface exec-level command.

The sample configuration for the above network topology is included below:

ip dhcp pool LAN
 import all
 network 10.0.0.0 255.255.255.0
 default-router 10.0.0.1
!
interface FastEthernet0/0
 description *** Internal LAN ***
 ip address 10.0.0.1 255.255.255.0
!
interface FastEthernet0/1
 description *** Public LAN interface ***
 ip address dhcp client-id FastEthernet0/1

OSPF graceful shutdown

Reloading a core router in a high-availability network is always a tricky proposition. Even if you tweak the routing protocol hello timers (or use fast L2 mechanisms to detect next-hop loss), it still takes a few seconds for the routing protocols to converge. For example, when using OSPF, the adjacent routers have to detect the neighbor loss, change their router LSAs, flood them (LSA flooding is rate-limited), the changed LSAs have to be propagated across the whole area and all routers in the area have to run SPF (which is also rate-limited). It's much better if you could gracefully take a router offline by increasing the OSPF cost on all its interfaces, thus forcing an OSPF SPF run while the router is still capable of forwarding the traffic (resulting in no packet loss).

The OSPF stub router advertisement (as this feature is officially called) documented in RFC 3137 is implemented in Cisco IOS release 12.2(4)T and 12.3. To force the router into stub status (prior to reboot/shutdown), use the max-metric router-lsa router configuration command. This command will change the OSPF metric for all non-stub interfaces in the router LSA to 65535.

Note: The infinite metric in the router LSA does not force the other routers to ignore the path, just nudge them into using alternate paths. The other routers in the network will thus select alternate OSPF paths (if they exist), but not the potential non-OSPF paths. Those will be selected only after the actual router reboot/shutdown.

This is a sample router LSA after the max-metric router-lsa has been configured:

b1#show ip ospf data router 172.16.0.21

OSPF Router with ID (172.16.0.21) (Process ID 1)

Router Link States (Area 0)

Exception Flag: Announcing maximum link costs
LS age: 18
Options: (No TOS-capability, DC)
LS Type: Router Links
Link State ID: 172.16.0.21
Advertising Router: 172.16.0.21
LS Seq Number: 80000003
Checksum: 0x88B2
Length: 72
Number of Links: 4

Link connected to: a Stub Network
(Link ID) Network/subnet number: 172.16.0.21
(Link Data) Network Mask: 255.255.255.255
Number of TOS metrics: 0
TOS 0 Metrics: 1

Link connected to: another Router (point-to-point)
(Link ID) Neighboring Router ID: 172.16.0.11
(Link Data) Router Interface address: 172.16.1.2
Number of TOS metrics: 0
 TOS 0 Metrics: 65535

Link connected to: a Stub Network
(Link ID) Network/subnet number: 172.16.1.0
(Link Data) Network Mask: 255.255.255.252
Number of TOS metrics: 0
TOS 0 Metrics: 50

Link connected to: a Transit Network
(Link ID) Designated Router address: 192.168.0.6
(Link Data) Router Interface address: 192.168.0.5
Number of TOS metrics: 0
 TOS 0 Metrics: 65535

In-depth presentation on Tcl, EEM, ERM, Kron ...

The comment made by Xavier sent me searching the Cisco documentation for the scripting tcl low-memory command. While I was not able to find anything useful (even links to some other scripting commands point to nowhere), I've discovered this gem - a great presentation explaining Tcl, EEM, ESM, ERM and Kron. Highly recommended reading :)

Update @ 2009-03-05: fixed the link to the presentation

Traffic engineering's role in next-generation networks

In the TechTarget article Traffic engineering's role in next-generation networks I'm describing why one would need traffic engineering in a Service Provider network, what options were available using the “legacy” WAN technologies and what you can do to deploy traffic engineering in a modern end-to-end IP network.

The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.

Default DHCP client-id

If you configure a Cisco router as a DHCP client, you'll notice that it uses weird client-id in its DHCP requests (assuming you care about client IDs on the DHCP server). Instead of using the interface MAC address as the client ID (as most workstations do), the client ID is the string 'cisco-dotted.mac.ascii-ifname' where the dotted.mac.ascii is the interface MAC address in ascii and the ifname is the short interface name.

Obviously, if your ISP checks your MAC address (and at least most cable operators do), you might have a problem. To make the router behave like a workstation, use the ip address dhcp client-id interface-name configuration command. The new client ID will be the MAC address of the specified interface (which can be different from the interface you're configuring).You can inspect the actual client ID in ASCII and hex with the debug dhcp detail. This is a sample default DHCP request packet:

DHCP: SRequest attempt # 1 for entry:
Temp IP addr: 172.18.0.3 for peer on Interface: FastEthernet0/1
Temp sub net mask: 255.255.255.0
DHCP Lease server: 172.18.0.1, state: 5 Renewing
DHCP transaction id: 2578
Lease: 600 secs, Renewal: 300 secs, Rebind: 525 secs
Next timer fires after: 00:03:46
 Retry count: 1 Client-ID: cisco-0016.c85e.fbc9-Fa0/1
Client-ID hex dump: 636973636F2D303031362E633835652E
666263392D4661302F31
Hostname: a1

... and this is a DHCP request packet after the client-id option has been attached to the ip address dhcp command.

Temp IP addr: 0.0.0.0 for peer on Interface: FastEthernet0/1
Temp sub net mask: 0.0.0.0
DHCP Lease server: 0.0.0.0, state: 9 Purging
DHCP transaction id: 5CD
Lease: 0 secs, Renewal: 0 secs, Rebind: 0 secs
Next timer fires after: 00:00:26
 Retry count: 0 Client-ID: 0016.c85e.fbc9
Client-ID hex dump: 0016C85EFBC9
Hostname: a1

You fix some, you break some ...

When Cisco fixed the tclsh bug in IOS release 12.4, they managed to break another nice feature: you can no longer execute tcl scripts within HTTP server on Cisco IOS. Previously you could use tcl scripts to generate customized outputs or reports that could be viewed through a web browser or even generate parts of HTML code that could be included in web pages served from the router. It's all gone in 12.4(15)T1 ...

Example: Tcl script with command-line parameters

In a comment to the “Execute multiple commands at once” post, Michal has asked for a complete Tcl-shell-with-parameter example. Here's a short script that shuts down the interface and displays its status:

• Variable ifname is set to the value of the first command-line parameter (in many other programming languages, this would be written as argv[0]);
• If the ifname is empty, the script aborts and prints the usage guidelines (again, in a more human-oriented programming language, this would be if (ifname == “”) ...);
• The show ip interface ifname command is executed. If it fails, the interface name is not correct and the script aborts.
• IOS configuration commands interface ifname and shutdown are executed.
• The show ip interface brief configuration command is executed and filtered with the interface name.

#
# ifname is set to first CLI parameter (interface name)
#
set ifname [lindex $argv 0]
if {[string equal $ifname ""]} { puts "Usage: shutdown ifname"; return; }
if { [ catch { exec "show ip interface $ifname" } errmsg ] } {
  puts "Invalid interface $ifname, show ip interface failed"; return}

ios_config "interface $ifname" "shutdown"
puts [ exec "show ip interface brief ¦ include $ifname" ]

If you store this Tcl script into your flash as shutdown.tcl and configure alias exec shutdown tclsh flash:shutdown.tcl, you can execute the command shutdown Serial0 to shut down the serial interface.

Notes:

• The last show command will display the interface status only if the specified interface name exactly matches the actual IOS interface name (whereas the rest of the script accepts shortcut names). The more generic matching algorithm is left as an exercise for the reader
• For more in-depth information on Tclsh implementation on Cisco IOS, read the IOS Tclsh resources.
• This article is part of You've asked for it series.

Re-enable debugging without EEM

In his comment to my post about re-enabling debugging after router reload, Mike pointed out an interesting IOS feature: you can execute the do command from a configuration file, not just from the user interface. To make his tip even more useful, you can store the do command(s) in an external file on a TFTP server, not in the startup configuration (which would have to be edited manually). With the boot host URL configuration command you'd then ensure that these commands are executed after the router reload.

Notes:

• The router expects a newline character at the end of the configuration file. The best way to ensure it's always there is to add a comment line at the end of the file
• The configuration file load usually fails immediately after the reboot, as the interfaces and IP routing processes are not yet fully operational. You might thus miss the first few seconds of the router's operations (unless you store the extra configuration file Flash or NVRAM).

Tclsh is broken in 12.4T

As Roddie pointed out in his comment, in some cases tclsh writes its output to the first line (usually console), not to the current line as expected. This “feature” has been introduced in IOS release 12.4(6)T1 (12.4(6)T is still OK) and fixed sometime before IOS release 12.4(11)T; 12.4(9)T is still broken (and I haven't tried all the other interim rebuilds ... or shall I call them rebreaks?).

Introduction to BGP

If you have to give someone a brief introduction to BGP, the Introduction to Border Gateway Protocol (BGP) article I wrote for TechTarget might come handy. It describes where (and why) you'd use BGP, the basic principles of operation (including BGP attributes), the BGP benefits and drawbacks as well as common deployment scenarios.

The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.

Sample configuration: periodic upload of router configuration

Pete Vickers sent me a very interesting configuration sample:

To get an IOS device to upload it’s configuration periodically to an external FTP server:

ip ftp source-interface loopback 0
ip ftp username ftp_username
ip ftp password ftp_password
file prompt quiet
!
kron policy-list backup
 cli copy running-config ftp://10.20.30.40
!
kron occurrence daily-backup at 0:30 recurring
 policy-list backup

The beauty of this example is that you can use it on platforms that don't support Embedded Event Manager (which has a very similar cron functionality) as the kron commands were introduced in 12.2T and 12.3 IOS releases.

Note: You have to use the file prompt quiet configuration command as the commands executed by kron cannot supply any user input

Conditional OSPF default route: tested configuration

One of my readers asked for a working configuration of the conditional OSPF default route advertisment feature. In my scenario, the OSPF default route would be announced whenever an Internet prefix (172.18.0.0/16) would be present in the IP routing table.

router ospf 1
 log-adjacency-changes
 network 0.0.0.0 255.255.255.255 area 0
 default-information originate always route-map FromInternet
!
router bgp 11
 bgp log-neighbor-changes
 neighbor 172.16.1.2 remote-as 21
!
ip access-list standard FromInternet
 permit 172.18.0.0
!
route-map FromInternet permit 10
 match ip address FromInternet

Caveats:

• The route map configured in the default-information originate command tests the IP prefixes in the IP routing table. You can thus match only on those attributes that are present in the IP routing table (IP prefix, metric, next-hop), not on additional BGP attributes (like AS-path), which would be really cool
• Contrary to what Sebastian wrote in his comment, you don't have to redistribute BGP route into OSPF to make it work in IOS release 12.4(11)T or 12.2SRC, but the IP prefix you test cannot be a subnet.

This article is part of You've asked for it series.