SNMP v3 users not shown in running-config

Ralf sent me a SNMPv3 question:

If I create a SNMPv3 user which has a password (snmp-server user userthree groupthree v3 auth md5 user3passwd), this user does not appear in the running- or startup-config. Cisco even documents this if you know what to look for.

I strongly suspect (although I did not test this) that these users are also missing from configuration exported to TFTP servers. What would be the recommended way to make usable config backups of routers with such users?

Like certificates, the SNMPv3 users are stored in private-config and thus never appear in the router configuration. If you want to have a backup of the user data, create a text file on one of your NMS servers, add SNMPv3 usernames and passwords in the text file and use the copy somewhere running-config to configure SNMPv3 users on the routers.

This article is part of You've asked for it series.

IOS scheduling parameters

Peter Weymann sent me a really intriguing question:

A few days ago I started reading the Ciscopress book End-to-End Network Security: Defense-in-Depth and stumbled over the scheduler command. This one could be used to allocate time that the cpu spends on fast switching packets or process switching packets, if I understand it correctly. They also mention interrupting CPU processes but honestly I don't really understand how it works.

Cisco routers support (at least) three forms of layer-3 switching (formerly known as routing). CEF switching and fast switching are performed entirely within the interrupt context (I/O adapter interrupts a process the CPU is currently executing and all the work is done before the process resumes). Process switching is performed in two steps: packet is briefly analysed within the interrupt context and requeued into the IP Input process where it's eventually switched. Almost all I/O adapters used these days use a concept of RX/TX rings to communicate with the CPU, meaning that the CPU potentially has to handle more than one packet for each interrupt.

Fast switching is gone starting with IOS release 12.4(20)T.

Under very high load, the packet arrival rate could be so high that the router would constantly service packets within the interrupt context without ever returning back to the IOS processes.

You can check the CPU load incurred by the interrupt context and IOS processes with the show process cpu command. The second number in the five seconds part of the first line tells you the amount of interrupt context activity in the last five seconds.

To prevent the starvation of IOS processes (which could result in keepalive and routing protocol problems, eventually leading to loss of routing protocol neighbors), the scheduler allocate command limits the amount of time that can be spent in the interrupt context and allocates some guaranteed time to the IOS processes. Very probably the routers have a mechanism to mask the requests from the I/O adapters during that period so that the CPU is not interrupted (BTW, this slightly increases the jitter).

A similar command is the scheduler interval command. IOS has high- and low priority processes. Whenever the CPU has to decide what process to run (usually following an interrupt or when a process decides it's done with its work), it will run a high-priority process if one is ready. This could lead to starvation of low-priority processes and the scheduler interval command specifies the maximum amount of time the higher-priority processes can consume before a low-priority process is given a chance to run.

Unless you have serious (and I mean __serious__) problems in your network, don't play with these commands. They are a last-resort things you can do if you're under very heavy load and still need access to the exec to
reconfigure the router. In most cases, you should not have to worry ... and anyhow, if the CPU load is close to 100%, you have other problems anyway.

Apart from the Inside Cisco IOS Software Architecture book that you absolutely must have if you're interested in (a bit outdated) view of the internals of Cisco IOS, you can get more information in these documents:

• Router performance tuning by Rob Thomas
• Troubleshooting router hangs from Cisco Systems

Is a label imposed in case of Penultimate Hop Popping?

Shivlu Jain sent me an interesting question:

I'm wondering whether a router performing penultimate hop popping (PHP) imposes an IGP label or not.The value of implicit null is 3; does it mean the router imposes this label (and adds four bytes to the packet)?

The penultimate router does not impose the IGP label (that's why this behavior is called penultimate hop popping). However, the egress router has to signal to its upstream neighbor (the penultimate router) that it should NOT impose a label, so it uses "implicit null" label (= 3) in TDP/LDP updates to signal that the top label should be popped, not rewriten.

This article is part of You've asked for it series.

Measure the cable lengths on a Catalyst switch

Ken McCoy sent me a short question:

At one point someone posted an article about a command you could run on the Catalyst switch that would give you back the distance of the cable between the switch and end device, but now I can't find it.

I remembered reading the same article and after I've figured out the underlying technology is called TDR (Time Domain Reflectometer), uncle Google immediately provided a reader tip from Csaba Farkas.

This article is part of You've asked for it series.

Public servers in a small multihomed site

If you want to deploy high-availability public servers within your network, you should implement proper multi-homing solution including BGP routing with the Service Providers. If you don't have your own public IP address space and your own AS number, you should try to become multihomed to one ISP (or change your ISP if they don't know what you're talking about). If you want to be multi-homed to two ISPs using techniques similar to the ones I've described in the Small-Site Multi-Homing article, you should be using a hosted service (they're probably cheaper than your time), not your own public server.

But if you still insist (like numerous readers of my articles) to deploy public servers on a site multi-homed via NAT, you'll find the design and implementation guidelines in my latest IP Corner article Servers in Small Site Multi-homing.

This article is part of You've asked for it series.

The “fallback global” VRF option does not exist in Cisco IOS

Cheng sent me an interesting question:

I'm reading your book MPLS and VPN Architectures and I've found the ip vrf forwarding name fallback global command in the “Additional Lookup in the Global Routing Table” section. I can only find this command in Junos, but not in IOS.

… and he was right. When we were writing the book, we described several features that were still in development as it looked like they would be in the production code by the time the book was published. Many of them made it into the public IOS releases (for example, the Carrier's Carrier architecture), but some of them (like this command) simply vanished from the surface.

However, it looks like the engineers that switched from Cisco to Juniper took the concept with them and implemented it in JunOS, so JunOS has this feature but IOS doesn't.

This article is part of You've asked for it series.

When “copy” actually means “merge”

Marcus Jensen asked me a very interesting question:

I want to send 3 lines of configuration to a remote router, but I know the first line will kill my connection. Can I save these 3 lines of code to a text file, and then issue a Tcl command to add those to the running config?

The solution is much simpler and does not have to involve Tcl at all. The copy something system:running-config command merges the configuration commands in the source file with the current running configuration.

You can store the configuration commands you want to execute in a local file (even in NVRAM) or you could execute them directly off a file server (using HTTP, FTP, TFTP or SCP protocol).

This article is part of You've asked for it series.

Interesting QoS problem on Catalyst 3750

Mohammad Faraz Rehan has encountered an interesting problem when using IP access-list based class/policy maps on Catalyst 3750:

When I try to apply the same service-policy to more than 15 interfaces, the configuration command fails and the switch generates the following syslog message:

%QOSMGR-4-POLICER_PLATFORM_NOT_SUPPORTED: Policer configuration has exceeded hardware limitation for policymap …

I've tried to help him with various TCAM-related information and in the end he found an interesting solution to the problem:

It looks like there is a limit related to using the same access-list/class-map/policy-map on multiple interfaces.

The first time I was applying the same policy-map (19 classes/19 ACLs/46 ACL lines) on all interfaces, but the switch would not accept it on more than 15 interfaces. Another test scenario had 18 classes/18 ACLs/52 ACL entries and the same policy-map would only work on 13 interfaces.

Now we implemented 24 different policy-maps, class-maps and ACLs remained the same, and the switch is happy.

OSPF default route based on IP SLA

Olivier Guillemain has asked an interesting question: “how could I originate a default route into OSPF based on IP SLA (for example, based on pinging a remote IP address)?”

This is very easy to do when the router originating the default route into OSPF needs an SLA-based default route itself:

1. Configure IP SLA and a corresponding track object;
2. Configure a default route using reliable static routing
3. Advertise the default route into OSPF with the default-information originate router configuration command

The solution is a bit more complex when the router originating the default route into OSPF should not have a default route. In this case, you could use a routing trick:

1. Configure IP SLA and a corresponding track object as before;
2. Use reliable static routing to configure a static host route for a bogus IP address (for example, 10.0.0.1/32) pointing to null0 (for example, ip route 10.0.0.1 255.255.255.255 null 0 track 100). Obviously this host route should not be redistributed into any routing protocol.
3. Conditionally advertise default route into OSPF based on presence of the static host route.

Note: This article is part of You've asked for it series.

The never-ending story of IP fragmentation

In the last few months I've run across a number of IP fragmentation issues, as you've probably noticed through my blog posts. I've also encountered a lot of misconceptions about IP fragmentation, its impact on GRE and IPSec as well as the fragmentation-related mechanisms, for example MTU Path Discovery. I hope that you'll find my January IP Corner article The Never-Ending Story of IP Fragmentation a good summary of the subject.

OSPF graceful shutdown

I've described the OSPF Graceful Shutdown functionality in an earlier post and got an excellent question about the implications of using OSPF stub router functionality with external routes.

The more I was digging into this issue, the more design questions I've got … and finally ended up writing a whole IP Corner article about it. You can read the in-depth discussion of design and implementation aspects of OSPF stub router functionality in the December IP Corner article: Bring your Network Closer to Five Nines with Graceful Shutdown.

mturoute: ping-mode output

Jeff West has asked me to document the printout produced by the mturoute utility. Here's the first part of the documentation.

mturoute works in two modes:

• Without the -t flag, it sends variable-lenght ICMP echo packets to the specified destination address, trying to figure out the largest packet that is successfully propagated to the destination.
• With the -t flag, it uses traceroute-like algorithm to find the hop-by-hop IP addresses (the source IP addresses of the ICMP TTL exceeded replies) and uses the same packet-size-calculating algorithm to measure the path MTU to each hop.

Today we'll focus on the non-trace mode. It tries to measure the path MTU with a bisection method varying the packet sizes between minimum MTU (92) and maximum MTU (specified with the -m parameter, default is 10000 bytes). The payload size of the first packet (without the -m flag) is thus 5046 bytes ((10000 + 92)/2).

On each iteration, the algorithm prints a “cryptic” sign indicating whether the ping with the current payload size succeeded. The following indications are given:

• '+': ICMP echo reply arrived
• '-': The ping failed (for various reasons, including exceeding the path MTU)
• 'u': ICMP destination unreachable response arrived, indicating blackhole or access-list.

ICMP unreachable is considered a successful response; at least we're measuring the path MTU up to the failure point

• '.': timeout. The ping will be retried up to three times (or the number specified with the -r option).

In the ping mode, mturoute prints the current ICMP payload size at each step, resulting in a printout similar to the one below. If you'd have specified the -d option, the printout would include detailed status codes from the IcmpSendEcho function.

$ mturoute 10.0.3.3
* ICMP Fragmentation is not permitted. *
* Maximum payload is 10000 bytes. *
- ICMP payload of 5046 bytes failed..
- ICMP payload of 2569 bytes failed..
+ ICMP payload of 1330 bytes succeeded.
- ICMP payload of 1949 bytes failed..
- ICMP payload of 1639 bytes failed..
- ICMP payload of 1484 bytes failed..
+ ICMP payload of 1407 bytes succeeded.
- ICMP payload of 1445 bytes failed..
+ ICMP payload of 1426 bytes succeeded.
+ ICMP payload of 1435 bytes succeeded.
+ ICMP payload of 1440 bytes succeeded.
+ ICMP payload of 1442 bytes succeeded.
+ ICMP payload of 1443 bytes succeeded.
+ ICMP payload of 1444 bytes succeeded.
+ ICMP payload of 1444 bytes succeeded.
Path MTU: 1472 bytes.

Note: To use the debug-enabled version of mturoute, or the version that does not need VC++ runtime, download the new ZIP archive from my web site.

You cannot start a Telnet session from Tcl shell

Recently, I've got a really interesting question: “If I start a telnet or SSH session from my Tcl shell with the exec command, I get the initial text, but then it all freezes. What's wrong?”

Let's start with the basic picture: When you open a telnet session to a router, the input and output is captured by the Exec process. When you open an outgoing telnet or SSH session to another router, the Exec process implements the telnet/ssh client functionality and switches the input/output data between the two TCP sessions:However, when you start the Tcl shell, the Exec CLI loses its control of the input/output streams, as they are taken over by Tclsh. The only means of communicating with the Exec CLI is through the typeahead and exec commands:So, if you start another Telnet session with the exec command, the Tclsh still controls the input/output of the user session and there is no way for the telnet client in the Exec CLI to communicate with the user.

This article is part of You've asked for it series.

Router as a TFTP server

Shaun needed an extra TFTP server in CCNP labs and asked whether you could use a router to act as one. The read-only (download only) TFTP functionality has been available in Cisco IOS for a long time, but the common wisdom was that you could only use the TFTP server function to serve current IOS image.

Fortunately, as of IOS 11.0, the function is more generic; you can serve any file residing on the router (you still cannot upload files), but you have to declare each file to be served with the tftp-server path global configuration command. You could even specify an alias to have the file available under a different name and attach an access list to each configured file to restrict its availability.

Note: This article is part of You've asked for it series.

Get creative: Is anyone using a default route?

I've got a great question from one of my readers: I have two central sites (primary data center and a disaster recovery data center) and I've inherited a situation where there was a lot of static routing and a default route pointing from the primary campus to the DRC. I've replaced most of the static routing with dynamic routing protocol, but as the documentation is scarce, I am afraid to remove the default static route (which I would need for proper Internet access). Is there a way to figure out whether the default route is still used?

Obviously, there are two types of solutions:

• The deterministic one: inspect the IP routing tables on primary router and DRC router and identify any IP prefix on the DRC router that has no matching or less specific prefix on the primary router.
• The non-deterministic ones: try to figure out if any packet is using the default route
This is where you can get really creative: how would you figure out if a packet going from the primary data center to the DRC site is using the default route?

Get creative: static routing with Catalyst 3750

I've probably got this question due to my IP Corner article Small site multi-homing, where I've been describing reliable static routing. Here's the scenario: we have two sites, each using a Catalyst 3750 switch, and routing between them using static routes. There's a primary fiber link between them and we're using twisted-pair-to-fiber converters due to port limitations on Cat3750. These converters do not report fiber link down status correctly (the carrier is still present on twisted pair even if fiber is down), so the primary Ethernet interfaces do not go down if the fiber link breaks and the primary static route is not removed, requiring manual action to switch over to the backup link.

My initial reaction was a polite answer explaining that the dynamic routing protocols were invented to handle scenarios like this one, but the poor guy responded that “his boss does not want to hear about a dynamic routing protocol.” The next idea was the reliable static routing, tracking next-hop availability over both interfaces, but the Catalyst IOS does not support that, as it's based on 12.2 release.

I've got a few other ideas in the meantime (at least one of them working perfectly), but let's hear it from you first ... what would be your solution to this problem?

Insert responses to command prompts in Tclsh

I have been aware of the typeahead Tcl command for months, but somehow I never got it to work. It works perfectly in IOS release 12.4(15)T (it might have something to do with other fixes to Tclsh), so to clear interface counters (as Michal would like to do), this is what you can do:

typeahead "y"
exec "clear counter dialer 0";

Warning: if the input is not consumed by the executed commands, it stays in the typeahead buffer; quite dangerous if you have a sequence of commands, as the wrong command could be acknowledged.

Note: This article is part of You've asked for it series.

Install a static route when an IP address is NOT reachable

In my February IP corner article, Small Site Multihoming, I've described how to install a static route based on reachability of a remote IP address (as measured with the IP SLA feature of Cisco IOS) and one of my readers recently asked an interesting question: “How do you install a static route when an IP address is not reachable?”

Without going into the design reasons that prompted the question, you can actually track when IP SLA measurement fails with an obscure configuration syntax of the track objects.In my example, the route to 1.0.0.0/8 would be inserted in the IP routing table when the ping to 172.16.0.22 fails:

!
! Define and start the IP SLA probe
!
ip sla 53
 icmp-echo 172.16.0.22
 timeout 500
 frequency 3
ip sla schedule 53 life forever start-time now
!
! Define an object that tracks the SLA probe
!
track 13 rtr 53 reachability
!
! Define another object that is the negation of the previous object
!
track 14 list boolean and
 object 13 not
!
! Insert a static route if the second object is UP (thus the 
! IP SLA probe failed)
!
ip route 1.0.0.0 255.0.0.0 Null0 track 14

Note: This article is part of You've asked for it series.

Reload a router from Tcl script

In his comment, Michal has asked about the ability to execute IOS commands with prompts from Tcl shell. I haven't found a generic solution yet, but you can reload a router from a Tcl script. First you have to define an EEM applet that reloads the router and can be triggered from command-line interface:

event manager applet forceReload
 event none
 action 1.0 reload

Now you can use the exec "event manager run forceReload" Tcl command in your Tcl script to run the applet (and reload the router).

Notes:

• To execute file management commands from Tcl shell, you have to disable prompts with file prompt quiet configuration command;
• This article is part of You've asked for it series.

Example: Tcl script with command-line parameters

In a comment to the “Execute multiple commands at once” post, Michal has asked for a complete Tcl-shell-with-parameter example. Here's a short script that shuts down the interface and displays its status:

• Variable ifname is set to the value of the first command-line parameter (in many other programming languages, this would be written as argv[0]);
• If the ifname is empty, the script aborts and prints the usage guidelines (again, in a more human-oriented programming language, this would be if (ifname == “”) ...);
• The show ip interface ifname command is executed. If it fails, the interface name is not correct and the script aborts.
• IOS configuration commands interface ifname and shutdown are executed.
• The show ip interface brief configuration command is executed and filtered with the interface name.

#
# ifname is set to first CLI parameter (interface name)
#
set ifname [lindex $argv 0]
if {[string equal $ifname ""]} { puts "Usage: shutdown ifname"; return; }
if { [ catch { exec "show ip interface $ifname" } errmsg ] } {
  puts "Invalid interface $ifname, show ip interface failed"; return}

ios_config "interface $ifname" "shutdown"
puts [ exec "show ip interface brief ¦ include $ifname" ]

If you store this Tcl script into your flash as shutdown.tcl and configure alias exec shutdown tclsh flash:shutdown.tcl, you can execute the command shutdown Serial0 to shut down the serial interface.

Notes:

• The last show command will display the interface status only if the specified interface name exactly matches the actual IOS interface name (whereas the rest of the script accepts shortcut names). The more generic matching algorithm is left as an exercise for the reader
• For more in-depth information on Tclsh implementation on Cisco IOS, read the IOS Tclsh resources.
• This article is part of You've asked for it series.