IOS scheduling parameters

Peter Weymann sent me a really intriguing question:

A few days ago I started reading the Ciscopress book End-to-End Network Security: Defense-in-Depth and stumbled over the scheduler command. This one could be used to allocate time that the cpu spends on fast switching packets or process switching packets, if I understand it correctly. They also mention interrupting CPU processes but honestly I don't really understand how it works.

Cisco routers support (at least) three forms of layer-3 switching (formerly known as routing). CEF switching and fast switching are performed entirely within the interrupt context (I/O adapter interrupts a process the CPU is currently executing and all the work is done before the process resumes). Process switching is performed in two steps: packet is briefly analysed within the interrupt context and requeued into the IP Input process where it's eventually switched. Almost all I/O adapters used these days use a concept of RX/TX rings to communicate with the CPU, meaning that the CPU potentially has to handle more than one packet for each interrupt.

Fast switching is gone starting with IOS release 12.4(20)T.

Under very high load, the packet arrival rate could be so high that the router would constantly service packets within the interrupt context without ever returning back to the IOS processes.

You can check the CPU load incurred by the interrupt context and IOS processes with the show process cpu command. The second number in the five seconds part of the first line tells you the amount of interrupt context activity in the last five seconds.

To prevent the starvation of IOS processes (which could result in keepalive and routing protocol problems, eventually leading to loss of routing protocol neighbors), the scheduler allocate command limits the amount of time that can be spent in the interrupt context and allocates some guaranteed time to the IOS processes. Very probably the routers have a mechanism to mask the requests from the I/O adapters during that period so that the CPU is not interrupted (BTW, this slightly increases the jitter).

A similar command is the scheduler interval command. IOS has high- and low priority processes. Whenever the CPU has to decide what process to run (usually following an interrupt or when a process decides it's done with its work), it will run a high-priority process if one is ready. This could lead to starvation of low-priority processes and the scheduler interval command specifies the maximum amount of time the higher-priority processes can consume before a low-priority process is given a chance to run.

Unless you have serious (and I mean __serious__) problems in your network, don't play with these commands. They are a last-resort things you can do if you're under very heavy load and still need access to the exec to
reconfigure the router. In most cases, you should not have to worry ... and anyhow, if the CPU load is close to 100%, you have other problems anyway.

Apart from the Inside Cisco IOS Software Architecture book that you absolutely must have if you're interested in (a bit outdated) view of the internals of Cisco IOS, you can get more information in these documents:

• Router performance tuning by Rob Thomas
• Troubleshooting router hangs from Cisco Systems

Goodbye fast switching & cell-mode MPLS

After leaving us in the dark for almost a year, Cisco finally released new functionality in IOS release 12.4(20)T. Support for a number of hardware platforms has been removed (dynamips fans are left with the 7200’s, everything else is gone). They also removed two switching features: fast switching and label-controlled ATM (cell-mode MPLS-over-ATM) together with Label Switch Controller (LSC).

I have no problem living without LC-ATM or LSC; this technology was primarily a retrofit for the old boxes by the time MPLS really took off with MPLS VPN. Fast switching is a different beast. Whenever you’d encounter bugs in more creative designs involving NAT, IPSec and GRE on low-end routers, you could turn off CEF (assuming you did not run NBAR) and things would (sometimes) miraculously start to work. Without fast switching, turning off CEF would bring you straight into process switching, resulting in an order-of-magnitude (or more) performance loss. On the other hand, it's obvious it makes no sense to maintain an obsolete switching method … and more bugs will probably get reported and fixed now that the easy escape route is gone.

CEF and MLS

Harold Arley Morales has asked an interesting question:

What's the difference between Cisco Express Forwarding and Cisco MLS? Is Cisco's implementation of MLS standardized?

CEF is a routing table lookup mechanism. Instead of doing a lookup in the main IP routing table (displayed with the show ip route), the router does a lookup in a fully computed non-recursive version of the IP routing table (Forwarding Information Base - FIB) with layer-2 next-hop information attached to it (adjacency table).

MLS is a caching mechanism (similar to Netflow) that offloads layer-3 processing from the routing component into layer-2 ASICs that cannot perform full-blown layer-3 switching. When the layer-2 engine detects a single IP packet traversing multiple VLANs, the MLS populates the cache with the flow details and the subsequent packets belonging to the same flow (same source/destination IP addresses and port numbers ...) are switched without going through all the layer-3 mechanisms (for example, access lists). The Multilayer Switching Overview document gives you additional details.

The MLS uses a proprietary protocol (MLSP) through which the layer-2 switches identify routers.

This article is part of You've asked for it series.

MAC addresses on VLAN interfaces

CCIE Pursuit writes about the usage of MAC addresses on SVI (VLAN) interfaces on various Catalyst switches. Some switches might reuse the same MAC address on multiple VLANs, leading to interesting problems if another box bridges those VLANs (for example, a transparent firewall).