IOS scheduling parameters

Peter Weymann sent me a really intriguing question:

A few days ago I started reading the Ciscopress book End-to-End Network Security: Defense-in-Depth and stumbled over the scheduler command. This one could be used to allocate time that the cpu spends on fast switching packets or process switching packets, if I understand it correctly. They also mention interrupting CPU processes but honestly I don't really understand how it works.

Cisco routers support (at least) three forms of layer-3 switching (formerly known as routing). CEF switching and fast switching are performed entirely within the interrupt context (I/O adapter interrupts a process the CPU is currently executing and all the work is done before the process resumes). Process switching is performed in two steps: packet is briefly analysed within the interrupt context and requeued into the IP Input process where it's eventually switched. Almost all I/O adapters used these days use a concept of RX/TX rings to communicate with the CPU, meaning that the CPU potentially has to handle more than one packet for each interrupt.

Fast switching is gone starting with IOS release 12.4(20)T.

Under very high load, the packet arrival rate could be so high that the router would constantly service packets within the interrupt context without ever returning back to the IOS processes.

You can check the CPU load incurred by the interrupt context and IOS processes with the show process cpu command. The second number in the five seconds part of the first line tells you the amount of interrupt context activity in the last five seconds.

To prevent the starvation of IOS processes (which could result in keepalive and routing protocol problems, eventually leading to loss of routing protocol neighbors), the scheduler allocate command limits the amount of time that can be spent in the interrupt context and allocates some guaranteed time to the IOS processes. Very probably the routers have a mechanism to mask the requests from the I/O adapters during that period so that the CPU is not interrupted (BTW, this slightly increases the jitter).

A similar command is the scheduler interval command. IOS has high- and low priority processes. Whenever the CPU has to decide what process to run (usually following an interrupt or when a process decides it's done with its work), it will run a high-priority process if one is ready. This could lead to starvation of low-priority processes and the scheduler interval command specifies the maximum amount of time the higher-priority processes can consume before a low-priority process is given a chance to run.

Unless you have serious (and I mean __serious__) problems in your network, don't play with these commands. They are a last-resort things you can do if you're under very heavy load and still need access to the exec to
reconfigure the router. In most cases, you should not have to worry ... and anyhow, if the CPU load is close to 100%, you have other problems anyway.

Apart from the Inside Cisco IOS Software Architecture book that you absolutely must have if you're interested in (a bit outdated) view of the internals of Cisco IOS, you can get more information in these documents:

• Router performance tuning by Rob Thomas
• Troubleshooting router hangs from Cisco Systems

Impact of CEF accounting

Jozef Janitor wrote a highly relevant comment to my post on CEF accounting: enabling it on a Catalyst switch drastically reduces its performance. The impact of CEF accounting (or other forwarding plane features) depends on switching implementation:

• There is almost no impact on single-CPU software platforms; the router has to perform CEF lookup anyway and increases the CEF accounting counters on-the-fly;
• Distributed software platforms are more complex, as the central CPU has to (at the very least) collect the switching statistics.
• The impact on hardware platforms is dependent on the layer 3 lookup implementation

In the case of a Catalyst 3550 switch, the L3 TCAM it uses to support IP routing, PBR and access-lists is so different from the CEF data structures that IOS has to fall back to software L3 switching to collect CEF accounting data.

Debugging cached CEF adjacencies

A while ago I wrote about cached CEF adjacencies and the impact they have on ARP caching. If you ever need to, you can debug them with the debug ip cef table command. As this command might produce a lot of output in a production network, always use it in combination with an access-list that limits the debugging to the selected address range.

Alternatively, you can use the debug arp adjacency command, but you cannot limit its output with an access-list

For example, to test cached CEF adjacencies in subnet 10.0.0.0/24, I've used the following commands:

rtr#show ip access-list 99
Standard IP access list 99
10 permit 10.0.0.0, wildcard bits 0.0.0.255 (26 matches)
rtr#debug ip cef table 99
IP CEF table debugging is on for access list 99
rtr#debug arp
ARP packet debugging is on
rtr#ping 10.0.0.10

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.10, timeout is 2 seconds:

08:57:27: IP ARP: creating incomplete entry for IP address: 10.0.0.10 interface FastE
thernet0/0
08:57:27: IP ARP: sent req src 10.0.0.6 0016.c876.8b38,
dst 10.0.0.10 0000.0000.0000 FastEthernet0/0
08:57:27: IP ARP: rcvd rep src 10.0.0.10 000c.29a7.8ade, dst 10.0.0.6 FastEthernet0/0
08:57:27: CEF-IP: Checking dependencies of 10.0.0.0 255.255.255.0
08:57:27: CEF-Table: Adjacency-prefix 10.0.0.10 255.255.255.255 add request -- succee
ded.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/4 ms
rtr#

CEF accounting

The last week's creative challenge was obviously too easy; a number of readers quickly realized that the CEF accounting can do what we need (and I have to admit I've completely missed it).

However, when I started to explore the various CEF accounting features, it turned out the whole thing is not as simple as it looks. To start with, the ip cef accounting global configuration command configures three completely unrelated accounting features: per-prefix accounting (that we need), traffic matrix accounting (configured with the non-recursive keyword) and prefix-length accounting.

The per-prefix accounting is the easiest one to understand: every time a packet is forwarded through a CEF lookup, the counters attached to the CEF prefix entry are increased. To clear the CEF counters, you can use the clear ip cef address prefix-statistics command. The per-prefix counters are also lost when the IP prefix is removed from the CEF table (for example, because it temporarily disappears from the IP routing table during network convergence process). The CEF per-prefix accounting is thus less reliable than other accounting mechanisms (for example, IP accounting).

Note: The CEF per-prefix counters are always present; if the CEF per-prefix accounting is not configured, they simply remain zero.

Last but not least, you don't need the detail keyword if you want to display the CEF accounting data for a particular prefix. The show ip cef address mask command is enough. And, finally, if you're running IOS release 12.2SB or 12.2XN, you can inspect the CEF counters with SNMP.

ARP entries are periodically refreshed if you use CEF switching

In a previous post I've been writing about the inability to clean the ARP cache due to cached CEF adjacencies. As it turns out, this behavior has another side effect: the router will automatically refresh all ARP entries (and CEF adjacencies) as they expire from the ARP cache. This might become a problem on high-end devices with a lot of directly connected hosts if you set the arp timeout to a low value.
Here is a sample debugging printout verifying this behavior:

Jun 7 16:34:49: IP ARP: sent req src 192.168.0.1 0016.c7fe.f150,dst 192.168.0.2 000c.293a.b455 FastEthernet0/0.100
Jun 7 16:34:49: IP ARP: rcvd rep src 192.168.0.2 000c.293a.b455, dst 192.168.0.1 FastEthernet0/0.100
Jun 7 16:34:49: IP ARP: creating entry for IP address: 192.168.0.2, hw: 000c.293a.b455

What is a cached CEF adjacency?

Whenever a router running CEF switching has LAN interfaces (or any other multi-access interfaces), you'll find cached adjacencies for active directly attached IP neighbors in its CEF table. These adjacencies ensure the smooth traffic flow toward the LAN-attached next-hops (preventing the initial packet drop symptom once the next-hop becomes active).

The cached adjacencies (for individual IP hosts) are created whenever the first packet is sent toward an IP destination covered by a glean adjacency and stay in the CEF table during the changes in IP routing table (even after the complete IP routing table is cleared with the clear ip route * command). The only way to remove them from the CEF table (although I can't see a good reason to do that) is to shutdown and re-enable the interface.To display the cached adjacencies use the show ip cef destination mask longer-prefixes detail command, for example:

a2#show ip cef 10.0.0.0 255.0.0.0 longer detail
... CEF statistics deleted ...

Adjacency Table has 3 adjacencies
10.0.0.0/24, version 14, epoch 0, attached, connected
0 packets, 0 bytes
  via FastEthernet0/0, 0 dependencies
    valid glean adjacency
10.0.0.0/32, version 5, epoch 0, receive
10.0.0.5/32, version 12, epoch 0, cached adjacency 10.0.0.5
0 packets, 0 bytes
  via 10.0.0.5, FastEthernet0/0, 0 dependencies
    next hop 10.0.0.5, FastEthernet0/0
    valid cached adjacency
10.0.0.6/32, version 4, epoch 0, receive
10.0.0.10/32, version 11, epoch 0, cached adjacency 10.0.0.10
0 packets, 0 bytes
  via 10.0.0.10, FastEthernet0/0, 0 dependencies
    next hop 10.0.0.10, FastEthernet0/0
    valid cached adjacency
10.0.0.255/32, version 6, epoch 0, receive

The receive adjacencies are the ones the router is listening to (its own IP address and both variants of subnet multicast address), the glean adjacency covers the directly connected IP subnet and the cached adjacencies are created on-demand for active IP next-hops.

Unequal cost load-sharing

One of the most commonly asked load-sharing-related questions is "can I load-share traffic across unequal-cost links". In general, the answer is no. In order to load-share the traffic, you need more than one path to the destination and the only way to get multiple routes toward a destination in the IP routing table is to make them equal-cost (the only notable exception being EIGRP that supports unequal-cost load-sharing with the variance parameter).

There are, however, two cases where you can force unequal traffic split across equal-cost paths toward a destination: when using inter-AS BGP with the link bandwidth parameter and when using unequal-bandwidth traffic-engineering tunnels.

Note: You can read more about load sharing with MPLS TE in my IP Corner article Perfect Load-Balancing: How Close Can You Get?Due to the way MPLS TE autoroute is implemented in Cisco IOS, all tunnels toward the same destination appear as equal-cost paths, even when their TE bandwidths are not the same. For example, using a simple TE configuration ...

interface Tunnel0
 ip unnumbered Loopback0
 tunnel destination 172.16.0.21
 tunnel mode mpls traffic-eng
 tunnel mpls traffic-eng autoroute announce
 tunnel mpls traffic-eng priority 7 7
 tunnel mpls traffic-eng bandwidth  300
 tunnel mpls traffic-eng path-option 1 explicit identifier 1
 no routing dynamic
!
interface Tunnel1
 ip unnumbered Loopback0
 tunnel destination 172.16.0.21
 tunnel mode mpls traffic-eng
 tunnel mpls traffic-eng autoroute announce
 tunnel mpls traffic-eng priority 7 7
 tunnel mpls traffic-eng bandwidth  500
 tunnel mpls traffic-eng path-option 1 explicit identifier 2
 no routing dynamic

... you get two equal-cost paths in your IP routing table even though the tunnel mpls traffic-eng bandwidths are different:

a1#show ip route ospf
  172.16.0.0 255.255.0.0 is variably subnetted, 6 subnets, 2 masks
O  172.16.0.21 255.255.255.255 [110/51] via 0.0.0.0, 00:11:06, Tunnel0
                               [110/51] via 0.0.0.0, 00:11:06, Tunnel1
O  172.16.0.22 255.255.255.255 [110/52] via 0.0.0.0, 00:11:06, Tunnel0
                               [110/52] via 0.0.0.0, 00:11:06, Tunnel1

When transferring the IP routing table into the CEF table, the router takes MPLS TE bandwidth in consideration, resulting in unequal traffic split proportional to the MPLS TE bandwidth:

a1#show ip cef 172.16.0.21 internal
172.16.0.21/32, version 55, epoch 1, per-destination sharing
0 packets, 0 bytes
  tag information set
    local tag: tunnel-head
  via 0.0.0.0, Tunnel0, 0 dependencies
    traffic share 3
    next hop 0.0.0.0, Tunnel0
    valid adjacency
    tag rewrite with Tu0, point2point, tags imposed: {}
  via 0.0.0.0, Tunnel1, 0 dependencies
    traffic share 5
    next hop 0.0.0.0, Tunnel1
    valid adjacency
    tag rewrite with Tu1, point2point, tags imposed: {}

  0 packets, 0 bytes switched through the prefix
  tmstats: external 0 packets, 0 bytes
           internal 0 packets, 0 bytes
  Load distribution: 0 1 0 1 0 1 0 1 0 1 0 1 1 1 1 1 (refcount 1)

  Hash  OK  Interface                 Address         Packets  Tags imposed
  1     Y   Tunnel0                   point2point           0    {}
  2     Y   Tunnel1                   point2point           0    {}
  3     Y   Tunnel0                   point2point           0    {}
  4     Y   Tunnel1                   point2point           0    {}
  5     Y   Tunnel0                   point2point           0    {}
  6     Y   Tunnel1                   point2point           0    {}
  7     Y   Tunnel0                   point2point           0    {}
  8     Y   Tunnel1                   point2point           0    {}
  9     Y   Tunnel0                   point2point           0    {}
  10    Y   Tunnel1                   point2point           0    {}
  11    Y   Tunnel0                   point2point           0    {}
  12    Y   Tunnel1                   point2point           0    {}
  13    Y   Tunnel1                   point2point           0    {}
  14    Y   Tunnel1                   point2point           0    {}
  15    Y   Tunnel1                   point2point           0    {}
  16    Y   Tunnel1                   point2point           0    {}

Note: this article is part of You've asked for it series.

CEF punted packets

The packets that cannot be CEF-switched in a box with CEF switching enabled are punted to the next switching level (fast switching or process switching). The incoming packets can be punted for a number of reasons, for example:

• If the destination is reachable over an interface that cannot use CEF-switching due to a feature not supported by CEF (for example, X.25 link), the packet has to be fast- or process-switched.

These destinations are easily discovered by inspecting the punt adjacencies).

• All packets destined for the router itself are process switched (thus punted).
• If the router needs to reply back to the source with an ICMP packet (redirect, unreachable ...), the reply can be generated only in the process-switching path.
• All packets with the IP options are punted to process switching.
• Fragments that have to be processed by the router are also process-switched.

You can inspect the amount of punted packets with the show cef not-cef-switched command.

This article is part of You've asked for it series.

CEF punt adjancency

In "border cases" you might find interesting CEF adjacencies in your CEF adjacency table (displayed with show ip cef adjacency). Most common one is the glean adjacency used for directly connected routes (this adjacency type is a placeholder that indicates the router it should perform the ARP table lookup and send the packet to directly connected neighbor). Discard, Drop, Noroute and Null adjacencies are obvious, the "weird" one is the Punt adjacency, which indicates that the router cannot CEF-switch the packet toward the destination (due to a feature being used that is not yet supported by CEF), thus the packet is punted to the next switching method (fast switching and ultimately process switching).

It's very easy to create a punt adjacency with a back-to-back X.25 link, for example

a1#show running interface serial 0/1/0
interface Serial0/1/0
 ip address 172.16.4.1 255.255.255.0
 encapsulation x25 dce
 x25 address 1
 x25 map ip 172.16.4.2 2
end
a1#show ip cef adjacency punt
Prefix              Next Hop             Interface
172.16.4.0/24       attached             Serial0/1/0

You can inspect the "special" CEF adjacencies in your router with the show ip cef adjacency discard¦drop¦glean¦noroute¦null¦punt command.

Per-port CEF load sharing

In designs with very low number of IP hosts, no per-destination load-sharing algorithm will work adequately. Consider, for example, an extranet design where a large number of IP hosts are NAT-ed to a single IP address which then accesses a single remote server.

In this design, all the traffic flows between a single pair of IP addresses, making per-destination load-sharing unusable.

Cisco has addressed this problem in IOS release 12.4(11)T with per-port CEF load sharing, which extends the CEF hashing function to include source and/or destination TCP or UDP port.

The global configuration command that enables per-port CEF load-sharing is ip cef load-sharing algorithm [ include-ports [source] [dest] ] seed. To test it, use the show ip cef exact-route command, which now supports source and destination port numbers. For example:

a1(config)#ip cef load-sharing algorithm include-ports source dest 22

a1#show ip cef exact-route 10.0.0.10 src-port 35 192.168.0.2 dest-port 80
10.0.0.10       -> 192.168.0.2    : Serial0/0/0.100 (next hop 172.16.1.2)
a1#show ip cef exact-route 10.0.0.10 src-port 36 192.168.0.2 dest-port 80
10.0.0.10       -> 192.168.0.2    : Serial0/0/0.200 (next hop 172.16.1.6)
a1#show ip cef exact-route 10.0.0.10 src-port 37 192.168.0.2 dest-port 80
10.0.0.10       -> 192.168.0.2    : Serial0/0/0.100 (next hop 172.16.1.2)

Per-destination or per-packet CEF load sharing?

Cisco Express Forwarding (CEF) can perform per-packet or per-destination (actually source/destination IP address pair) load-sharing with no performance degradation (without CEF, per-packet load-sharing requires process switching). Even though there is no performance impact on the router, per-packet load sharing will almost always result in out-of-order packets. The packet reordering might degrade TCP throughput in high-speed environments (in low-speed/few-flows scenarios, per-packet load-sharing actually improves the per-flow throughput) or severely impact applications that cannot survive out-of-order packet delivery, such as Fast Sequenced Transport for SNA over IP or voice/video streams.

To configure per-packet load-sharing, use the ip load-sharing per-packet interface configuration command (default is per-destination). This command has to be configured on all outgoing interfaces over which the traffic is load-shared.

Note: The switch between the load-sharing modes is not immediate; sometimes you have to wait a few seconds for the ip load-sharing command to take effect, worst case a manual clearing of the CEF table (clear ip cef address) is required.

Which switching path does an IOS feature use

I've got an excellent question recently: Which switching path is used in Zone-based firewalls when a packet is dropped? As usual, IOS documentation was not very helpful (which is understandable as the answer might depend on hardware platform, interface encapsulation, other features configured on the router etc.). However, there is a great tool to use - the show interface stats command.The show interface stats command displays the number of packets switched per switching mechanism (unfortunately, CEF is lumped together with fast switching). This command thus helps us to figure out if the packets are process switched. The show cef not-cef-switched command displays all packets punted from CEF to other switching mechanisms, so together they give us a pretty strong indication of what's going on.

To answer the question, I've configured a test lab router as follows:

class-map type inspect match-all ICMP
 match protocol icmp
!
!
policy-map type inspect InToOut
 class type inspect ICMP
  drop log
 class class-default
  inspect
!
zone security inside
zone security outside
zone-pair security InToOut source inside destination outside
 service-policy type inspect InToOut
!
interface FastEthernet0/0
 ip address 10.0.0.1 255.255.255.0
 zone-member security inside
!
interface Serial0/0/0.100 point-to-point
 description Link to the Internet
 ip address 192.168.201.6 255.255.255.252
 zone-member security outside

Note: it's best to perform these tests in very "quiet" and controlled environment to ensure the statistics are not skewed by other traffic.

I've started a continuous ping on a LAN-attached workstation (as expected, all pings failed), cleared the interface counters on the router and took a snapshot of them a while later. The counter values clearly indicate that the dropped packets are not process-switched.

fw#clear counters
...
*Dec 15 18:14:28: %FW-6-DROP_PKT: Dropping icmp pkt 10.0.0.2:8 => 
172.16.0.1:0 with ip ident 17758 due to DROP action found in policy-map
...
fw#show interface FastEthernet0/0 stats
FastEthernet0/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 4 553 10 919
Route cache 32 2368 0 0
Total 36 2921 10 919

fw#show cef not-cef-switched
CEF Packets passed on to next switching layer
Slot No_adj No_encap Unsupp'ted Redirect Receive Options Access Frag
RP   0      0        0          0        0       0       0      0

Fine-tuning CEF load-sharing

In environments with a low number of IP hosts you have to fine-tune the CEF load-sharing algorithm to ensure that the traffic is spread between all parallel paths. A typical scenario is a primary-backup data center setup with pairs of replicating servers, as shown in the figure below.

In these cases, you have to try different values of seed parameter of the CEF universal algorithm.
For example, if you have two equal-cost paths between networks 10.0.0.0/24 and 192.168.0.0/24 ...

a1#show ip cef 192.168.0.0 detail
192.168.0.0/24, version 33, epoch 0, per-destination sharing
0 packets, 0 bytes
 via 172.16.1.6, Serial0/0/0.200, 0 dependencies
   traffic share 1
   next hop 172.16.1.6, Serial0/0/0.200
   valid adjacency
 via 172.16.1.2, Serial0/0/0.100, 0 dependencies
   traffic share 1
   next hop 172.16.1.2, Serial0/0/0.100
   valid adjacency

... you might want the traffic between 10.0.0.1 and 192.168.0.1 to flow over a different link than the traffic between 10.0.0.2 and 192.168.0.2. The command that will help you is the show ip cef exact-route source destination. In our example, both traffic flows would go over the same serial link:

a1#show ip cef exact-route 10.0.0.1 192.168.0.1
10.0.0.1 -> 192.168.0.1 : Serial0/0/0.100 (next hop 172.16.1.2)
a1#show ip cef exact-route 10.0.0.2 192.168.0.2
10.0.0.2 -> 192.168.0.2 : Serial0/0/0.100 (next hop 172.16.1.2)

However, by changing the seed parameter of the ip cef load-sharing algorithm universal command, you can influence the CEF hashing function, eventually reaching a state where the traffic flows are spread between both WAN links:

a1(config)#ip cef load-sharing algorithm universal 1
a1(config)#^Z
a1#show ip cef exact-route 10.0.0.1 192.168.0.1
10.0.0.1 -> 192.168.0.1 : Serial0/0/0.100 (next hop 172.16.1.2)
a1#show ip cef exact-route 10.0.0.2 192.168.0.2
10.0.0.1 -> 192.168.0.2 : Serial0/0/0.200 (next hop 172.16.1.6)

CEF per-destination load sharing algorithms

According to the Cisco IOS documentation, you can select between the original and the universal CEF load sharing algorithm with the ip cef load-sharing algorithm name parameter global configuration command (we'll leave the tunnel algorithm aside for the moment). Of course, they don't tell you what you select.

The original algorithm used only the source and destination IP addresses to get the 4-bit hash entry (see the CEF Load Sharing Details for more information), which could result in suboptimal network utilization in some border cases (if anyone wants to know why, leave me a comment). The universal algorithm adds a router-specific value to the hash function, ensuring that the same source-destination pair will hash into a different 4-bit value on different boxes. If you really want to fine-tune the hash function, you can even specify the value to be added with the last option of the ip cef load-sharing algorithm command.

CEF load sharing details

I had to investigate the details of CEF load sharing for one of my upcoming article and found (yet again) that the details are rather undocumented in official documentation. So, this is how it works (in case you ever need to know):

• For every CEF entry (IP route) where there are multiple paths to the destination, the router creates a 16-row hash table, populating the entries with pointers to individual paths. The hash table can be inspected with the show ip cef prefix internal command.
• The load balancing ratio is approxiated by number of entries in the hash table belonging to each path. If you have unequal-cost load balancing (EIGRP based on composite metrics and MPLS TE tunnels based on requested bandwidth), individual paths will be associated with different number of rows.
• If you configure per-destination load balancing, the source and destination IP address in the incoming IP packet are hashed into a 4-bit value that selects the outgoing path in the CEF has table.

If this sounds confusing, here are two examples to make it easier: if you have two equal-cost paths to the same destination, each path will have eight entries in the hash table.

a1#show ip route 192.168.0.0
Routing entry for 192.168.0.0 255.255.255.0
  Known via "ospf 1", distance 110, metric 51, type intra area
  Last update from 172.16.0.21 on Serial0/0/0.100, 00:00:05 ago
  Routing Descriptor Blocks:
  * 172.16.0.21, from 172.16.0.22, 00:00:05 ago, via Serial0/0/0.100
      Route metric is 51, traffic share count is 1
    172.16.0.21, from 172.16.0.22, 00:00:05 ago, via Serial0/0/0.200
      Route metric is 51, traffic share count is 1
a1#show ip cef 192.168.0.0 internal
192.168.0.0/24, version 33, epoch 0, per-destination sharing
0 packets, 0 bytes
  via 172.16.0.21, Serial0/0/0.100, 0 dependencies
    traffic share 1
    next hop 172.16.0.21, Serial0/0/0.100
    valid adjacency
  via 172.16.0.21, Serial0/0/0.200, 0 dependencies
    traffic share 1
    next hop 172.16.0.21, Serial0/0/0.200
    valid adjacency

  0 packets, 0 bytes switched through the prefix
  tmstats: external 0 packets, 0 bytes
           internal 0 packets, 0 bytes
  Load distribution: 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 (refcount 1)

  Hash  OK  Interface                 Address         Packets
  1     Y   Serial0/0/0.100           point2point           0
  2     Y   Serial0/0/0.200           point2point           0
  3     Y   Serial0/0/0.100           point2point           0
  4     Y   Serial0/0/0.200           point2point           0
  5     Y   Serial0/0/0.100           point2point           0
  6     Y   Serial0/0/0.200           point2point           0
  7     Y   Serial0/0/0.100           point2point           0
  8     Y   Serial0/0/0.200           point2point           0
  9     Y   Serial0/0/0.100           point2point           0
  10    Y   Serial0/0/0.200           point2point           0
  11    Y   Serial0/0/0.100           point2point           0
  12    Y   Serial0/0/0.200           point2point           0
  13    Y   Serial0/0/0.100           point2point           0
  14    Y   Serial0/0/0.200           point2point           0
  15    Y   Serial0/0/0.100           point2point           0
  16    Y   Serial0/0/0.200           point2point           0

However, if you have three equal-cost paths to the destination, each path will have only five entries and the hash table will have 15 rows instead of 16.

a1#show ip route 192.168.0.0
Routing entry for 192.168.0.0 255.255.255.0
  Known via "ospf 1", distance 110, metric 51, type intra area
  Last update from 10.0.0.6 on FastEthernet0/0, 00:00:02 ago
  Routing Descriptor Blocks:
  * 172.16.0.21, from 172.16.0.22, 00:00:02 ago, via Serial0/0/0.100
      Route metric is 51, traffic share count is 1
    172.16.0.21, from 172.16.0.22, 00:00:02 ago, via Serial0/0/0.200
      Route metric is 51, traffic share count is 1
    10.0.0.6, from 172.16.0.22, 00:00:02 ago, via FastEthernet0/0
      Route metric is 51, traffic share count is 1
a1#show ip cef 192.168.0.0 internal
192.168.0.0/24, version 44, epoch 0, per-destination sharing
0 packets, 0 bytes
  via 172.16.0.21, Serial0/0/0.100, 0 dependencies
    traffic share 1
    next hop 172.16.0.21, Serial0/0/0.100
    valid adjacency
  via 172.16.0.21, Serial0/0/0.200, 0 dependencies
    traffic share 1
    next hop 172.16.0.21, Serial0/0/0.200
    valid adjacency
  via 10.0.0.6, FastEthernet0/0, 0 dependencies
    traffic share 1
    next hop 10.0.0.6, FastEthernet0/0
    valid adjacency

  0 packets, 0 bytes switched through the prefix
  tmstats: external 0 packets, 0 bytes
           internal 0 packets, 0 bytes
  Load distribution: 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 (refcount 1)

  Hash  OK  Interface                 Address         Packets
  1     Y   Serial0/0/0.100           point2point           0
  2     Y   Serial0/0/0.200           point2point           0
  3     Y   FastEthernet0/0           10.0.0.6              0
  4     Y   Serial0/0/0.100           point2point           0
  5     Y   Serial0/0/0.200           point2point           0
  6     Y   FastEthernet0/0           10.0.0.6              0
  7     Y   Serial0/0/0.100           point2point           0
  8     Y   Serial0/0/0.200           point2point           0
  9     Y   FastEthernet0/0           10.0.0.6              0
  10    Y   Serial0/0/0.100           point2point           0
  11    Y   Serial0/0/0.200           point2point           0
  12    Y   FastEthernet0/0           10.0.0.6              0
  13    Y   Serial0/0/0.100           point2point           0
  14    Y   Serial0/0/0.200           point2point           0
  15    Y   FastEthernet0/0           10.0.0.6              0