Next-hop fixup in partially-meshed NBMA networks

Shahid has sent me an interesting observation: he was running RIP in partially-meshed Frame Relay network and wanted to use static host routes to fix next-hop problems, but somehow they didn't work as expected. I ran a series of scenarios testing RIP, OSPF, EIGRP and BGP and found that some of the old tricks don't work any more. Furthermore, it's almost impossible to run RIP over partially-meshed NBMA network, as the default RIP next-hop processing messes up your routing table. The details are explained in the Next-hop fixup in partially-meshed NBMA networks article in the CT³ wiki.

This article is part of You've asked for it series.

Frame Relay mapping to 0.0.0.0

Someone has recently asked me what could cause a dynamic Frame Relay map to address 0.0.0.0. Before I had time to start investigating this weird behavior, Internetwork Expert's CCIE blog published the answer. Great job, Brian :)

You might have to use Firefox to see the blog post, it didn't appear in my IE window.

PPP default route

One of those readers that prefer to remain anonymous has left an interesting comment to my post “Almost-dynamic routing over ADSL interfaces”:

You do not need the route "ip route 0.0.0.0 0.0.0.0 Dialer0 10 track 100" and the tracking if you configure "ppp ipcp route default" on the dialer interface. Works the same way... :-)

You might be wondering why Cisco's engineers decided to pollute IOS with yet another feature. The problem they had was the way PPP over Frame Relay is implemented: it uses virtual interfaces and although you have a very static connection, you cannot bind a static interface name to it. A dynamic interface (with potentially changing name) is cloned from the virtual template every time the PPP-over-Frame-relay session is started. Obviously you cannot configure a static default route pointing to it in advance, so you need yet another feature to do it (I'll not even try to figure out how to create non-default static routes pointing to cloned interface).

X.25 is still kicking

The results of the X.25 poll amazed me. I thought X.25 was long dead, but around 10% of people responding to the survey indicated they are still using it (primarily to connect the legacy equipment) and more than half of the respondents had to configure it in the past (I didn't know we were all going so far back :).Obviously some very big networks are still heavy X.25 users, as Cisco has been implementing more and more X.25-related features (primarily in the XOT space) in recent years … if only they would have been available when I really needed them 15 years ago :(.

By now you should be asking yourself “Well, so what's the latest and greatest X.25 feature?” Brace yourselves: 12.4(15)T has introduced support for X.25 accounting and the Call Detail Records (CDRs) are generated as syslog messages.

The impact of tx-ring-limit

Setting the size of the hardware output queue in Cisco IOS with the (then undocumented) tx-ring-limit (formerly known as tx-limit) has been a big deal when I was developing the first version of the QoS course that eventually became the initial release of the Implementing Cisco Quality of Service training. However, while it's intuitively clear that the longer hardware queue affects the QoS, years passed before I finally took the time to measure the actual impact. The results are available in my wiki.

Almost-dynamic routing over ADSL interfaces

Recently I had to implement Internet access using ADSL as the primary link and ISDN as the backup link. Obviously the most versatile solution would use the techniques described in my IP Corner article Small Site Multi-homing, but the peculiarities of Cisco IOS implementation of the ADSL technology resulted in a much simpler solution.

IOS implementation of PPPoE links uses dialer interfaces. However, the “dialing” on these interfaces is activated as soon as the underlying PPPoE session is active (before the first interesting packet is routed to the interface). When the simulated dial-out occurs, the router starts PPP negotiations including the IPCP handshake, which usually results in an IP address assigned to the dialer interface. Net result: if the dialer interface has an IP address, the PPPoE session is obviously active (and vice versa).

As my ADSL link and the ISDN backup used the same service provider (and very probably the same Radius servers), it made no sense to define additional IP SLA measurements to figure out if the service provider's network is operational; the IP route to the primary dialer interface is installed as soon as the interface is ready to route IP packets. The relevant parts of the router's configuration are included below.

interface FastEthernet0
 description outside LAN
 no ip address
 ip virtual-reassembly
 duplex auto
 speed auto
 pppoe enable group global
 pppoe-client dial-pool-number 3
!
interface BRI0
 description ISDN line
 encapsulation ppp
 dialer pool-member 1
 isdn switch-type basic-net3
!
interface Dialer0
 description ADSL primary uplink
 ip address negotiated
 ip mtu 1492
 ip nat outside
 encapsulation ppp
 dialer pool 3
!
interface Dialer1
 description ISDN backup
 ip address negotiated
 ip nat outside
 dialer pool 1
!
track 100 interface Dialer0 ip routing
 delay down 10 up 10
!
ip route 0.0.0.0 0.0.0.0 Dialer0 10 track 100
ip route 0.0.0.0 0.0.0.0 Dialer1 250

IOS Queuing 101

A while ago I've been a co-mentor to a graduate student who investigated the difference between shaping and policing for his thesis. While doing the lab tests, he stumbled across an interesting problem: he could not configure fancy queuing (for example, CB-WFQ) on a point-to-point Frame Relay subinterface. The solution was to configure a QoS hierarchy, doing shaping first and queuing within the shaping queues.

A few days ago, Arden Packeer covered the same topic in one of his blog posts, so it looks like the underlying problem (and the solution) is not as widely known as I had assumed. Time to write a tutorial on Queuing Principles in Cisco IOS.

The “IP fragmentation” quick learning module

In the IP Corner article The Never-Ending Story of IP Fragmentation, I've described the various challenges posed by IP fragmentation in modern IP networks. If you want a more in-depth look at that topic, check the new IP fragmentation e-lesson. It includes a recorded presentation and a remote lab exercise where you can test various fragmentation scenarios, including MPLS VPN backbone, GRE tunnels and GRE+IPSec combination.

E-lessons are subscription-based; you can repeat each module in the lesson (including the lab) as many times as needed.

WAN IP addresses and subnet masks

Whisper asked an interesting question …

“What I would like to know is, on my PPP negotiated ADSL connection, how the ISP assigns me a /32 ip address.”

… which prompted me to test various WAN encapsulations and address assignment rules. Here are the results:

• On all WAN encapsulations you can configure subnet masks down to /31 (/30 in old IOS releases).
• The same IP address can be used on more than one interface as long as both IP address and subnet mask match.
• Two WAN interfaces can have different IP addresses but still belong to the same IP subnet. You would use this on Frame Relay when you have multiple interfaces into the same FR cloud for bandwidth reasons.
• If you configure IP address with IPCP (with the ip address negotiated command), the subnet mask becomes /32 as IPCP does not carry subnet mask (and you get the host route toward the PPP peer unless you turn off the PPP peer route option)
• If you configure IP address with SLARP (Serial Line ARP) on HDLC, the subnet mask is inherited from the peer (HDLC SLARP carries subnet mask) and the IP address is determined by flipping the low-order bits in the neighbor's IP address.

This article is part of You've asked for it series.

Reduce IP addressing errors in lab environment

One of the most tedious tasks in the initial lab setup (at least for me) is the IP address configuration, which usually includes a number of typos and mixups on the WAN links. You can simplify then WAN address configuration if you configure only one end of the WAN link and let PPP do the rest. For example, you could use the following configuration to configure WAN link on your core router …

hostname Core-2
!
interface Serial1/0
 description link to POP
 ip address 10.0.2.1 255.255.255.252
 encapsulation ppp
 peer default ip address 10.0.2.2

… and use IPCP negotiation on the POP router to pick up the WAN IP address:

hostname POP
!
interface Serial1/0
 description link to Core-1
 ip address negotiated
 encapsulation ppp

You should not configure no peer neighbor-route on the router that gets dynamic IP address, as the subnet mask is not assigned with IPCP; you need the IPCP-generated host routes if you want to do hop-by-hop telnet between the routers.

Running OSPF across Frame Relay

The various models you can use to run OSPF over partially-meshed WAN (usually Frame Relay) are complex enough to make you scream ... or switch to EIGRP :) Arden Packeer did an excellent job in his three-part tutorial describing broadcast and non-broadcast network types, point-to-multipoint network type and point-to-point links over Frame Relay.

If you still have problems with OSPF after reading all three tutorials, use point-to-point subinterfaces.

Remove unwanted PPP peer route

When IOS started supporting dynamic allocation of IPCP (IP over PPP) addresses, it also got the mechanism to insert a dynamic host route toward the neighbor's IP address once the IP addresses were negotiated. This mechanism makes perfect sense in dynamic address allocation environments, as the subnet mask is not negotiated with IPCP. Without a host route toward the other end of the PPP link, there would be no easy way to reach the IP prefixes reachable via the PPP link.However, the PPP code got way too aggressive in the recent IOS releases. For example, in the 12.4T train, you get a connected host route toward the IP address of the PPP peer even on a leased line where both addresses are in the same IP subnet. Here's a sample printout from my lab router that illustrates this behavior:

R1#show run interface serial 1/0
Current configuration : 107 bytes
!
interface Serial1/0
 ip address 10.1.0.1 255.255.255.252
 encapsulation ppp
 serial restart-delay 0
end
 
R1#show ip route 10.1.0.0 255.255.255.252 longer | begin Gateway
Gateway of last resort is not set
 
     10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
C       10.1.0.2/32 is directly connected, Serial1/0
C       10.1.0.0/30 is directly connected, Serial1/0

To disable the automatic insertion of the connected host route, use the no peer neighbor-route interface configuration command.

You have to clear the IP routing table or flap the interface to remove the PPP-generated host route

WAN emulation toolkit

In one of his posts, Joe Harris describes a great Linux-based WAN emulation toolkit that you can use to introduce latency, bandwidth constraints or packet loss in your lab environment.

Layer-2 or Layer-3 VPN services?

Occasionally someone would try to persuade me that the layer-2 VPN services are like aspirin (you know, totally harmless plus it could get rid of all your headaches). OK, that might be true if you take the layer-2 VPN offering as a pure transport solution and plug in an extra router (sometimes also called a layer-3 switch by marketing people) between the Service Provider's Ethernet (or whatever they give you) and your LAN. But there are people who don't know the details and plug the SP Ethernet straight into their L2 switch … and things might even work for a while … until the whole network collapses.

In my opinion, we need both L2 and L3 VPN services, but it's important that they are positioned and deployed correctly. You can read more about my views on this topic in the SearchTelecom article Making the case for Layer 2 and Layer 3 VPNs.

Configure DNS servers through IPCP

After I've fixed the default routing in my home office, I've stumbled across another problem: the two ISPs I'm using for my primary and backup link have DNS servers that reply solely to the DNS requests sent from their own IP address range:

When the traffic is switched from the primary to the backup ISP, I therefore also need to switch the DNS servers. Fortunately, this is quite easy to do on a router; you just need to configure ppp ipcp dns request on the dialer interface and the router starts asking for the DNS server address as part of the IPCP negotiation.
The negotiation process can be debugged with the debug ppp negotiation command; it's a bit more complex than usual in my case since the access server has no secondary DNS (only the primary DNS is configured):

Se1/0 IPCP: O CONFREQ [Closed] id 1 len 22
Se1/0 IPCP:    Address 0.0.0.0 (0x030600000000)
Se1/0 IPCP:    PrimaryDNS 0.0.0.0 (0x810600000000)
Se1/0 IPCP:    SecondaryDNS 0.0.0.0 (0x830600000000)
Se1/0 PPP: Process pending ncp packets
Se1/0 IPCP: Redirect packet to Se1/0
Se1/0 IPCP: I CONFREQ [REQsent] id 1 len 10
Se1/0 IPCP:    Address 10.0.0.33 (0x03060A000021)
Se1/0 IPCP: O CONFACK [REQsent] id 1 len 10
Se1/0 IPCP:    Address 10.0.0.33 (0x03060A000021)
Se1/0 IPCP: I CONFREJ [ACKsent] id 1 len 10
Se1/0 IPCP:    SecondaryDNS 0.0.0.0 (0x830600000000)
Se1/0 IPCP: O CONFREQ [ACKsent] id 2 len 16
Se1/0 IPCP:    Address 0.0.0.0 (0x030600000000)
Se1/0 IPCP:    PrimaryDNS 0.0.0.0 (0x810600000000)
Se1/0 IPCP: I CONFNAK [ACKsent] id 2 len 16
Se1/0 IPCP:    Address 10.0.0.34 (0x03060A000022)
Se1/0 IPCP:    PrimaryDNS 10.0.0.10 (0x81060A00000A)
Se1/0 IPCP: O CONFREQ [ACKsent] id 3 len 16
Se1/0 IPCP:    Address 10.0.0.34 (0x03060A000022)
Se1/0 IPCP:    PrimaryDNS 10.0.0.10 (0x81060A00000A)
Se1/0 IPCP: I CONFACK [ACKsent] id 3 len 16
Se1/0 IPCP:    Address 10.0.0.34 (0x03060A000022)
Se1/0 IPCP:    PrimaryDNS 10.0.0.10 (0x81060A00000A)
Se1/0 IPCP: State is Open

The results can be inspected only with the show host command:

GW#show host
Default domain is not set
Name/address lookup uses domain service
Name servers are 10.0.0.10

The access server receiving the call requires no special configuration; the first IP address configured with the ip name-server command is used as the primary DNS and the second one as the secondary. Alternatively, you can configure a different set of DNS servers to pass to the client with the ppp ipcp dns primary-DNS-address secondary-DNS-address interface configuration command.

Unfortunately, the integration with LAN clients is not as seamless as with DHCP; to make the whole solution work, you have to configure the router as a forwarding DNS server and make the LAN clients use the router as the default gateway and DNS server with the DHCP pool configuration:

ip dns server
!
ip dhcp pool LAN
   import all
   network 192.168.0.0 255.255.255.240
   default-router 192.168.0.1
   dns-server 192.168.0.1

Install default route with PPP

In my home office, I'm using DSL access to the Internet with ISDN backup to another ISP, as shown on the next figure:

Obviously, I would like the ISDN backup to kick in whenever the primary connection goes down; two static default routes and reliable static routing on the primary default seem like a perfect solution. However, as I'm using PPP encapsulation on the primary connection, there's another option: PPP can insert a dynamic default route whenever IPCP negotiations succeed (and remove it when the line protocol goes down). To configure this feature (introduced in IOS releases 12.3(11)T and 12.4), use the ppp ipcp route default interface configuration command on the primary dialer interface.Contrary to the DHCP-installed default route, the PPP-installed default route has administrative distance 1 (and is thus impossible to override):

GW#show ip route | begin Gateway
Gateway of last resort is 10.0.0.33 to network 0.0.0.0
 
     10.0.0.0/32 is subnetted, 2 subnets
C 10.0.0.34 is directly connected, Serial1/0
C 10.0.0.33 is directly connected, Serial1/0
     192.168.0.0/28 is subnetted, 1 subnets
C 192.168.0.0 is directly connected, FastEthernet0/0
S* 0.0.0.0/0 [1/0] via 10.0.0.33

Emulate dialup links with serial lines

I had to figure out various PPP parameters (and associated Cisco IOS behavior) and didn't have real dial-up equipment in my lab setup. I could have gone with PPPoE, but it turned out it's way simpler to emulate dialup connections (at least the PPP negotiations work as expected) on fixed serial lines. This is the minimum setup you need on the “caller” side …

interface Serial1/0
 ip address negotiated
 encapsulation ppp
 ppp authentication pap optional
 ppp pap sent-username client password 0 client

… and this is the “server”-side configuration:

interface Serial1/0
 ip address 10.0.0.33 255.255.255.252
 encapsulation ppp
 peer default ip address 10.0.0.34
 ppp authentication pap callin
!
username client password client

To trigger PPP negotiations, shut down and re-enable the serial interface on either side.

Note: As I'm using PAP authentication, I could use the more secure username secret configuration command, which would not work with CHAP.

Ones are slower than zeroes

Thinking about the implications of bit stuffing I wrote about in the SDLC post, I realized that long sequences of ones would be transmitted slower than long sequences of zeroes due to an extra bit being inserted after every fifth consecutive one. The theory would predict a 20% decrease in transmission speed.

Of course I wanted to test this phenomenon immediately. I needed real equipment with low-speed serial links (that would make the difference more pronounced and less dependent on other intra-router delays), so I started one of the BGP lab exercises; Basic BGP Setup looked like a perfect choice. We're using 64 kbps Frame Relay links in the lab with a Frame Relay switch in the middle (makes the task of designing an arbitrary WAN topology quite simple), so it's a perfect environment to test this thing. And, not surprisingly, the results confirmed the theory:

Internal-Core#ping 197.1.1.49 data 0000 size 1200 repeat 50
Sending 50, 1200-byte ICMP Echos to 197.1.1.49, timeout is 2 seconds:
Packet has data pattern 0x0000
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (50/50), round-trip min/avg/max = 608/608/632 ms
 
Internal-Core#ping 197.1.1.49 data FFFF size 1200 repeat 50
Sending 50, 1200-byte ICMP Echos to 197.1.1.49, timeout is 2 seconds:
Packet has data pattern 0xFFFF
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (50/50), round-trip min/avg/max = 724/724/728 ms

The results are almost too close to the predicted ones, but they are real :)

Back to the roots: it all started with SDLC

My recent post about problems with old modems has generated a lot of comments with some very useful ideas, but nobody addressed the question “why was a long string of ones not a problem?”, so let's start there. Almost all WAN synchronous protocols in use today are descendants of venerable SDLC invented by IBM more than 30 years ago. SDLC was later extended to support connectionless and balanced modes, resulting in HDLC. PPP is just an extension of HDLC, adding support for negotiations and standard layer-3 protocol demultiplexing. In SDLC, IBM also solved the frame delimiting and associated escape character problem inherent in previous protocols like BSC (DLE was used in BSC) by introducing bit stuffing: a zero would be inserted after five consecutive ones (and silently removed by the receiver) to differentiate the regular data stream from framing (six consecutive ones) and abort (more than six consecutive ones) sequences. Thus, the HDLC (or PPP) data stream can never contain more than six consecutive ones and the long sequences of ones never cause synchronization loss.

IBM obviously also had problems with bad modems and solved it with the NRZI encoding that was part of SDLC standard (and a major pain in the good old days when the appliques on the old Cisco routers did not support it and we've been trying hard to penetrate IBM accounts). You can still configure NRZI encoding on most routers' serial links (it might depend on the actual hardware platform) with the nrzi-encoding interface configuration command (you had to do it with jumpers in the AGS+). Incidentally, changing interface encoding to NRZI was really helpful when you had to break things in the preparation for the troubleshooting part of the original CCIE lab).

Enough theory, let's summarize the proposed solutions:

• The nrzi-encoding (if available) is the best one, as it reliably solves the problem, is transparent and does not incur additional overhead.
• Compression or encryption are OK, but they result in significant CPU overhead (unless you have hardware encryption/compression modules) and might (at least in theory) still produce a long sequence of zeroes, although with a very low probability. IPSec also introduces overhead due to additional IPSec headers.
• LFI (effectively multilink PPP over a single link) is also a good solution, as the PPP framing and MLPPP headers break the long sequences of zeroes (you might have to fine-tune the fragment size with ppp multilink fragment size configuration command), but it introduces overhead on the WAN link.
• IP fragmentation would work, but would be quite bandwidth-consuming. If the fragmentation would be performed by the router, the overhead would be 20 bytes per fragment (IP header), if the sending host performs the fragmentation, the overhead is 40 bytes per fragment for TCP sessions. For example, if we reduce the IP MTU size to 256 bytes, the TCP session overhead is over 18% (and we were scoffing at the ATM designers that made us live with 10% overhead).

There were also a few suggestions that would not work very well:

• The invert data command would only help if the modem has problems with long strings of zeroes, not with long strings of the same value.
• The tunnel key command just sets a 4-byte field in the GRE header but does not affect the encapsulated data at all.

React to excessive jitter with EEM

William Chu sent me a working configuration he uses to measure jitter with the IP SLA tool and react to excessive jitter on the primary link. First you have to create the jitter probe with the IP SLA commands:

ip sla monitor 3000
 type jitter →
   dest-ipaddr 199.11.18.168 dest-port 12333 →
   source-ipaddr 199.11.18.169 codec g729a →
   codec-numpackets 100
 tos 184
 frequency 10

Note: The continuation character (→) indicates that the configuration command spans multiple lines

Next you have to define the IP SLA reaction to excessive jitter. William configured his router to react when the jitter exceeds 300 milliseconds and returns back to normal when the jitter falls below 290 milliseconds (some hysteresis is always a good thing).

ip sla monitor reaction-configuration 3000 →
  react MOS threshold-value 300 290 →
  threshold-type consecutive →
  action-type trapOnly

As the last step in the SLA configuration, you have to start the probe:

ip sla monitor schedule 3000 →
  life forever start-time now

After the SLA probe and out-of-bounds reaction have been configured, the router will generate syslog messages whenever the jitter gets above the threshold as well as when it falls below the second threshold. You can then use the EEM applets to act on the syslog messages:

event manager applet MOS-Below
 event syslog occurs 1 period 120 →
   pattern "Threshold below for MOS"
 ... actions ...
!
event manager applet MOS-Above
 event syslog occurs 1 period 120 →
   pattern "Threshold exceeded for MOS"
 ... actions ...