If you want to study for your CCSP certification and have partner-level access to Cisco's web, you can schedule Securing Networks with Cisco Routers and Switches remote labs free-of-charge straight from Partner E-learning connection by clicking this link (partner-level CCO username required).
If you're not a Cisco partner, you can buy the same labs from our web site.
We all know that BGP is meant to converge slowly ... well, the MPLS/VPN service providers tend to disagree, as their users are not used to minute-long convergence times. One of the major components of slow BGP convergence is the time it takes a router to discover that a neighbor has disappeared. Traditionally, the BGP keepalive packets were sent every minute and it took up to three minutes to discover that a neighbor is down. Of course you could fine-tune those times with the neighbor timers configuration command, but the reduced timers resulted in increased TCP traffic and consequently increased CPU load, which could reach tens of percents if the timers were set to a few seconds and the router had lots of BGP neighbors.
The neighbor loss detection has improved dramatically in 12.3T and 12.0S with the introduction of the fast session deactivation, where a BGP session is dropped as soon as the route to the BGP neighbor is lost. You can configure this feature with the ominous-sounding neighbor fall-over configuration command. Obviously, this feature does not work well if you use default routing (or summaries), since the path to the BGP neighbor is never completely lost. In that case, you can use a route-map option of the neighbor fall-over command (introduced in 12.4(4)T) to select which less specific route is still a valid route to the BGP neighbor.Here are the logging and debugging printouts from a router that lost a BGP neighbor and discovered it after the BGP hold time has expired:
00:00:48: %BGP-5-ADJCHANGE: neighbor 10.0.3.3 UpAs you can see, there is more than a two minute gap between the time the OSPF route to the BGP neighbor was lost and the time the BGP session went down (and the BGP routes were recalculated). When the neighbor 10.0.3.3 fall-over is configured, the BGP session is disconnected as soon as the OSPF route to the neighbor is gone:
00:01:23: RT: del 10.0.3.3/32 via 10.0.1.2, ospf metric [110/129]
00:01:23: RT: delete subnet route to 10.0.3.3/32
00:03:49: %BGP-3-NOTIFICATION: received from neighbor 10.0.3.3 4/0 (hold time expired) 0 bytes
00:03:49: %BGP-5-ADJCHANGE: neighbor 10.0.3.3 Down BGP Notification received
00:08:12: RT: del 10.0.3.3/32 via 10.2.0.2, ospf metric [110/75]
00:08:12: RT: delete subnet route to 10.0.3.3/32
00:08:12: RT: NET-RED 10.0.3.3/32
00:08:12: RT: Try lookup less specific 10.0.3.3/32, default 1
00:08:12: RT: Failed found subnet on less specific
00:08:12: RT: return NULL
00:08:12: %BGP-5-ADJCHANGE: neighbor 10.0.3.3 Down Route to peer lost
I will cover the route-map option as well as the design implications of this feature in an upcoming IP corner article.
After I've fixed the default routing in my home office, I've stumbled across another problem: the two ISPs I'm using for my primary and backup link have DNS servers that reply solely to the DNS requests sent from their own IP address range:
When the traffic is switched from the primary to the backup ISP, I therefore also need to switch the DNS servers. Fortunately, this is quite easy to do on a router; you just need to configure ppp ipcp dns request on the dialer interface and the router starts asking for the DNS server address as part of the IPCP negotiation.
The negotiation process can be debugged with the debug ppp negotiation command; it's a bit more complex than usual in my case since the access server has no secondary DNS (only the primary DNS is configured):
Se1/0 IPCP: O CONFREQ [Closed] id 1 len 22The results can be inspected only with the show host command:
Se1/0 IPCP: Address 0.0.0.0 (0x030600000000)
Se1/0 IPCP: PrimaryDNS 0.0.0.0 (0x810600000000)
Se1/0 IPCP: SecondaryDNS 0.0.0.0 (0x830600000000)
Se1/0 PPP: Process pending ncp packets
Se1/0 IPCP: Redirect packet to Se1/0
Se1/0 IPCP: I CONFREQ [REQsent] id 1 len 10
Se1/0 IPCP: Address 10.0.0.33 (0x03060A000021)
Se1/0 IPCP: O CONFACK [REQsent] id 1 len 10
Se1/0 IPCP: Address 10.0.0.33 (0x03060A000021)
Se1/0 IPCP: I CONFREJ [ACKsent] id 1 len 10
Se1/0 IPCP: SecondaryDNS 0.0.0.0 (0x830600000000)
Se1/0 IPCP: O CONFREQ [ACKsent] id 2 len 16
Se1/0 IPCP: Address 0.0.0.0 (0x030600000000)
Se1/0 IPCP: PrimaryDNS 0.0.0.0 (0x810600000000)
Se1/0 IPCP: I CONFNAK [ACKsent] id 2 len 16
Se1/0 IPCP: Address 10.0.0.34 (0x03060A000022)
Se1/0 IPCP: PrimaryDNS 10.0.0.10 (0x81060A00000A)
Se1/0 IPCP: O CONFREQ [ACKsent] id 3 len 16
Se1/0 IPCP: Address 10.0.0.34 (0x03060A000022)
Se1/0 IPCP: PrimaryDNS 10.0.0.10 (0x81060A00000A)
Se1/0 IPCP: I CONFACK [ACKsent] id 3 len 16
Se1/0 IPCP: Address 10.0.0.34 (0x03060A000022)
Se1/0 IPCP: PrimaryDNS 10.0.0.10 (0x81060A00000A)
Se1/0 IPCP: State is Open
GW#show hostThe access server receiving the call requires no special configuration; the first IP address configured with the ip name-server command is used as the primary DNS and the second one as the secondary. Alternatively, you can configure a different set of DNS servers to pass to the client with the ppp ipcp dns primary-DNS-address secondary-DNS-address interface configuration command.
Default domain is not set
Name/address lookup uses domain service
Name servers are 10.0.0.10
Unfortunately, the integration with LAN clients is not as seamless as with DHCP; to make the whole solution work, you have to configure the router as a forwarding DNS server and make the LAN clients use the router as the default gateway and DNS server with the DHCP pool configuration:
ip dns server
!
ip dhcp pool LAN
import all
network 192.168.0.0 255.255.255.240
default-router 192.168.0.1
dns-server 192.168.0.1
It's amazing how many options (most of them still undocumented) the show interfaces command accepts in IOS release 12.4T (I won't even start guessing when each one was introduced, if you're running old IOS releases, please feel free to comment):
- show interfaces description displays interface names, L1 and L2 status (line and line-protocol status) and interface description. Extremely handy if you want to check which interfaces are up/down.
- show interfaces counters protocol status displays the L3 protocols active on each interface.
- show interfaces summary displays the state of various interface queues and related drop counters in a nice tabular format.
- show interfaces accounting displays per-protocol in/out counters.
a1#show interfaces description
Interface Status Protocol Description
Fa0/0 up up Central LAN
Fa0/1 admin down down
Se0/0/0 up up Frame Relay
Se0/0/0.100 up up Link to B1
Se0/1/0 admin down down
Se0/1/1 admin down down
Lo0 up up
a1#show interfaces counters protocol status
Protocols allocated:
FastEthernet0/0: Other, IP, DEC MOP, ARP, CDP
FastEthernet0/1: Other, IP
Serial0/0/0: Other, IP, CDP
Serial0/1/0: Other, IP
Serial0/1/1: Other, IP
Loopback0: Other, IP
VoIP-Null0: Other, IP
a1#show interfaces summary
*: interface is up
IHQ: pkts in input hold queue IQD: pkts dropped from input queue
OHQ: pkts in output hold queue OQD: pkts dropped from output queue
RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)
TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)
TRTL: throttle count
Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL
------------------------------------------------------------------
* FastEthernet0/0 0 0 0 0 0 0 0 0 0
FastEthernet0/1 0 0 0 0 0 0 0 0 0
* Serial0/0/0 0 0 0 0 0 0 0 0 0
* Serial0/0/0.100 - - - - - - - - -
Serial0/1/0 0 0 0 0 0 0 0 0 0
Serial0/1/1 0 0 0 0 0 0 0 0 0
* Loopback0 0 0 0 0 0 0 0 0 0
NOTE:No separate counters are maintained for subinterfaces
Hence Details of subinterface are not shown
a1#show interfaces accounting
FastEthernet0/0 Central LAN
Protocol Pkts In Chars In Pkts Out Chars Out
Other 0 0 490 29400
IP 2737 216847 3052 424422
DEC MOP 0 0 8 616
ARP 5 316 12 720
CDP 82 30914 84 29563
Interface FastEthernet0/1 is disabled
Serial0/0/0 Frame Relay
Protocol Pkts In Chars In Pkts Out Chars Out
Other 0 0 490 6370
IP 515 43748 1034 87608
CDP 83 26477 168 55272
When I've been describing the limitations of kron, ??? quickly asked an interesting question: “as I cannot insert extra input keystrokes with EEM applet, can I run a Tcl script from it with the action sequence cli command "tclsh script" command and use the typeahead function call to get around the limitation?” The only answer I could give at that time was “maybe” … and obviously it was time for a more thorough test. The short result is: YES, you can do it (at least in IOS release 12.4(15)T1).
… and here is the long description of the test. I've started by creating a small Tcl script (see below) that clears the counters on Loopback 0. As the clear counters command requires keyboard input and generates a syslog message, it was a perfect test case.
typeahead "y"
exec "clear counter loop 0"
I've copied this script into the flash:tcl/clearL0.tcl and tested it:
R1#tclsh flash:tcl/clearL0.tcl
%CLEAR-5-COUNTERS: Clear counter on interface Loopback0 by console
So far, so good. Next, I've created an EEM applet with no trigger …
event manager applet Clear
event none
action 1.0 cli command "enable"
action 1.1 cli command "tclsh flash:tcl/clearL0.tcl"
… enabled the EEM CLI debugging and started it:
R1#debug event man action cli
Debug EEM action cli debugging is on
R1#event man run Clear
R1#
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : CTL : cli_open called.
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT : R1>
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : IN : R1>enable
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT : R1#
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : IN : R1#tclsh flash:tcl/clearL0.tcl
%CLEAR-5-COUNTERS: Clear counter on interface Loopback0 by on vty0 (EEM:Clear)
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT : R1#
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : CTL : cli_close called.
Great. It works. Let's move on: I've inserted a trigger into the EEM applet that ran the applet when an OSPF neighbor reached FULL adjacency:
event manager applet Clear
event syslog pattern "OSPF-5-ADJCHG.*to FULL"
action 1.0 cli command "enable"
action 1.1 cli command "tclsh flash:tcl/clearL0.tcl"
And now for the final test: after I've enabled the serial interface, OSPF neighbors established adjacency …
R1(config-if)#interface ser 1/0
R1(config-if)#no shutdown
R1(config-if)#
%LINK-3-UPDOWN: Interface Serial1/0, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0, changed state to up
%OSPF-5-ADJCHG: Process 1, Nbr 10.0.1.2 on Serial1/0 from LOADING to FULL, Loading Done
%CLEAR-5-COUNTERS: Clear counter on interface Loopback0 by on vty0 (EEM:Clear)
… and the counters on Loopback0 were cleared. Test completed :)
You can find more Tclsh-related information in the Tclsh on Cisco IOS tutorial. Sample Tclsh scripts are available in the Tclsh script library. If you need expert help in planning, developing or deploying Tclsh scripts in your network, contact the author.
This article is part of You've asked for it series.
I've written a lot about MPLS Traffic Engineering (not nearly as much as I would like, but there are always time constraints), as I believe this technology has interesting applications in Enterprise networks (and we all know that a lot of Service Providers are using it anyway). You might have seen my 10 MPLS Traffic Engineering Myths or the Perfect Load Balancing article … and if you don't know what I'm talking about, there's always the introductory Traffic Engineering the Service Provider Network.
The major problem of MPLS TE is that it's complex and that networking engineers usually lack the hands-on skills, and this is where we can help you: we've just rolled out the revised MPLS TE lab exercises. Compared to remote lab offerings from other sources, these lab exercises are very focused: you get step-by-step instructions (but no recipes, that would spoil the learning process), preconfigured equipment (so you don't have to configure IP addresses or IP routing protocols to get the job done) and detailed solutions explaining which task is achieved using a specific set of configuration commands.
I was able to get a discount for my readers: if you click this link and type in the promotion code 42B078 (expires on January 15th, 2008), you'll get a one week subscription to the MPLS TE remote lab bundle for €56. As this is a subscription offering, you can run the lab exercises as often as you like within a week of the purchasing date. And if you need one more argument to be persuaded, check the lab topology; you can experiment in a preconfigured nine router network :)
Unfortunately, I've started discovering the downsides of blogging: someone is actively stealing my content and publishing it on a number of blogs. I don't mind people quoting me (even larger passages of my text) as long as the proper credits appear somewhere in the post, but straight copy from my blog feed is too much.
Obviously, the easy stopgap measure would be to change the feed format from full to short, which would only pass the first few lines of every blog post into the feed, resulting in a bit more effort on your end, as you'd have to click on the post link in the feed reader to view the whole text. Would this be a major nuisance for anyone?
When two groups within Cisco needed time-based command execution in Cisco IOS, they (in a typical big-corporation fashion) decided to implement the same wheel from two different sets of spokes and rims. One group built the Embedded Event Manager with its event timer cron command (introduced in 12.2(25)S and 12.3(14)T), the other group created the more limited kron command set (introduced in 12.3(1)).
EEM is almost a perfect superset of kron, both can trigger a set of CLI commands at reload, at periodic intervals or at certain time in the future. The only extra functionality offered by kron is the ability to specify a different username for each event (whereas all EEM applets have to run under a common username) … and kron is available in older IOS releases.
Similar to EEM applets, CLI commands executed within kron cannot expect extra input (so you cannot execute clear counters or reload from kron) and the output they generate is lost unless you use output filters to redirect it to an external file.
Here is a simple configuration that archives the router's running configuration every sunday half an hour before midnight:
kron policy-list archiveConfig
cli archive config
!
kron occurrence archiveConfig at 23:30 Sun recurring
policy-list archiveConfig
In my home office, I'm using DSL access to the Internet with ISDN backup to another ISP, as shown on the next figure:
Obviously, I would like the ISDN backup to kick in whenever the primary connection goes down; two static default routes and reliable static routing on the primary default seem like a perfect solution. However, as I'm using PPP encapsulation on the primary connection, there's another option: PPP can insert a dynamic default route whenever IPCP negotiations succeed (and remove it when the line protocol goes down). To configure this feature (introduced in IOS releases 12.3(11)T and 12.4), use the ppp ipcp route default interface configuration command on the primary dialer interface.Contrary to the DHCP-installed default route, the PPP-installed default route has administrative distance 1 (and is thus impossible to override):
GW#show ip route | begin Gateway
Gateway of last resort is 10.0.0.33 to network 0.0.0.0
10.0.0.0/32 is subnetted, 2 subnets
C 10.0.0.34 is directly connected, Serial1/0
C 10.0.0.33 is directly connected, Serial1/0
192.168.0.0/28 is subnetted, 1 subnets
C 192.168.0.0 is directly connected, FastEthernet0/0
S* 0.0.0.0/0 [1/0] via 10.0.0.33
I've stumbled across a blog post that indicates there's still confusion on some fundamental configuration issues. I will not even try to guess whether there is a wide consensus on how to configure a router, but these are the facts (and here is a ten year old position from Cisco):
- Type-7 encryption used in enable password has been broken. Source code for the decrypt program and cracker programs are available online, or you could use a router to do it for you.
- The type-7 encryption is reversible (and easily breakable due to a weak algorithm), whereas type-5 encryption is a one-way encryption that probably requires a dictionary attack to break.
- Based on the previous two facts, you should never use enable password. Use enable secret.
- The service password-encryption encodes passwords attached to local usernames with type-7 encryption. The usage of type-7 encryption is necessary as you might need the cleartext passwords in some authentication mechanisms (for example, CHAP). However, it's still better to have scrambled passwords than cleartext ones; at least a casual observer will not be able to read them. Conclusion: use service password-encryption.
- If your authentication methods don't need cleartext passwords (examples: local username/password authentication, local AAA authentication or PAP authentication), use username secret configuration command (available from IOS releases 12.2T, 12.3 and 12.0S).
I had to figure out various PPP parameters (and associated Cisco IOS behavior) and didn't have real dial-up equipment in my lab setup. I could have gone with PPPoE, but it turned out it's way simpler to emulate dialup connections (at least the PPP negotiations work as expected) on fixed serial lines. This is the minimum setup you need on the “caller” side …
interface Serial1/0… and this is the “server”-side configuration:
ip address negotiated
encapsulation ppp
ppp authentication pap optional
ppp pap sent-username client password 0 client
interface Serial1/0To trigger PPP negotiations, shut down and re-enable the serial interface on either side.
ip address 10.0.0.33 255.255.255.252
encapsulation ppp
peer default ip address 10.0.0.34
ppp authentication pap callin
!
username client password client
Note: As I'm using PAP authentication, I could use the more secure username secret configuration command, which would not work with CHAP.
When I was testing the SMTP client on the routers (configured with action … mail command or as part of EEM SMTP library), I didn't want the messages with weird addresses and content to circulate through our e-mail system (plus I wanted to see the results immediately), so I wrote a simple SMTP server in Perl that prints the messages it receives. It accepts a single parameter: the IP address to listen on (only needed on workstations with multiple IP addresses).
You need the Net::SMTP::Server module on top of the standard Perl distribution to make it work.
You can download the PERL source code or compiled EXE file (for Windows users) from CT3 wiki.
A while ago I was asked to write an article about IPv6 training. I could just cover the training aspect, like what's offered (answer: not much) and whether someone can train the whole operations team like you could in the IPv4 or MPLS/VPN world (answer: no), but I wanted to understand whether anyone is really using IPv6 in a production network. I found a few academic networks (after all, there are about 2000 IPv6 prefixes assigned and someone should be doing something with them), but not much of what I would call a real production environment, which is a bad thing, as it looks like the IPv4 address space will get saturated in a few years.
Update 2010-03-12: Numerous commercial ISPs now offer native IPv6 connectivity, but they also face significant deployment challenges. You will find an overview of those in my Market trends in Service Provider networks workshop (register for the online webinar). Advanced backbone designs and configurations are explained in the Building IPv6 Service Provider core workshop (register for the online webinar).
Update 2011-12-07: You might also want to read the Responsible generation of BGP default route article describing the ISP side of the solution.
Update 2008-08-10: IOS behavior has changed; fixed the article.
Martin Kluge sent me an interesting BGP question: he has two upstream links and runs BGP on both. Since his router is low on RAM, he cannot accept full routing, so he's just announcing his IP prefix and using static default routing toward upstream ISPs.
The relevant configuration on the GW router is somewhat similar to the configuration I've used as a staring point in my lab:
Tim Riegert sent me an interesting hint: you don't need password crackers to decode type-7 passwords, you just need access to a router. Here's how you do it:
We'll turn on type-7 encryption for local passwords and generate a test username
R1(config)#service password-encryption
R1(config)#username test password t35t:pa55w0rd
Next we'll inspect the generated username with the show running command
R1(config)#do show run | include username
username test password 7 08351F1B1D431516475E1B54382F
Now we'll create a key chain and enter the type-7 encrypted password as the key string …
R1(config)#key chain decrypt
R1(config-keychain)#key 1
R1(config-keychain-key)#key-string 7 08351F1B1D431516475E1B54382F
… and the show command does the decryption for us.
R1(config-keychain-key)#do show key chain decrypt
Key-chain decrypt:
key 1 -- text "t35t:pa55w0rd"
accept lifetime (always valid) - (always valid) [valid now]
send lifetime (always valid) - (always valid) [valid now]
You can use the show process cpu sorted command in combination with an output filter to display only those IOS processes that consumed noticeable amount of CPU time in the last five minutes, last minute or last five seconds. Use the following patterns to construct your regular expression:
- The [0-9.]+% pattern will match any non-zero percentage;
- The 0.00% pattern will obviously match the zero-percentage display;
- As the percentage figures are separated by various amounts of whitespace characters, we have to use the ' +' pattern to match those;
alias exec cpu1min show process cpu sorted 1min | exclude [0-9.]+% +0.00%A sample printout from one of my routers is included:
+[0-9.]+%
alias exec cpu5sec show process cpu sorted 5sec | exclude 0.00% +[0-9.]+% +[0-9.]+%
alias exec cpu5min show process cpu sorted 5min | exclude [0-9.]+% +[0-9.]+% +0.00%
rtr#cpu1min
CPU utilization for five seconds: 4%/0%; one minute: 2%; five minutes: 2%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5 27260472 1470452 18538 0.00% 1.74% 1.78% 0 Check heaps
62 536 226 2371 3.27% 0.52% 0.15% 2 Virtual Exec
30 248000 230369 1076 0.16% 0.07% 0.02% 0 IP Input
25 617780 25736 24004 0.00% 0.03% 0.00% 0 Per-minute
43 32 485 65 0.00% 0.01% 0.00% 0 TCP Timer
Someone has asked me a while ago whether it's possible to retain variable values between invocations of an EEM policy. Since a new copy of Tcl interpreter is started for each event, global variables obviously won't work; they are lost as soon as the Tcl policy is finished. A potential solution is to modify the router's configuration and save the values you wish to preserve in event manager environment, but that's a time-consuming process that interferes with whatever router configuration management process you have.
The real solution is based on the appl_setinfo and appl_reqinfo calls. They work, but like many other Tcl-related IOS features they are … well … weird.This time, the programmers managed to implement WORO (Write-Once-Read-Once) memory:
- The value you want to preserve is saved with appl_setinfo key name data value function call. Keys must be unique; you can only set the same key once. If you try to set the value of a key multiple times, the function does not overwrite the previous value but fails.
- You can read the value with appl_reqinfo key name function call. If the key value hasn't been set, it returns an empty string and sets the $_cerrno variable, otherwise it returns a list with 'data' as the first element and your value as the second list element (I have to admit I've seen simpler APIs :).
- Once you read the key value, it's gone. You cannot read it twice.
::cisco::eem::event_register_cli sync no skip no pattern "show"
namespace import ::cisco::eem::*
namespace import ::cisco::lib::*
Set the variable value to zero (in case we haven't saved the value before) and read the previous value
set lastCnt 0
set getLastCnt [ appl_reqinfo key "showCounter" ]
If the first element in the list is 'data', then the second element is our value.
if { [ lindex $getLastCnt 0 ] == "data" } {
set lastCnt [ lindex $getLastCnt 1 ]
}Increase the counter and generate a syslog message
incr lastCnt
action_syslog priority info msg "Show command was executed $lastCnt times"
Save the new value of the counter to be retrieved by the next invocation of the same policy.
appl_setinfo key "showCounter" data $lastCnt
This article is part of You've asked for it series.
Thinking about the implications of bit stuffing I wrote about in the SDLC post, I realized that long sequences of ones would be transmitted slower than long sequences of zeroes due to an extra bit being inserted after every fifth consecutive one. The theory would predict a 20% decrease in transmission speed.
Of course I wanted to test this phenomenon immediately. I needed real equipment with low-speed serial links (that would make the difference more pronounced and less dependent on other intra-router delays), so I started one of the BGP lab exercises; Basic BGP Setup looked like a perfect choice. We're using 64 kbps Frame Relay links in the lab with a Frame Relay switch in the middle (makes the task of designing an arbitrary WAN topology quite simple), so it's a perfect environment to test this thing. And, not surprisingly, the results confirmed the theory:
Internal-Core#ping 197.1.1.49 data 0000 size 1200 repeat 50
Sending 50, 1200-byte ICMP Echos to 197.1.1.49, timeout is 2 seconds:
Packet has data pattern 0x0000
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (50/50), round-trip min/avg/max = 608/608/632 ms
Internal-Core#ping 197.1.1.49 data FFFF size 1200 repeat 50
Sending 50, 1200-byte ICMP Echos to 197.1.1.49, timeout is 2 seconds:
Packet has data pattern 0xFFFF
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (50/50), round-trip min/avg/max = 724/724/728 ms
The results are almost too close to the predicted ones, but they are real :)
A few days ago I was describing the impact of CEF accounting on a router (based on its architecture). The picture is clearer with Netflow: Cisco has published a white paper detailing the impact of various types of Netflow accounting on a large variety of platforms, from an 1800 ISR to the GSR (12000).
The link to this white paper has been published in Joe Harris' blog.
A while ago, people believed you cannot do load balancing with BGP (they also believed the Earth was flat a few years before that). While that's no longer true, designing good BGP load balancing is still a complex undertaking. In the November IP Corner article, Load Balancing in BGP Networks I'm describing almost all options you have to implement BGP-based load balancing, both within your autonomous system as well as across an AS boundary.
Speaking of Joe Harris … he just wrote a great post about the IP routing table profiling available in Cisco IOS and the ways it can be used to monitor the stability of your network.
Joe Harris published an excellent post detailing how you can use Flexible Packet Matching to recognize (and potentially block) Skype traffic. The solution depends on recognizing the first four bytes sent by the Skype application in a TCP session. While this is a great idea, you have to be aware that there's always a non-zero chance of false positives, more so as the described filter is testing the beginning of the payload in every TCP packet (not just the first data packet in the session).
Jozef Janitor wrote a highly relevant comment to my post on CEF accounting: enabling it on a Catalyst switch drastically reduces its performance. The impact of CEF accounting (or other forwarding plane features) depends on switching implementation:
- There is almost no impact on single-CPU software platforms; the router has to perform CEF lookup anyway and increases the CEF accounting counters on-the-fly;
- Distributed software platforms are more complex, as the central CPU has to (at the very least) collect the switching statistics.
- The impact on hardware platforms is dependent on the layer 3 lookup implementation
My recent post about problems with old modems has generated a lot of comments with some very useful ideas, but nobody addressed the question “why was a long string of ones not a problem?”, so let's start there. Almost all WAN synchronous protocols in use today are descendants of venerable SDLC invented by IBM more than 30 years ago. SDLC was later extended to support connectionless and balanced modes, resulting in HDLC. PPP is just an extension of HDLC, adding support for negotiations and standard layer-3 protocol demultiplexing. In SDLC, IBM also solved the frame delimiting and associated escape character problem inherent in previous protocols like BSC (DLE was used in BSC) by introducing bit stuffing: a zero would be inserted after five consecutive ones (and silently removed by the receiver) to differentiate the regular data stream from framing (six consecutive ones) and abort (more than six consecutive ones) sequences. Thus, the HDLC (or PPP) data stream can never contain more than six consecutive ones and the long sequences of ones never cause synchronization loss.
IBM obviously also had problems with bad modems and solved it with the NRZI encoding that was part of SDLC standard (and a major pain in the good old days when the appliques on the old Cisco routers did not support it and we've been trying hard to penetrate IBM accounts). You can still configure NRZI encoding on most routers' serial links (it might depend on the actual hardware platform) with the nrzi-encoding interface configuration command (you had to do it with jumpers in the AGS+). Incidentally, changing interface encoding to NRZI was really helpful when you had to break things in the preparation for the troubleshooting part of the original CCIE lab).
Enough theory, let's summarize the proposed solutions:
- The nrzi-encoding (if available) is the best one, as it reliably solves the problem, is transparent and does not incur additional overhead.
- Compression or encryption are OK, but they result in significant CPU overhead (unless you have hardware encryption/compression modules) and might (at least in theory) still produce a long sequence of zeroes, although with a very low probability. IPSec also introduces overhead due to additional IPSec headers.
- LFI (effectively multilink PPP over a single link) is also a good solution, as the PPP framing and MLPPP headers break the long sequences of zeroes (you might have to fine-tune the fragment size with ppp multilink fragment size configuration command), but it introduces overhead on the WAN link.
- IP fragmentation would work, but would be quite bandwidth-consuming. If the fragmentation would be performed by the router, the overhead would be 20 bytes per fragment (IP header), if the sending host performs the fragmentation, the overhead is 40 bytes per fragment for TCP sessions. For example, if we reduce the IP MTU size to 256 bytes, the TCP session overhead is over 18% (and we were scoffing at the ATM designers that made us live with 10% overhead).
- The invert data command would only help if the modem has problems with long strings of zeroes, not with long strings of the same value.
- The tunnel key command just sets a 4-byte field in the GRE header but does not affect the encapsulated data at all.
I've started publishing posts on a regular basis about a year ago … and this week you've raced by another milestone: more than 1000 page loads/day (on average; weekdays are more active and weekends are a bit slow). For those of you who like statistics, here's the weekly report …
… and the last few months:
William Chu sent me a working configuration he uses to measure jitter with the IP SLA tool and react to excessive jitter on the primary link. First you have to create the jitter probe with the IP SLA commands:
ip sla monitor 3000
type jitter →
dest-ipaddr 199.11.18.168 dest-port 12333 →
source-ipaddr 199.11.18.169 codec g729a →
codec-numpackets 100
tos 184
frequency 10
Note: The continuation character (→) indicates that the configuration command spans multiple lines
Next you have to define the IP SLA reaction to excessive jitter. William configured his router to react when the jitter exceeds 300 milliseconds and returns back to normal when the jitter falls below 290 milliseconds (some hysteresis is always a good thing).
ip sla monitor reaction-configuration 3000 →
react MOS threshold-value 300 290 →
threshold-type consecutive →
action-type trapOnly
As the last step in the SLA configuration, you have to start the probe:
ip sla monitor schedule 3000 →
life forever start-time now
After the SLA probe and out-of-bounds reaction have been configured, the router will generate syslog messages whenever the jitter gets above the threshold as well as when it falls below the second threshold. You can then use the EEM applets to act on the syslog messages:
event manager applet MOS-Below
event syslog occurs 1 period 120 →
pattern "Threshold below for MOS"
... actions ...
!
event manager applet MOS-Above
event syslog occurs 1 period 120 →
pattern "Threshold exceeded for MOS"
... actions ...
If you're new to Tcl and would like to start using it on Cisco IOS, here's what worked for me:
- I've downloaded the ActiveTcl from ActiveState. It's always good to have development environment on your workstation.
- The documentation that comes with ActiveTcl is quite cryptic, but once you know what you're looking for, it's OK.
- To get to the "I know what I'm looking for" stage, I've used Tcl/Tk: A Developer's Guide.
If you know better beginner books/sources, let us all know.
You can find more Tclsh-related information in the Tclsh on Cisco IOS tutorial. Sample Tclsh scripts are available in the Tclsh script library. If you need expert help in planning, developing or deploying Tclsh scripts in your network, contact the author.
This article is part of You've asked for it series.
In the pre-DSL days, you had two options to get a short-haul high-speed link (at least in Europe): take E1 (or fractional E1) from a telecom (which was more expensive than a highway robbery, as the cost was recurring) or use baseband modems with proprietary encoding techniques on physical copper wires (assuming you could get them). As it turned out, some of these encoding techniques were not as good as the others (but the equipment was relatively cheap, so the budget limits usually forced the decision). We had our own share of modem-related problems, but they were never as bad as what I've heard from one of my students: his modems would lose synchronization when transmitting a long string of zeroes over a regular synchronous serial interface; ping ip 1.2.3.4 size 1000 data 0000 would be enough to bring down the link.
And now two questions for you:
- What could you do on the router to fix this problem?
- Why was the synchronization retained when transmitting a long string of ones?
Did you believe MPLS TE was a quality-of-service feature? Did someone persuade you it's mandatory to run OSPF or IS-IS if you want to deploy MPLS TE? I've collected a few more myths like these two and explained the actual facts behind them in an article published by SearchTelecom.
The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.
In a previous post, I've described how you can turn your router into a TFTP server. As you can configure the router to serve any file residing on it, you can also pull startup and running configuration from it with TFTP, providing that you configure:
tftp-server nvram:startup-config
tftp-server system:running-config
Warning: Due to total lack of any security features in TFTP protocol, use this functionality only in lab environment.
Posts
