Internetworking perspectives by Ivan Pepelnjak
For over 15 years NIL has been offering custom-tailored security training solutions (for example, we ran bootcamps for a major system integrator that brought their engineers to CCSP certification within two weeks) and developing security courses for Cisco Systems.
Throughout April 2009, Cisco is promoting one of our e-learning security solutions on Cisco Learning Network.
The Wake-on-LAN discussion we had a while ago brought us nowhere; there's simply no way to generate UDP packets on the router. I thought I could use Application Performance Monitor's Tcl scripts to generate the packet, but it looks like APM has been removed from recent IOS releases (and it's not clear whether you can use APM without a peer router).
The discussion nonetheless had an interesting side effect. Robert Turnšek sent me an interesting trick: with static ARP you can generate layer-2 broadcasts with a layer-3 unicast packet.
Let's assume your LAN has IP prefix 10.2.3.0/24 and you want to use 10.2.3.77 as the IP address that will generate MAC-layer broadcasts (you need to give this address to a WOL program). Configure arp 10.2.3.77 FFFF.FFFF.FFFF and you're done.
You might wonder how this technique differs from directed subnet broadcast. The important difference is that although every IP host on the subnet will receive the layer-2 broadcast, they will ignore the packet since it's not addressed to them. This solution is thus not vulnerable to the smurf attack.
Network World just published an article on LISP, including a quote attributed to Geoff Huston:
LISP relies on tunneling, and tunneling is not 100% perfect. At times, the tunnel passes a packet that's too big and it disappears without a trace to the sender or the recipient…That's really bad.
We all know what can go wrong with the tunnels: a combination of:
I sincerely doubt that the authors of the LISP draft would not be aware of those problems (Cisco's gear is one of largest source of tunnels in the IP world) or that mr. Huston would not know what he's talking about (read the MTU articles on his personal blog).
The only explanation for the fairy-tale quote in the article is thus the process of multi-tier simplification until facts became adjusted to the technology awareness of someone in the publishing pipeline.
Julian is faced with an interesting challenge:
In the real world, many customers using DSL solutions have their Internet connection disrupted by one internal user performing a large download. On a typical DSL solution, implementing quality of service on outbound traffic is trivial (you can use PQ, CBWFQ, policing or shaping). However, how does one rate-limit inbound traffic in a sensible fashion? Turnkey solutions like packeteer allow inbound classes of traffic like HTTP to be rate limited per flow by dynamically changing window sizes.
Cisco IOS has three basic QoS mechanisms: queuing, shaping and policing. It cannot intercept a TCP session and slow it down by reducing its window size (like PacketShaper).
A queuing mechanism works only if the (temporary) packet arrival rate is higher than the interface transmission rate and thus an output queue forms. When the router receives packets from a DSL link and sends them to an Ethernet link, the output queue remains almost empty and thus you cannot use any queuing mechanism.
Policing works regardless of the size of the output queue, but it cannot be fine-tuned to react to interface congestion (unless you use EEM interface triggers to detect high interface utilization). Furthermore, IOS does not have proportional policing of individual traffic flows.
The only remaining mechanism is shaping: you have to create an artificial bottleneck on the outbound (LAN) interface with the shape MQC command. The shaping rate should be less than the useful speed of your DSL link. The shape command introduces a bottleneck: the packets arrive through the DSL interface faster than they can be sent to the (shaped) LAN interface. Now you have an output queue that you can influence with further QoS policies (you need to configure hierarchical MQC). Without further configuration, the output queue created with the shape command uses per-flow weighted fair queuing, giving each layer-4 session a fair share of the bandwidth.
The output rate configured with the shape command has to be significantly smaller than the DSL rate, otherwise the output queue forms only sporadically. By shaping the inbound DSL packets on the LAN interface you reduce the maximum throughput, slightly increase latency and introduce jitter.
This article is part of You've asked for it series.
Yesterday I’ve described the difference between line rate and bit rate (actually physical layer gross bit rate and physical layer net bit rate). Going to the other extreme, we can measure goodput (application-level throughput), which obviously depends on multiple factors, including the TCP window sizes and end-to-end delays. There are numerous tools to test the goodput from/to various locations throughout the world (speedtest.net worked quite nicely for me) and you’ll soon discover that the goodput on your DSL line differs significantly from what the ISP is advertising.
Apart from obvious marketing ploys, the difference arises due to the baroque Russian dolls encapsulation stack used on ADSL (see the ADSL-based protocol stacks section in this document). A typical PPPoE connection runs IP over PPP (2 bytes) over PPPoE framing (6 bytes) over Ethernet framing (14 bytes) over RFC 1483 bridged connection (8 bytes) over AAL5 (8 bytes) over ATM cells (5 byte overhead for every 48 bytes of payload). Every IP packet is thus burdened with 38 bytes of DSL protocol stack overhead plus 10.4% ATM cell tax. Assuming the average IP packet size in the Internet is around 500 bytes, the DSL protocol stack overhead is 6%, resulting in a total of over 16% of lost bandwidth just to accommodate the committees that constructed this monster.
Update @ 2009-03-27: I forgot yet another encapsulation technique: AAL5. I also need to investigate whether the RFC 1483 bridged encapsulation really uses the 8-byte header (there are numerous options in the RFC).
Rajendra had an interesting problem:
Recently I got confused with the term line-rate. Is it the packets being switched across the switch fabric or control packets destined to the protocol tasks or both or something else?
The line rate is a physical layer term that has nothing to do with the line cards or switching fabrics. It indicates the actual speed with which the bits are sent onto the wire (and is thus also known as physical layer gross bit rate). The data transfer rate (commonly known as bit rate) is the transfer rate offered by the physical layer to the data link layer. If you want to be precise, you should call it physical layer net bit rate.
Two well-known physical layer technologies with different line rate and data transfer rate are ISDN (actually the I.430 recommendation) with 160192 kbps line rate and 144 kbps data transfer rate and Gigabit Ethernet (the 802.3z recommendation) with 1.25 Gbps line rate (due to 8b/10b encoding).
This article is part of You've asked for it series.
Sebastian sent me an interesting question:
I have read that we can only use 75% of the bandwidth for the custom queues as 25% is reserved for the keepalives and routing protocol updates. If I want to set 50% of bandwidth for a particular queue should it 50% of the total bandwidth or 50% of the available 75% of the bandwidth?
Before going into the details, it’s important to remember that the WFQ (CB-WFQ is only a mechanism to sort packets into output queues) uses relative ratios (percentages) between queues to determine which packet to send (the absolute bandwidths are used just to compute the ratios).
To prevent the classified traffic from starving the routing protocols and other important packets (ARP, CDP, ILMI, Frame Relay keepalives …), the router does not allow the class queues to consume more than 75% of the total interface bandwidth. The remaining 25% are used for the default class (all non-classified traffic) as well as all non-IP packets. You can change the limit with the max-reserved-bandwidth interface configuration command; just make sure you have plenty of bandwidth left for the control traffic.
When you configure the class-based queuing with the bandwidth [remaining] percent percent command, the number you specify is the percentage of the total interface bandwidth and is used to determine the ratios in the WFQ system. Obviously, the sum of all percentages and the max-reserved-bandwidth should not be more than 100.
When using the bandwidth kbps command, the traffic in the class you’re configuring will get at least the specified amount of bandwidth. The bandwidth percentage is computed as the specified bandwidth divided by the interface bandwidth.
The interface bandwidth can be set based on the actual hardware (it’s not hard to guess what the bandwidth of a 100BASE-TX interface is), computed for multilink bundles or configured with the bandwidth interface configuration command.
This article is part of You've asked for it series.
One of my regular readers asked this question:
I want to shut my particular interface after 6:oo pm every day. Right now I have just the general idea as to how EEM works. What I guessed is that when the router's clock ticks 6:00 pm a syslog message might be generated and event manager will catch it and execute the particular command. But I don’t understand how EEM monitors the clock so that it knows its 6:00 pm now and it has to generate a syslog message.
If you want a quick answer, you’ll find the exact solution in the Time-based wireless interface activity article in the EEM category in the CT3 wiki. However, you might want to get a bit more out of this question.
If you want to use EEM, the best investment of your time would be to spend an hour or two reading its introductory documentation, particularly the section describing the event detectors. It nicely describes four time-based event detectors available in EEM. You can also try searching my blog and CT3 wiki. For example, I’ve written about EEM timer cron event more than two years ago. The blog labels and wiki categorization should be of great help: start with the EEM label in the blog and the EEM category in the CT3 wiki.
Last but not least, there’s always more than one way to solve a problem with Cisco IOS. If you’re not happy with EEM, you can use Kron.
This article is part of You've asked for it series.
Two weeks ago Douglas Crockford wrote that his blog generated more than a million page views. Since he's an icon in the mainstream Web industry with his own Wikipedia listing, I simply had to check how my blog compares to that. Not bad, StatCounter.com claims I had 1.2 million page loads in two and a half years.
The reader who was concerned about making a loop while connecting a switch to itself was also facing “customer-installed” DHCP servers in his LAN. He wrote:
Some users have installed their own Linksys routers and plug our cable in router's LAN ports, so there is DHCP servers fight in our LAN. How can I sort this out (I cannot physically find the location of the Linsys routers)?
The ideal solution is DHCP snooping (assuming your switch supports it), well documented on www.cisco.com. The basic configuration takes only a few minutes:
This article is part of You've asked for it series.
I’ve always thought that Cisco introduced the BGP Local-AS feature into IOS to support complex MPLS VPN design scenarios. Obviously I was wrong, the early documentation always describes an ISP AS merging scenario. Unfortunately, all the articles I’ve found skip some important details: they describe the basics and the configuration commands, but forget to mention the impact on the AS paths received by the ISP customers. Obviously we need a more thorough description of this feature.
Read the whole article in the CT3 wiki
Pappyar has sent me an interesting password recovery technique, which can be used in those weird circumstances when you cannot force the router to go to ROMMON (for example, you’ve configured no service password-recovery and the break signal does not work as expected). Unfortunately, his trick works only if you can remove the flash memory from the router (it’s soldered in low-end models):
A comment to my post “Video:Small remote site using BGP as PE-CE routing protocol” has made me think very hard about the PE-CE routing protocol selection. Justin was right: although it’s best for the Service Provider to try to push BGP as far out as possible, it might make customer’s life more complex in some scenarios. I’ve decided to write an article about these PE-CE routing protocol issues, but as I’ve started writing it, it dawned on me: this is not a technical issue, but a business one. You can find my reasoning in the “Choosing customer MPLS VPN routing protocols” article published by SearchTelecom.com.
The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.
StatCounter, the free web tracking service I'm using has recently launched global statistics tool that allows you to analyze the statistics of the data they're collecting from thousands of participating web sites. An interesting one is the "Search engine" graph:
A reader sent me an interesting question:
Do you have any advice for resetting/logging into a router (2821) where the one time user of cisco:cisco has already been used?
I couldn't offer any better advice than performing the regular password recovery procedure. Is there another solution?
This article is part of You've asked for it series.
If someone asks you to implement load balancing on equal-bandwidth parallel links between a pair of BGP routers, your first thought should be multihop EBGP session between loopback interfaces. Although this design is ancient and excruciatingly boring, it still has a few interesting twists, for example:
Last but not least, an EBGP multihop session is vulnerable. You can use the neighbor disable-connected-check command to have a single-hop EBGP session with an IP address that is not directly connected.
Read the whole article in the CT3 wiki
I’ve already described the problems you’re facing when running IS-IS over partially meshed Frame Relay and provided sample working configurations. To conclude this topic, I’ve also written a “technical solution” document that describes the symptoms, verification steps and the necessary configuration changes.
Read the article in the CT3 wiki
Karsten Iwen made an interesting comment to my “Don't let a lab rat anywhere near a production box” post: you should avoid the SSH/VPN key generation mistakes by using key labels. He also wrote a post explaining the concept but since it’s in German, let me rephrase it in English.
Cisco IOS release 12.2(8)T added the label parameter to the crypto key generate rsa command. You can use this parameter to assign a label to your VPN key, for example
Rtr(config)#crypto key generate rsa label VPN modulus 2048
To use the labeled key to generate your certificate, use the rsakeypair command in the CA-trustpoint configuration mode:
crypto pki trustpoint example.com
enrollment retry count 100
enrollment mode ra
enrollment url http://ca.example.com/certsrv/mscep/mscep.dll
rsakeypair VPN
One of our customers had to provide end-to-end IP transport across their enterprise network for an outsourced video surveillance solution. We implemented a true VPN solution for them (the hosts in the enterprise network cannot access the surveillance equipment and vice versa) using the Multi-VRF feature available in all recent Cisco IOS releases. The solution is described in the March IP Corner article “Add a VPN to an Enterprise Network with Multi-VRF Functionality”.
SearchTelecom is starting a series on SP network optimization (you can already read the first article). If you have experience in the area of the SP infrastructure optimization (for example, large-scale caching or video distribution) and would like to share it, please get in touch with Kate Gerwig, the site editor.
I tried to do a few simple NETCONF tests yesterday (I wanted to see how the router's configuration looks like when it's encoded in XML). I didn't want to start a lab for such a simple task and decided to use my home router. SSH was the only reasonable transport (you can't run BEEP with standard Linux tools), but it was not working on the router.
Obviously I've configured too many lab devices in the past. As soon as I've realized I had SSH problems, my fingers automatically typed crypto key generate rsa. A few milliseconds after I've hit ENTER it dawned on me that my router uses PKI certificates for the VPN connection to our network ... and I've managed to invalidate the router's certificate, which is one of the few things that a reload will not solve. Although our IT guru was fantastic and approved my (router's) certificate request late in the evening, I still felt bad about the whole experience.
And the moral? Don't let a lab wizard with fast fingers too close to a production box :)
Jan Bervar, our security/VPN expert, has started series of short articles in Fragments focusing on design/best practices aspects of PKI-enabled IPSec VPNs. His first article deals with the basics: how do you choose the key length, where do you store the keys and how do you protect them.
I found this figure in an introduction to a "Marketing 2.0" book:
It's almost guaranteed to be a winning combination in a Bullshit Bingo.
If you use Tcl to write Embedded Event Manager policies, you could use the action_syslog command to generate syslog messages. In all other Tcl-based environments (including tclsh), this API is not available, but you could use the syslog: file system to generate debugging messages.
Read more in the CT3 wiki article Generating Syslog messages from Tclsh.
The article is part of Tclsh on Cisco IOS tutorial.
Last week IOS release 12.4(24)T appeared on CCO. One of the significant improvements in that release (I can’t manage to get enthusiastic about new kludges to support the SIP morass) is the support for 4-byte BGP AS numbers.
Finally an enterprise network that uses Cisco routers to connect to the Internet can use the new AS numbers distributed by the regional registries (assuming you’re brave enough to run 12.4(24)T on your production gear). The Service Providers using 7600 routers will have to wait … corresponding 12.2SR release is not yet available.
Hat tip to Tassos @ CCIE in 3 months: he was the first to write about this feature.
Just in case you're not reading Dilbert on a daily basis ... Scott Adams focused on marketectures. So far he's covered basic concepts and benchmarking. It looks like Dilbert is (yet again) becoming a documentary.
I thought I knew all there is to know about the AS-path prepending before last month’s incident, which prompted me to focus on this particular Cisco IOS feature.
For example, did you know you could do inbound AS-path prepending? I didn’t, until Rodney Dunn from Cisco mentioned it in an e-mail exchange. Did you ever wonder whether the AS-path prepending affects inbound or outbound AS-path filters? I had a hunch it doesn’t, but was never sure.
You can find everything you never wanted to know about AS-path prepending in the AS-path prepending (technical details) article in the CT3 wiki.
A post by CiscoSubnet has pointed me to the official IntelliShield alert describing the AS-path incident. While the alert is very well written (I wouldn’t expect less) and the associated Protecting Border Gateway Protocol for the Enterprise is excellent, I was highly amused by the following text:
Cisco Security Intelligence Operations has identified a method through which administrators can modify device configurations to mitigate the effects of the AS path processing issue. Administrators can limit the amount of AS path segments that are associated with any route by using the bgp maxas-limit feature …
The bgp maxas-limit command was suggested by some members of the NANOG mailing list as early as few hours after the incident (which happened on February 16th; the IntelliShield alert has been released on February 20th) and I’ve released a detailed article on February 17th. I know Cisco’s engineers did a great job in this particular case; why did someone have to run their results through the /usr/local/marketese | /usr/bin/bowdlerize?
Posts
All Comments
