Decent DNS, DHCP and HTTP server on an ISR router

Readers of my blog have probably noticed that I’m occasionally documenting the shortcomings of DNS and DHCP servers built into Cisco IOS (I will not even mention the HTTP server, this one gets constantly degraded). On the other hand, although you could centralize all these services, the centralization makes the branch offices completely dependent on the availability of WAN uplinks; without a working uplink, a branch office stops completely.

When Cisco introduced AXP (Linux blade for the ISR routers), my first idea was: “now, that’s a platform on which you could implement proper DNS and HTTP server”. Not surprisingly, I was not the only one with the “brilliant” idea: Infoblox has partnered with Cisco to offer the set of IP servers one might need in a branch office on the AXP.

The only problem bugging AXP is its price: the low-end model costs $3500 (list price, without the Infoblox software). Cisco and Infoblox have flashy whitepapers “documenting” huge TCO savings, but both of them failed to share with us the input parameters, assumptions and calculations. All that’s left is a small graph showing the desired results. I’m not saying AXP is not more cost-effective than the alternative; I’m just not easily persuaded by nice-looking colorful horizontal bands. What’s your perspective? Would you replace branch servers with AXP?

NIL Monitor performance tools help uncover SDH configuration errors

Grega Modrijan wrote an excellent article in Fragments, describing how he was able to use the functionality of NIL Monitor to pinpoint the configuration errors in an SDH ring.

The article provides an in-depth analysis of what went wrong and why (so you might find it interesting from pure SDH perspective), but the most important part is the moral of the story: NIL Monitor measures the metric that’s important to the end-customer (TCP performance), whereas the test tools measure what the performance labs care about (blasts of UDP packets). Another caveat to keep in mind when reading test reports from “independent” labs.

IPv6 over IPv4 tunneling

The CT3 wiki is starting to grow organically. A few days ago, an excellent article “IPv6 over IPv4 tunneling” appeared out of the blue sky. Michael Aschwanden did a great job describing the IPv6 tunneling mechanisms and various free public services. He also included sample router configurations used to connect to the public tunneling services.

The hidden wealth of IOS Tcl

Another undocumented (and thus very probably unsupported) Tcl-on-IOS detail: numerous Tcl packages are bundled with IOS and available in the tmpsys:lib/tcl directory (the tmpsys: is a virtual file system mapped to a part of the IOS image).

The list of Tcl packages seems to be pretty stable; 12.2SRC and 12.4T have the same contents of the tmpsys:lib/tcl directory.

router#dir tmpsys:lib/tcl
Directory of tmpsys:lib/tcl/

   13  -r--       19226                    <no date>  auto.tcl
   23  -r--        2549                    <no date>  base.tcl
   25  -r--        7655                    <no date>  cli_lib.tcl
   26  -r--        2589                    <no date>  context_lib.tcl
    3  drw-           0                    <no date>  eem_scripts
   28  -r--         185                    <no date>  email_template_cfg.tm
   29  -r--         147                    <no date>  email_template_chs.tm
   30  -r--         154                    <no date>  email_template_cmd.tm
   31  -r--         156                    <no date>  email_template_dmp.tm
   32  -r--         144                    <no date>  email_template_sl.tm
   33  -r--         325                    <no date>  email_template_sm.tm
   34  -r--         135                    <no date>  email_template_wd.tm
   14  -r--        9135                    <no date>  history.tcl
   15  -r--       23558                    <no date>  http.tcl
   19  -r--       17725                    <no date>  init.tcl
   21  -r--        6932                    <no date>  ldAout.tcl
   24  -r--       33266                    <no date>  optparse.tcl
   16  -r--       19415                    <no date>  package.tcl
   20  -r--        1014                    <no date>  parray.tcl
   17  -r--       32649                    <no date>  safe.tcl
   27  -r--       10367                    <no date>  smtp_lib.tcl
   22  -r--       13659                    <no date>  tclIndex
   18  -r--        4499                    <no date>  word.tcl

For whatever weird incomprehensible reason, these packages are not made available to the Tclsh interpreter started from the command line. For example, if you want to use the HTTP package, you cannot execute the package require http command like you would in any normal Tcl environment, but have to use the source "tmpsys:lib/tcl/http.tcl" command.

If you want to safeguard against a potential change-of-mind of IOS Tclsh developers, use this construct:

if {[catch {package require http}]} { source "tmpsys:lib/tcl/http.tcl" }

I need to slow down :)

I’ve just opened the January Technical Services News from Cisco. Nothing in there that would really interest me. Almost no routing protocols (one OSPF article), no BGP, no MPLS VPN. Based solely on this newsletter, one could get the feeling that I’m producing more documents covering core IP routing in a month than Cisco (I am positive that’s not the case).

But maybe Cisco’s engineers are refocusing on the new Support Wiki. Not really. After I’ve filtered out sequential changes to a single document, there were only 11 significantly changed documents in the Support Wiki in the last 30 days.

So I’m left wondering … what’s going on? Has everything already been written about the core IP routing features and the productive minds have shifted to voice and wireless? Are the engineers focused on IP routing becoming the dinosaurs? What’s your perspective?

But one thing is clear: I need to slow down.

Interactions between IP routing and QoS

One of my readers sent me an interesting question a while ago:

I reviewed one of your blog posts "Per-Destination or Per Packet CEF Load Sharing?" and wondered if you had investigated previously on how MQC QoS worked together with the CEF load-sharing algorithm (or does it interact at all)? For example, let's say I have two equal cost paths between two routers and the routing table (as well as CEF) sees both links as equal paths to the networks behind each router. On each link I have the same outbound service policy applied with a simple LLQ, BW, and a class-default queues. Does CEF check each IP flow and make sure both link's LLQ and BW queues are evenly used?

Unfortunately, packet forwarding and QoS are completely uncoupled in Cisco IOS. CEF performs its load balancing algorithm purely on source/destination information and does not take in account the actual utilization of outbound interfaces. If you have bad luck, most of the traffic ends on one of the links and the packets that would easily fit on the other link will be dropped by the QoS mechanisms.

You could use multilink PPP to solve the problem in low-speed environments. With MLPPP, CEF sends the traffic to a single output interface (the Multilink interface) and the queuing mechanisms evenly distribute packet fragments across the links in the bundle.

In high-speed environments, you can only hope that the number of traffic flows traversing the links will be so high that you’ll get a good statistical distribution (which is usually the case).

Flash-based DHCP database

Pete sent me an interesting question a while ago:

It might be interesting to write an article about ip dhcp database flash:dhcp-db command, documenting the pros of surviving a reboot versus cons of wear on the flash device.

I’ve already written about a few problems that can be solved with the DHCP database (but obviously a longer text is warranted … already stored in my to-do list) and it took me a while to find the time to dig out the relevant information on the flash device wear.

It looks like there’s nothing to worry about. To start with, most modern flash devices can survive millions of write cycles (read the article published on StorageResearch.com). The increase in flash reliability caused the manufacturers to stop quoting write cycles in their specifications and focused on MTTF or lifetime. I don’t know which flash cards Cisco ships with the routers, but if you’re really concerned, open the box, find the flash chip and contact the chip manufacturer.

Furthermore, almost all flash devices with embedded microcontrollers (for example, CF cards, USB flash drives or SSD) use wear leveling algorithms to reduce the impact of someone continuously rewriting the same sector (DHCP database in Cisco IOS or FAT table in Windows). Wear leveling can significantly increase the usable lifetime of a flash device.

It looks like the flash devices can be treated as disks if you don’t over-abuse them, so you might wonder why I wrote a post describing how to store DHCP database in NVRAM. At that time, a lot of low-end routers still shipped with flash formatted in the “old” Cisco format and the flash was not really usable to store ever-changing files (of course you can reformat the flash but you have to be careful not to lose the IOS image in the process).

EBGP multipath load sharing and CEF

When I was discussing the details of the BGP troubleshooting video with one of my readers, he pointed out that I should mention the need for CEF switching in EBGP multipath scenario. My initial response was “Why would you need CEF? EBGP multipath is older than CEF” and his answer told me I should turn on my gray cells before responding to emails: “Your video as well as Cisco’s web site recommends CEF for EBGP multipath design … but interestingly, it does work without CEF”.

The real reason we need CEF in EBGP load sharing designs is the efficacy of load distribution. Without CEF, the router will send all traffic toward a single BGP prefix over one of the links (fast switching performs per-destination-prefix load sharing). With CEF, the load is distributed based on the source-destination IP address pair combinations. Even if multiple clients send the traffic toward the same server, the load is spread across available links.

Obviously, I should write about CEF and load sharing once a month to refresh my failing memory.

Video: Simple BGP troubleshooting

One of the BGP aspects beginners find most frustrating is the BGP troubleshooting. The simple BGP troubleshooting video covers basic BGP troubleshooting techniques, from EBGP session troubleshooting to route origination and route propagation troubleshooting. The scenario used in the video is a two site MPLS/VPN-based network; obviously you can apply the same procedures to any BGP network.

The Wiki article contains the scenario description, the video and the router configurations, or you can watch the video served from Vimeo.

I guess the BGP beginners are not regular readers of my blog, so I would appreciate if you could spread the word.

Generate HTTP(S) requests from Tcl shell

A few days ago, a reader sent me an e-mail titled “Telnet Automation from a Cisco Router” and complained that IOS Tcl does not support the expect commands (spawn, send and expect). Since Expect is a Tcl extension, not part of the core Tcl, it’s not included in Cisco IOS, which was the only answer I could give.

You might be able to port Expect to IOS as a Tcl package if it doesn’t require external libraries.

However, it turned out that the reader actually wanted to trigger HTTP requests from the router. IOS Tcl has some built-in client-side HTTP support, but it’s far simpler to rely on the built-in http: file system. For example, to do a HTTP GET request, use the following Tcl command:

rtr#tclsh
rtr(tcl)#set result [exec {more http://webServer/index.html}]
<h1>Empty</h1>

As always, there are a few caveats:

• You can trigger HTTP GET requests, but not the PUT or POST requests.
• The server-side script should always return the HTTP status 200 (successful), otherwise the more command will fail. The actual status can be passed in the HTML response.

You can use the same trick to trigger HTTPS requests from Tcl.

You can find more Tcl-related information in the Tclsh on Cisco IOS tutorial. Sample Tclsh scripts are available in the Tclsh script library. If you need expert help in planning, developing or deploying Tclsh scripts in your network, contact the author.

This article is part of You've asked for it series.

Based on a true story: Remote workstation troubleshooting

Dragan, one of our support engineers, recently encountered an interesting scenario: a workstation in a remote office (located in a different country to make troubleshooting even more interesting) could not access the corporate applications although it was able to surf the Internet. It looked like a usual case of inter-site VPN problems, until we’ve discovered that the connectivity between the remote site router and the central office worked as expected.

Read more in Fragments

Five hottest telecom topics

Twenty seconds of shameless self-promotion: SearchTelecom.com has published its hot topics and editor’s picks for 2008 and my articles got the first (MPLS) and the second (BGP) spots.

My articles are a bit hard to find on SearchTelecom (unless you’re willing to sift through the site search results), so I’ve put together a Wiki page listing them all.

Proactive monitoring: turn an incident into a solution

A few weeks ago Andraž described how the NIL Monitor team caught someone running BitTorrent client on the infrastructure of one of our clients. The anomaly that triggered their suspicion was periodic nighttime high bandwidth utilization, but they finally caught the perpetrator by looking at the number of NAT sessions: BitTorrent client usually opens a large number of TCP sessions, whereas remote backup or software upgrade process does not.

Based on the incident, the NIL Monitor team decided to implement another monitoring object: now they monitor the number of NAT translations and raise alarm if an unusual spike is detected. Obviously you can do that with any network monitoring tool; the added value of NIL Monitor is our customer base: all our customers benefit from the new functionality, even though the incident happened only on one site.

Read more in Fragments

Multiple BGP daemons on a Linux host

An article I wrote a few months ago explained how to use Quagga on a Linux host to emulate BGP-speaking router inserting test routes into your lab network. If you need multiple BGP sources, you can use a few simple tricks to run numerous BGP daemons on the same host.

Read the full article in the CT3 wiki

Can brain dumps be stopped?

Last year Cisco finally decided to crack down on the major issues that allowed certification candidates to cheat on the certification exams. On top of numerous sensible measures that should have been implemented decades ago (living in central Europe, one cannot comprehend that the testing center does not verify your identity using a hard-to-spoof government-issued document), Cisco also decided to implement Forensic Security for certification exams. The geek hidden (not-so-very)deep inside every engineer should be delighted by this idea, but there is a simpler (and probably more effective) approach.

Read more in Fragments

IS-IS on partially-meshed Frame Relay subnet: sample configuration

In the IS-IS on multi-access partially-meshed Frame Relay interface article I’ve described the design rules you have to follow when implementing IS-IS over partially-meshed multi-access network. The IS-IS on partially-meshed Frame Relay subnet: sample configuration article contains step-by-step example, including initial router configuration, IS-IS configuration, verification steps and complete final configurations.

Read the article in the CT3 wiki

Video:Small site using BGP on two uplinks into an MPLS VPN network

Last week I’ve published a video describing the simplest possible BGP-in-MPLS/VPN scenario: a single-router site with one uplink. Today’s video covers a slightly more complex setup: there is still a single router on the site, but it has two links that should be used in load-sharing mode.

The Wiki article contains the scenario description, the video and the final router configurations, or you can watch the video served from Vimeo.

The most popular posts in 2008

The traffic statistics for 2008 are really interesting: the blog’s home page gets almost 20 times as many hits as the first blog post (even visitors using the search engines are seven times more likely to land on the first page than on any other page). This is clearly a side-effect of the platform I use: if you’re a regular visitor, you can read all the content on the blog’s home page (there’s no other way to do it with Blogger).

The most popular individual posts were:

• Upgrade 2800 series router to support USB boot
• BGP Essentials: AS-path prepending
• BGP default route
• Type 7 decryption in Cisco IOS
• Copy the text files into router's flash through a Telnet session
• Assigning server IP addresses with DHCP

The message is clear: write about easy hacks. Let’s look at some of the most commented posts:

• Should VTP be disabled by default?
• OSPF bypasses Control Plane Host Subinterface
• Stuffing the polls: the adventures of a convoluted mind
• Off-topic: disappointed by the antivirus industry
• Disable the wireless interface during the night
• Almost-dynamic routing over ADSL interfaces
• Technology is supposed to be simple, right?
• The never-ending story of IP fragmentation
• I’m too old … I prefer CLI over GUI

And the winner is: Why I'm no longer an active CCIE. Another clear message: write (preferably controversial) posts about problems or easy hacks.

Don’t worry; the focus of my blog will stay unchanged. I’m more interested in having fun writing it than in attracting large crowds.

Stop the finger-pointing with NIL Monitor

If you’ve never been involved in finger-pointing with your Service Provider, you’re either incredibly lucky, have a fantastic service provider or haven’t been in this industry long enough. In most cases, it’s quite hard to prove where the fault lies … but our NIL Monitor customers get armed with a very powerful weapon: weeks or even months of independent monitoring data.

Recently, one of our customers experienced site-to-site delays that affected only their IPSec traffic. The Service Provider engineers claimed that the delays were not their fault until our customer showed them very regular periodic graphs spanning several weeks … and the problem got fixed almost immediately.

Read the full story in Fragments

When was the ip ospf area command implemented?

One of my readers tried to implement my OSPF Best Practices and found out that ...

The ip ospf area does not work on all platforms/IOS versions. I noticed that it works on 7600s(12.2SRB4), but not on 7200(12.2(23)). Is it IOS/platform specific or a newly introduced command?

There are two reasonably accurate ways to figure out which IOS release contains the command you're interested in: the Feature Navigator and the IOS reference documentation.

It's sometimes hard to discover how the IOS marketing called the feature implementing your command, so the IOS reference manuals usually yield a faster answer. However, you still have to select the correct reference manual to open ... unless you use the Command Lookup Tool, which quickly finds the relevant part of the documentation. In my case, I easily figured out that the ip ospf area command became available in 12.0S, 12.3T (and therefore 12.4), 12.2SB and 12.2SRB.

To add icing to the cake, you can add the Command Lookup Tool to your browser’s search toolbar.

The death of VoIP?

In another great example of “investigative journalism”, Network World is asking whether the VoIP is dead (and I guess I’ll never make it to their top-20 list again). Regardless of their sensationalistic approach, take your time and read the original articles they quote (Part 1, Part 2). What the original author claims (and I don’t think you can disagree with him) is that VoIP has turned from hot technology to plumbing faster than some people would like. Whether that’s bad or not depends on the perspective … what’s yours?

Extranet with overlapping addresses

The idea to write an IP Corner article describing how you can use MPLS VPN-enabled NAT to implement flexible extranets that allow participants to retain their existing (and sometimes overlapping) IP address space has been sitting in my to-do list for over a year. After I’ve finally written it (without even hinting what I’ve been working on), I got several e-mails from my readers asking the questions this article answers, so it looks like the topic has suddenly become very hot. Do you have any ideas why that would be the case?

Read the Flexible Extranet
Implementation article in IP Corner

Looking for engineers working on MPLS/VPN networks

I have a few ideas that I would like to discuss with engineers working on the Service Provider side of the MPLS VPN networks. If you’re one of them and have a few spare minutes and the necessary willpower, please get in touch with me.

Test the real-life skills of your job candidates

In my previous certification-related posts, I’ve described the limitations of the current Cisco’s certification system and the ways some employers use certifications to screen the job applications. By now, we probably all agree that an unfortunately large number of certification holders do not have the experience you look for in your future employees. So what can you do? The answer is simple … test them in a real-life lab setup (that’s what we do). We can also help you get there: you can use our standard remote labs or we can work with you to identify the critical skills you need in your network and design the test lab exercise around them. To get in touch with our training department, use the Training Request Form and make sure you write a short note in the Customized courses at the bottom of the form with a reference to this post.

Video:Small remote site using BGP as PE-CE routing protocol

With everyone (and their cats) having videos on YouTube, the challenge to make a short BGP-related video was simply irresistible. I’ve tried to address the BGP beginners (maybe Cisco marketing would call them BGP associate candidates), as they probably benefit most from the video format (I know I would always prefer reading about a complex topic over watching a video about it). The video is focused on a scenario anyone could encounter: you want to move to an MPLS VPN service and the Service Provider is trying to persuade you to use BGP (which is a very good idea).

The Wiki article contains the scenario description, the video and the final router configurations, or you can watch the video served from Vimeo.

New Year Resolution#1: Fix the blog feed

More than a year ago, I got extremely upset by the SEO spammers that copied content from my blog feed and decided to reduce the feed into article summaries. However, I didn’t want to have half-finished sentences in my feed the way Blogger or Wordpress implement their short feed format. As I have already implemented something similar to the Wordpress’ more tag in Blogger, that would be a natural cutoff point. For shorter posts, I would like to retain whole paragraphs … or, as I’ve summarized the dilemma: I would like to give you enough information to decide whether you want to read the article or not.

Finally I found time to implement a reasonably good feed filter that:

• Does not touch the purely introductory posts (for example, those that point you to a Wiki article). If I followed a link from my feed reader to a post whose single purpose is to send me to another article, I know I would be mightily annoyed.
• Cuts off the posts that have the more tag at that point, giving me total control over what I want to have in the summary of a lengthy post.
• Reduces the post to approximately 350 characters (HTML markup not included in the count) while retaining its structure (a top-level paragraph, quote or DIV is never truncated).

If you experience any problems with the new feed or would like to use my solution (warning to OpenSource zealots: it’s written in VBScript), please do let me know.