Much ado about rootkits

Ten days ago, the industry press was buzzing with the news of the IOS rootkit developed by Sebastian Muniz. At that time I wrote “Personally I doubt it would go beyond Tcl scripts that we already know about” … and now it's time to admit that:

1. I was wrong.
2. I'm really impressed.

Although the rootkit was just a proof of concept (which is usually enough for a white-hat researcher), it does demonstrate that you can (with proper skills, tools and lots of patience) reverse-engineer IOS, write your own code and insert it into IOS image.

The rootkit presentation prompted Cisco to generate an excellent document describing how to detect patched IOS images and the precautions you can take to ensure an intruder does not get access to your devices.

On the other hand, I was bitterly disappointed by the lack of coverage from the "industry press". There was speculation that Cisco released three patches in anticipation of the presentation (anyone who looked into what those patches were would easily find out that two of them were not IOS related) and a few notable exceptions correctly describing the situation, but some publications that were very loud before the presentation forgot to tell their readers that the threat was "slightly" over-rated. Of course, the lack of interest in non-sensational news has already started conspiracy theories.

If you want to have more details, read a down-to-earth description of the presented rootkit by Nicolas Fischbach.

Policy Routing with BGP Quick Learning Module

Our developers have just released the BGP Policy Routing E-Lesson that accompanies the Scalable Policy Routing IP Corner article. The lesson contains an instructor-led presentation that explains the concepts and a remote lab where you configure and test the solution without endangering the operations of your live network.

If you'd like to review the e-lesson and write/blog about it and the concept of short e-learning modules covering very specific technologies or design scenarios, send me a message describing what you'd like to do and I'll organize the access to the e-lesson for you.

Display CPU utilization with every show command

Xavier has mentioned an interesting undocumented command in his comment to the “Continuous display of top CPU processes” post: after you execute terminal exec prompt timestamp, every show command displays current time and CPU utilization before the requested printout.Here is a short example:

R1#terminal exec prompt timestamp 
R1#show ip interface brief
Load for five secs: 4%/0%; one minute: 1%; five minutes: 0%
Time source is NTP, 17:31:14.456 UTC Wed May 28 2008

Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet0/0            10.0.0.1        YES NVRAM  up                    up      
FastEthernet0/1            192.168.200.205 YES DHCP   up                    up      
Serial1/0                  unassigned      YES NVRAM  administratively down down    
Serial1/1                  unassigned      YES NVRAM  administratively down down    
Serial1/2                  unassigned      YES NVRAM  administratively down down    
Serial1/3                  unassigned      YES NVRAM  administratively down down

Conditional BGP route origination

Sebastian Majewski has found an interesting feature: if you use the network route-map BGP configuration command to originate BGP prefixes and use the match conditions within the route-map, BGP inserts the IP prefix in the BGP table only if the source route in the IP routing table satisfies the route-map conditions. More details and a sample router configuration are available in CT³ wiki.

How big is your network?

Based on the somewhat surprising results of the routing protocols poll (the high percentage BGP got), it would be nice to know the network type that you design/manage/operate. Please respond to this week's readers' poll; if you work for Cisco, other equipment vendors or a system integrator/professional services company, simply select the network type that you work with most often.

Routing protocol usage

The results of the routing protocol poll are in. As expected, OSPF is more popular than EIGRP (60:40 ratio), with IS-IS having significant presence (primarily large SP networks, I would assume) and some people using RIP in the edges of the network.

What really surprised me was the very large percentage of users using BGP. Either BGP is becoming very popular in the enterprise networks or most of the readers are coming from Service Provider environments.And last but not least, it looks like I'm focusing on the topics (BGP and OSPF) that you actually use in your networks (or maybe it's a self-fulfilling prophecy and you read my blog because I write about topics that you're interested in :).

Continuous display of top CPU processes

When you have to monitor which processes consume router's CPU over a period of time, a Tcl script that emulates the Unix top command might come handy. Download it, install it and enjoy the view :)

If anyone has discovered a reliable technique that detects a keypress event (= character available on stdin) in the Tcl loop, please let me know. The Ctrl/C solution is a kludge.

You can find more Tclsh-related information in the Tclsh on Cisco IOS tutorial. Sample Tclsh scripts are available in the Tclsh script library. If you need expert help in planning, developing or deploying Tclsh scripts in your network, contact the author.

Router architecture books

Another interesting question I've received:

Can you please recommend me literature explaining the architecture of Cisco routers and switches (buffers, control-plane, forwarding-plane, process switching …)

The best one I've found so far is the Inside Cisco IOS Software Architecture, but it's a bit old, so if you've found something better, please comment.

Full disclosure: if you click on the link above and buy the book, I might eventually get $1.76 from Amazon.

NTP Implementation Quick Learning Module

The NTP Implementation Quick Learning Module we've developed teaches you how to use Network Time Protocol (NTP) on Cisco IOS devices to synchronize with a reliable time source.

The presentation part of this e-lesson describes NTP basics and IOS implementation details. The hands-on remote exercise enables you to configure multi-tier NTP implementation on Cisco routers.

Running OSPF across a PIX/ASA firewall: TTL details

Sharath Samanth has recently asked an interesting question:

I have seen the post on running OSPF across a PIX firewall. Since I did not have a PIX, I tested the solution by replacing PIX with a router.

I had configured the neighbor statements on both routers, but the OSPF was failing to come up. The debug indicated that the router emulating PIX was sending time exceeded ICMP to both OSPF-speaking routers.

The OSPF hello by default has a TTL of 1 which I think is an issue with this scenario. Is there anything special thats done on PIX to get OSPF working?

The answer is quite simple: PIX is not behaving like a router, but rather like a bridge with additional IP features (NAT and traffic filters). It does not decrement the TTL of a transit packet (which could lead to interesting loops if you badly mess up a redundant topology) … and I have to congratulate Sharath for an excellent diagnosis of the problem.

This article is part of You've asked for it series.

Typo of the month

Xabrouck found the following gem in my “Multihoming to one ISP” article:

If a customer already owns provider-independent addresses but does not have a registered AS number, it can use a private AS number to establish BGP peering with the ISP to which it’s multi-hoped.

I obviously had some very bad experiences in the past and my subconsciousness has expressed itself :)

Cable modem problems with Cisco 871

The undesired intermittent bridging behavior of Cisco 871 using old ROMMON software can lead to hard-to-diagnose problems if you're connected to an Internet access network through a cable modem that accepts only a single MAC address. The right sequence of events can leave the router/modem combination in a state with no external connectivity requiring a modem power-cycle:

1. The router and the cable modem are power-cycled.
2. The router starts to bridge between all LAN interfaces, effectively connecting inside workstations directly to the cable modem.
3. One of the workstations could detect a LAN failure (due to router reload) and restart the DHCP process (a Windows XP host would definitely do that).
4. The DHCP requests from the workstation are bridged straight to the cable modem which caches the workstation's MAC address and forwards the DHCP request.
5. The workstation is assigned a public IP address (at this time, the workstation is connected directly to Internet and thus vulnerable).
6. The router loads Cisco IOS and reinitializes the Ethernet interfaces. Bridging between internal and external interfaces is stopped.
7. The router sends DHCP request on the outside interface, but the modem ignores it, as the MAC address of the DHCP request differs from the previously cached one.

In most cases, the cable modem has to be power-cycled to lose the cached MAC address.

This behavior can be observed only if the router and the cable modem are reset at the same time and the cable provider doesn't care much about MAC security and allows the modem to learn the MAC address. If you reset only the cable modem, the router is not bridging (no problem); if you reset just the router, the cable modem still caches the router's MAC address and ignores the DHCP request from the inside workstation(s).

Multihoming to a single ISP

A while ago Greig asked an interesting question:

Would it be possible to explain in detail a scenario where dual-as and as-overide is being used and another scenario where dual-as (using no-prepend / replace-as), as-overide and remove-private-as are used?

I decided to start from the last item. The neighbor remove-private-as option is used in scenarios where you run BGP between public and private AS numbers to collect IP prefixes and advertise these prefixes to the rest of the world as belonging to the public AS. The most common design in this category is multihoming to a single ISP.

This article is part of You've asked for it series.

Create structured e-mails from EEM applets

A few weeks ago I've described how to use the append show filter and more command to send e-mails containing multiple printouts from an EEM applet. A few hours after I've published the post, David Houser sent me a great EEM applet that used texts stored in flash: files to generate headings between various show commands. While his solution works perfectly (and gives you all the flexibility you want), it's a bit verbose and requires lots of small files that clutter your flash: memory. I've thus decided to write a small Tcl script that executes the Cisco IOS command specified in the command line and appends the command results together with a heading in an output file.

Guide to Harden Cisco IOS Devices

In the last days, industry journalists have started to make a big fuzz about a Cisco IOS rootkit that someone is going to present in a few days. Personally I doubt it would go beyond Tcl scripts that we already know about (OK, maybe it's EEM-based so it doesn't need a VTY and maybe it starts at router reload) … but I might be really surprised.

However, the Cisco's response to this announcement (which was basically saying "we haven't seen anything new yet") included a nice gem: a link to the Cisco Guide to Harden Cisco IOS Devices document.

Private domain names

I'm positive the IP prefixes reserved for private use by RFC 1918 are well-known to anyone building private IP networks. Likewise, you should be familiar with reserved AS numbers documented in RFC 1930 if you're building private networks running BGP. But if you know there are reserved DNS domains that can be used to write sample configurations and test code, you're smarter than I was a few weeks ago.

I was writing the June IP Corner article and needed to set up DNS servers within the lab. I used example.com as the domain name and decided to check what would happen if you'd visit the actual www.example.com web site (try it out). It politely referenced me to RFC 2606, which documents the reserved domain names you can use.

As a rule, you should use private IP addresses, AS numbers and domain names in all technical documentation you're producing (unless, of course, you're describing an actual network). If you're forced to use public addresses or AS numbers (for example, to illustrate how the neighbor remote-private-as command works), you should clearly state that the AS numbers are imaginary.

Cisco Advanced Services training offered by NIL

Recently we've started offering advanced courses developed by Cisco Advanced Services (AS) group. These courses cover high-end platforms (7600, CRS, IOS XR) and advanced solutions (Service Control Application). You can view the currently-offered list of courses and enroll into them in our online course schedule.

Control Plane Protection inbound packet classification

The inability of Control Plane host interface to detect inbound OSPF packets (and the flurry of comments that followed my blog post) has prompted Sebastian and myself to search for more documentation and conduct further tests. Sebastian already had a working configuration from which he could infer most of the configuration rules and he also found the well-written Understanding CPPr document on CCO. Together with the tests I ran in my router lab, we're pretty confident the CPPr inbound packet classification rules are (approximately) as follows:

Use the latest 12.4T software (at least 12.4(15)T5) if you want reliable CPPr operation.

• control-plane aggregate service-policy disables any control-plane subinterface service policies.
• If you want to use the per-subinterface (host, transit and cef-exception) policies, you have to remove the inbound service policy from the control-plane aggregate path.
• Routed packets that cannot be CEF-switched (have to be punted to another switching mechanism) are classified as transit packets.
• Local multicast packets with destination IP addresses within IP prefix 224.0.0.0/24 and packets with TTL <= 1 are classified as transit packets in 12.4(15)T5. These packets will be classified as cef-exception packets in the future (see the Understanding CPPr document).
• Unicast packets without IP options addressed to the router and having TTL > 1 are classified as host packets.
• Non-IP traffic (ARP, Frame Relay keepalives, CDP ...) is classified as cef-exception.

The TTL-related rules explain why the router classifies IBGP packets as host packets and EBGP packets as transit packets. As soon as you configure neighbor ebgp-multihop on the router router, inbound EBGP packets become host packets.

Which routing protocol do you use?

Years ago EIGRP and OSPF had strong presence in large enterprise networks, BGP was used solely by Internet Service providers and IS-IS was a rarity (and there were people using Banyan Vines).

The situation has probably changed over the last years, I would (sadly) expect EIGRP to decline and (happily) BGP to grow. Let's figure it out; please respond to this week's readers' poll. Of course you can choose more than one routing protocol.

BGP load sharing across multiple AS-paths

Sebastian Majewski has contributed a short article to the CT³ Wiki explaining a hidden Cisco IOS router configuration command that can bypass the limitations of the IBGP multipath load sharing in scenarios where the paths across which you want to load-share cross different upstream autonomous systems.

Cisco 851 and 871 bridge between LAN and WAN interfaces during boot process

Euphrates Greene sent me a report of a very annoying “functionality” of Cisco 851/871: they're bridging between the inside (LAN) ethernet and outside (WAN) ethernet interfaces while they're running the ROMMON code (from the reload/power-up throughout the software decompression process until the control is transferred to the Cisco IOS). It's worth mentioning that these routers are commonly used as SOHO firewalls and that the internal LAN is exposed while the router is in the bridging mode.

Our security experts have replicated the behavior and reported it to Cisco PSIRT. Fortunately it's a known vulnerability, documented as CSCsd60259 (release note is available on CCO to registered users) and fixed with a ROMMON upgrade.

New routers are shipped with new ROMMON version, so you shouldn't be seeing this behavior on brand new boxes … but one cannot help but wonder why such a nasty behavior was not documented as a field notice/security advisory.

RTBH links (and thanks for the acronym :)

One of the comments to my Sunday post mentioned RTBH. Obviously I'm not geeky enough, as I had to ask uncle Google for help (but don't worry, I'll work on my geekiness factor :).

The search results produced a few very interesting links, among them a well-structured presentation on RTBH that refers to a paper describing how you can detect remote DoS attacks with the backscatter analysis (assuming the attackers are randomly spoofing source IP addresses).

X.25 is still kicking

The results of the X.25 poll amazed me. I thought X.25 was long dead, but around 10% of people responding to the survey indicated they are still using it (primarily to connect the legacy equipment) and more than half of the respondents had to configure it in the past (I didn't know we were all going so far back :).Obviously some very big networks are still heavy X.25 users, as Cisco has been implementing more and more X.25-related features (primarily in the XOT space) in recent years … if only they would have been available when I really needed them 15 years ago :(.

By now you should be asking yourself “Well, so what's the latest and greatest X.25 feature?” Brace yourselves: 12.4(15)T has introduced support for X.25 accounting and the Call Detail Records (CDRs) are generated as syslog messages.

How do you know you're an SP-geek

1. You're creating a multi-AS BGP test lab on Sunday evening;
2. The core AS is running 12.2SRC code;
   
3. You insert a P-router in the core network ... because every large network has P-routers;
4. You create BGP session templates instead of configuring two parameters of a few IBGP neighbors;
5. You configure MPLS in the core network instead of using BGP on all routers ... because it saves you a few BGP sessions ... and that's the way things should be done anyway;
6. When configuring OSPF, you define inter-AS links as passive interfaces ... not because you're running OSPF in the other AS but for security reasons :)
   
7. ... add your comment here ...
   

Please comment: Is asymmetric routing harmful?

We've always been trying to minimize asymmetric routing, in both design and implementation phase, as it impacts a number of IP services/features, including:

• Network Address Translation;
• Content-based Access Control (CBAC);
• Reflexive access lists;
  
• Redundant firewalls (at least until recently);
• IP Multicast;

In some scenarios, asymmetric routing can impact delay/jitter and consequently the perceived quality of service.

However, asymmetric routing is a reality within the Internet (it's close to impossible to guarantee symmetric routing even for multi-homed end users) and it might even help in some scenarios (low-speed/low-delay upstream link with high-speed/high-delay downstream link).

What's your opinion? Is asymmetric routing harmful? Should we strive to avoid it ... or do you just accept it as one of facts of life?

Scalable Policy Routing

If “policy routing” reminds you of nightmarish spaghetti solutions of hop-by-hop route-maps, read my latest IP Corner article, Scalable Policy Routing. It describes how you can use a distance vector routing protocol (BGP is used in the article, but you could do the same with EIGRP or even RIP) to solve common policy routing problems in a highly scalable way.

OSPF bypasses Control Plane Host Subinterface

I wanted to implement a mechanism that would automatically (using EEM) block unstable OSPF neighbors. Once you identify the neighbors to block (this should be the hard part), blocking them is easy if you're running point-to-point interfaces (you just make the interface passive), but blocking a single neighbor on a multi-access interface is a royal pain. I didn't want to use the access lists, as it would be very hard to integrate OSPF-specific filters with existing incoming access-lists configured on the interfaces. Control Plane Protection looked like the ideal tool to use; if I could drop certain inbound IP packets (OSPF hello packets) based on their source IP address (= unstable neighbor) and IP protocol (= OSPF), they would never get to the OSPF process and the adjacency would not form, resulting in a more stable network.

Update: After the comment from William Chu, I've tested 12.4 mainstream release. OSPF is blocked as configured. Next I've re-read the documentation … and found that one of the documented restrictions is that the host subinterface only filters UDP and TCP traffic. Configuring the service policy on the aggregate path (the control-plane keyword with no options) worked.

Before trying to figure out the integration between the SYSLOG messages and router configuration changes, I performed an easy test: I tried blocking all OSPF traffic in the host control-plane (the one controlling packets received by the IOS processes) with the following configuration:

class-map match-all BlockOSPF
 match access-group name BlockOSPF
!
!
policy-map ControlPlane
 class BlockOSPF
   drop
!
ip access-list extended BlockOSPF
 permit ospf any any
!
control-plane host
 service-policy input ControlPlane

However, according to the show commands, the service policy did not identify any packets as belonging to the BlockOSPF class and the OSPF adjacencies were not affected:

C1#show policy-map control-plane host
 Control Plane Host 

  Service-policy input: ControlPlane

    Class-map: BlockOSPF (match-all)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: access-group name BlockOSPF
      drop

    Class-map: class-default (match-any)
      5 packets, 400 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any 
C1#show ip ospf neighbor 

Neighbor ID     Pri   State           Dead Time   Address         Interface
10.0.0.12         1   FULL/DR         00:00:30    10.0.1.2        FastEthernet0/0
10.0.0.2          0   FULL/  -        00:00:33    10.0.0.2        Serial1/0.101
10.0.2.2          0   FULL/  -        00:00:33    10.0.0.1        Serial1/0.100

After a few more tests, I had to conclude that the Control Plane Protection using host subinterface does not work on OSPF packets (and it might does not work on other non-TCP/UDP traffic either). Consequently you cannot protect your router from a DoS attack coming through an interface on which you have to run OSPFTo filter non-TCP/UDP traffic, use the aggregate path control plane protection.

The “fallback global” VRF option does not exist in Cisco IOS

Cheng sent me an interesting question:

I'm reading your book MPLS and VPN Architectures and I've found the ip vrf forwarding name fallback global command in the “Additional Lookup in the Global Routing Table” section. I can only find this command in Junos, but not in IOS.

… and he was right. When we were writing the book, we described several features that were still in development as it looked like they would be in the production code by the time the book was published. Many of them made it into the public IOS releases (for example, the Carrier's Carrier architecture), but some of them (like this command) simply vanished from the surface.

However, it looks like the engineers that switched from Cisco to Juniper took the concept with them and implemented it in JunOS, so JunOS has this feature but IOS doesn't.

This article is part of You've asked for it series.

A bug in the IOS “section” filter

The section filter of the show commands contains a nasty bug (at least in the IOS release 12.4T) in IOS release 12.2SRC: if a line in the section matches the same regular expression as the section header, the rest of the section is not printed.I guess this sounds a bit perplexing, so here's an example. When using the router bgp regular expression in a section filter appended to the show running command, the whole BGP configuration is displayed:

PE-A#show run ¦ section router bgp
router bgp 65000
 template peer-policy Internal
  send-community both
 exit-peer-policy
 !
 template peer-session Internal
  remote-as 65000
  update-source Loopback0
 exit-peer-session
 !
 no synchronization
 bgp log-neighbor-changes
 neighbor 10.0.1.5 inherit peer-session Internal
 neighbor 10.0.1.5 description PE-C(RR)
 neighbor 10.0.1.5 inherit peer-policy Internal
 no auto-summary
 !
 address-family vpnv4
  neighbor 10.0.1.5 activate
  neighbor 10.0.1.5 send-community extended
 exit-address-family

However, if you use bgp as the regular expression, the printout starts with the router bgp command, but stops abruptly after the first line containing the string bgp, skipping the rest of the section:

PE-A#show run ¦ section bgp
router bgp 65000
 template peer-policy Internal
  send-community both
 exit-peer-policy
 !
 template peer-session Internal
  remote-as 65000
  update-source Loopback0
 exit-peer-session
 !
 no synchronization
 bgp log-neighbor-changes

Web Citation Archive

The blogging world is an interestingly dynamic place ... sometimes a page would vanish moments after you would mention it in a blog post :) Google Cache might be a short term solution, but you could use a much better tool: Webcite gives you the ability to store a snapshot of a web page at the time you've cited it.

IOS scripting with Tcl available on almost all platforms

The Tcl scripting (invoked with tclsh command) is now available on almost all IOS-based platforms (apart from the low-end Catalyst switches). For high-end distributed platforms, use the 12.2(33)SRC release, for Catalyst 6500 the 12.2(33)SXH release. On most other platforms, you can use mainstream 12.4 release.

You can find more Tclsh-related information in the Tclsh on Cisco IOS tutorial. Sample Tclsh scripts are available in the Tclsh script library. If you need expert help in planning, developing or deploying Tclsh scripts in your network, contact the author.

Labor day

Central Europe has a venerable tradition of celebrating Labor Day (May 1st in this part of the world) by not working … and while this might sound like an oxymoron, we definitely have to respect traditions, don't we? See you on Monday :)