Showing posts with label network management. Show all posts
Showing posts with label network management. Show all posts

SNMP v3 users not shown in running-config

Ralf sent me a SNMPv3 question:

If I create a SNMPv3 user which has a password (snmp-server user userthree groupthree v3 auth md5 user3passwd), this user does not appear in the running- or startup-config. Cisco even documents this if you know what to look for.

I strongly suspect (although I did not test this) that these users are also missing from configuration exported to TFTP servers. What would be the recommended way to make usable config backups of routers with such users?
Like certificates, the SNMPv3 users are stored in private-config and thus never appear in the router configuration. If you want to have a backup of the user data, create a text file on one of your NMS servers, add SNMPv3 usernames and passwords in the text file and use the copy somewhere running-config to configure SNMPv3 users on the routers.
This article is part of You've asked for it series.

Introduction to Embedded Menu Manager

One of the great new features introduced in IOS release 12.4(20)T is the Embedded Menu Manager (EMM), which allows you to build dynamic hierarchical menus with a combination of static definitions in XML and Tcl scripts. The Cisco documentation on this feature is quite cryptic, so I wrote an introduction to EMM, describing the major XML elements and the ways to use it.

Read more in CT3 wiki.

Disable optional IOS features on high CPU load

One of my readers has submitted an interesting EEM applet in a comment to the Generate SNMP trap on high CPU load post. The applet monitors the CPU load (using SNMP variable from the CISCO-PROCESS-MIB) and disables WCCP when the 1-minute average load exceeds 75%. You can change the thresholds or disable/enable other IOS features by modifying the applet's source code.

Control Plane Protection logging does not work on transit subinterface

When I was trying to test how the router running IOS release 12.4(15)T5 classifies inbound IP packets into various CPPr subinterfaces, I wanted to use the log action in the MQC classes I've defined. This approach worked perfectly for the host and cef-exception interface (I've even seen ARP packets logged), but the packets classified as transit generated no log messages. While this makes perfect sense (after all, all punted packets are processed by the transit service-policy), the IOS should generate a warning when you apply a policy-map with the log option as service-policy on the control-plane transit interface.

IOS auto-upgrade

I've noticed the IOS auto-upgrade functionality when the IOS software release 12.4(15)T was launched, but it was missing from the 1800 images, so I wrote a note in the "to-test" folder and forgot about it. In the meantime, the code obviously appeared in IOS images, as Joe Harris managed to get the auto-ugprade from CCO to work. However, the IOS documentation lacks "a few" details, while Joe's post has a step-by-step explanation.

Another way to generate SNMP trap on high CPU load

Yesterday ago I've described how you can use the ERM functionality together with an EEM applet to generate SNMP traps whenever the CPU load exceeds predefined thresholds. When testing this solution, I started to wonder what the snmp-server enable traps cpu threshold command does. After lenghty conversation with uncle Google and Cisco documentation, I found that there's another way to detect and report high CPU load in Cisco IOS: the CPU threshold notification introduced in IOS release 12.3T (and Tassos pointed that out before I had the time to write a post about it :).To use this feature, you have to configure the thresholds with the process cpu threshold configuration command and enable related SNMP traps with the snmp-server enable traps cpu threshold. For example, to send SNMP traps whenever the total CPU load measured over a 30-second interval exceeds 40%, use the following configuration:

snmp-server enable traps cpu threshold
process cpu threshold type total rising 40 interval 30

Generate SNMP trap on high CPU load

Gernot Nusshall has asked an interesting question:

How could I configure the EEM to send an SNMP trap when the cpu load (interval=30sec) is higher than 30%?
My first solution was to enable resource policy traps with the snmp-server enable traps resource-policy, but this feature was introduced in 12.4(15)T and I am not sure everyone is willing to run the latest-and-greatest IOS code. Furthermore, it looks like the traps are sent only for resource policies defined through the ERM MIB; I was not able to generate a trap from a manually configured resource policy. Obviously it was time for another EEM applet.The EEM version 2.0 (available in 12.2S, 12.3T and 12.4) includes the action snmp-trap command, which can generate a trap from an EEM applet. To generate CPU utilization traps, configure the desired resource policy and an EEM applet that is triggered on the ERM policy event. The simplest EEM applet would just report a change in ERM policy …
event manager applet ReportHighCPU
 event resource policy "HighGlobalCPU"
 action 1.0 snmp-trap strdata "High CPU"

… but as the applet would be run on rising and falling events, it would make sense to include a few _resource_* environment variables in the SNMP trap data. Last but not least, don't forget to enable EEM traps with the snmp-server enable traps event-manager configuration command.

This article is part of You've asked for it series.

Use UDP flood to increase router's CPU load

If you want to test the ERM policies in a controlled environment, it's almost mandatory to have tools that allow you to overload the router. One of these tools is the UDP flood: if you flood a router's IP address, you're guaranteed to raise the CPU to 100%, with majority of the process CPU being used by the IP Input process (the interrupt CPU load will also be significant).

This phenomenon illustrates very clearly why it's so important to have inbound access lists protecting the router's own IP addresses on all edge interfaces.

If you want to stress-test the router's forwarding functionality, you could use the host route to the null0 interface; all packets sent to that IP address will be CEF-switched, so the only impact of the UDP flood to the unreachable IP address will be the increased interrupt CPU load. I was able to increase the interrupt CPU load to close to 50% on a 2800 router using a virtual PC with a Fast Ethernet interface.

And just in case you need it, here is the configuration from my test router. All packets sent to 10.0.0.22 are CEF-switched and dropped (the CPU load from the IP input process is negligible).

interface FastEthernet 0/0
ip address 10.0.0.1 255.255.255.0
!
ip route 10.0.0.22 255.255.255.255 null 0

Predefine your own Tcl functions

If you want to have your own Tcl functions available when you start tclsh, you could use the scripting tcl init file configuration command that I've briefly mentioned in one of the previous posts. This command specifies a source file that is executed every time you start Tcl shell. The source file can contain function definitions, package declarations or any other Tcl code.

If you need to, you can specify multiple initialization files.

For example, if you'd like to implement a comfortable Tcl-based pinger (similar to the one Ethan Banks found in the Sadikhov forums, store the following Tcl code into the file flash:pinger.tcl

proc pinger { iplist } {
  foreach ip $iplist {
    if { [regexp "(!!!)" [exec "ping $ip timeout 1" ]] } {
      puts "$ip"
    } else { puts "$ip **** failed ***" }
  }
}
… and configure scripting tcl init flash:pinger.tcl. Now you can ping a number of hosts in a single operation:
R1#tclsh
R1(tcl)#pinger { 10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4 }
10.0.0.1
10.0.0.2
10.0.0.3 **** failed ***
10.0.0.4 **** failed ***

When “copy” actually means “merge”

Marcus Jensen asked me a very interesting question:

I want to send 3 lines of configuration to a remote router, but I know the first line will kill my connection. Can I save these 3 lines of code to a text file, and then issue a Tcl command to add those to the running config?

The solution is much simpler and does not have to involve Tcl at all. The copy something system:running-config command merges the configuration commands in the source file with the current running configuration.

You can store the configuration commands you want to execute in a local file (even in NVRAM) or you could execute them directly off a file server (using HTTP, FTP, TFTP or SCP protocol).

This article is part of You've asked for it series.

Display open TCP and UDP ports

With the introduction of Control Plane Policing features (available from 12.3(4)T), you can easily inspect all the open ports (servers and clients) on a router with the show control-plane host open-ports command, resulting in a printout very similar to the netstat -a printout on a Unix/Windows workstation.For example, on the router where I've configured BGP, HTTP server, NTP and DHCP, this command produces the following output (a session to a BGP neighbor as well as a telnet session was established):

R1#show control-plane host open-ports
Active internet connections (servers and established)
Prot Local Address Foreign Address Service State
 tcp *:23 *:0 Telnet LISTEN
 tcp *:80 *:0 HTTP CORE LISTEN
 tcp *:179 *:0 BGP LISTEN
 tcp *:179 10.0.7.2:43962 BGP ESTABLIS
 tcp *:23 10.0.7.2:18036 Telnet ESTABLIS
 udp *:67 *:0 DHCPD Receive LISTEN
 udp *:68 *:0 BootP client LISTEN
 udp *:123 *:0 NTP LISTEN
Notes:
  • This show command does not display non-TCP/UDP servers (OSPF, EIGRP, RSVP) or even some UDP-based services (RIP).
  • Although I was considering writing about CPP for a long time, Artur Szymanski was the one that brought this command to my attention. Thanks!

OSPF graceful shutdown

I've described the OSPF Graceful Shutdown functionality in an earlier post and got an excellent question about the implications of using OSPF stub router functionality with external routes.

The more I was digging into this issue, the more design questions I've got … and finally ended up writing a whole IP Corner article about it. You can read the in-depth discussion of design and implementation aspects of OSPF stub router functionality in the December IP Corner article: Bring your Network Closer to Five Nines with Graceful Shutdown.

Enhanced show interfaces command

It's amazing how many options (most of them still undocumented) the show interfaces command accepts in IOS release 12.4T (I won't even start guessing when each one was introduced, if you're running old IOS releases, please feel free to comment):

  • show interfaces description displays interface names, L1 and L2 status (line and line-protocol status) and interface description. Extremely handy if you want to check which interfaces are up/down.
  • show interfaces counters protocol status displays the L3 protocols active on each interface.
  • show interfaces summary displays the state of various interface queues and related drop counters in a nice tabular format.
  • show interfaces accounting displays per-protocol in/out counters.
Here are a few sample printouts:
a1#show interfaces description
Interface Status Protocol Description
Fa0/0 up up Central LAN
Fa0/1 admin down down
Se0/0/0 up up Frame Relay
Se0/0/0.100 up up Link to B1
Se0/1/0 admin down down
Se0/1/1 admin down down
Lo0 up up
 
a1#show interfaces counters protocol status
Protocols allocated:
 FastEthernet0/0: Other, IP, DEC MOP, ARP, CDP
 FastEthernet0/1: Other, IP
 Serial0/0/0: Other, IP, CDP
 Serial0/1/0: Other, IP
 Serial0/1/1: Other, IP
 Loopback0: Other, IP
 VoIP-Null0: Other, IP
 
a1#show interfaces summary
 
 *: interface is up
 IHQ: pkts in input hold queue IQD: pkts dropped from input queue
 OHQ: pkts in output hold queue OQD: pkts dropped from output queue
 RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)
 TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)
 TRTL: throttle count
 
  Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL
------------------------------------------------------------------
* FastEthernet0/0 0 0 0 0 0 0 0 0 0
  FastEthernet0/1 0 0 0 0 0 0 0 0 0
* Serial0/0/0 0 0 0 0 0 0 0 0 0
* Serial0/0/0.100 - - - - - - - - -
  Serial0/1/0 0 0 0 0 0 0 0 0 0
  Serial0/1/1 0 0 0 0 0 0 0 0 0
* Loopback0 0 0 0 0 0 0 0 0 0
NOTE:No separate counters are maintained for subinterfaces
     Hence Details of subinterface are not shown
 
a1#show interfaces accounting
FastEthernet0/0 Central LAN
      Protocol Pkts In Chars In Pkts Out Chars Out
         Other 0 0 490 29400
            IP 2737 216847 3052 424422
       DEC MOP 0 0 8 616
           ARP 5 316 12 720
           CDP 82 30914 84 29563
Interface FastEthernet0/1 is disabled
 
Serial0/0/0 Frame Relay
      Protocol Pkts In Chars In Pkts Out Chars Out
         Other 0 0 490 6370
            IP 515 43748 1034 87608
           CDP 83 26477 168 55272

Kron: poor-man's cron

When two groups within Cisco needed time-based command execution in Cisco IOS, they (in a typical big-corporation fashion) decided to implement the same wheel from two different sets of spokes and rims. One group built the Embedded Event Manager with its event timer cron command (introduced in 12.2(25)S and 12.3(14)T), the other group created the more limited kron command set (introduced in 12.3(1)).

EEM is almost a perfect superset of kron, both can trigger a set of CLI commands at reload, at periodic intervals or at certain time in the future. The only extra functionality offered by kron is the ability to specify a different username for each event (whereas all EEM applets have to run under a common username) … and kron is available in older IOS releases.

Similar to EEM applets, CLI commands executed within kron cannot expect extra input (so you cannot execute clear counters or reload from kron) and the output they generate is lost unless you use output filters to redirect it to an external file.

Here is a simple configuration that archives the router's running configuration every sunday half an hour before midnight:

kron policy-list archiveConfig
 cli archive config
!
kron occurrence archiveConfig at 23:30 Sun recurring
 policy-list archiveConfig

Download router configurations via TFTP

In a previous post, I've described how you can turn your router into a TFTP server. As you can configure the router to serve any file residing on it, you can also pull startup and running configuration from it with TFTP, providing that you configure:

tftp-server nvram:startup-config
tftp-server system:running-config

Warning: Due to total lack of any security features in TFTP protocol, use this functionality only in lab environment.

Send an e-mail when an interface goes down

John S. Pumphrey recently asked an interesting question: “Can the router send an e-mail when an interface goes down?” The enterprisey solution is obvious: deploy a high-end EMS to collect SNMP traps and use its API to write a custom module that would use a MQ interface to alert the operator. Fortunately, Event Manager applets in Cisco IOS provide action mail command (available in 12.3(14)T and 12.4) that can send an e-mail to a SMTP server straight from the router.

There are two ways you can detect that an interface went down with EEM: either you track the interface status with a track object and start an EEM applet when the track object changes state or you catch the syslog messages reporting that the interface line protocol changed state to down. The second approach is obviously more generic, as a single applet can act on multiple interfaces.

event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1

Notes:

  • If you want to limit the applet to serial interfaces only, you could change the pattern to LINEPROTO-5-UPDOWN.*Serial.*to down.
  • The → continuation character is used to indicate that a single configuration line has been split to increase readability.

The action mail command specifies the mail server's address (use a hostname and DNS lookup or ip host configuration command to make the EEM applet more generic), from and to address, message subject and body. In each of these fields, you can use EEM environment variables that you can define with the event manager environment configuration command. Each EEM event also defines a few environment variables that you can use (see the table of EEM system-defined variables on CCO). For example, you can define the e-mail recipient in the router's configuration and use the _syslog_msg variable to include the syslog message in the e-mail body:

event manager environment _ifDown_rcpt admin@lab.com
!
event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1
 action 1.0 mail server "mail-gw" →
    to "$_ifDown_rcpt" from "R1@lab.com" →
    subject "Interface down on R1" →
    body "$_syslog_msg"

You can make the applet even more generic with the help of action info type routername command, which stores the current router's name into the $_info_routername environment variable:

event manager environment _ifDown_rcpt admin@lab.com
!
event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1
 action 1.0 info type routername
 action 2.0 mail server "mail-gw" →
    to "$_ifDown_rcpt" from "$_info_routername@lab.com" →
    subject "Interface down on $_info_routername" →
    body "$_syslog_msg"

Note: This article is part of You've asked for it series.

SNMP with Tcl

Looking from the outside, it looks like Tcl SNMP routines in Cisco IOS were designed by a commitee or came straight from Dilbert. The snmp_getone function that reads a single SNMP value does not return an array or a list (as one would expect), but a string representation of something that looks like an XML object (but is not, since its attributes are not properly quoted). As Tcl on Cisco IOS has no built-in XML support, parsing the return values is a pure joy (and a nice exercise in writing regular expressions).

The following excerpt of a telnet session shows how to extract a single SNMP value in Tcl (I've used extra steps and an interactive tclsh session for illustration purposes). The SNMP community has to be configured in advance with the snmp-server community test ro configuration command.

rtr#tclsh
rtr(tcl)#set value [snmp_getone test system.3.0]
{<obj oid='sysUpTime.0' val='14886'/>}
rtr(tcl)#regexp -inline {oid='(.*)'.*val='(.*)'} $value
{oid='sysUpTime.0' val='14886'} sysUpTime.0 14886
rtr(tcl)#regexp {oid='(.*)'.*val='(.*)'} $value ignore oid result
1
rtr(tcl)#puts $result
14886
And now for a complete example: the following script prints the router uptime.
#
# Simple Tcl script to print system uptime
#
set value [snmp_getone test system.3.0]
regexp {oid='(.*)'.*val='(.*)'} $value ignore oid result
set result [expr $result / 100]
puts "Router uptime is $result seconds"

Turn your flash card into an ATA drive

The flash memory available in newer router platforms (at the very minimum the ISR routers and 37xx series) is capable of being used as a regular disk drive (for example, to store system logging information), but it might be formatted as a traditional Low-End File System (LEFS) flash card (more likely if the router was not manufactured recently). To change the flash card format to disk-like FAT32 format, use the format flash: privileged-level command (and don't forget to store the IOS image to another location before formatting the flash). After the format process is complete, you can create subdirectories on the flash: memory and use it as a regular disk device.A sample formatting operation is displayed below:

fw#format flash:
Format operation may take a while. Continue? [confirm]
Format operation will destroy all data in "flash:". Continue? [confirm]
Enter volume ID (up to 64 chars)[default flash]:
Current Low End File System flash card in flash will be formatted into DOS File System flash card!
Continue? [confirm] y

Format: Drive communication & 1st Sector Write OK...
Writing Monlib sectors.
........................................................
Monlib write complete

Format: All system sectors written. OK...

Format: Total sectors in formatted partition: 125297
Format: Total bytes in formatted partition: 64152064
Format: Operation completed successfully.

Format of flash complete

fw#show file system | include flash
* 64012288 27734016 disk rw flash:#

Show IP access lists attached to an interface

When developing yet another Tcl script, I've stumbed across an interesting show command: the show ip access-list interface name introduced in IOS release 12.4(6)T displays the contents of the inbound and outbound IP access-list applied to the specified interface. The really nice part is that the ACL statistics (number of matches displayed next to the ACL lines) are kept and displayed per-interface.For example, this is the printout from one of my test routers:

R2#show ip access-list 101
Extended IP access list 101
10 permit ip any any (1900 matches)
R2#show ip access-list interface tunnel 0
Extended IP access list ICMP in
10 deny icmp any host 10.0.1.2 echo
20 deny icmp any host 10.2.0.2 echo
30 permit ip any any (2279 matches)
Extended IP access list 101 out
10 permit ip any any (10 matches)

Router as a TFTP server

Shaun needed an extra TFTP server in CCNP labs and asked whether you could use a router to act as one. The read-only (download only) TFTP functionality has been available in Cisco IOS for a long time, but the common wisdom was that you could only use the TFTP server function to serve current IOS image.

Fortunately, as of IOS 11.0, the function is more generic; you can serve any file residing on the router (you still cannot upload files), but you have to declare each file to be served with the tftp-server path global configuration command. You could even specify an alias to have the file available under a different name and attach an access list to each configured file to restrict its availability.

Note: This article is part of You've asked for it series.