EEM violates RFC 822

Justin sent me an interesting problem description:

Emails that come from EEM don't have a body, it just puts anything after the body keyword in the subject.

I quickly wrote a simple EEM applet ...

event manager applet MailTest
 event none
 action 1 mail server "10.0.0.10" to "c@d" from "a@b" →
  subject "eeee" body "ffff"

... and started it with event manager run MailTest. The simple SMTP server I'm using for debugging EEM printed out the following message content:

Incoming mail ... from 10.0.0.1 received
From: <a@b>
To:   <c@d>

Date: Fri, 01 Mar 2002 00:01:34 +0000
Message-ID: <00002002000135003@R1>
From: a@b
To: c@d
Subject: eeee
ffff

As you can see, there is no empty line between the Subject header and the message body. A quick look in RFC 822 confirmed that there should be an empty line separating headers from the message body:

message =  fields *( CRLF *text )

Conclusion: EEM is not RFC822-compliant e-mail client.

Hint to EEM developers: if it works with sendmail, it's not necessarily correct.

This article is part of You've asked for it series.

DNS views work with EEM

If you've tried to use DNS view/view-list configuration commands in EEM applets, the applets failed as the EEM did not recognize DNS-specific configuration prompts (you could work around this problem with the prompt option of the action cli command). This bug was fixed in 12.4(15)T5, now you can configure DNS views from EEM.

Disable optional IOS features on high CPU load

One of my readers has submitted an interesting EEM applet in a comment to the Generate SNMP trap on high CPU load post. The applet monitors the CPU load (using SNMP variable from the CISCO-PROCESS-MIB) and disables WCCP when the 1-minute average load exceeds 75%. You can change the thresholds or disable/enable other IOS features by modifying the applet's source code.

Change the source IP address of an EEM SMTP session

I've got the following question from Levi:

I have a Tcl script that is used in conjunction with EEM to send email whenever the amount of CRC errors on a particular interface increases above a certain threshold. My problem is that the router uses the IP of the outgoing interface as the source IP when it communicates with the SMTP server. This particular interface happens to have a private IP. There's another interface with a public IP and I wanted to know how to get the router to use the public IP on the other interface when it's sending email generated by the TCL script.

There are several ways to solve this problem. If you use Tcl, you could write your own SMTP client and use the -myaddr parameter in the socket call to specify the source IP address. Those of us who prefer EEM applets are not so lucky, you have to use NAT to change the source IP address before the packet is sent toward the SMTP server.

This article is part of You've asked for it series.

Generate SNMP trap on high CPU load

Gernot Nusshall has asked an interesting question:

How could I configure the EEM to send an SNMP trap when the cpu load (interval=30sec) is higher than 30%?

My first solution was to enable resource policy traps with the snmp-server enable traps resource-policy, but this feature was introduced in 12.4(15)T and I am not sure everyone is willing to run the latest-and-greatest IOS code. Furthermore, it looks like the traps are sent only for resource policies defined through the ERM MIB; I was not able to generate a trap from a manually configured resource policy. Obviously it was time for another EEM applet.The EEM version 2.0 (available in 12.2S, 12.3T and 12.4) includes the action snmp-trap command, which can generate a trap from an EEM applet. To generate CPU utilization traps, configure the desired resource policy and an EEM applet that is triggered on the ERM policy event. The simplest EEM applet would just report a change in ERM policy …

event manager applet ReportHighCPU
 event resource policy "HighGlobalCPU"
 action 1.0 snmp-trap strdata "High CPU"

… but as the applet would be run on rising and falling events, it would make sense to include a few _resource_* environment variables in the SNMP trap data. Last but not least, don't forget to enable EEM traps with the snmp-server enable traps event-manager configuration command.

This article is part of You've asked for it series.

Create structured e-mails from EEM applets

A few weeks ago I've described how to use the append show filter and more command to send e-mails containing multiple printouts from an EEM applet. A few hours after I've published the post, David Houser sent me a great EEM applet that used texts stored in flash: files to generate headings between various show commands. While his solution works perfectly (and gives you all the flexibility you want), it's a bit verbose and requires lots of small files that clutter your flash: memory. I've thus decided to write a small Tcl script that executes the Cisco IOS command specified in the command line and appends the command results together with a heading in an output file.

Cookbook: collect router printouts in an EEM applet

Quite often, you'd like to include results of various show printouts as the body of the e-mail sent by an EEM applet. It's easy to include a single printout: execute the show command with the action cli command and include the $_cli_result variable in the e-mail body (more details can be found in the Send email from EEM applet article).

It's trickier to include outputs from several show commands in the same e-mail: you have to collect the outputs into a file in the router's flash, list the contents of the file and send the result.

Cookbook: send e-mail from EEM applet

I've described the action mail command that allows you to send e-mails from EEM applets in several posts. However, the posts were always focused on a particular task and the various action mail options were mentioned in passing. The Send e-mail from EEM applet article contains a step-by-step recommendations on portable and scalable usage of the action mail command.

Use EEM to respond to ERM events

In a previous post, I've described how you can detect high CPU load with the Embedded Resource Manager (ERM). If you want to respond to these events, you could use the syslog event detector within EEM, but it's more reliable to use the new event resource detector available in EEM version 2.2 (introduced in IOS release 12.4(2)T). The resource detector is best used in Tcl policy; if you use it in EEM applet, the same applet is triggered every time a resource policy threshold (minor/major/critical, rising or falling) is crossed. Within the EEM applet it's almost impossible to detect which threshold was crossed.

However, even EEM applet could solve some immediate problems. For example, if you want to store a snapshot of processes on a TFTP server every time the global CPU load crosses a policy threshold, you could use the following applet:

event manager applet ReportHighCPU
 event resource policy "HighGlobalCPU"
 action 1.0 cli command "show process cpu sorted 5sec | redirect tftp://10.0.0.10/highCPU$_resource_time_sent.txt"

To differentiate the snapshots, I've appended the _resource_time_sent variable set by the EEM before the applet is started to the file name, guaranteeing that the snapshot files will have unique names (at least until the router reload).

As an alternative, you could send the show process output in an e-mail:

event manager environment _ifDown_rcpt admin@lab.com
!
event manager applet ReportHighCPU
 event resource policy "HighGlobalCPU"
 action 1.0 cli command "show process cpu sorted 5sec"
 action 1.1 info type routername
 action 2.0 mail server "mail-gw" →
    to "$_ifDown_rcpt" from "$_info_routername@lab.com" →
    subject "CPU @ $_resource_current_value" →
    body "$_cli_result"

This article is part of You've asked for it series.

How do I detect router restarts?

Mike Nipp has wondered which syslog message to use to reliably detect router reload under all circumstances:

The problem I had with the SYS-5-RESTART message is I don't think you will get one if the power is suddenly pulled from the router. It does do a SNMP-5-COLDSTART and SYS-6-BOOTTIME on boot up.

I did an actual power-cycle test of a router (we can do that remotely in our labs, so I didn't have to touch the box :) and the SYS-5-RESTART message is reliably generated at every startup, be it from the power cycle or the reload command (I was not able to provoke an on-demand crash ;).

This article is part of You've asked for it series.

Copy file to an FTP server with EEM applet

cpmf14 has left an interesting comment documenting how to perform a periodic back up of a file in router's flash to an FTP server:

event manager applet backup-crl
 event timer watchdog time 86400 maxrun 4294967295
 action 1.0 cli command "enable"
 action 2.0 cli command "copy flash:/iosca.crl ftp://username:passwd@a.b.c.d/" pattern "a.b.c.d"
 action 3.0 cli command "a.b.c.d" pattern "iosca.crl"
 action 4.0 cli command "iosca.crl"
 action 5.0 syslog msg "FTP backup successful"

Time-based BGP policy routing

Petr Lapukhov describes an interesting scenarion in his post BGP Time-Based Policy Routing: a multi-homed customer that uses one upstream link (for example, more reliable but slower one) during the work hours, switching to the other upstream link (faster, less reliable) after that.

He uses BGP communities to achieve the switch (perfect solution if your ISP supports them) and time-based ACL in a route-map to set the community based on time-of-day. As Cisco changed the way BGP imports local routes in IOS release 12.3T, he then devises an ingenious solution based on reliable static routing to trigger a change in the IP routing table.

The optimum solution is way simpler: you just configure two EEM applets to perform clear ip route network command at appropriate times.Here is a tested configuration that changes BGP community of a locally sourced route one minute after the time-range changes state:

router bgp 65001
 network 172.18.1.0 mask 255.255.255.0 route-map TimeBasedCommunity
!
ip access-list extended MatchWorkingHours
 permit ip any any time-range WorkDay
!
time-range WorkDay
 periodic weekdays 9:00 to 16:59
!
route-map TimeBasedCommunity permit 10
 match ip address MatchWorkingHours
 set community 10:25 additive
!
route-map TimeBasedCommunity permit 15
!
event manager applet WeekdayStart
 event timer cron name "WeekdayStart" cron-entry "1 9 * * 1-5"
 action 1.0 cli command "enable"
 action 2.0 cli command "clear ip route 172.18.1.0"
!
event manager applet WeekdayEnd
 event timer cron name "WeekdayEnd" cron-entry "1 17 * * 1-5"
 action 1.0 cli command "enable"
 action 2.0 cli command "clear ip route 172.18.1.0"

The cron-entry parameter in the event timer cron command specifies that the applet is run one minute after 9AM or 5PM (the reverse time/date order makes it particularly confusing) on every day of every month (the two asterisks) but only on weekdays (the 1-5 parameter for the day-of-week).

I've decided to create two EEM applets to be able to perform different actions on the start/stop events. If you don't need that functionality, just merge them into a single applet; the cron-entry then becomes "1 9,17 * * 1-5".

Environment variables set by EEM applet action commands

I've finally found the EEM reference documentation that specifies the side effects (changes in environment variables) of all action commands. You can use the changed environment variables in subsequent action commands by prefixing the variable name with the $ sign (similar to the EEM applet where I've included router's name in an outgoing e-mail).

Fix bugs in EEM action cli implementation

Every now and then, EEM applets fail to recognize a new configuration prompt generated by the router and abort due to timeout (or hang-up forever if you're using IOS release prior to 12.4(15)T). You can use the new pattern keyword of the action cli configuration command to fix the bug.

For example, the DNS view configuration is not recognized by the EEM code, so the following applet fails to complete:

event manager applet Test
 event none
 action 1.0 cli command "enable"
 action 1.1 cli command "configure terminal"
 action 1.2 cli command "ip dns view default"
 action 1.3 cli command "dns forwarder 10.0.0.2"

… as you can test quite easiliy with the EEM CLI debugging (note the highlighted times that indicate the EEM applet timeout) …

Rtr#event man run Test
:13.343: %HA_EM-6-LOG: Test : DEBUG(cli_lib) : : CTL : cli_open called.
:13.451: %HA_EM-6-LOG: Test : DEBUG(cli_lib) : : OUT :
:13.455: %HA_EM-6-LOG: Test : DEBUG(cli_lib) : : OUT : GW-B>
:13.459: %HA_EM-6-LOG: Test : DEBUG(cli_lib) : : IN  : GW-B>enable
:13.499: %HA_EM-6-LOG: Test : DEBUG(cli_lib) : : OUT :
:13.499: %HA_EM-6-LOG: Test : DEBUG(cli_lib) : : OUT : GW-B#
:13.499: %HA_EM-6-LOG: Test : DEBUG(cli_lib) : : IN  : GW-B#configure terminal
:13.519: %HA_EM-6-LOG: Test : DEBUG(cli_lib) : : OUT :
:13.519: %HA_EM-6-LOG: Test : DEBUG(cli_lib) : : OUT : Enter configuration commands, one per line.  End with CNTL/Z.
:13.523: %HA_EM-6-LOG: Test : DEBUG(cli_lib) : : OUT : GW-B(config)#
:13.523: %HA_EM-6-LOG: Test : DEBUG(cli_lib) : : IN  : GW-B(config)#ip dns view default
:33.395: %HA_EM-6-LOG: Test : DEBUG(cli_lib) : : OUT :
:33.399: %HA_EM-6-LOG: Test : DEBUG(cli_lib) : : OUT : GW-B(cfg-dns-view)#
:33.403: %HA_EM-6-LOG: Test : DEBUG(cli_lib) : : CTL : cli_close called.

To fix this bug, use the pattern "#" option of the action cli command to tell the EEM applet what prompt to expect:

event manager applet Test
 event none
 action 1.0 cli command "enable"
 action 1.1 cli command "configure terminal"
 action 1.2 cli command "ip dns view default" pattern "#"
 action 1.3 cli command "dns forwarder 10.0.0.2" pattern "#"

Telnet/SSH session cannot be started from EEM applet

The chances that you would be able to start SSH or Telnet session from an EEM applet were pretty slim, but the comment from melwong triggered my curiosity and I simply had to try it. After all, as the action cli command uses a VTY line (like a regular user session), you might be able to use the pattern option of the action cli command to write something similar to an expect script. This was my best shot at getting it done:

event manager applet SSH
 event none
 action 0.9 cli command "enable"
 action 1.0 cli command "ssh -l ssUser R2" pattern "word:"
 action 1.1 cli command "ssPassword" pattern "#"
 action 2.0 cli command "clear ip route *" pattern "#"
 action 3.0 cli command "exit" pattern "#"

My applet got past the SSH authentication (debugging on R2 confirmed that the SSH session was started) but could not send data through the session itself (it hung on the clear ip route command).

This article is part of You've asked for it series.

Configure the default route based on the presence of a BGP session

You've probably already heard the phrase "When the only tool you have is a hammer, everything looks like a nail" (and seen people acting according to it). Likewise, if you have an IOS release with EEM support, a lot of things that would require smart design could be solved in a brute-force way with a few EEM applets. For example, the problem of the BGP default route could be solved “easily” with a few applets that track syslog messages reporting when the BGP neighbors go up/down.I've set up the following scenario:

• My router has two BGP neighbors: 10.0.7.2 and 10.0.7.6;
• Internet access through10.0.7.2 is the primary path;
• The default route through 10.0.7.6 should be used as a backup only.

The solution shown below is probably a bit over-engineered, as it would be sufficient to track solely the availability of the primary BGP peer and insert/remove the primary static default route (leaving the floating one intact) … or you could use yet another floating default route as the backup-of-last-resort. It's important, though, that you remove the default routes when the router is restarted, as there will be no BGP-related syslog messages if the BGP neighbor is not available after the reload.

event manager applet BGP_A_Up
 event syslog pattern "BGP-5-ADJCHANGE.*10.0.7.2 Up"
 action 1.0 cli command "enable"
 action 1.1 cli command "configure terminal"
 action 1.2 cli command "ip route 0.0.0.0 0.0.0.0 10.0.7.2"
 action 2.0 syslog msg "Primary BGP peer available"
event manager applet BGP_A_Down
 event syslog pattern "BGP-5-ADJCHANGE.*10.0.7.2 Down"
 action 1.0 cli command "enable"
 action 1.1 cli command "configure terminal"
 action 1.2 cli command "no ip route 0.0.0.0 0.0.0.0 10.0.7.2"
 action 2.0 syslog msg "Primary BGP peer lost"
event manager applet BGP_B_Up
 event syslog occurs 1 pattern "BGP-5-ADJCHANGE.*10.0.7.6 Up" period 20
 action 1.0 cli command "enable"
 action 1.1 cli command "configure terminal"
 action 1.2 cli command "ip route 0.0.0.0 0.0.0.0 10.0.7.6 250"
 action 2.0 syslog msg "Alternate BGP peer available"
event manager applet BGP_B_Down
 event syslog pattern "BGP-5-ADJCHANGE.*10.0.7.6 Down"
 action 1.0 cli command "enable"
 action 1.1 cli command "configure terminal"
 action 1.2 cli command "no ip route 0.0.0.0 0.0.0.0 10.0.7.6 250"
 action 2.0 syslog msg "Alternate BGP peer lost"
event manager applet BGP_Restart
 event syslog pattern "SYS-5-RESTART"
 action 1.0 cli command "enable"
 action 1.1 cli command "configure terminal"
 action 1.2 cli command "no ip route 0.0.0.0 0.0.0.0 10.0.7.2"
 action 1.3 cli command "no ip route 0.0.0.0 0.0.0.0 10.0.7.6 250"
 action 2.0 syslog msg "Default routes removed following the system restart"

EEM CLI patterns are not context sensitive

When writing EEM applets or policies that act on CLI commands, keep in mind that the pattern matching is not context sensitive. For example, if you want to disable the reload command and use the EEM applet …

event manager applet NoReload
 event cli pattern "reload" sync no skip yes

… you cannot enter the action x.y reload configuration command any more (or any other command that includes the string reload).

To distinguish the reload command from other appearances of the same string, use the ^reload pattern (reload occuring at the beginning of the line).

Trivia: this actually occured to me when I was testing the setup described in the December IP Corner article. Sometimes we have to learn the hard way :)

Mandatory EEM CLI commands

The action cli commands used in EEM applets as well as the cli* Tcl functions used in EEM Tcl policies open a virtual Telnet session to a VTY line to execute the CLI commands. The first command you have to execute in the EEM applet is thus the enable command to ensure the next commands will be executed with privilege level 15.

You don't have to specify the enable password.

Likewise, if you want to configure the router, the next command to execute is the configure terminal command, followed by the configuration commands.

The actual execution of the EEM CLI commands can be debugged with the debug event manager action cli command. For example, when the following EEM applet is executed …

event manager applet TEST
 event none
 action 1.0 cli command "enable"
 action 1.1 cli command "configure terminal"
 action 2.0 cli command "interface loopback 3"
 action 2.1 cli command "no shutdown"

… it produces this debugging output:

%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : CTL : cli_open called.
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW>
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : IN : GW>enable
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW#
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : IN : GW#configure terminal
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : Enter configuration commands, one per line.
 End with CNTL/Z.
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW(config)#
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : IN : GW(config)#interface loopback 3
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW(config-if)#
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : IN : GW(config-if)#no shutdown
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW(config-if)#
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : CTL : cli_close called.

Execute CLI commands with prompts in EEM

In response to my post about combining Tcl shell with EEM to get around the “no prompts” limitation of EEM action cli command, Xavier proposed using the undocumented pattern option of the action cli command, which changes the string the EEM script is expecting to indicate that the current command has been executed.

By default, the EEM action cli command waits until it receives exec-level prompt from the VTY (Router> or Router#), resulting in an endless wait and aborted EEM applet in IOS release 12.4(15)T (earlier releases would hang a VTY line forever) if a CLI command returns an additional prompt. With the pattern option, you can change the expected reply to whatever prompt the CLI command is outputting.For example, to clear interface counters, you could use the following EEM applet:

event manager applet test
 event none
 action 0.9 cli command "enable"
 action 1.0 cli command "clear counter loopback 0" pattern "confirm"
 action 1.1 cli command "y"

When it's executed with the event manager run test command and the event manager CLI debugging is enabled, you'll see the following printout:

%HA_EM-6-LOG: test : DEBUG(cli_lib) : : CTL : cli_open called.
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT : R1>
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : IN  : R1>enable
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT : R1#
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : IN  : R1#clear counter loopback 0
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : IN  : y
%CLEAR-5-COUNTERS: Clear counter on interface Loopback0 by  on vty0 (EEM:test)
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT : y
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT : R1#
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : CTL : cli_close called.

Can I combine EEM applets with Tcl shell?

When I've been describing the limitations of kron, ??? quickly asked an interesting question: “as I cannot insert extra input keystrokes with EEM applet, can I run a Tcl script from it with the action sequence cli command "tclsh script" command and use the typeahead function call to get around the limitation?” The only answer I could give at that time was “maybe” … and obviously it was time for a more thorough test. The short result is: YES, you can do it (at least in IOS release 12.4(15)T1).

… and here is the long description of the test. I've started by creating a small Tcl script (see below) that clears the counters on Loopback 0. As the clear counters command requires keyboard input and generates a syslog message, it was a perfect test case.

typeahead "y"
exec "clear counter loop 0"

I've copied this script into the flash:tcl/clearL0.tcl and tested it:

R1#tclsh flash:tcl/clearL0.tcl
%CLEAR-5-COUNTERS: Clear counter on interface Loopback0 by console

So far, so good. Next, I've created an EEM applet with no trigger …

event manager applet Clear
 event none
 action 1.0 cli command "enable"
 action 1.1 cli command "tclsh flash:tcl/clearL0.tcl"

… enabled the EEM CLI debugging and started it:

R1#debug event man action cli
Debug EEM action cli debugging is on
R1#event man run Clear
R1#
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : CTL : cli_open called.
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT : R1>
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : IN  : R1>enable
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT : R1#
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : IN  : R1#tclsh flash:tcl/clearL0.tcl
%CLEAR-5-COUNTERS: Clear counter on interface Loopback0 by  on vty0 (EEM:Clear)
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT : R1#
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : CTL : cli_close called.

Great. It works. Let's move on: I've inserted a trigger into the EEM applet that ran the applet when an OSPF neighbor reached FULL adjacency:

event manager applet Clear
 event syslog pattern "OSPF-5-ADJCHG.*to FULL"
 action 1.0 cli command "enable"
 action 1.1 cli command "tclsh flash:tcl/clearL0.tcl"

And now for the final test: after I've enabled the serial interface, OSPF neighbors established adjacency …

R1(config-if)#interface ser 1/0
R1(config-if)#no shutdown
R1(config-if)#
%LINK-3-UPDOWN: Interface Serial1/0, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0, changed state to up
%OSPF-5-ADJCHG: Process 1, Nbr 10.0.1.2 on Serial1/0 from LOADING to FULL, Loading Done
%CLEAR-5-COUNTERS: Clear counter on interface Loopback0 by  on vty0 (EEM:Clear)

… and the counters on Loopback0 were cleared. Test completed :)

This article is part of You've asked for it series.