Catching all syslog messages

If you use UDP-based syslog servers, you might have noticed that they miss a message or two generated during a router reload (particularly when the syslog server is on a directly connected LAN). The reason is simple - when the first message is sent to the syslog server, its MAC address is not yet in the router's ARP cache and the message is dropped. To prevent the message loss, you can use the logging server-arp configuration command (introduced in IOS release 12.3T), making sure that the router sends ARP request to the configured syslog server(s) before generating the first syslog messages.

Note: this issue does not occur when using TCP as the syslog transport mechanism.

Log IP SLA failures

The IP Service Level Agreement (SLA) measurement feature of Cisco IOS is extremely SNMP-oriented. For example, there is no easy way to generate a logging message when an SLA operation fails (or exceeds the threshold). However, Embedded Event Manager reading SNMP variables can help you get there.Before checking the status of an SLA operation, you need to schedule it (remember the SLA entry#, you'll need it when forming the SNMP object ID). For example:

ip sla 42
 icmp-echo 172.16.1.2 source-interface Serial0/0/0.100
 timeout 500
 threshold 100
 frequency 1
ip sla schedule 42 life forever start-time now

When the SLA measurement has been defined, create an EEM applet to monitor the status of the rttMonCtrlOperTimeoutOccured variable in the CISCO-RTT-MON MIB (note that the SLA entry# is the last number in the OID). The EEM applet triggers on value 1 (true: timeout) and re-arms on value 2 (false: no timeout).

event manager applet SLA_Timeout
event snmp oid 1.3.6.1.4.1.9.9.42.1.2.9.1.6.42 get-type exact entry-op eq entry-val 1 exit-op eq exit-val 2 poll-interval 5
action 1.0 syslog msg "Ping to 172.16.1.2 timeout"

Similarly, you can write an applet that triggers on SLA response time being over threshold by monitoring SNMP OID 1.3.6.1.4.1.9.9.42.1.2.9.1.7.sla# (the rttMonCtrlOperOverThresholdOccurred variable).
Note: this article is part of You've asked for it series.

Reporting a state of an IP route

Have you ever wanted to be notified when your router loses a specific route (for example, the default route toward the Internet)? In IOS release 12.4T you can do it with a combination of Enhanced Object Tracking and Embedded Event Manager 2.2.As the first step, you configure a track object that will monitor the state of an IP route with the track ip route command.

track 100 ip route 0.0.0.0 0.0.0.0 reachability

After the track object has been created, you can monitor its state with an EEM applet (in the configuration example it only generates a syslog message, but you could also send an email to the network administrator).

event manager applet TrackRoute_0.0.0.0
 event track 100 state any
 action 1.0 syslog msg "route is $_track_state"

And now a test run ...

a1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
a1(config)#ip route 0.0.0.0 0.0.0.0 loopback 0
a1(config)#^Z
01:08:47: %SYS-5-CONFIG_I: Configured from console by console
01:08:51: %HA_EM-6-LOG: TrackRoute_0.0.0.0: route is up
a1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
a1(config)#interface loopback 0
a1(config-if)#shutdown
01:09:10: %SYS-5-CONFIG_I: Configured from console by console
01:09:11: %LINK-5-CHANGED: Interface Loopback0, changed state to administratively down
01:09:21: %HA_EM-6-LOG: TrackRoute_0.0.0.0: route is down

As you can see from the example, there is a significant delay before the track object responds to a route change. You can adjust this delay with the track timer ip route seconds global configuration command (default is 15 seconds).

Cisco IOS Login Enhancements

Cisco has in IOS release 12.3(4)T (integrated into 12.4) finally introduced features (long available in Unix and Windows) that slow down dictionary attacks on a router. On top of logging of login failures, you can also slow down the login process by delaying the router response after a login failure with the login delay seconds command.

On top of that, the you can configure the router to enter quiet mode after several login failures have been detected in specified timeframe with the login block-for seconds attempts tries within seconds configuration command.The quite mode is implemented by applying an access-list to the VTY lines. You can specify the access-list yourself with the login quiet-mode access-class { acl-name-or-number } command, otherwise the router generates an access-list named sl_def_acl ...

Extended IP access list sl_def_acl
10 deny tcp any any eq telnet log
20 deny tcp any any eq www log
30 deny tcp any any eq 22 log
40 permit tcp any any eq 22 log

and applies it with the access-class in command to the VTY lines.

Warning: If you save router configuration during the quiet period, the access-class command will be saved to NVRAM.

The router also logs the entering and exiting of the quiet mode with log messages similar to the ones below:

1d03h: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: ] [Source: 192.168.0.6] [localport: 23] [Reason: Login Authentication Failed] at 19:20:17 UTC Sat Dec 2 2006
1d03h: %SEC_LOGIN-1-QUIET_MODE_ON: Still timeleft for watching failures is 43 secs, [user: ] [Source: 192.168.0.6] [localport: 23] [Reason: Login Authentication Failed] [ACL: sl_def_acl] at 19:20:17 UTC Sat Dec 2 2006
1d04h: %SEC_LOGIN-5-QUIET_MODE_OFF: Quiet Mode is OFF, because block period timed out at 19:21:17 UTC Sat Dec 2 2006

Count the logging messages

I haven't figured out why I would need this particular IOS feature (IOS documentation says you can at least get a glimpse of what was happening before the logging buffer wrapped over), but it's such a cool one I simply have to mention it ... starting with release 12.2(8)T and 12.3, IOS can tabulate the occurence of each log message.You configure this feature with the logging count global configuration command and inspect its results with the show logging count command:

a1#show logg count
Facility       Message Name      Sev Occur      Last Time
============================================================
SYS            CONFIG_I          5    4 *Jan 16 11:32:57.130
-------------  -------------- ------------------------------
SYS TOTAL                             4

OSPF           ADJCHG            5    2 *Jan 16 11:31:26.434
-------------  -------------- ------------------------------
OSPF TOTAL                            2

Enhanced OSPF adjacency logging

The log-adjacency-changes OSPF configuration command got improved (IOS documentation claims it's happened in release 12.1) with the detail command that logs every step of OSPF adjacency establishment (sample printout below). Great troubleshooting tool :)

%OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.21 on Serial0/0/0.100 from DOWN to INIT, Received Hello
%OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.21 on Serial0/0/0.100 from INIT to 2WAY, 2-Way Received
%OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.21 on Serial0/0/0.100 from 2WAY to EXSTART, AdjOK?
%OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.21 on Serial0/0/0.100 from EXSTART to EXCHANGE, Negotiation Done
%OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.21 on Serial0/0/0.100 from EXCHANGE to LOADING, Exchange Done
%OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.21 on Serial0/0/0.100 from LOADING to FULL, Loading Done

How do I stop all logging done by the router

Although you probably don't ever want to stop all router's logging activities (at the very minimum you should collect the messages in a memory buffer with the logging buffered command), the global configuration command to do it is no logging on.

Note: this article is part of You've asked for it series.

Disable console logging

Large amount of logging output (most often produced in a debugging process) sent to a router's console can significantly increase the router's CPU load and even stop the box from forwarding packets (high-end routers with distributed forwarding architecture are obviously an exception, but even they can lose routing adjacencies). The reason is very simple - console interrupt is one of the highest-priority interrupts on the router (otherwise you wouldn't be able to get a response to the BREAK key on a hung box).

Update January 9th 2007: The router does not check if a user is logged into the console port or a device (for example, a terminal) is attached to it; if console logging is enabled, messages are always sent to the console port (causing CPU load).

To stop the console logging, use the no logging console global configuration command (highly recommended for routers that are not usually accessed through the console port) or you might want to limit the amount of messages sent to the console with the logging console level configuration command (for example, logging console notifications).

Note: this article is part of You've asked for it series.

Remove timestamps from syslog messages

The ability to replace router uptime with date and time in the logging messages with the service timestamps log datetime command was present in IOS for a long time, but I was always annoyed at timestamps when collecting syslog messages for demonstration purposes. The command to turn them off has also been available "forever", but was too obvious for me to try out ... the no service timestamps log command.

Configuration Change Logging ignores the configuration downloads

The Configuration Change Notification and Logging feature is supposed to log changes to the router's configuration. While it does a great job of logging commands entered in the router configuration mode, it completely ignores configuration changes due to configuration download (for example, with configure network or copy tftp running-config command).Here is an example:

fw#configure terminal
fw(config)#archive
fw(config-archive)#log config
fw(config-archive-log-cfg)#logging enable
fw(config-archive-log-cfg)#^Z
fw#
fw#configure network
Host or network configuration file [host]?
This command has been replaced by the command:
         'copy  system:/running-config'
Address or name of remote host [10.0.0.2]?
Source filename [fw-confg]?
Configure using tftp://10.0.0.2/fw-confg? [confirm]
Loading fw-confg from 10.0.0.2 (via FastEthernet0/0): !
[OK - 858 bytes]
fw#
%SYS-5-CONFIG_I: Configured from tftp://10.0.0.2/fw-confg by console
fw#show archive log config all
 idx   sess           user@line      Logged command
    1     1        console@console    logging enable

Log terminal access to your router

In a previous post, I've shown how you can log the changes in interactive user's privilege level. With the Cisco IOS Login Enhancements (introduced in IOS release 12.3(4)T, integrated in 12.4), you can also log all login successes and failures, even when using local user database (a similar functionality was previously achievable only when using central TACACS+ or RADIUS server).

The configuration commands to enable terminal access logging are login on-success log and login on-failure log. You can also specify that you want send SNMP traps in these circumstances (with the trap option) or that you only want to log every Nth attempt with the every n option.After you've configured terminal access logging, the router will start to generate syslog messages similar to the ones below (localport: 23 indicates the user was using telnet to access the router, localport: 80 that she was using HTTP):

%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: test] [Source: 172.16.1.1] [localport: 23] at 19:10:27 UTC Sat Dec 2 2006
1d04h: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: a] [Source: 192.168.0.10] [localport: 80] [Reason: Login Authentication Failed - BadPassword] at 19:35:53 UTC Sat Dec 2 2006

If the user accesses the router through the console port, both the source and localport are set to all zeroes:

%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: ] [Source: 0.0.0.0] [localport: 0] at 19:10:48 UTC Sat Dec 2 2006

Router Configuration Management … Too Good to be True?

In the Router Configuration Management … Too Good to be True?, the latest IP Corner article, I'm describing two of the router configuration management features introduced in Cisco IOS release 12.4: Configuration Change Notification and Contextual Configuration Diff utility. While the first one behaves as expected, the second one produced unexpected results under the stress test.

Log user privilege level changes

The logging userinfo configuration command introduced in IOS release 12.3T (integrated into 12.4) starts logging of all changes in user privilege levels (as requested by enable or disable commands).
Sample printout:

fw>enable
Password:
03:00:50: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by unknown on console
fw#disable
fw>
03:00:52: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 1 by unknown on console

Log configuration commands entered on your Cisco router

As part of Configuraton Change Notification and Logging feature, Cisco IOS stores the most recent configuration commands in a circular buffer and (optionally) sends them to syslog streams.

This feature is configured under the archive configuration mode with the log config command, which brings you to yet another configuration mode where you can fine-tune the parameters (they are obvious, on-router help is sufficient), for example:

archive
 log config
  logging enable 100
  notify syslog
  hidekeys

After you've enabled configuration command logging, you can use the show archive log config all command to inspect the logging buffer. You can also display commands entered in a particular session or by a selected user.

If you've configured notify syslog, every configuration command also triggers a syslog message similar to this one:

3d03h: %PARSER-5-CFGLOG_LOGGEDCMD: User:console logged command:interface loopback 0

Note: This feature logs only the configuration commands, if you want to log all commands, use TACACS+ or Embedded Event Manager.

CLI command logging without TACACS+

The AAA architecture of Cisco IOS contains a number of very useful features, including the ability to authorize and log every CLI command executed on the router. Unfortunately, the AAA command accounting only supports TACACS+ as the AAA transport protocol, making it unusable in environments using RADIUS.

You can use Embedded Event Manager as a workaround. The following configuration commands will log every command executed on the router.

event manager applet CLIaccounting
event cli pattern ".*" sync no skip no
action 1.0 syslog priority informational msg "$_cli_msg"
set 2.0 _exit_status 1

The log messages generated by this EEM applet have the following format:

%HA_EM-6-LOG: CLIaccounting: command

As the EEM uses standard IOS logging mechanisms, you can use the show logging command to examine the command execution history or store the messages on a syslog server.

Note: As a side effect, all commands executed on a router will be echoed to the router's console, unless you disable console logging with no logging console command or use TCL-based syslog filters (more about them in an upcoming post).

Running syslog over TCP

Syslog has always been considered an undependable means of reporting network problems by serious network administrators as it runs over unreliable UDP transport. Sometime in the twilight zone between IOS releases 12.3T and 12.4, Cisco IOS got the capability to transport syslog messages over TCP with the logging host ip-address transport tcp port configuration command (the command is documented in 12.4 manuals but missing in 12.3T manuals).

Note: IOS implements standard syslog stream over TCP, not the more complex RFC 3195.

To support syslog over TCP, you also need TCP-capable syslog server. In Unix environments, you can use syslog-ng, on Windows, Kiwi syslog daemon is a perfect choice.

Note: to enable syslog over TCP in Kiwi Syslog Daemon, go to File/Setup/Inputs/TCP, click Listen for TCP Syslog messages and enter the desired TCP port number.

Subinterface link status logging

If you're still stuck with frame-relay connections (or use them in test environment, because it's easy to set up any-to-any connectivity between a larger number of routers), you were probably used to subinterface logging events reporting when the line protocol on a point-to-point subinterface would go up or down based on LMI DLCI status.

Very quietly, these logging events disappeared, first on 7500-series routers in IOS release 12.1(14), now they're gone by default on all platforms. If you still want to see what's going on with your frame-relay subinterfaces, you have to enter logging event subif-link-status configuration command on every subinterface.

I can only guess that some people that used the syslog events for network management were very surprised by the first (undetected) frame-relay failure following an IOS upgrade :)