How do I detect router restarts?

Mike Nipp has wondered which syslog message to use to reliably detect router reload under all circumstances:

The problem I had with the SYS-5-RESTART message is I don't think you will get one if the power is suddenly pulled from the router. It does do a SNMP-5-COLDSTART and SYS-6-BOOTTIME on boot up.

I did an actual power-cycle test of a router (we can do that remotely in our labs, so I didn't have to touch the box :) and the SYS-5-RESTART message is reliably generated at every startup, be it from the power cycle or the reload command (I was not able to provoke an on-demand crash ;).

This article is part of You've asked for it series.

EEM syslog messages look like debugging messages

If you use EEM applets with action syslog commands, the messages generated by those commands will be displayed in the same format as the debugging messages, regardless of their severity. This might be a problem if you decide to use different timestamps for debugging and regular syslog messages (or disable debugging timestamps completely).For example, the following configuration ...

service timestamps debug uptime
service timestamps log datetime
 
event manager applet Serial_Down
 event track 1 state any
 action 1.0 syslog priority errors msg "IP routing on Serial 0/1/0 is $_track_state"

... produces these messages after an interface failure ...

Aug 14 11:18:32: %LINK-3-UPDOWN: Interface Serial0/1/0, changed state to up
1d16h: %HA_EM-3-LOG: Serial_Down: IP routing on Serial 0/1/0 is up

Logging to flash disk

Cisco IOS release 12.4(15)T brought (among a plethora of voice features) the logging to non-volatile storage, a nice-sounding name for the ability to write syslog messages into files on your flash memory (or an embedded disk, if you have one). To configure it, use the logging persistent [url directory] [size filesystem-size] [filesize logging-file-size] global configuration command:

• The directory argument specifies where you want the files to be stored (for example, flash:/logging).
• The filesystem-size specifies the maximum disk space the logging files can consume (once you exceed the limit, the oldest file is deleted)
• The logging-file-size parameter specifies the maximum size of each file (once the file grows too large, a new file is created).

Note: You can store the log files on the router's flash memory if it appears as a disk file system (check with the show file systems command). Wouldn't it be great if this feature would also work on USB drives ...

Cisco IOS Login Enhancements

Cisco has in IOS release 12.3(4)T (integrated into 12.4) finally introduced features (long available in Unix and Windows) that slow down dictionary attacks on a router. On top of logging of login failures, you can also slow down the login process by delaying the router response after a login failure with the login delay seconds command.

On top of that, the you can configure the router to enter quiet mode after several login failures have been detected in specified timeframe with the login block-for seconds attempts tries within seconds configuration command.The quite mode is implemented by applying an access-list to the VTY lines. You can specify the access-list yourself with the login quiet-mode access-class { acl-name-or-number } command, otherwise the router generates an access-list named sl_def_acl ...

Extended IP access list sl_def_acl
10 deny tcp any any eq telnet log
20 deny tcp any any eq www log
30 deny tcp any any eq 22 log
40 permit tcp any any eq 22 log

and applies it with the access-class in command to the VTY lines.

Warning: If you save router configuration during the quiet period, the access-class command will be saved to NVRAM.

The router also logs the entering and exiting of the quiet mode with log messages similar to the ones below:

1d03h: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: ] [Source: 192.168.0.6] [localport: 23] [Reason: Login Authentication Failed] at 19:20:17 UTC Sat Dec 2 2006
1d03h: %SEC_LOGIN-1-QUIET_MODE_ON: Still timeleft for watching failures is 43 secs, [user: ] [Source: 192.168.0.6] [localport: 23] [Reason: Login Authentication Failed] [ACL: sl_def_acl] at 19:20:17 UTC Sat Dec 2 2006
1d04h: %SEC_LOGIN-5-QUIET_MODE_OFF: Quiet Mode is OFF, because block period timed out at 19:21:17 UTC Sat Dec 2 2006

Enhanced OSPF adjacency logging

The log-adjacency-changes OSPF configuration command got improved (IOS documentation claims it's happened in release 12.1) with the detail command that logs every step of OSPF adjacency establishment (sample printout below). Great troubleshooting tool :)

%OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.21 on Serial0/0/0.100 from DOWN to INIT, Received Hello
%OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.21 on Serial0/0/0.100 from INIT to 2WAY, 2-Way Received
%OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.21 on Serial0/0/0.100 from 2WAY to EXSTART, AdjOK?
%OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.21 on Serial0/0/0.100 from EXSTART to EXCHANGE, Negotiation Done
%OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.21 on Serial0/0/0.100 from EXCHANGE to LOADING, Exchange Done
%OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.21 on Serial0/0/0.100 from LOADING to FULL, Loading Done

Remove timestamps from syslog messages

The ability to replace router uptime with date and time in the logging messages with the service timestamps log datetime command was present in IOS for a long time, but I was always annoyed at timestamps when collecting syslog messages for demonstration purposes. The command to turn them off has also been available "forever", but was too obvious for me to try out ... the no service timestamps log command.

Log terminal access to your router

In a previous post, I've shown how you can log the changes in interactive user's privilege level. With the Cisco IOS Login Enhancements (introduced in IOS release 12.3(4)T, integrated in 12.4), you can also log all login successes and failures, even when using local user database (a similar functionality was previously achievable only when using central TACACS+ or RADIUS server).

The configuration commands to enable terminal access logging are login on-success log and login on-failure log. You can also specify that you want send SNMP traps in these circumstances (with the trap option) or that you only want to log every Nth attempt with the every n option.After you've configured terminal access logging, the router will start to generate syslog messages similar to the ones below (localport: 23 indicates the user was using telnet to access the router, localport: 80 that she was using HTTP):

%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: test] [Source: 172.16.1.1] [localport: 23] at 19:10:27 UTC Sat Dec 2 2006
1d04h: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: a] [Source: 192.168.0.10] [localport: 80] [Reason: Login Authentication Failed - BadPassword] at 19:35:53 UTC Sat Dec 2 2006

If the user accesses the router through the console port, both the source and localport are set to all zeroes:

%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: ] [Source: 0.0.0.0] [localport: 0] at 19:10:48 UTC Sat Dec 2 2006

Router Configuration Management … Too Good to be True?

In the Router Configuration Management … Too Good to be True?, the latest IP Corner article, I'm describing two of the router configuration management features introduced in Cisco IOS release 12.4: Configuration Change Notification and Contextual Configuration Diff utility. While the first one behaves as expected, the second one produced unexpected results under the stress test.

Log user privilege level changes

The logging userinfo configuration command introduced in IOS release 12.3T (integrated into 12.4) starts logging of all changes in user privilege levels (as requested by enable or disable commands).
Sample printout:

fw>enable
Password:
03:00:50: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by unknown on console
fw#disable
fw>
03:00:52: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 1 by unknown on console

Log configuration commands entered on your Cisco router

As part of Configuraton Change Notification and Logging feature, Cisco IOS stores the most recent configuration commands in a circular buffer and (optionally) sends them to syslog streams.

This feature is configured under the archive configuration mode with the log config command, which brings you to yet another configuration mode where you can fine-tune the parameters (they are obvious, on-router help is sufficient), for example:

archive
 log config
  logging enable 100
  notify syslog
  hidekeys

After you've enabled configuration command logging, you can use the show archive log config all command to inspect the logging buffer. You can also display commands entered in a particular session or by a selected user.

If you've configured notify syslog, every configuration command also triggers a syslog message similar to this one:

3d03h: %PARSER-5-CFGLOG_LOGGEDCMD: User:console logged command:interface loopback 0

Note: This feature logs only the configuration commands, if you want to log all commands, use TACACS+ or Embedded Event Manager.

CLI command logging without TACACS+

The AAA architecture of Cisco IOS contains a number of very useful features, including the ability to authorize and log every CLI command executed on the router. Unfortunately, the AAA command accounting only supports TACACS+ as the AAA transport protocol, making it unusable in environments using RADIUS.

You can use Embedded Event Manager as a workaround. The following configuration commands will log every command executed on the router.

event manager applet CLIaccounting
event cli pattern ".*" sync no skip no
action 1.0 syslog priority informational msg "$_cli_msg"
set 2.0 _exit_status 1

The log messages generated by this EEM applet have the following format:

%HA_EM-6-LOG: CLIaccounting: command

As the EEM uses standard IOS logging mechanisms, you can use the show logging command to examine the command execution history or store the messages on a syslog server.

Note: As a side effect, all commands executed on a router will be echoed to the router's console, unless you disable console logging with no logging console command or use TCL-based syslog filters (more about them in an upcoming post).

Running syslog over TCP

Syslog has always been considered an undependable means of reporting network problems by serious network administrators as it runs over unreliable UDP transport. Sometime in the twilight zone between IOS releases 12.3T and 12.4, Cisco IOS got the capability to transport syslog messages over TCP with the logging host ip-address transport tcp port configuration command (the command is documented in 12.4 manuals but missing in 12.3T manuals).

Note: IOS implements standard syslog stream over TCP, not the more complex RFC 3195.

To support syslog over TCP, you also need TCP-capable syslog server. In Unix environments, you can use syslog-ng, on Windows, Kiwi syslog daemon is a perfect choice.

Note: to enable syslog over TCP in Kiwi Syslog Daemon, go to File/Setup/Inputs/TCP, click Listen for TCP Syslog messages and enter the desired TCP port number.

Subinterface link status logging

If you're still stuck with frame-relay connections (or use them in test environment, because it's easy to set up any-to-any connectivity between a larger number of routers), you were probably used to subinterface logging events reporting when the line protocol on a point-to-point subinterface would go up or down based on LMI DLCI status.

Very quietly, these logging events disappeared, first on 7500-series routers in IOS release 12.1(14), now they're gone by default on all platforms. If you still want to see what's going on with your frame-relay subinterfaces, you have to enter logging event subif-link-status configuration command on every subinterface.

I can only guess that some people that used the syslog events for network management were very surprised by the first (undetected) frame-relay failure following an IOS upgrade :)