Follow-up: P-to-P router encryption

The “P-to-P router encryption” post has generated numerous comments. One of the readers suggested using dedicated Ethernet encryption devices, which is probably the best option if you’ve realized you need encryption in the network acquisition phase when there’s still some budget left (too bad the vendor recommended in the comments does not want to admit how expensive the boxes are).

However, assuming you have high-speed IPSec encryption modules and you have to implement P-to-P encryption in existing network, the only option left to you is GRE tunnel. Here’s why:

IP and MPLS are distinct protocols at the boundary between layer-2 and layer-3 (let’s forget the question whether MPLS is a layer-3 or layer-2.5 protocol at the moment). If you want to transport both of them across a shared connection, you have to include the layer-2 protocol ID (L2-PID) in the payload to ensure the receiving router is able to distinguish between incoming IP packets (which need FIB lookup) and MPLS packets (which need LFIB lookup).

As described in the previous post, IPSec encryption works only on IP packets. If you want to transport multiprotocol data (IP and non-IP traffic) across an IPSec-encrypted session, you need to encapsulate them into IP datagrams using an encapsulation method that includes L2-PID. The only such method supported in Cisco IOS is GRE.

One of the readers suggested using Virtual Tunnel Interface (VTI). The VTI is just another conceptual implementation of IPSec (using point-to-point links instead of crypto maps). It does not introduce non-IPSec encapsulation headers and thus cannot transport L2-PID and multiprotocol data. Although you can configure mpls ip on the VTI interface, it doesn’t work.

GET VPN is completely unusable for our task; it’s a scalable way of key distribution and crypto map creation. It offers no encapsulation mechanism and works only for IP traffic.

Last but not least, someone started walking down the Rube Goldberg path and proposed adding extra boxes between the P-routers and using MPSL-over-GE-over-L2TPv3-over-IPSec-over-GE-(over-VPLS). The extra devices (your Cisco AM would love this option) and different encapsulation method (L2TPv3 instead of GRE) add unnecessary complexity … unless, of course, you cannot implement hardware encryption on your P-router, in which case it’s comparable to the “dedicated Ethernet encryption devices” proposal.

DHCP logging in Cisco IOS is a nightmare

One of the readers sent me an interesting question: he’d like to know the IP address of his home router (to be able to connect to it from the office), but its IP address is assigned through DHCP and changes occasionally.

I wanted to solve the problem by hooking an EEM applet onto the DHCP-6-ADDRESS_ASSIGN syslog message. No good; as it turns out, Cisco IOS generates the logging message only when a DHCP-acquired IP address is assigned to an interface without one. If the IP address is changed via DHCP, the change is not logged.

One could understand the faulty programmers’ logic if the initial address assignment would be different from the address change, but DHCP is such a simple protocol that any change in client’s IP address requires the client to enter the INIT state, so acquiring a new IP address is no different from changing an existing one. I guess they had to take special precautions not to log the address change (and ensure we have another interesting challenge to chew on).

Fortunately, the IP routing table changes after every change in interface IP address … more about that in a few days.

Deploying IPv6 in Enterprise Networks

I was invited to present my views on the IPv6 deployment in enterprise networks during the local IPv6 summit. Instead of joining the cheering few or the dubious crowds, I’m trying to present a realistic view answering questions like “what do I have to do”, “when should I start” and “where should I focus my efforts”.

Here’s the outline of my presentation, any feedback, additional thoughts or insightful critique is most welcome.

Background information

Scenario: Enterprise network connected to the Internet. No need for internal IPv6 (RFC 1918 is good enough).

Question: where shall I focus my IPv6 efforts?

Facts of life:

• IPv6 is a reality, get used to it.
• Migration is supposed to be easy, but you will get stuck on details.
• Start small, but start now.

Phases of public IPv6 deployment:

Phase#1: Dual stack content (starting now)

Phase#2: IPv6-only Internet clients (in a few years)

Phase#3: IPv6-only major content providers (10+ years from now)

Obviously this is just my perception of the critical milestones, as they apply to enterprise network deployment.

Proposed action plan

Phase#1 has already started, get ready for it:

• Establish IPv6 connectivity with all the upstream providers
• Deploy IPv6 on your public servers. Start with small, non-critical applications to get hands-on experience.
• Change your whole DMZ into dual-stack DMZ.

As an enterprise network, you don't care about Phase #2:

• Your content is reachable over IPv4 and IPv6
• Interesting content is reachable over IPv4.
• Use this time to plan your internal IPv6 deployment.

When the public content becomes available only over IPv6 (phase #3) you might be in a morass if your internal network is not yet dual-stack (you’ll have to face ugly 4to6 NAT). Deploy dual-stack throughout your network:

• RFC1918 + 4to4 NAT
• Public IPv6 address space

Encrypting P-to-P-router traffic

Rob sent me a really good question:

I have an enterprise MPLS network. Two P routers are connected via carrier point-to-point Gigabit Ethernet and I would like to encrypt the MPLS traffic traversing the GE link. The PE-routers don't have hardware crypto accelerators, so I would like to keep the MPLS within the buildings running in cleartext and only encrypt the inter-site (P-to-P) MPLS traffic.

The only solution I could imagine would nicely fit the motto of one of our engineers: »Any time you have a problem, use more GRE tunnels« (if you have a better solution, please post it in the comments).

As far as I understand Cisco's IPSEC implementation, IOS can encrypt only IP traffic. If you want to encrypt MPLS frames forwarded by a P-router, you have to convert them back to IP ... and the only way to convert MPLS frames back into IP without losing the whole label stack which is vital to proper operation of MPLS VPN is to encapsulate them in a GRE tunnel and encrypting the MPLS-over-GRE traffic.

The "obvious" problem of this setup (apart from performance hit) is the MTU issue: the MTU on the GE link has to be high enough to support all the extra overhead on top of the 1500-byte payload. If that's not the case, you have to lower the MTU on the tunnel interface to ensure the GRE packets are not fragmented - that would kill the receiving router.

There are also "a few" intricacies of using MPLS-over-GRE/IPSec when the P-router is a 6500 or 7600. Don't ask me what they are, I'm a conceptual person and try to stay away from the specific hardware implementations, but our engineers built several large networks using this architecture and got a lot of hands-on experience. If you feel you could benefit from their expertise, get in touch with our Professional Services team.

The tunneling Kool-Aid

In a comment to my Data Center Interconnect article my friend Ronald (who once certified me as a Bovine Professional) wrote:

I don't drink this Cisco Kool Aid about interconnecting data centres using an IP backbone. Rather use FC directly over DWDM instead of FCIP on MPLS.

This time I could agree with him wholeheartedly ... assuming you already have DWDM gear (or infinite budget to buy some) and you can get dark fiber when and where you need it. Unfortunately not everyone is so lucky and/or rich, so we have to compromise.

His comment also reminded me of the comments I’ve heard from SNA gurus more than 15 years ago when Cisco introduced SDLC-over-IP and later LLC2-over-IP tunneling. We’ve spent years trying to persuade them to drink the DLSw Kool-Aid … and in the end they all gave up their precious SDLC leased lines and Front-End Processors as the market realities forced them to consolidate their infrastructure and migrate to routed networks. Not to mention that most of the gear they supported (including ATM machines) got replaced by newer models supporting IP (or even having IP as the only connectivity option).

A similar story played out in the Service Provider world, where the idea of offering Frame Relay or ATM service on top of an IP backbone was a heresy … until it became cheaper to migrate the legacy services to a router-based network than to pay the maintenance fees for the Frame Relay gear.

Moral of the story: whenever you propose to tunnel a well-established protocol using a new entrant in a particular technology area, people will tell you that it’s all a marketing plot and that they prefer to stick with the tested scenario … until the business pressures force them to change their mind. The tricky part is figuring out the right time to jump: if you’re too early, you could get burnt; if you’re too late, you might have lost a fortune.

Last but not least, don’t forget that not all tunneling scenarios were worth pursuing: IP-over-APPN never really took hold.

Fishing for free information: the ultimate experience

A while ago the amount of queries I’ve been receiving has reached a threshold where I felt the need to be very honest about the type of questions I will answer (after all, we’re in business of providing networking-related services and if I want to continue blogging there has to be some revenue to pay the bills). Some people don’t mind and still send me requests for free information they need to implement the projects they’re paid to do. Recently I’ve got this shopping list …

My customer is asking me to do a reverse DNS entry in the DNS for his domain. But I do not have any DNS.

1. What is the suggested DNS server hardware for an ISP?
2. What is the procedure to be followed to inform the rest of the Internet that I own this pool?
3. Do I need to register my company name and the allocated pool with any of IANA accredited registrars?
4. Shall we use a single IP for registration or will our entire IP address pool be used?
5. If I ask the registrar to make forward and reverse entry in their root server, do I also need to make the same entries in my DNS server?
6. Is it mandatory to maintain master and slave DNS servers from the beginning? Do I need to allocate one of the allocated public IP to these servers and inform the register to make entries for this DNS server?

This is precisely the type of work that we’re best at: helping our clients plan and design their implementation, so I’ve politely offered our professional services. This is the reply I’ve received:

I appreciate your business mind. But I do feel, you need to spare some knowledge for free of cost like this.

Go figure …

Carrier Ethernet service from customer’s perspective

As the Carrier Ethernet services are becoming more popular, people are starting to wonder how to use it in a router-based network. I’ve got the following question from one of my readers:

I was wondering if it was possible to design a redundant network where the core uses L2 MPLS, the provider edge uses L2 for access but the customer edge equipment uses L3 Routers. We don't want to customer to see any STP at their routers.

Of course you can do that. There are two scenarios to consider:

(A) The Service Provider is offering point-to-point Ethernet service (pseudowire). In this case, two of the customer routers would be connected with what looks like a point-to-point Ethernet link. Usually the remote site would have just one "outside" Ethernet connection while the hub site would have numerous links bundled in a trunked (VLAN) link.

(B) The SP is offering VPLS service. In this case, all customer routers appear as being connected to the same Ethernet segment.

In both cases, the customer edge (CE) routers should treat the SP Ethernet link as a simple LAN segment, in case (A) connecting two routers, in case (B) connecting many routers.

Expired DHCP lease bounces the interface

You would think that an expired DHCP lease is not a big deal for a DHCP client. Although the interface IP address is lost, you can always try to get a new address from the DHCP server.

IOS has a different opinion: when the DHCP lease expires on a router configured with ip address dhcp interface configuration command, the interface is administratively shut down and re-enabled. Here’s a sample printout taken from a router running 12.4(24)T software:

%DHCP-5-RESTART: Interface FastEthernet0/1 is being restarted by DHCP
%LINK-5-CHANGED: Interface FastEthernet0/1, changed state to administratively down
%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down
%LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up
%DHCP-6-ADDRESS_ASSIGN: Interface FastEthernet0/1 assigned DHCP address 10.17.1.15, mask 255.255.255.0, hostname Client

You might wonder how you could ever end up with an expired lease when there’s a working DHCP server on the network. It’s simple if your DHCP server runs on IOS: when you clear the DHCP bindings on a router running a DHCP server, it stops responding to lease extension requests (DHCPREQUEST packets) from unknown clients.

Back from vacation

Some of you might have noticed that I was somewhat quieter than usual the last few days … and I had an excellent reason: finally I’ve managed to sneak a week of climbing vacations into my schedule. There’s a fantastic eco lodge (chalet) in a nice valley between Chamonix and Geneva and they’re offering combined climbing-yoga classes.

If you’re interested in one or the other, you simply have to try them … and in the meantime you can enjoy a few pictures I’ve taken during the week.

Display the rejected BGP routes

Jernej sent me an interesting question: “does Cisco IOS have an equivalent to the Extremeware’s show bgp neighbor a.b.c.d rejected-routes command which displays all routes rejected by inbound filters?”

Short answer: it doesn’t.

Somewhat longer explanation

If you want to display routes rejected by an inbound BGP filter, you have to store every route you ever received from the BGP neighbor, increasing the memory consumption of your BGP process. You can do that in Cisco IOS if you configure neighbor a.b.c.d soft-reconfiguration in.

Configuring per-neighbor inbound soft reconfiguration significantly increases BGP memory consumption. On top of BGP routes inserted in the main BGP table, the BGP process has to store every route received from the BGP neighbor in a neighbor-specific table (two copies of the accepted routes are needed because an inbound route-map might have changed the route attributes).

Workaround

Use this procedure to find rejected routes:

• Within the BGP process configuration, enter neighbor a.b.c.d soft-reconfiguration in (this command might clear the BGP session).
• Populate the per-neighbor table with the clear ip bgp a.b.c.d soft in.
• Display the routes received from the neighbor with the show ip bgp neighbor a.b.c.d received-routes.
• Display the routes sent by the neighbor and accepted by the inbound BGP filters with the show ip bgp neighbor a.b.c.d routes.
• Do a diff of the two printouts (if you’ll write a short Tclsh script that does that, please feel free to send it to me or submit it in the comments).

Cisco IOS is not an authoritative DHCP server

Imagine you have to move your DHCP clients to a different range within the same IP subnet. Can you do it if you run the DHCP server on a router running Cisco IOS? Sure, there’s the ip dhcp excluded-address command.

Not so fast … the ip dhcp excluded-address command does not affect the existing bindings. Sounds weird? It is. You tell the router to avoid some of the IP addresses in the DHCP pool and it will happily extend the leases for those IP addresses, it just won’t allocate new IP addresses from the excluded range. To change the IP address assignment of the existing clients, you have to clear the DHCP bindings with the clear ip dhcp binding command.

OK, so you clear the bindings. The next time the clients try to extend the lease, their requests will be rejected. Wrong – Cisco IOS is not an authoritative DHCP server. It ignores DHCPREQUEST messages coming from unknown clients in a correct subnet.

Read the DHCP client address change article in the CT3 wiki