Cisco’s Data Center megalaunch is probably old news by now. They announced numerous new Nexus products, linecards for Catalyst 6500 (will that box ever retire?), UCS servers, tons of FCoE-related features and few extra goodies I’m most interested in (LISP and MPLS/VPN on Nexus 7000). Scott Lowe and Chad Sakac already wrote extensive reviews of the new products and features, so I’d just like to say: Congratulations! Well done! Keep surprising me with good news ;)
Some of the biggest buyers of the networking gear have decided to squeeze some extra discount out of the networking vendors and threatened them with open-source alternative, hoping to repeat the Linux/Apache/MySQL/PHP saga that made it possible to build server farms out of low-cost commodity gear with almost zero licensing costs. They formed the Open Networking Foundation, found a convenient technology (OpenFlow) and launched another major entrant in the Buzzword Bingo – Software-Defined Networking (SDN).
Networking vendors, either trying to protect their margins by stalling the progress of this initiative, or stampeding into another Wild West Gold Rush (hoping to unseat their bigger competitors with low-cost standard-based alternatives) have joined the foundation in hordes; the list of initial members reads like Who’s Who in Networking.
I’m getting questions like this one all the time: “Where are we with NAT-PT? It was implemented in IOS quite a few years ago but it has never made it into ASA code.”
Bad news first: NAT-PT is dead. Repeat after me: NAT-PT is dead. Got it? OK.
More bad news: NAT-PT in Cisco IOS was seriously broken after they pulled fast switching code out of IOS. Whatever is left in Cisco IOS might be good enough for a proof-of-concept or early deployment trials, but not for a production-grade solution.
Short answer: yes, it does.
During the geeky chat we had just after we’d finished recording the Data Center Fabric Packet Pushers podcast, Kurt (@networkjanitor) Bales asked me whether the MPLS/VPN-over-DMVPN scenarios I’m describing in Enterprise MPLS/VPN Deployment webinar (register here) really work (they do seem a bit complex).
I always test the router configurations I use in my webinars and I usually share them with the attendees. Immediately after registering for the Enterprise MPLS/VPN Deployment webinar you get access to complete sets of router configurations covering 10 scenarios, including five different MPLS/VPN-over-DMVPN designs, so you can easily test them in your lab and verify that they do work. But what about a live deployment?
Tip-of-the-week
The Tip-of-the-Week award goes to Jeremy (@packetlife) Stretch for his excellent Tips for starting an IT blog advice. He’s also experimenting with a three-year-delayed blog feed. Marvelous idea, I have to do something similar.
IPv6
Shortly after my Where's my AAAA record post, RIPE Labs published an excellent explanation of the broken-IPv6 phenomena. Most of the problems are caused by 6to4 tunnels; obviously it’s time to get rid of them (but unfortunately they’re built into operating systems like Windows Vista).
Geoff Huston published a two-part series on transitioning to IPv6. Part 1 describes dual stack and tunnels and part 2 focuses on Service Provider technologies like CGN, DS-Lite, 6rd and 6PE.
Jason Braddy posted an interesting link in one of the comments to my IPv6-to-IPv4 load balancing post: check out Zeus Traffic Manager; supposedly it can rewrite web pages on-the-fly.
"Which Cisco IOS services work in a VRF?" is the question I get in almost any VRF-related discussion, so I made sure it’s covered very early in my Enterprise MPLS/VPN deployment webinar (register here). This is the explanation I usually give in the webinar:
Every Data Center fabric technology has to integrate seamlessly with legacy equipment running the venerable Spanning Tree Protocol (STP) or one of its facelifted incarnations (for example, RSTP or MST). The alternative, called rip-and-replace when talking about other vendors’ boxes or synchronized upgrade when promoting your wares (no, I haven’t heard it yet, but I’m sure it’s coming), is simply indigestible to most data center architects.
TRILL and Cisco’s proprietary Fabric Path take a very definitive stance: the new fabric is the backbone of the network routing TRILL-encapsulated layer-2 frames across bridged segments (TRILL) or contiguous backbone (Fabric Path). Both architectures segment the original STP domain into small chunks at the edges of the network as shown in the following figure:
In every enterprise-focused IPv6 presentation, including my Enterprise IPv6 – the first steps webinar (buy the recording or register for an online session), I’m telling the attendees that they can easily make their legacy applications reachable over IPv6 with a little help from F5 load balancers. After all, Facebook is doing exactly that, so it should work (in theory) ... but as we all know, in practice, the theory and practice are wildly different.
Cisco recently launched two very interesting products: layer-3 routing for the Nexus 5000 switch and the Virtual Security Gateway (which is a fantastic solution that you’ll hear more about in my future posts). Sadly, both products support only IPv4.
I’m in this industry long enough to understand the need for “baby steps” and “focusing on what customers want” (and I know there are hundreds of great engineers within Cisco who know what needs to be done, but still have to read blog posts like this one), but launching critical products without IPv6 support after the IPv4 global address pool has already been depleted definitely doesn’t look futuristic (just for fun, you might want to watch John Chambers talking about Cisco’s IPv6 thought leadership).
Martin sent me an interesting challenge: he needs to connect an HP switch in a blade enclosure to a pair of Catalyst 3500G switches. His Catalysts are not stackable and he needs the full bandwidth between the switches, so he decided to fake the multi-chassis link aggregation functionality by configuring static LAG on the HP switch and disabling STP on it (the Catalysts have no idea they're talking to the same switch):
Advice of the Week
Straight from Scott Adams: true to his (and Dilbert’s) nature, he writes about Happiness Engineering.
Virtualization
Brad Hedlund is back from the corporate overload writing about inverse virtualization (isn’t that a great name for a server farm – Brad has some serious future in corporate marketing if he ever decides to change jobs again ;)
I’ll conclude this week’s IPv6 saga with a fair question I’ve received several times during the last few days: “Where’s your AAAA record?” The snappy answer would be “if you can’t see it, your ISP is not ready for production-grade IPv6”; the reality is a bit more complex.
Let’s assume we’re all past the IPv6 myths phase and know that IPv6 does not offer more (or less) inherent security than IPv4. Will the IPv6 networks be as secure as IPv4 ones? Not necessarily, because we’re lacking feature parity and implementation experience. As I explained in the “IPv6 security issues: Fixing implementation problems” I wrote for SearchTelecom:
Until equipment vendors fill in the gaps and offer true feature parity between IPv4 and IPv6 security features, we can expect the IPv6 networks to be less secure that today’s IPv4 networks -- not because IPv6 is insecure, but because today’s IPv6 implementations still lag behind their IPv4 counterparts.
Read more @ SearchTelecom (or consider the excellent IPv6 Security book by Eric Vyncke).
We know the world will eventually run out of IPv4 addresses, but while at least some service providers got the message and already deployed IPv6, it seems like most enterprise IT departments still practice the denial strategy. It’s worrisome to read articles from Jeff Doyle describing the ignorance of his enterprise clients or Terry Slattery wondering about the incentives of moving to IPv6, so I’ll try (yet again) to explain why you should start IPv6 planning NOW.
I got an interesting question from Andrew: “Would you say that bridge assurance makes UDLD unnecessary? It doesn't seem clear from any resource I've found so far (either on Cisco's docs or on Google).”
It’s important to remember that UDLD works on physical links whereas bridge assurance works on top of STP (which also implies it works above link aggregation/port channel mechanisms). UDLD can detect individual link failure (even when the link is part a LAG); bridge assurance can detect unaggregated link failures, total LAG failure, misconfigured remote port or a malfunctioning switch.
A while ago Matthew Norwood wrote an excellent article describing the troubleshooting process they used to figure out why a particular web application worked way too slowly. Greg Ferro was quick to point out that it doesn’t make sense to assume the network is the problem and work through the whole chain slowly eliminating every potential networking device as the source of the problem when you might be facing an application design issue. However, there’s an even more important consideration: your network troubleshooting toolbox lacks the right troubleshooting tools for this job.
Collected in my Inbox and Twitter feed throughout last week:
To Tell the Truth: Multihop FCoE. An excellent and very precise explanation. It’s so nice to see a vendor’s blogger joining my efforts to take confusion out of multihop FCoE (it probably helps that Cisco finally has a full-blown multihop FCoE code).
One of the interesting problems I was facing in the recent weeks was multi-tenant security. Combine it with fuzzy all-encompassing vapor-based terminology and you have a perfect mix that can fit anything you want to sell. In the Ensuring multi-tenant security in cloud services I wrote for SearchTelecom.com I tried to structure the cloudy visions a bit: let’s figure out which type of service we’re talking about, then we can discuss what security mechanisms make sense.
As you might expect, I find IaaS the most challenging as you’re bound to hit a number of roadblocks, from VLAN limitations to architectural limitations of virtual security appliances.
Chris Pollock from io Networks was kind enough to share yet another method of implementing DHCPv6 prefix delegation on PPP interfaces in his comment to my DHCPv6-RADIUS integration: the Cisco way blog post: if you tell the router not to use the Framed-IPv6-Prefix passed from RADIUS in the list of prefixes advertised in RA messages with the no ipv6 nd prefix framed-ipv6-prefix interface configuration command, the router uses the prefix sent from the RADIUS server as delegated prefix.
This setup works reliably in IOS release 15.0M. 12.2SRE3 (running on a 7206) includes the framed-IPv6-prefix in RA advertisements and DHCPv6 IA_PD reply, totally confusing the CPE.
As expected, the demand for the Building IPv6 Core webinar is slowly diminishing – those ISPs that know what IPv6 is all about are already implementing it. The session on March 10th is thus the last live session in the first half of 2011; if you’d like to attend a live session, now is the time to register for it.
Please note that the webinar is not going away: you’ll still be able to buy the webinar recording or watch the recordings as part of the yearly subscription package.
Last week I described how Cisco IOS uses two RADIUS requests to authenticate an IPv6 user (request#1) and get the delegated prefix (request#2). The second request is sent with a modified username (-dhcpv6 is appended to the original username) and an empty password (the fact that is conveniently glossed over in all Cisco documentation I found).
FreeRADIUS server is smart enough to bark at an empty password, to force the RADIUS server to accept a username with no password you have to use Auth-Type := Accept:
Site-A-dhcpv6 Auth-Type := Accept
cisco-avpair = "ipv6:prefix#1=fec0:1:2400:1100::/56"
Regardless of the current creed preached by Gartner, multi-vendor networks are a fact of life. I’m therefore trying to make my webinars as multi-vendor as possible and as every vendor has its own Data Center Interconnect solutions, I’ve reached out to my various contacts trying to collect as much information as possible while developing the materials for my latest Data Center Interconnect webinar. I was already familiar with Cisco’s DCI solutions and Juniper’s BGP MPLS-based MAC VPN technology, but knew almost nothing about what F5 (BIG-IP with EtherIP) and others are doing. To make matters worse, I had no contacts with F5.
The best-discovery-of-the-week award undoubtedly goes to CCIE Routing and Switching Review Kit: Fantastic 63 pages of mind maps covering all CCIE R&S technologies, from Frame Relay and LAN switching to BGP and MPLS/VPN (link found on The Quest for CCIE blog by @mfp).
The rest of the collection is primarily focused on IPv6. I can’t possibly fathom why that would be the case ...
A few days ago I used Google to search for an article I’d written. My article was among the top results, but there was also another web site with very similar text. I’m used to blockheads publishing content stolen from my RSS feed (which is one of the primary reasons you won’t see a full feed of my blog any time soon), but this guy seemed to be copying the whole articles ... only they sounded somewhat crazy. For example,
Yesterday I described how the IPv6 architects split the functionality of IPCP into three different protocols (IPCPv6, RA and DHCPv6). While the split undoubtedly makes sense from the academic perspective, the service providers offering PPP-based services (including DSL and retrograde uses of PPP-over-FTTH) went berserk.
... became ...
Yesterday we dеѕсrіbеd hοw thе IPv6 architects rip thе functionality οf IPCP іntο 3 odd protocols (IPCPv6, RA аnd DHCPv6). Whіlе thе rip positively mаkеѕ clarity frοm thе educational perspective, thе use providers charity PPP-based services (counting DSL аnd opposing uses οf PPP-over-FTTH) wеnt berserk.
A few months ago Brocade launched its own version of Data Center Fabric (VCS) and the VDX series of switches claiming that:
The Ethernet Fabric is an advanced multi-path network utilizing an emerging standard called Transparent Interconnection of Lots of Links (TRILL).
Those familiar with TRILL were immediately suspicious as some of the Brocade’s materials mentioned TRILL in the same sentence as FSPF, but we could not go beyond speculations. The Brocade’s Network OS Administrator’s Guide (Supporting Network OS v2.0, December 2010) shows a clear picture.
Have you noticed how quickly fabric got as meaningless as switching and cloud? Everyone is selling you data center fabric and no two vendors have something remotely similar in mind. You know it’s always more fun to look beyond white papers and marketectures and figure out what’s really going on behind the scenes (warning: you might be as disappointed as Dorothy was). I was able to identify three major architectures (at least two of them claiming to be omnipotent fabrics).
Business as usual
Each networking device (let’s confuse everyone and call them switches) works independently and remains a separate management and configuration entity. This approach has been used for decades in building the global Internet and thus has proven scalability. It also has well-known drawbacks (large number of managed devices) and usually requires thorough design to scale well.
Yesterday I described how the IPv6 architects split the functionality of IPCP into three different protocols (IPCPv6, RA and DHCPv6). While the split undoubtedly makes sense from the academic perspective, the service providers offering PPP-based services (including DSL and retrograde uses of PPP-over-FTTH) went berserk. They were already using RADIUS to authenticate PPP users ... and were not thrilled by the idea that they should deploy DHCPv6 servers just to make the protocol stack look nicer.
Last week I got an interesting tweet: “Hey @ioshints can you tell me what is the radius parameter to send ipv6 dns servers at pppoe negotiation?” It turned out that the writer wanted to propagate IPv6 DNS server address with IPv6CP, which doesn’t work. Contrary to IPCP, IPv6CP provides just the bare acknowledgement that the two nodes are willing to use IPv6. All other parameters have to be negotiated with DHCPv6 or ICMPv6 (RA/SLAAC).
The following table compares the capabilities of IPCP with those offered by a combination of DHCPv6, SLAAC and RA (IPv6CP is totally useless as a host parameter negotiation tool):

