Cisco IOS Hints and Tricks

RIBs and FIBs (aka IP routing table and CEF table)

2010-09-02T07:14:00.002+02:00

Every now and then, I’m asked about the difference between Routing Information Base (RIB), also known as IP Routing Table and Forwarding Information Base (FIB), also known as CEF table or IP forwarding table.

We’ve discussed this topic during the Enterprise MPLS Packet Pushers Podcast, so you might want to listen to that one first before going into details.

Let’s start with an overview picture (which does tell you more than the next thousand words I’ll write):

A router has numerous ways of learning the best paths toward individual IP prefixes: they might be directly connected, configured as static routes or learned through dynamic routing protocols.

Each dynamic routing protocol (including RIP) has its own set of internal data structures, known as OSPF/IS-IS database, EIGRP topology table or BGP table. The routing protocol updates its data structures based on routing protocol updates exchanged with its neighbors, eventually collecting all the relevant information. Throughout this article we’ll work with 10.0.1.1/32 learned through OSPF and 10.0.11.11/32 learned through BGP, so let’s inspect the relevant OSPF/BGP data structures.

RR#show ip bgp | begin Network
   Network          Next Hop            Metric LocPrf Weight Path
r>i10.0.1.1/32      10.0.1.1                 0    100      0 i
r>i10.0.1.2/32      10.0.1.2                 0    100      0 i
*>i10.0.11.11/32    10.0.1.1                 0    100      0 i

RR#show ip ospf database router 10.0.1.1

            OSPF Router with ID (10.0.1.5) (Process ID 1)

                Router Link States (Area 0)

  LS age: 1612
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.1.1
  Advertising Router: 10.0.1.1
  LS Seq Number: 80000003
  Checksum: 0xC764
  Length: 60
  Number of Links: 3

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 10.0.1.1
     (Link Data) Network Mask: 255.255.255.255
      Number of MTID metrics: 0
       TOS 0 Metrics: 1

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 10.0.1.6
     (Link Data) Router Interface address: 10.0.7.9
      Number of MTID metrics: 0
       TOS 0 Metrics: 64

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 10.0.7.8
     (Link Data) Network Mask: 255.255.255.252
      Number of MTID metrics: 0
       TOS 0 Metrics: 64

Each routing protocol runs its own route selection algorithm (SPF algorithm in case of OSPF or IS-IS or pretty complex set of rules in case of BGP), deriving a set of IP prefixes reachable through that routing protocol and IP next hops that should be used to reach them. You can view the results of these route selection algorithms with protocol-specific show commands (for example, show ip bgp prefix for BGP and show ip ospf rib prefix for OSPF).

RR#show ip bgp 10.0.11.11
BGP routing table entry for 10.0.11.11/32, version 6
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  Local
    10.0.1.1 (metric 66) from 10.0.1.1 (10.0.1.1)
      Origin IGP, metric 0, localpref 100, valid, internal, best
RR#show ip ospf rib 10.0.1.1

            OSPF Router with ID (10.0.1.5) (Process ID 1)

OSPF local RIB
Codes: * - Best, > - Installed in global RIB

*>  10.0.1.1/32, Intra, cost 66, area 0
     SPF Instance 2, age 00:48:15
     Flags: RIB, HiPrio
      via 10.0.2.1, FastEthernet0/0, flags: RIB
       LSA: 1/10.0.1.1/10.0.1.1

Both BGP and OSPF associate IP next hops with IP prefixes, but BGP simply uses the value of the next-hop attribute attached to the BGP route, whereas OSPF computes the IP address of the next-hop OSPF router with the SPF algorithm.

The results of intra-routing-protocol route selection are inserted in the IP routing table (RIB) based on administrative distance (and there are interesting consequences if two routing protocols have the same AD). Most routing protocols don’t complain when their routes are not used in the IP routing table; BGP has a special show command that can display RIB failures. In our scenario, the 10.0.1.1/32 prefix is received via OSPF and BGP and the OSPF route wins as OSPF has lower AD than internal BGP route.

RR#show ip bgp rib-failure
Network      Next Hop    RIB-failure            RIB-NH Matches
10.0.1.1/32  10.0.1.1    Higher admin distance  n/a
10.0.1.2/32  10.0.1.2    Higher admin distance  n/a

Ideally, we would use RIB to forward IP packets, but we can’t as some entries in it (static routes and BGP routes) could have next hops that are not directly connected.

Compare an IBGP route in the IP routing table (RIB) with an OSPF route:

RR#show ip route 10.0.11.11
Routing entry for 10.0.11.11/32
  Known via "bgp 65000", distance 200, metric 0, type internal
  Last update from 10.0.1.1 00:00:55 ago
  Routing Descriptor Blocks:
  * 10.0.1.1, from 10.0.1.1, 00:00:55 ago
      Route metric is 0, traffic share count is 1
      AS Hops 0
      MPLS label: none

RR#show ip route 10.0.1.1
Routing entry for 10.0.1.1/32
  Known via "ospf 1", distance 110, metric 66, type intra area
  Last update from 10.0.2.1 on FastEthernet0/0, 00:33:47 ago
  Routing Descriptor Blocks:
  * 10.0.2.1, from 10.0.1.1, 00:33:47 ago, via FastEthernet0/0
      Route metric is 66, traffic share count is 1

OSPF route has an outgoing interface; it’s computed by the SPF algorithm and transferred in the IP routing table. BGP route has no outgoing interface and its next hop is not directly connected; the router has to perform recursive lookups to find the outgoing interface (recursive lookups are also used to implement EBGP load balancing with loopback interfaces).

Early IOS releases performed a recursive lookup on the first packet sent to a new destination (process switching) and cached the result for subsequent packets (fast switching). Fast switching worked well in early Internet (with few global IP prefixes), but as the Internet grew and address-spraying DoS attacks became common, core routers frequently experienced cache trashing. Large number of packets were being process switched, resulting in very high CPU utilization and occasional router meltdown. It was time to move from cache-assisted forwarding to deterministic forwarding.

Forwarding Information Base (FIB) and Cisco Express Forwarding (CEF) switching were introduced to make layer-3 switching deterministic. When IP routes are copied from RIB to FIB, their next hops are resolved, outgoing interfaces are computed and multiple entries are created when the next-hop resolution results in multiple paths to the same destination.

For example, when the BGP route from the previous printout is inserted into FIB, its next-hop is changed to point to the actual next-hop router. The information about the recursive next-hop is retained, as it allows the router to update the FIB (CEF table) without rescanning and recomputing the whole RIB if the path toward the BGP next-hop changes.

RR#show ip cef 10.0.11.11 detail
10.0.11.11/32, epoch 0, flags rib only nolabel, rib defined all labels
  recursive via 10.0.1.1
    nexthop 10.0.2.1 FastEthernet0/0 label 19

Fully-evaluated FIB (CEF table) can then be used directly for layer-3 switching.

The IPv6 “experts” strike again

2010-09-01T07:08:00.000+02:00

IT World Canada has recently published an interesting “Disband the ITU's IPv6 Group, says expert” article. I can’t agree more with the title or the first message of the article: there is no reason for the IPv6 ITU group to exist. However, as my long-time readers know, that’s old news ... and the article is unfortunately so full of technical misinformation and myths and that I hardly know where to begin. Trying to be constructive, let’s start with the points I agree with.

IPv6 was designed to meet the operational needs that existed 20 years ago. Absolutely true. See my IPv6 myths for more details.

ITU-T has spun up two groups that are needlessly consuming international institutional resources. Absolutely in agreement (but still old news). I also deeply agree with all the subsequent remarks about ITU-T and needless politics (not to mention the dire need of most of ITU-T to find some reason to continue existing). That part of the article should become a required reading for any standardization body.

And now for (some of) the blunders:

IPv6 was a geek response to IAB’s adoption of CLNP. First of all, I don’t remember IAB every adopting CLNP. It did adopt IS-IS (which is the routing protocol used by CLNP) – an excellent decision that resulted in the only standardized multi-protocol IGP available today (that’s why all Shortest Path Bridging proposals including TRILL and 802.1aq use IS-IS). I can only hope that the loss of distinction between IS-IS and CLNP was a result of the editorial process, not a statement by the author (in which case I would have to start doubting his expertise).

Next, one of the proposals for next-generation IP (TUBA – TCP and UDP over Big Addresses) was indeed based on CLNP, but it was decided to move forward in a way that was more familiar to the IP world (as you know, regardless of window dressing, IPv6 is really IPv4 with minor fixes and longer addresses).

To put my statements in perspective: I was a devote supporter of TUBA and was deeply disappointed when IETF decided to reinvent the wheel instead of using a protocol that was already available, implemented by numerous vendors and field-tested. However, with 20 years of layer 8-10 experience behind me, I can understand that the totally different routing paradigm of CLNP and its variable-length addresses scared the IP gurus ... and it’s always nicer to reinvent the wheel and get the credit for the shiny new toy than to adopt someone else’s work.

Last, I wouldn’t use a pop book that was published 15 years after the events as a reliable source. When I was stupid enough to believe an “expert” who explained in vivid details what a catastrophe FTP is, numerous readers pointed out that FTP was, in fact, a well designed tool for the job it was supposed to be solving.

When IPv6 was finally adopted, many in industry made it clear they were not going to use the protocol. Of course, what else would you expect? It was much simpler to create private IP addresses and add NAT on top of IPv4 (all of which was solved within the network layer) than to wait for the rest of the IT industry to implement new protocol stack on all the hosts, persuade all the developers to change the way they open sockets (using the broken API) and fix all the outstanding applications. It took OS vendors more than a decade to finally implement IPv6 properly and several more years for the off-the-shelf applications to become reasonably IPv6-aware. Who knows what’s lurking in hidden depths of home-baked enterprise code?

After 16 years of evangelization, the "father of the Australian Internet," Geoff Huston, who became tired of the endless IPv6 hype, demonstrated in 2008 that only 0.4 per cent of the TCP/IP traffic was IPv6. What else would you expect? See above. And, just as an aside, if someone would truly invest 16 years in evangelizing IPv6, we might have seen some results. The sad fact was that IPv6 was largely forgotten after everyone implemented RFC 1918 and NAT.

Governments ... seem obsessed with continuing to drive IPv6 as some kind of panacea. It would be really good to hear from the same self-professed expert what his solution to the IPv4 public address depletion is. Carrier-grade NAT? CLNP? X.25?

Unfortunately, those few government agencies that can see beyond the next elections had to start the IPv6 push because the whole IT industry focused on quarterly results was in a state of denial (not dissimilar to the car industry’s handling of global warming). Without the government-sponsored push, there would be little (if any) IPv6 deployment today and we would be even more unprepared for the IPv4 address depletion.

20 years ago governments pushed CLNP because everyone believed it was the right protocol (IP was a geek toy). History proved them wrong. Lesson learned: don’t trust a protocol designed by a committee? Now they are pushing IPv6 because the only alternative is the breakdown of the end-to-end Internet.

LISP ... provides a compelling direction that IPv4 or IPv6 lack. LISP is a necessary hack because we can’t change what should truly be changed: the end hosts. I like LISP and the approach it takes to solving multi-homing and traffic engineering problems, but all those problems wouldn’t exist if someone would have taken the time to fix the broken TCP/IP architecture. What we’re doing today is introducing another layer of indirection at the network layer because we can’t change the things that are really broken.

A trusted implementation of LISP will provide a level of attribution and routing security that does not exist in the present architecture. Wishful dreams. To start with, LISP relies on underlying IPv4 or IPv6 global transport, which will be still implemented with current tools (BGP) and will thus stay as “secure” as it is today. Second, if we would be truly interested in routing security, we could have implemented secure BGP a long time ago, but it didn’t happen for a simple reason: those people that would benefit from it (content providers) were in no rush to pay those people that would have to implement it (ISPs).

To give you another example: it took almost a decade to develop and implement DNSSEC, even though we’ve experienced some very interesting DNS-based attacks in the meantime. Why would LISP be any different?

BGP: time to grow up

2010-08-31T07:26:00.003+02:00

If you’re in the Service Provider business, this is (hopefully) old news: on Friday, RIPE decided to experiment with the Internet causing routers running IOS-XR to hiccup. They stopped the experiment in less than half an hour and only 2% of the Internet was affected according to Renesys analysis (a nice side effect: Tassos had great fun decoding the offending BGP attribute from hex dumps).

My first gut reaction was “something’s doesn’t feel right”. A BGP bug in IOS-XR affects only 2% of the Internet? Here are some possible conclusions:

Most other intermediate routers (IOS and JunOS based, one would assume) decided to silently drop the offending attribute and thus only those IOS XR routers directly peering with RIPE were exposed. Not likely, that would be a direct violation of current BGP standards.
IOS XR is not widely used (read: not many people have CRS routers). Not likely, at least some very big providers have them.
Most people don’t run BGP on IOS XR and use the high-end boxes only in their IP+MPLS core.
IOS XR is typically not in the BGP update propagation path. If your core routers are receiving BGP updates solely from the BGP route reflector, there’s nobody behind them and nobody would notice the malformed updates (yet another reason to have good network design).

On a more serious note: the experiment has unintentionally exposed another long-term problem we’re facing: anyone can obviously attach any garbage to a BGP prefix and cause global memory consumption. The only thing you’d notice is increased BGP memory utilization that would be extremely hard to troubleshoot manually. Cisco IOS and IOS XR have no relevant filtering or scrubbing mechanisms (like they have for BGP communities) that you could use to protect yourself (and JunOS is probably no better).

The first line of defense could be BGP monitoring services like bgpmon.net. They could detect unknown transitive BGP attributes and report all memory-consuming attributes.

However, it’s high time we get away from “everyone is a trusted good guy” model BGP uses today and (at least) get a knob in BGP implementations that allows us to drop unknown attributes (today, unknown transitive attributes are silently propagated). Ideally, we would have a route-map/route-policy mechanism that would allow us to match BGP attributes based on their ID and accept BGP routes with select unknown attributes based on the attribute ID and its length.

Last but not least (before someone starts yelling at me): I know the “drop unknown attributes” knob will make all the future extensions to BGP harder to deploy, but the alternative is worse.

UPDATE (2010-09-01): Russell Heilling makes a very good point in his Unexpected Consequences post: it would be better to drop IP prefixes with unknown (or oversized) attributes than to silently scrub the attributes. In any case, we need conditions in the route maps/route policies that can match unknown attributes and the size of unknown (or all) attributes of a BGP route; the action to take (drop/permit/scrub) can then be specified in the route map/route policy.

Multihop FCoE 101

2010-08-30T07:19:00.015+02:00

The FCoE confusion spread by networking vendors has reached new heights with contradictory claims that you need TRILL to run multihop FCoE (or maybe you don’t) and that you don’t need congestion control specified in 802.1Qau standard (or maybe you do). Allow me to add to your confusion: they are all correct ... depending on how you implement FCoE.

Before going into details, you need to know some FC and FCoE port terminology:

FC/FCoE term	Translated into plain English
N_Port	Fiber channel port on a server or storage
F_Port	Fiber channel port on a switch
E_Port	FC port on a switch that can be used to interconnect switches
Domain	FC switch
VN_Port	Virtual N_Port. Created on a FCoE node (server or storage) to enable FCoE communication with a FCoE switch
VF_Port	Virtual F_port on a switch, created as needed to establish connection with an end-node (N_Port)
VE_Port	Virtual E_port, created on an FCoE switch to link it with another FCoE switch

In the simplest FCoE topology, a server with CNA (converged network adapter; a card that can send both Ethernet and FCoE traffic over the same gigabit Ethernet uplink) is connected to an FCoE-enabled switch, which has a direct connection into the legacy FC network.

The muddy waters start to appear when you have to insert intermediate switches between the servers (VN_ports) and legacy FC fabric. HP and NetApp work with the assumption that the intermediate switches don’t run FC protocol stack and only support Data Center Bridging (DCB) standards. Congestion control becomes mandatory in large networks and network stability is of paramount importance (thus NetApp’s recommendation to use TRILL).

Cisco, on the other hand, is pushing another design: every intermediate switch is a full-blown FC switch, running full FC protocol stack and participating in FSPF (FC routing). In this case, you don’t need congestion control (congestion is handled by the FC protocol stack) and you totally bypass bridging, so you don’t care whether bridging uses spanning tree, TRILL or something else.

The truly confusing part of the whole story: both designs (and any combination of them) are valid according to the FC-BB-5 standard; I’ll try to point out some of their benefits and drawbacks in future posts.

If you liked this explanation and would like to get a more thorough introduction to new LAN, storage and server virtualization technologies, register for my Data Center 3.0 for Networking Engineers.

Interesting links (2010-08-29)

2010-08-29T07:34:00.000+02:00

In his HSRP, vPC and the vPC peer-gateway command post Jeremy Filliben documents how the storage vendors ignore RFCs and implement what they think is proper ARP handling, causing havoc in a redundant network.

Andrew Vonnagy writes about another extreme stupidity customer convenience Microsoft managed to implement: you can turn any Windows 7 into a rogue Access Point. Like we didn’t have enough problems already.

And then there’s Charles Stross, taking the “where we went wrong” rants observations to a completely new level. While I’m complaining about lack of session layer in TCP/IP and broken socket API, he’s taking on Von Neumann architecture.

Storage networking is different

2010-08-26T07:07:00.001+02:00

The storage industry has a very specific view of the networking protocols – they expect the network to be extremely reliable, either by making it lossless or by using a transport protocol (TCP + embedded iSCSI checksums) that was only recently made decently fast.

Some of their behavior can be easily attributed to network-blindness and attempts to support legacy protocols that were designed for a completely different environment 25 years ago, but we also have to admit that the server-to-storage sessions are way more critical than the user-to-server application sessions.

Storage session loss can result in large-scale data corruption. If an end-user’s application session fails, you’ll hear some foul language, but the data will remain in a consistent state (assuming, of course, your application uses a decent database server with rollback capabilities). If a storage session is lost, the disk data could be left in some indeterminate state and might be permanently corrupted. Databases are quite good at recovering data; for whatever weird reason file systems are sometimes less robust.

Loss of a disk device can crash the server. If your server loses its network connection, all the user sessions are gone, there will be some data loss, but you’ll probably end with consistent data (transactions that have not been completed will be rolled back). The server will happily continue its (now largely non-existent) work.

If your server loses its disk, a panicky crash is almost unavoidable. Dual HBA (storage adapters) and dual paths to the storage are thus a requirement in a decent data center.

Impact of lost storage is extremely high. Imagine a web server with thousands of concurrent users, tens of web server worker processes and a database server (or you could have the components distributed if you so wish). If you lose an end-user session, a single user will be impacted. If you lose the session with the database server, some web worker processes will be impacted (others might continue working if you have a redundant setup). If you lose connectivity to your disk, all bets are off.

Storage is, well, permanent. If there’s an undetected error in your application session, your program might crash; if the incorrect data is written to a disk, it stays there indefinitely.

However, every decent layer-2 protocol has checksums that should detect transmission errors. IP and TCP also try to detect gross negligence in routers (although these attempts are pretty lame). One has to wonder why the storage people insist on another layer of checksums, be it in iSCSI or FCoE.

The answer is simple: the only end-to-end error detection mechanisms in an IP network are IP and TCP checksums and these are not good enough to detect potential router problems.

Anything else? I’ve probably missed a hundred other reasons why we have to treat storage networks more carefully. Please feel free to add them in the comments.

And the usual addendum: storage networking is just one of the topics described in my Data Center 3.0 for Networking Engineers webinar (register here).

FCoE and DCB standards

2010-08-25T07:17:00.001+02:00

The debate whether the DCB standards are complete or not and thus whether FCoE is a standard-based technology are entering the metaphysical space (just a few more blog posts and they will join the eternal angels-on-a-hairpin problem), but somehow the vendors are not yet talking about the real issues: when will we see the standards implemented in shipping products and will there be a need to upgrade the hardware.

Read more ... (yet again @ etherealmind.com)

Tweak the Search Engine rankings to push IPv6

2010-08-24T07:16:00.003+02:00

We all know that IPv6 deployment is a chicken-and-egg problem: Service Providers are slow to adopt IPv6 (there are some notable exceptions, including Comcast) because they can’t charge for it and the content providers don’t care because there are no IPv6 customers.

My good friend Jan Žorž got a great idea during the Google IPv6 Implementers Conference and finally managed to write it down: all we need is a slight search engine preference for sites reachable over IPv4 and IPv6. A small well-publicized tweak in Google’s scoring algorithm would push the content providers toward IPv6 and force web hosting companies to roll out IPv6 support immediately.

Enterprise IPv6: still not ready for prime time

2010-08-23T15:00:00.000+02:00

Almost a year ago I wrote a blog post listing various enterprise networking devices from Cisco lacking IPv6 support. As Jessica Scarpati found out (quoting, among other sources, my articles), not much has changed in the meantime and Cisco is not the only myopic company. For example, according to her article, Riverbed spokesman has “declined a request to discuss how the WAN optimization vendor will help customers with IPv6 migrations, saying the company was not ready to speak publicly about its strategy.” No wonder enterprises are not rushing to join the IPv6 crowd (although there’s no excuse for not having an IPv6-enabled public web site).

To make matters worse, there’s another roadblock to enterprise IPv6 deployment: MPLS VPN services supporting native IPv6 are still rare (and in most cases you can’t have any other routing protocol but EBGP).

Interesting links (2010-08-21)

2010-08-21T20:32:00.003+02:00

Two interesting QoS-focused posts from last week: Brad Hedlund was explaining the difference between UCS and HP Virtual Connect QoS (short summary: one does queuing, the other one rate-limiting) and Russell Heilling nicely described the QoS problems encountered in a Service Provider network (he’s coming to the same conclusion as I did: we need per-user queuing, but describes it way more eloquently).

After I’d stumbled upon Russell’s blog, I started reading his older posts and found a nice in-depth explanation of one potential road to Net Neutrality hell paved with good intentions.

And, in a blitz of late news, ATAoE has resurrected like Phoenix from its ashes.

More IPv6 FUD being thrown @ CFOs

2010-08-20T07:00:00.023+02:00

The CFO magazine has recently published a FUDful article “Trouble Looms for Company Websites” (read it to see what CFOs have to deal with). Obviously, some people think it’s a good idea to throw FUD at CFOs to get the budget to implement IPv6. Long term, it’s a losing strategy; your CFO will become immune to anything coming from the IT department and ignore the real warnings.

Yes, it's time to make your content reachable over IPv4 and IPv6, more so if you’re in the eyeballs business. Google knows that. So does Facebook. Twitter doesn’t seem to care. Maybe because they’re not selling ads?

Yes, you should go and implement IPv6 in your DMZ. Yes, you should start planning and budgeting today. But today the problems are as real as the Y2K ones were on Cisco IOS.

Eventually (in a decade) you’ll be dead if you won’t have IPv6, as every Service Provider will stop caring about your obsolete protocol and discontinue the translation services. In the meantime, the problems will appear gradually ... faster for people that have fancy two-way applications or broken web sites (for example, tracking users by their IP addresses or doing sloppy load balancing), slower for those that have more traditional applications using HTTP the way it was designed to be used (including, interestingly, Netflix).

The article is also full of NATty FUD. Without ever calling NAT64 by its name, it implies that a “connection through a gateway probably won't perform nearly as well as users have come to expect”. Come on, almost every enterprise user accessing the Internet is using some form of NAT. Why would NAT work well today and so badly in the future when it has to convert IPv6 addresses into IPv4 addresses? Admittedly it’s true that my (or their) claims are not verifiable since there’s no publicly-available large-scale production-grade NAT64 device (but there are some in field trials).

On a more positive note, you’ll find a universal truth in the article (courtesy of Martin Levy): “You can either do a planned, careful migration, or you can do it in a panic. And you should know full well that panicking is more expensive.” I’ll definitely use it in my presentations.

Last but not least, offering early IPv6 access to your content is not a big deal: Facebook did it pretty quickly and so can you ... assuming you have an IPv6-capable load balancer (some companies are still pretending you don’t need them). If you don’t, why don’t you go over to F5 web site and download their 90-day trial virtual machine? Or use NAT-PT in a standalone Cisco IOS box (I’m positive you have a few lying around) for a trial deployment? Just make sure you understand the caveats.

Would you like to hear more about my (potentially biased) perspective on enterprise IPv6 deployments? Get in touch with me and let’s organize an Enterprise IPv6 Deployment workshop for your team.

I don’t need no stinking firewall ... or do I?

2010-08-19T07:04:00.007+02:00

Brian Johnson started a lively “I don’t need no stinking firewall” discussion on NANOG mailing list in January. I wanted to write about the topic then, but somehow the post slipped through the cracks (thank you, Pavel, for a kindly reminder!) ... and I’m glad it did, as I’ve learned a few things in the meantime, including the (now obvious) fact that no two data centers are equal (the original debate had to do with protecting servers in large-scale data center).

First let’s rephrase the provocative headline from the discussion. The real question is: do I need a stateful firewall or is a stateless one enough?

Stateless firewalling is implemented quite easily with router access lists and thus works at line speed in most routers. Stateful firewalls implemented in routers are usually suitable for low-speed remote offices; you should use dedicated firewall devices in data centers.

Technology issues

Stateful firewall is the only option if you’re trying to tightly protect applications that use dynamic port numbers, including everything from peer-to-peer applications (including SIP) to RPC-based applications (let’s try not to call them broken ... how about unpredictable applications).

You can limit the dynamic port range for some of these applications and allow all ports in that range through the firewall ... while hoping that some other service on your server won’t grab one of those ports and expose itself unnecessarily.

If your applications use only well-known fixed port numbers (let’s call them fixed-port applications), you don’t have to inspect the application data stream and can match the applications with access lists; stateless solutions seem appropriate.

However, some stateful firewalls can add value even in fixed-port environments: they can delay the commitment of server resources to TCP sessions with TCP SYN cookies (also available in numerous server operating systems) and check the validity of TCP sessions.

You might think that there are no vulnerabilities left in TCP that could be exploited. A long while ago, everyone thought it was impossible to establish one-way spoofed TCP sessions even though there were known vulnerabilities in TCP sequence number generation ... until Kevin Mitnick proved them wrong.

Last but not least, stateful firewall in front of a server can block TCP fingerprinting attempts. Sometimes you simply don’t want the attacker to know too much about your infrastructure.

Design choices

There are two extremes you can be facing in a data center:

Unified large-scale infrastructure using fixed-port applications, including Google, Yahoo, Facebook, Twitter and a few others. You would expect to see from tens to thousands of almost-identical servers with standardized and identically-configured operating system in these environments. It’s quite manageable to harden and patch these servers and combined with fixed-port applications these environments usually offer, it makes perfect sense to be satisfied with router ACLs (and that was the case the proponents of “get rid of the firewall” line of thinking were promoting in the NANOG discussion).

Dynamic hodgepodge of servers, operating systems and application encountered in a usual enterprise data center. Server hardening is mission impossible (as you have so many different operating systems and/or versions of the same operating system), patching is the responsibility of individual server administrators (and they might not even be a unified team) and the applications are a nightmare from the security perspective. For example, early Outlook Web Access was easy to firewall but even then some people advocated putting OWA in the inside network (I wish I were blogging at that time, I would have had so much fun debunking that article). I was not able to find anything from Microsoft telling me how to configure my firewall to support OWA for Exchange Server 2010; the cynical response I got from a (non-Microsoft) security engineer a few days ago was “nobody can figure out which ports it uses”.

Now what?

Obviously you’re the only one who can decide where your Data Center environment is and which measures you’d like to implement, but as a generic rule, the more exposed (and the worse managed) a server is, the more you should lean toward a stateful firewall in front of it. Most enterprise data centers make heavy use of stateful firewalls and that’s also what we’re recommending to most of our customers.

To get insight into typical Data Center architectures (including load balancing and firewalling) and emerging DC technologies, register for my Data Center 3.0 for Networking Engineers webinar. For an in-depth solution, you might want to consider the benefits of our Hypercenter architecture.

Port or Fabric Extenders?

2010-08-18T07:16:00.001+02:00

Among other topics discussed during the Big Hot and Heavy Switches (Part 1) podcast (if you haven’t listened to it yet, it’s high time you do), we’ve mentioned port extenders. As our virtual whiteboard is not always clearly visible during the podcast (although we scribble heavily on it), here’s the big-picture architecture:

After the podcast I wanted to dig into a few minor technical details (just in case someone throws me a curveball during the Data Center webinar) and stumbled into a veritable confusopoly.

Port or Fabric? Cisco uses the term fabric extender; port extender is used in the IEEE 802.1Qbh standard. Obviously one is a technology term, the other one a marketing term (or product name). It looks like everything in the Data Center is *Fabric*, everything in WAN is *Borderless*. Slowly I might get used to it.

Another explanation I found says “Fabric Extender is a standalone implementation of a Port Extender”. I’m not buying it; see the 802.1Qbh introduction (from Cisco) for details; it's very clear they are not talking about a virtualized environment.

Standardized or not? Cisco brought the idea of port extenders to IEEE 802.1 and effectively started the 802.1Qbh working group. However, 802.1Qbh specifies only the control protocol between the controlling bridge and the port extender. It refers to 802.1Qbg for the specification of the tagging on the link between them. 802.1Qbg (which is close to VNTag/VNLink idea from Cisco) refers to 802.1Qbc standard (well hidden behind the paywall) which specifies the actual encapsulation mechanism. The claim that “Cisco has pioneered the VNTag scheme and IEEE 802.1Qbh will use it as a base for the standard” (Cisco Unified Computing System, Cisco Press 2010, page 85) is thus somewhat dubious.

VNTag or not? I was not able to find any authoritative information on the encapsulation format used between fabric extender and controlling bridge. There are plenty of allusions floating around, but nothing very explicit.

There’s also a slight nesting problem: if a fabric extender (Cisco UCS 2100 Fabric Extender) uses VNtag toward the controlling bridge and the UCS blade NIC (Cisco Palo) uses VNtag toward the controlling bridge, we might end with two stacked VNtags.

Fabric Extender limitations. Compared to IEEE 802.1Qbh draft that Cisco started, fabric extenders are extremely limited. For example, the standard aims to provider:

Autodiscovery of port extenders. Not available in the Nexus software; you have to manually configure the switchport type and FEX ID on the interface to which the fabric extender is connected.
Chaining of port extenders. A port extender can be connected to another port extender, creating a hierarchy. Not available with Nexus 2000 series.
Port extender transparency. You can connect anything (host or another bridge) to a port extender physical port. Nexus 2000 products allow only hosts to be connected to the downstream ports; BPDU guard is automatically configured on those ports.

Fabric Extender enhancements. As always, Cisco offers more than the standard it has launched. For example, you can use Virtual Port Channel to have dual-homed fabric extenders.

More Data Center-related technologies

Data Center 3.0 for Networking Engineers webinar (register here) covers a wide range of Data Center technologies, including port extenders, VNTags, FabricPath and TRILL.

If you'd enjoy a live Data Center design session hosted by two top-notch CCIEs, why don't you join us @ IOSHints Live in San Jose on September 15th?

Packet Filters on a Nexus 7000

2010-08-17T06:56:00.001+02:00

We’re always quick to criticize ... and usually quiet when we should praise. I’d like to fix one of my omissions: a few days ago I was trying to figure out whether Nexus 7000 supports IPv6 access lists (one of the presentations I was looking at while researching the details for my upcoming Data Center webinar implied there might be a problem) and was pleasantly surprised by the breadth of packet filters offered on this platform. Let’s start with a diagram.

Every packet traversing a Nexus 7000 goes through a number of filters:

Inbound port filters. You can specify IPv4, IPv6 and MAC filters.

VLAN filters. Whenever a packet is bridged within a VLAN, vlan access-map is applied. VLAN access maps can match on IPv4, IPv6 and MAC addresses (using already-defined access-lists) and even redirect a packet to another interface for further analysis. VLAN filters can be applied to a range of VLANs to improve configuration scalability.

Obviously this step is skipped if the inbound interface is a layer-3 interface.

Routed port ACL. Whenever a packet enters or exits the layer-3 forwarding engine, it can be filtered with inbound/outbound ACL. Yet again, both IPv4 and IPv6 ACL are supported. If the packet is forwarded between two VLANs, the destination VLAN filter is also applied.

TrustSec Security Group. Last but not least, every packet traversing the Nexus 7000 is checked by TrustSec security group ACL (SGACL).

I would love to go into so many details in my Data Center 3.0 for Networking Engineers webinar (register here) but it’s quite impossible due to timing constraints and the breadth of technologies we’re covering. The “further reading” list I’m already preparing should help you find in-depth information on the topics discussed in the webinar. If, however, you'd enjoy a live design session hosted by two top-notch CCIEs, why don't you join us @ IOSHints Live in San Jose on September 15^th?

WAF musings ... not again?

2010-08-16T06:50:00.000+02:00

Following my obituary for Cisco’s WAF, Packet Pushers did a really great WAF-focused podcast with Raven Alder, appropriately named Saving the Web with Dinky Putt Putt Firewalls. If you have more than a fleeting interest in protecting business web applications, you should definitely listen to it. Just as an aside: when they were recording the podcast, I was writing my To WAF or not to WAF post ... and it’s nice to see we’re closely aligned on most points.

There’s just a bit I’d like to add to their ponderings. What Raven describes is the “proper” (arduous, time-consuming and labor-intensive) use of WAF that we’re used to from the layer-3/4 firewalls: learning what your web application does (learning because the design specs were never updated to reflect reality) and then applying the knowledge to filter everything else (what I sometimes call the fascist mode – whatever is not explicitly permitted is dropped).

You could also use a WAF in a “catch the known exploits” mode, similar to what we do with IDS and virus scanners: most WAF products come preloaded with patterns matching well-known exploits (like SQL injections or XSS attacks). These patterns will stop most of the attacks that use “common knowledge” exploits; they will be (obviously) unable to stop those attacks that are crafted with the knowledge of the business logic of your web application.

Did I just embrace the “quick-and-dirty just good enough” mentality? I must be going nuts.

Last but not least, now that I’ve mentioned IDS: some people think that you can program your IDS to provide functionality similar to WAF. Forget it; as I’ve explained more than a year ago, HTTP+HTML provide too many ways of obfuscating the content you want to sneak past the IDS, or as Raven put it (approximately): “the moment you start writing IDS rules that emulate WAF, you’ve started developing a completely new product”.

Interesting links (2010-08-14)

2010-08-14T16:20:00.000+02:00

A few days ago I wrote that you should always strive to understand the technologies beyond the reach of your current job. Stephen Foskett is an amazing example to follow: although he’s a storage guru, he knows way more about HTTP than most web developers and details of the web server architecture that most server administrators are not aware of. Read his High-Performance, Low-Memory Apache/PHP Virtual Private Server; you’ll definitely enjoy the details.

And then there’s the ultimate weekend fun: reading Greg’s perspectives on storage and FCoE. It starts with his Magic of FCoTR post (forget the FCoTR joke and focus on the futility of lossless layer-2 networks) and continues with Rivka’s hilarious report on the FCoTR progress. Oh, and just in case you never knew what TR was all about – it was “somewhat” different, but never lossless, so it would be as bad a choice for FC as Ethernet is.

Last but not least, there’s Kevin Bovis, the veritable fountain of common sense, this time delving with the ancient and noble art of troubleshooting. A refreshing must-read.

Deploying IPv6 article @ SearchTelecom

2010-08-13T10:05:00.000+02:00

Following my “Transition to IPv6” articles, Jessica Scarpati from SearchTelecom.com wrote a series of articles covering the telecom transition plans and the problems they’re experiencing with the vendors and content providers.

In the second article of the series, “Deploying IPv6? Demand responsiveness from vendors, content providers”, she’s quoting John Jason Brzozowski from Comcast, John Curran from ARIN, Matt Sewell from Global Crossing and myself. My key message: vote with your money and take your business elsewhere if the vendors don’t get their act together.

IOSHints Live San Jose 2010 with Etherealmind

2010-08-12T13:14:00.001+02:00

I have fantastic news for the engineers attending the IOSHints Live event in San Jose: Greg Ferro (of Etherealmind and Packet Pushers fame) will join me as co-presenter in the Data Center session. This will ensure a lively discussion as we sometimes have different perspectives on Data Center technologies and architectures (one of us supposedly lives in an ivory tower and the other one is a perpetual skeptic) and we’ll enjoy analyzing various network designs you’ll bring with you from multiple angles.

With Greg on board, we’ll definitely jump deep into FCoE and DCB waters, hear a few lovely words about Checkpoint firewalls and discuss the potential merits of vFabric.

Vendor performance tests

2010-08-11T13:41:00.000+02:00

Any comment would be superfluous.

Custom-written DNS servers: Learn to love them

2010-08-11T07:15:00.001+02:00

A cryptic e-mail message from one of my readers telling me that blog.ioshints.info does not resolve into an IP address (yet again, thank you Igor!) set just the right mood for my Monday morning. As I had about 30 minutes before rushing off to a day-long countryside family event, all I could do was run a few quick tests (of course it worked for me) and take my iPad with me (luckily, it’s a 3G model and the coverage in Slovenia is decent).

During the day, Igor was able to supply more details: the DNS hosting company I was using suddenly decided to return NXDOMAIN error code on every CNAME record unless the query type was CNAME, while providing the results in the response at the same time. Here’s a sample of their bizarre reasoning:

$ dig @ns1.domaindiscover.com xml.ioshints.info
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 14206
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;xml.ioshints.info.             IN      A

;; ANSWER SECTION:
xml.ioshints.info.      0       IN      CNAME   ghs.google.com.

Correct answer is obviously somewhere along the lines of the following printout (I will not waste my time reading the RFCs to figure out all the potential correct answers ;). The authoritative name server supplies just the CNAME, not the final answer (it has no authority to do it) and obviously has to return the success code (as it found the name we were looking for).

$ dig @ns.nil.si blog.ioshints.info
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59148
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 13, ADDITIONAL: 12
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;blog.ioshints.info.            IN      A

;; ANSWER SECTION:
blog.ioshints.info.     0       IN      CNAME   ghs.google.com.

Be liberal in what you accept

The worst part of the puzzle was that I was able to reach the blog throughout the day (ensuring the DNS TTL of the CNAME record in the server I was using did expire) even using different Service Providers (and DNS servers). In the afternoon, as I was finally able to get hold of a reasonable PC (for whatever reason, I don’t think iPad is the proper platform to write longer texts) I tried to figure out the difference between Igor and myself and polled my Twitter friends.

Only a few of them had problems (weird) and Ethan was kind enough to help me with my troubleshooting efforts. It looks like most recursive DNS servers (including Google’s 8.8.8.8 and OpenDNS) don’t bark at NXDOMAIN error if the answer is included in the reply and happily continue the resolution process, while a few servers decide NXDOMAIN means what it should mean and return an error to the client.

Let’s fake authority

Digging further, Igor uncovered another stupidity: the programmers working for DomainDiscover decided to make their life extremely easy and generate fake SOA records. This is what they claim the authority for google.com is:

$ dig @ns1.domaindiscover.com xml.ioshints.info

... deleted ...

;; AUTHORITY SECTION:
google.com.         3600    IN      SOA     ns1.tierra.net.
                    hostmaster.tierra.net. 2010080701 7200 1800 604800 28800

One must wonder what these guys will do when DNSSEC comes knocking on their doors. I decided I don’t want to be there when that happens and moved my DNS server today.

Conclusions

I am sad to be forced to leave DomainDiscover. They were working perfectly for the last 5+ years I was using them, but obviously the load they were handling (and the pricing wars in the domain registration business) pushed them to cut a few corners too many.

Unfortunately, we can only expect to see more failures like this one as the industry goes through endless cycles of doing more with less.

First ever Ioshints Live event (San Jose, September 2010)

2010-08-10T13:44:00.002+02:00

Webinars are great (say most of the attendees ;) but I miss meeting people. Webinars are just the right mechanism for brief technology updates, overviews or in-depth configuration discussions, but it’s nearly impossible to do a lively design discussion involving a reasonably-sized group of engineers over the web. The desire to do a live event was thus slowly brewing for almost a year and finally turned into reality: on September 15th you can join me in San Jose for the first ever Ioshints Live event.

We’ll discuss two hot topics: data centers (I simply had to use the vapor word in the title) and VPN solutions. Both sessions will be focused on design issues and your real-life needs. The registration is open; if you decide to join us for the whole day, just register for both sessions.

To WAF or not to WAF?

2010-08-09T07:23:00.003+02:00

Extremely valid comment made by Pavel Skovajsa in response to my “Rest in peace, my WAF friend” post beautifully illustrates the compartmentalized state some IT organizations face; before going there, let’s start with the basic questions.

Do we need WAF ... as a function, not as a box or a specific product? It’s the same question as “do we need virus scanners” or “do we need firewalls” in a different disguise. In an ideal world where all the developers would be security-conscious and there would be no bugs, the answer is “NO”. As we all know, we’re in a different dimension and getting further away from the heavens every time someone utters “just good enough” phrase or any other such bingo-winning slogan.

It’s popular to bash IT vendors’ lack of security awareness (Microsoft comes to mind immediately), but they’re still far ahead of a typical web application developer. At least they get huge exposure, which forces them to implement security frameworks.

The exposure a typical web application gets is low (unless you’re working for Google, Facebook, Twitter, Amazon or a few others) and the vulnerabilities are thus hidden until someone takes advantage of them. Not surprisingly, hacking accounts for more than 90% of stolen data records in 2009 and within that category, SQL injection is responsible for almost 90% of the stolen goods (source: Verizon’s 2010 Data Breach Investigation Report). All this has been known years ago, but still most web sites don’t use a WAF. The situation is similar to the behavior of early Internet adopters around the time the Firewalls and Internet Security book was published, but with infinitely more money at stake.

Web server module or standalone solution? When you mention standalone WAF to the web community, the typical response is “we don’t need it; there’s mod_security for Apache”. They might be partially correct: if you write your application in an open-source language running within an open-source web server on an open-source operating system it makes sense to use yet another open-source firewall product ... just make sure you understand the support model. The good-enough open-source solutions also explain (at least partially) Cisco’s failure in this market segment. However, the choice of the product (and its licensing methodology) should have no impact on the embedded or standalone question.

There are tons of good reasons favoring standalone solution and only a single one favoring the embedded module (it’s cheaper). In the world of virtual servers and virtual appliances deploying another virtual machine carries little incremental cost, so even the cost argument is gone.

And now let’s get to the behavior problems described in Pavel’s comment.

Application folks usually get to choose stuff protecting their application. True, but completely wrong. It’s the same idea as server people choosing the firewall to protect them. BTW, don’t even try to launch the “if they knew something about security, we wouldn’t need a WAF” argument; it will not make you popular.

WAF security should be a Data Center-wide function like firewalls and load balancers and whoever is made responsible for it should choose the best solution for the job, not the one the application people are most familiar with.

Last but not least, there’s always the “checks and balances” question.

Application people usually choose server-based protection. Of course they do, as they already have the web server up and running (see above). One should also consider that they’ll usually deploy tons of web servers for isolation, scalability and redundancy reasons ... and some web servers (or Web UI interfaces embedded in other software products) might not even have a WAF module.

Now step back a bit and look at the bigger picture: is it easier to manage, configure, patch and upgrade tens of modules on individual web servers (with different users having administrative rights) or a single centralized device?

Putting L7 devices managed by network guys in front of public facing servers is a bad idea due to the "it's not our fault" tendency. If this applies to your IT organization, it might be time to polish your resume and consider looking around. If your managers don’t understand where the industry is going, and what they should do to adapt, your users will eventually give up, try to bypass you and move to public cloud services. Definitely a bad idea, but it will happen eventually.

BTW, second part of The Future is Here post by Chuck Hollis has some useful ideas; I know I’ve read something even better, but (as usual) can’t find it when needed.

Interesting links (2010-08-08)

2010-08-08T12:01:00.000+02:00

Several interesting blog posts I’ve stumbled upon in the last few days:

Should developers have access to production? A really good analysis of pros and cons (including a pinch of “this is how it’s always been done”).

Migrating from Catalyst to Nexus. As usual, there are hidden gotchas and varying default settings that make the migration more interesting than expected.

Justifying VDI. VMware has decided to deploy virtual desktops; they describe what they’ve learnt and how you should approach VDI projects.

How many large-scale bridging standards do we need?

2010-08-06T13:32:00.001+02:00

Someone had a “borderless data center mobility” dream a few years ago and managed to infect a few other people, resulting in a networking industry pandemic that is usually exhibited by the following “facts”:

Unhindered Virtual Machine mobility across the globe is the absolute prerequisite for any business agility. Wrong. There are other field-proven solutions and although inter-site VM mobility has been demonstrated, it’s still a half-baked idea and has many caveats.
You can only reach that Holy Grail by extending your layer-2 domains across vast distances. Totally wrong. It would be easier to fix L3 routing and signaling protocols than to invent completely new technologies trying to fix L2 problems. Users of Microsoft NLB are might disagree ... in which case I wish them luck in scaling their architecture.
Large-scale bridging is absolutely mandatory if you want to build cloud solutions with tens of thousands of servers. Not sure about that. Google is there, Facebook, Twitter and Amazon are (at least) close, large web hosting providers have been around for years ... and yet they somehow managed to survive with existing technologies and good network designs.

Just today XKCD published a very relevant comic, so I can skip my usually sarcastic comments and focus on the plethora of emerging large-scale bridging standards and implementations. Let’s walk through them:

Traditional Service Provider solutions: pseudowires (with EoMPLS or L2TPv3) and VPLS. Both of them are covered from the SP perspective in the Market Trends in Service Provider Networks webinar and from the user perspective in the Choose the Optimal VPN Service webinar.

Do-it-yourself solutions relying on IP core: L2TPv3 or OTV.

Shortest-path bridging: TRILL and 802.1aq. Within 802.1aq, you have two incompatible implementations of the forwarding plane: SPBV and SPBM. And then there’s Cisco’s FabricPath, which looks to be enhanced 802.1aq at the moment. I’ll briefly describe these technologies in the Data Center 3.0 for Networking Engineers webinar.

While these technologies supposedly target large-scale LAN networks, they’ll soon escape into the WAN (why couldn’t you run 802.1aq SPBM or TRILL over VLPS, for example).

There’s more to come. As if the existing solutions wouldn’t be enough, Juniper & Alcatel-Lucent created their own BGP MPLS Based MAC VPN, which looks to be their response to OTV. As Andrew S. Tanenbaum said “The nice thing about standards is that you have so many to choose from.”

While it’s wonderful to see the explosive bursts of innovation, it might be high time for the IT vendors to stop their market-share-focused jostling and me-too technology launches and focus on what their customers need: stable field-proven interoperable standardized solutions. Everything else will just increase the FUD level ... or maybe that’s not a bad idea considering that I work for a consulting/system integration company.

Look beyond your cubicle

2010-08-05T15:02:00.002+02:00

The Packet Pushers Episode 11 (If You Can’t Be Replaced, You Can’t Be Promoted) contains numerous highly valuable career advices. I won’t spoil the fun by telling you what they are (listen to the podcast if you haven’t done so already); I’ll just add one to their long list: always look beyond what you’re doing at the moment. For example, a networking engineer working anywhere near a Data Center environment should be very familiar with the server and storage technologies.

Read more ... (this time @ etherealmind.com)