Internetworking perspectives by Ivan Pepelnjak
Given the following OSPF configuration …
interface Loopback 0 ip ospf 1 area 0 ! interface Ethernet 0 ip ospf 1 area 0 ! interface Serial 1 ip ospf 1 area 0 ! router ospf 1 passive-interface Ethernet 0
… how many LSAs will the router originate? Leave your opinions in the comments.
Soon after I wrote the Designing Fast Converging BGP Networks article, one of my regular readers sent me an interesting problem: BGP sessions would be lost in his (IS-IS based) core network if he would use fall-over on IBGP neighbors and the BGP router would have a primary and a backup path to the IBGP neighbor.
We’ve identified the source of the problem (delayed SPF run) and I decided to document it. It took “a while” (and a kind nudge from Mateusz following a discussion on Cisco-NSP mailing list) to find time to set up the test lab and document the behavior. You can find the (undesired) results in the Aggressive BGP fall-over behavior in the CT3 wiki.
Just in case someone from Cisco wants to put this on a wish list: please add a configurable delay to the neighbor fall-over command like you did with the bgp nexthop trigger delay command.
You might have noticed that your IOS release supports a ctunnel interface (hint: your image has to support CLNS) and wondered what it could do. Well, it’s a GRE tunnel between a pair of NSAPs, so you can transport IP traffic across your well-engineered CLNS network without ever exposing the core routers to the dangers of IP.
But wait, it gets better: starting with IOS releases 12.3(7)T and 12.2(33)SRA, you can transport IPv6 across the ctunnel interface. Unfortunately, they haven’t implemented MPLS over GRE over CLNS yet (the mpls ip command is present, but does not work).
It looks like there's at least one potentially very large-scale application that could use this feature.
As we all know, OSPF area ID is a 32-bit quantity that can be written as a decimal number or as a dotted decimal (although I never really understood the need for dotted decimal support). You would expect both representations to behave identically (after all, it’s a 32 bit number we’re talking about) and most often they do. However, Murali reported an interesting bug: some show commands do not understand that 4 times zero still equals zero.
If you execute the show ip ospf process 0 database self-originate command, you’ll get the list of LSAs the router originates into the backbone area. The show ip ospf process 0.0.0.0 database self-originate command does not display any LSAs (tested on 12.2(33)SRC4).
C2#show ip ospf 1 0 database self-originate
OSPF Router with ID (10.0.1.7) (Process ID 1)
Router Link States (Area 0)
Link ID ADV Router Age Seq# Checksum Link count
10.0.1.7 10.0.1.7 203 0x80000003 0x009303 4
Summary Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum
10.0.1.5 10.0.1.7 204 0x80000001 0x003EA9
10.0.1.6 10.0.1.7 170 0x80000002 0x009645
10.0.7.8 10.0.1.7 204 0x80000001 0x00B7F8
10.0.7.16 10.0.1.7 204 0x80000001 0x007169
10.0.7.32 10.0.1.7 204 0x80000001 0x00D0F9
10.2.1.0 10.0.1.7 204 0x80000001 0x00B22F
Type-10 Opaque Link Area Link States (Area 0)
Link ID ADV Router Age Seq# Checksum Opaque ID
1.0.0.0 10.0.1.7 215 0x80000001 0x005EAF 0
1.0.0.3 10.0.1.7 35 0x80000002 0x00765B 3
1.0.0.4 10.0.1.7 212 0x80000001 0x00B54A 4
C2#show ip ospf 1 0.0.0.0 database self-originate
OSPF Router with ID (10.0.1.7) (Process ID 1)Many thanks to those of you that responded with Netflix details (special thanks to Volcker for sending me the packet capture). Immediately after someone mentioned firewalls, I knew what the most sensible answer should be: to get across almost anything, use HTTP. No surprise, Netflix chose to use it. However, they’ve managed to deploy streaming video over TCP, which is not a trivial task. So, how did they do it?
The first trick is relatively easy: instead of streaming the whole content, the player reads smaller chunks of video files from the server. If the next chunk is downloaded before the previous one is completely played, you have an uninterrupted viewing experience.
They encode the chunks in the URL. They could have used the Range HTTP headers, but obviously decided to go with a more straightforward approach.
The next trick is the variable bit rate: they can switch between different encodings of the same content on-the-fly, making it easy to adapt to bandwidth fluctuations. You can view different video sources being fetched for the same movie in the summary of capture I’ve received.
The content is hosted within Level-3 network. I checked the IP address of the CDN server in the capture with ARIN WHOIS, but even the HTTP response headers tell the whole story:
HTTP/1.1 200 OK
Cache-Control: no-store
Pragma: no-cache
Content-Type: video/x-ms-wmv
ETag: "e8020c2-36c741-45f94afbc62c0"
Last-Modified: Sat, 03 Jan 2009 14:15:15 GMT
Accept-Ranges: bytes
Server: Level-3 Origin Storage/1.0
Date: Mon, 13 Jul 2009 18:50:59 GMT
Content-Length: 24
Connection: keep-alive
To summarize: nothing revolutionary (but you wouldn’t expect new protocols in residential environment where you have to keep a tight control on the support costs), but a lot of good ideas and sound engineering. I’m impressed.
I’m switching to the “traditional” summer schedule: until mid-August, I’ll post two to three shorter articles per week. I don’t want to spend too much of my vacation time writing, but I also don’t want to see you bored with dormant blogosphere.
Some of my projects will simply have to wait for the temperatures to drop, including a few selected Service Provider issues I’ve started writing about in the last weeks and the ADSL QoS topics (don’t worry, I haven’t forgotten them).
A few weeks ago I’ve been sitting in a meeting with a Wimax service provider. I was there for a completely different reason (they were interested in our outsourced network management solution), but after the meeting one of the Wimax technical gurus was kind enough to give me a quick walkthrough. I was more interested in the backhaul issues than in the wireless part (which is a total mystery to me) and I really liked what I’ve seen: pure IP-based backhaul, reusing exiting well known protocols like GRE, DHCP, RADIUS and Mobile IP.
Even the wireless part looks promising: they can map a CPE-side VLAN into a channel with its own QoS settings – a perfect solution to separate voice from data (and avoid the issues that choke the DSL world).
So, I started to wonder: will Wimax be the next disruptive technology (like IP was) or is it a dead branch on the wireless evolutionary tree?
If you have a Kindle, you can subscribe to the Kindle edition of the Cisco IOS Hints blog. The monthly subscription charged by Amazon is $0.99 (the price is set by them and I cannot influence it) and (in the spirit of full disclosure :) I get 30% of the revenue.
Thanks to my readers, I often learn something completely new about the intricacies of Cisco IOS. The “VLAN Interface Status” post resulted in a comment about the SVI autostate concept, which is (not surprisingly) a somewhat muddy topic:
The important part of the SVI autostate calculation is the “port is in STP forwarding state for the VLAN” requirement. If a VLAN is not carried in a trunk port (for example, due to switchport trunk allowed configuration command), the trunk port’s status does not influence the autostate.
Ying would like to have a PE-to-PE IPSec protection for traffic within a single VRF. For example, all traffic in VRF-A sent between PE-1 and PE-2 should be protected with IPSec and the PE-routers should be the endpoints of the IPSec session (CE-to-CE IPSec is trivial).
My first response was “hard to do”, then I started hallucinating about MPLS-over-GRE-over-IPSec-over-IP-over-MPLS tunnels between the PE-routers with tunnel-specific IGP and per-VRF BGP next hops. It can be done (we’ve implemented numerous large-scale MPLS/GRE/IPSec designs), but is there a simpler alternative? Please share your ideas in the comments.
Vijay sent me this question a while ago:
I have configured a L3 VLAN interface on a Cisco 3750 switch and assigned an IP address to it. I haven't assigned any ports to this VLAN. Why am I not able to ping the IP address of the VLAN interface from the switch itself?
The VLAN interface (like any other interface) has layer-1 and layer-2 state.
The layer-1 state is displayed in the Status column of the show ip interface brief command, the layer-2 state in the Protocol column.
A VLAN interface is always up, but its line protocol state tracks the state of attached ports: if at least one port is operational, the line protocol of the VLAN interface is up, otherwise it’s down. With no ports assigned to a VLAN, the line protocol of the VLAN interface is down, its IP address is not in the IP routing table and thus you cannot ping it.
This article is part of You've asked for it series.
I received numerous complaints about the inability to write comments to my blog in the last few weeks. I had comment-related problems for months, but the situation was getting out of control: initially only the Internet Explorer users had (manageable) problems, then it spread to some Firefox users, later it became impossible to submit comments at all and the last marvel in this buggy saga was the “automatic” conversion from Publish request to preview screen. I had enough, Google has yet again proven that you usually get what you pay for.
Today I stumbled across a blog post shared by one of my Facebook friends that mentioned JS-Kit services. I also happened to have a few spare creative hours (one of my appointments was pushed forward) and tried the JS-Kit comments on one of my rarely-visited blogs. After the initial crashes I got it to work and implemented it on the IOS Hints blog. If you’ll experience any problems with the new system, please let me know (get in touch with me via my e-mail web form if you can’t leave a comment).
Notice to NOSCRIPT users: you have to allow scripts from JS-KIT.COM if you want to leave a comment; you should have no problems seeing the already posted ones.
On July 1st, Cisco Systems ran the annual International NetRiders competition and Andraž Piletič (whom you might remember from his NIL Monitor posts published in our blog) was the top score winner.
Congratulations, Andraž!
I was absolutely delighted to hear the news (the best part was that almost nobody knew Andraž took part in this event). I always knew Andraž was a fantastic engineer and his achievement also shows that our student internship program rocks.
In one of our discussions, Stretch provided an excellent graph illustrating that the ISP competition seems to reduce prices almost linearly and asked me in a later comment to justify the inverse relation between subscription charges and consumer choice.
You might consider this debate to be purely between Stretch and myself, but it’s an interesting example of what you might need to do in daily your job. If you want to be a great networking engineer, you have to be prepared to question everything, including common wisdoms, “well-known truths”, “common practices” and facts that look too good to be true. Ready? Let’s go …
Start with the source. The graph came from ars technica article US broadband report: more popular, more expensive, which used the data from Pew Internet’s Home Broadband Adoption 2009 report. I had a “somewhat” biased opinion (I’m playing the devil’s advocate here), but this clearly falls into “looks too good to be true” category. Anyhow, if you want to understand what’s being reported, you have to go as far back to the original data as possible; in our case, read the Pew Internet’s report, not the summary provided by ars technica (although they did a good job).
The obvious conclusion. Looking at the graph, the message is simple: whenever there’s competition, the prices go down. Are you happy with this explanation? Do you question its validity? Do you understand what the underlying unknown variables might be?
Find the alternatives. Whenever something looks too good, consider the alternatives. How about this: it’s easy to have low prices in urban environment with very high population density (Manhattan, London, Paris, Vienna or Singapore). It’s also obvious that there’s a lot of competition there (everyone tries to cherry-pick these customers). The rural places have low population density, therefore higher prices (assuming the prices somewhat reflect the actual cost of providing services) and almost no competition. The low population density would thus automatically correlate higher prices (assuming they are related to higher costs) with low competition. Are there any other alternative explanations? Give them in the comments.
Evaluate the alternatives. Do you find the second scenario viable? I’m not claiming it’s correct, I’m trying to nudge you to think. Can you find enough information in the graph, the article or the report to decide which explanation is valid? Document you findings in the comments.
Why does this matter? Why should you as a network engineer invest your time in this process? You’re constantly bombarded with “facts” nicely laid out in media articles, design guides, performance charts or whitepapers. Do you accept the “fact” that a router should belong to at most three OSPF areas and that an area should have at most 50 routers … or are you trying to understand what the actual limitations are? Are you happy to be mediocre and use simple recipes (without knowing where they come from) or will you try to become innovative by understanding the in-depth details and using them to your advantage? Are you willing to digest whatever media throws at you or do you want to form your own opinion? The choice is yours.
I'm looking for details on how Netflix streams videos over the Internet. I've found their description of encoding and bit rates, but was not able to find lower-layer details (I can only assume they use UDP, but I would like to verify that with someone who's actually using the service).
I would also appreciate any information on whether they work with Service Providers (for example, using local direct peering) to ensure the upstream Internet connections are not clogged with streamed video.
Thanks in advance for your responses!
Ivan
Every so often, someone asks me what tools I use to draw the diagrams. Years ago I was perfectly happy with Visio, but since Microsoft bought it, it became so bloated that I’ve been forced to drop it (it would take minutes to start on my laptop) and revert back to PowerPoint.
Cisco provides great icon libraries (including the visionary “space router” icon shown on the right) in Visio and PowerPoint format and I’m lucky enough to have an older version where the colors of the devices are not light blue but a darker shade of blue/green/gray. Drawing connections between the devices is obviously easier in Visio than in PowerPoint, but if you keep the diagrams simple, you can work around the limitations.
Export from PowerPoint to JPEG/PNG has always been a nightmare with dubious results (although it looks like the copy/paste from PowerPoint to Paint Shop Pro produces reasonable quality in some cases). To work around this, I’m using SnagIT from Techsmith to capture the screen (Paint Shop Pro also has a screen capture utility, but SnagIT is so much easier to use) and trim/resize the images.
After the image is trimmed and resized, I need to add the final touch: replace the white background with PNG transparency so the diagrams look good in our Wiki, where the images are shown in a light gray frame. I use Paint Shop Pro as I happen to have it installed (SnagIT does not have this functionality), but any other decent image manipulation tool or even a PERL script with ImageMagick (which I am too lazy to write) would do.
Following the ADSL QoS discussions, I decided to test “a few” things in the lab. I didn’t want to build a huge lab with DSL modems and DSLAM and decided to emulate an end-to-end DSL network with routers.
Step 1: Create a PPPoE session between a client (SOHO router) and a server (NAS). I’d never configured it before, so I’ve visited uncle Google first. It gave me tons of useless hits (no wonder) and a few somewhat useful ones. All of them used the old VPDN-centric PPPoE syntax. It seems I’ve stumbled across another “niche spot”: PPPoE testbed implemented with recent IOS configuration syntax (bba-group).
Read more in the “PPPoE Testbed” article in the CT3 wiki.
A few days ago a member of the cisco-nsp mailing list asked an interesting question: “the passive-interface command is not available in a VRF OSPF process. What can I do?”
It turned out he stumbled across CSCeb86068, which is already fixed in a later software release for his platform.
The passive-interface command tells the routing process to ignore packets received from the specified interface. In case of OSPF, the relevant packets are the hello packets, as an OSPF router will not exchange routing updates without an established adjacency. You can get the same results by deploying an inbound access list on the interface (which is the functionally equivalent workaround for this bug), although this method generates more configuration overhead than the OSPF-specific solution.
On the other hand, it’s a good practice to have inbound access lists dropping unnecessary protocols on the stub (client-only) interfaces, so it might be very simple to add OSPF to the list of undesired protocols.
You could also use another workaround: declare only the transit interfaces in the OSPF process (or use per-interface ip ospf area configuration command) and redistribute stub interface prefixes into OSPF, but this solution unnecessarily pollutes the OSPF database with type-4 and type-5 LSAs.
Anderson sent me an interestion question:
I'd like to use the snmpset command to get my router to execute an EEM script. Are there OIDs that are associated with EEM scripts that could help me achieve this?
Although EEM has associated MIB, it has a single read-write variable: the size of the history table. It's thus not possible to use EEM MIB to trigger EEM events. However, EEM 2.4 added support for SNMP notification events, which you can use to trigger EEM applets based on incoming SNMP traps/informs.
You can therefore use the event snmp-notification command on a router and the snmptrap command on a Linux host to remotely trigger EEM applets.
Read more in the Trigger EEM applets with SNMP Informs article in the CT3 wiki.
This article is part of You've asked for it series.
The “All-I-can-eat-mentality” article has triggered (as expected) numerous responses. Some of them provided useful data, links to more information or informative perspectives – many thanks to those readers. A few others were unfortunately following the “I-am-right” line without considering facts. Most of the readers from the Service Provider community decided to stay anonymous (when you read all the comments, it becomes obvious they made a wise decision) or respond off-line.
Whatever your position in this issue, I would like to ask you to keep your comments focused on the topic. Although you were all infinitely more polite than the usual forum/blog crowd and provided some really good arguments, writing angry replies does not help. What’s happening with Internet is (like it or not) our common problem … or you could take the blue pill and continue bashing the other side.
I particularly liked the summary of our discussion posted on Slashdot (where someone included the link to my blog):
Whoa, whoa, whoa, that article seems to be promoting a balanced viewpoint that denies a) that telcos are totally evil and b) that we should all be allowed to have as much bandwidth as we want and not have to pay for it. We'll have none of that nonsense on /.
It’s also painfully obvious that the current handling of ISP service offerings is often dismal. For example:
In the next few weeks, I’ll try to cover a few of the topics raised in the comments, including:
According to FeedBurner statistics, my RSS feed had exactly 5000 subscribes on June 30th.
Disclaimer: The Feedburner subscriber count is a notoriously unreliable measure, as it tries to estimate the actual number of subscribers for on-line services that cache the feeds (for example, Google Reader) … but the result is nonetheless quite nice.
Red Pineapple was kind enough to share his 15-year-old ATM slides. They include interesting claims like:
ATM has the potential to displace all existing internetworking technologies
One single network handles all traffic types: Bursty data and Time-sensitive continuous traffic (voice/video).
All these claims are still true if you just replace »ATM« with »IP«. So what went wrong with ATM (and why did the underdog IP win)? I can see the following major issues:
The shift that really made ATM obsolete was the changing data networking landscape: voice and long-lived low-bandwidth data sessions which dominated the world at the time when ATM was designed were dwarfed by the short-lived bursty high-bandwidth web requests. ATM was (in the end) a perfect solution to the wrong problem.
Two days ago I’ve managed to write aGenuineStupidity™ (OK, maybe I cannot get a trademark on this concept): the MQC shaping actions cannot be attached to a Dialer interface; they have to be specified on the underlying physical interface (in case of PPPoE link, the outside Ethernet interface).
The reason for my stupidity (apart from the obvious one: writing without testing) is the difference between true logical interfaces and dialer templates. A tunnel interface or a VLAN interface is a true logical interface; it behaves like any other interface (with a few exceptions; for example, tunnel interface does not have an output queue) and you can use most QoS actions (including shaping) on it. A dialer interface is even more “conceptual”. It can never be operational on its own – as soon as the link is established, it’s bound to a physical (for example, BRI0:1) or virtual access interface (which is yet again bound to a physical interface) and the shaping is performed on the final physical interface.
This behavior (on top of being unexpectedly inconsistent) results in interesting quirks. For example, you have to shape PPPoE packets (based on their IP characteristics) on the physical Ethernet interface which usually doesn’t even have an IP address.
… and let’s hope that the late hour hasn’t resulted in another blunder.
Last week I happened to be copied on an e-mail which included a PDF describing a “SP Services and Solutions” bootcamp I knew nothing about. As I was slowly reading the document, I’ve realized it’s a perfect course for Service Provider (or channel partner) engineers who need an overview of the various technologies used in modern (I hate to use the marketing phrase “next-generation”) SP networks and the ways they can be used to create customer-focused solutions.
My next question (to our training manager) was: “when is this going to be available, I want to blog about it” and the answer astonished me: “we’re doing the first customer teach this week”. Our SP gurus managed to develop a course they always wanted to teach almost undercover.
Read the rest of my thoughts about this fantastic course in Fragments
Posts
All Comments
