Blog Posts in June 2023

Ignas Bagdonas sent a phenomenal summary of recent BGP developments to the RIPE Routing WG mailing list. Enjoy!

Found an interesting article describing the shenanigans of a biotech startup. Admittedly, it has nothing to do with networking apart from the closing paragraph…

But people will find all sorts of ways to believe what they want to believe, to avoid hearing things that they don’t want to hear, and to avoid thinking about things that are too worrisome to contemplate.

… which is a perfect description of why people believe in centralized control planes, flow-based forwarding, or long-distance vMotion.

Long story short: it’s time for another summer break, as people reporting my bloopers – THANK YOU!!! – know only too well. I plan to be back in early autumn rolling out tons of new content.

I’ll do my best to reply to support requests (it will take longer than usual), and probably won’t be able to resist publishing a few lightweight netlab-related blog posts. If you get bored there’s still over 400 hours of existing content, over 100 podcast episodes, and thousands of blog posts.

In the meantime, get away from work, turn off the Internet, and enjoy a few days in your favorite spot with your loved ones!

An anonymous commenter asked this highly relevant question about my Internet routing security lab:

What are the smallest hardware requirements to run the lab.

TL&DR: 2 GB RAM, 2 vCPU

Now for the more precise answer (aka “it depends”).

After I published the Source IP Address in Multicast Packets blog post, Erik Auerswald sent me several examples of network devices sending IP packets with source IP address set to 0.0.0.0:

• Cisco wireless access points using 0.0.0.0 as the source IP address in VRRP packets.
• Extreme (formerly Avaya) switches sending IGMP queries with source IP address 0.0.0.0 on VLANs on which they have no IP address.

I created a netlab topology you can use to practice BGP security tools I described in the Internet Routing Security webinar:

• The lab topology mirrors the sample topology I described in the Classification of BGP Route Leaks (RFC 7908) blog post with one router per autonomous system
• BGP is configured on all devices, and EBGP sessions are set up between all directly-connected devices.

Julia Evans published another phenomenal blog post, this time focused on blogging myths including:

• You need to be original
• You need to be an expert
• Posts need to be 100% correct
• Writing boring posts is bad
• You need to explain every concept
• Page views matter
• More material is always better

Dip Singh published an excellent primer on communication fundamentals including:

• Waves: frequency, amplitude, wavelength, phase
• Composite signals, frequency domain and Fourier transform
• Bandwidth, fundamental and harmonic frequency
• Decibels in a nutshell
• Transmission impairments: attenuation, distortion, noise
• Principles of modern communications: Nyquist theorem, Shannon’s law, bit and baud rate
• Line encoding techniques, quadrature methods (including QPSK and QAM)

Even if you don’t care about layer-1 technologies you MUST read it to get at least a basic appreciation of why stuff you’re using to read this blog post works.

Andrei Robachevsky asked me to spread the word about the new MANRS+ customer survey:

MANRS is conducting a survey for organizations that contract connectivity providers to learn more about if and how routing security fits into their broader supply chain security strategy. If this is your organization, or if it is your customers, we welcome you to take or share the survey at https://www.surveymonkey.com/r/BDCWKNS

I hope you immediately clicked on the link and completed the survey. If you’re still here wondering what’s going on, here’s some more information from Andrei:

Years ago I wrote an article describing how EIGRP stub routers work and how you should use them in redundant remote sites to make sure link- or node failures don’t result in partial connectivity. That article is now available on ipSpace.net; I hope at least someone will find it useful. I know it’s about ancient technology, but then people are still running COBOL on mainframes.

While preparing the Internet Routing Security webinar, I stumbled upon RFC 7908, containing an excellent taxonomy of BGP route leaks. I never checked whether it covers every possible scenario¹, but I found it a handy resource when organizing my thoughts.

Let’s walk through the various leak types the authors identified using the following sample topology:

Stefano Sasso added two new Junos platforms in netlab release 1.5.4:

• vJunos Evolved (vPTX) – a new VM-based product Juniper released a few weeks ago
• vSRX running as a vrnetlab-packaged container.

There are also the usual bug fixes, and we moved the documentation to netlab.tools.

Large language models (LLM) – ChatGPT and friends – are one of those technologies with a crazy learning curve. They look simple and friendly (resulting in plenty of useless demoware) but become devilishly hard to work with once you try to squeeze consistent value out of them.

Most people don’t want to talk about the hard stuff (sexy demoware results in more page views), but there’s an occasional exception, for example All the Hard Stuff Nobody Talks About when Building Products with LLMs describing all the gotchas Honeycomb engineers discovered when creating a LLM-based user interface.

Security researches found another ICMP redirect SNAFU: a malicious wireless client can send redirects on behalf of the access point redirecting another client’s traffic to itself.

I’m pretty sure the same trick works on any layer-2 technology; the sad part of this particular story is that the spoofed ICMP packet traverses the access point, which could figure out what’s going on and drop the packet. Unfortunately, most of the access points the researchers tested were unable to do that due to limitations in the NPUs (a fancier word for SmartNIC) they were using.

After discussing the backend and CPE architecture in a typical SD-WAN solution in the SD-WAN Overview webinar, Pradosh Mohapatra mentioned a few SD-WAN security aspects, focusing on typical attack vectors and the usual mitigations.

Got this question from a networking engineer attending the Building Next-Generation Data Center online course:

Has anyone an advice on LACP fast rate? When and why should you use it instead of normal LACP?

Apart from forming link aggregation groups, you can use LACP to detect link- and node failures (more details). However:

Distributed systems are complicated. Add networking to the mix, and you get traumatic challenges like the CAP theorem and Byzantine fault tolerance. Most of those challenges are unknown to engineers who have to suffer through the vendor marketing presentations, making it hard to determine whether the latest shiny gizmo works outside of PowerPoint.

I started collecting articles describing distributed-system gotchas years ago, wrote numerous blog posts on the topic in the heydays of the SDN Will Save the World lemming run, and organized them into the Distributed Systems Resources page.

EIGRP routing updates have always contained the next hop field (similar to BGP updates), which was unused until Cisco IOS release 12.3 when the no ip next-hop-self eigrp AS-number interface configuration command was implemented.

EIGRP does not set the next hop field by default. An EIGRP router receiving a routing update thus assumes that the next hop of the received routes is the sending router. This behavior usually works well, but prevents site-to-site shortcuts to be established in DMVPN networks, and results in suboptimal routing in some route redistribution scenarios.

One of the most common causes of Internet routing leaks is an undereducated end-customer configuring EBGP sessions with two (or more) upstream ISPs.

Without basic-level BGP knowledge or further guidance from the service providers, the customer network engineer¹ might start a BGP routing process and configure two EBGP sessions, similar to the following industry-standard CLI² configuration:

Brandon Hitzel published a detailed document describing various Internet WAN edge designs. Definitely worth reading and bookmarking.

Another phenomenal detective story published on Cloudflare blog: Unbounded memory usage by TCP for receive buffers, and how we fixed it.

TL&DR: Moving TCP window every time you acknowledge a segment doesn’t work well with scaled window sizes.

The interesting takeaways:

After introducing the routing protocols and explaining the basics of link-state routing it was time for implementation considerations including:

• Collecting local endpoint reachability information
• Finding neighbors and exchanging the collected information (hint: a link-state topology database is just a distributed key-value store)
• Running the SPF algorithm (including partial SPF details) and installing the results

One of my readers sent me this (paraphrased) question:

What I have seen in my network are multicast packets with the IP source address set to 0.0.0.0 and source port set to 0. Is that considered acceptable? Could I use a multicast IP address as a source address?

TL&DR: **** NO!!!

It also seemed like a good question to test ChatGPT, and this time it did a pretty good job.