FrOSCon 13 Network Track – Videos and Slides

A month ago the 13th Free and Open Source Software Conference (FrOSCon) took place in St. Augustin, Germany.  At this years event I organized a two day Network Track designed for a broad audience of Linux folks, system administrators and developers to answer questions about networking topics they were afraid to ask or didn’t realize they wanted or had to know! As the lines between system engineering and network engineering keep on blurring it’s getting more and more important to broaden the focus in both worlds, keywords being things like SDN, IP-Fabric, Segment Routing etc. here.

The track started with a lecture about networking basics and over both days advanced to technically more sophisticated topics following a red line.

On Saturday the focus was on Layer 2 and Layer 3 fundamentals (Ethernet switching and routing), dynamic routing protocols as well as the Linux packet-filter. Sundays track started gently with VLANs, Bonding and Bridging and advanced to more sophisticated topics like policy-based routing, VRFs, Open vSwitch, Segment Routing and Software Defined Networking. The track concluded with an overview about Best Current Operational Practices and a Q&A sessions.

All these talks are – thanks to the nice folks at CCC-VOC – available on Video at media.ccc.de (german audio) as are the slides (english):

Day 1

Day 2

  • Adv. topics in Layer2/3 – Light and dark magic with the Linux network stack  (Video) (Slides)
  • Segment Routing  (Video) (Slides)
  • Open vSwitch – The switch within your machine  (Video) (Slides)
  • Best Current Opertional Practices – Dos, Don’ts and lessons learned
    (Video) (Slides)
  • Building your own SDN – The penguin orchestrates the network  (Video) (Slides)
  • Grand Q&A  (Video)

Topology of a Freifunk network

This post is part of the series Building your own Software Defined Network with Linux and Open Source Tools and covers the re-designed topology of the distributed infrastructure.

From a birds eye perspective, the Freifunk Hochstift infrastructure mainly consists of three building blocks:

  1. Distributed servers hosted in data centers in Paderborn as remotely in Germany providing infrastructure services
  2. Wireless backbones within the city centers of Paderborn, Warburg, etc.
  3. Freifunk nodes at homes, shops, enterprises, or elsewhere

This post will focus on the distributed servers as well as the wireless backbones and will only cover the around 1.000 client nodes from the perspective of connecting them to the backbone (“gateways“).

With all the things mentions in Specifics and history of a Freifunk network in mind I got back to the drawing board and thought about a new design.

Continue reading Topology of a Freifunk network

GulaschProgrammierNacht 2018 – Awesome talks, fiber cuts and a lot of fun

As every year some weeks after the EasterHegg, the GulaschProgrammierNacht (GPN18) took place at  Hochschule für Gestaltung (HFG) and Zentrum für Kunst und Medien (ZKM) in Karlsruhe. It’s a four day event with a lot of lectures, workshops, Gulasch, a lounge and other amazing things; like any chaos event. As usual the C3VOC did an amazing job streaming and recording (nearly) all sessions!

CC-BY 4.0 by Flo Köhler

 

Awesome talks

The GPN had a huge programm with so many technical, cultural, social, … sessions. I would like to especially highlight Alles was ihr schon immer über Glasfasern wissen wolltet (de) by Marc & Momo, Moderne Kommandozeilen-Werkzeuge (de) + Standards – Gut, dass so Viele zur Auswahl stehen (de) by Martin as well as One Brain, One Keyboard, One Editor (en) by Miro.

The network

At the NOC – as always – we had some fun with shiny hardware and running a 4,5 day conference ISP for fun and Tschunk. This year Juniper and Arista provided some nice boxes which made up the core of the GPN network. An Arista DCS-7280SR was the core and border router and connected the GPN18 to the world with 120Gbit/s, a Juniper QFX5100 distributed all that bandwidth within the building, which appeared to be a setup we might want to adopt for future GPNs. As it turns out Aristas Cisco-like CLI is not quite cisco, but better 🙂

As it happens, not everything went after plan and we have a major downtime in the middle of the night due to a fiber cut and a misconfiguration coming together – Murphy must have been at the GPN, too. We had built a 2x40Gb/s LAG with two redundant paths from core to distribution, but one interfaces was in 4x10G mode instead of 1x40G and therefor not part ofthe LAG (so check your ports kids!) and one fiber patch of the active path was taped to the wall and broken by accident. Luckily it was in the middle of the night.

Arista’s not Cisco – Nifty CLI features

At GPN18 we had an Arista DCS-7280SR as our core and border router. The Cisco-like CLI made it easy to configure the system as I know my way around IOS*.

While setting up the final BGP sessions to our upstreams at the GPN18 we by accident found out that Arista supports watch on their CLI which is quite awesome when you want to see if your peering are coming up.

router> watch sh ip bgp summary

The next thing Arista can do which Cisco can’t is show active. When you are in a config stanza like an interface, access-list or router bgp 13020, you can print the current configuration of this sub-tree; in configure mode! This is something every Cisco admin would love to have as it’s not possible to do a show running-config <thingy> for most parts of the config tree and you have to fiddle around with show running-config | section <something>.

As we grew fond of show active really fast we wanted to use it to verify our access-list changes etc. and were wondering why they didn’t show up in show active directly after attemping the chance.

core-hfg#sh ip access-lists bgp
IP Access List bgp
 10 permit tcp host 1.2.3.4 any eq bgp
 20 permit tcp host 2.3.4.5 any eq bgp
 30 permit tcp host 3.4.5.6 any eq bgp

core-hfg#conf t
core-hfg(config)#ip access-list bgp
core-hfg(config-acl-bgp)#40 permit tcp host 9.8.7.6 any eq bgp
core-hfg(config-acl-bgp)#show active 
ip Access List bgp
 10 permit tcp host 1.2.3.4 any eq bgp
 20 permit tcp host 2.3.4.5 any eq bgp
 30 permit tcp host 3.4.5.6 any eq bgp

It turns out some config changes only get active when you leave the block you were editing, which is another subtle difference to Ciscos behavior!

core-hfg(config-acl-bgp)#exit
core-hfg(config)#ip access-list bgp
core-hfg(config-acl-bgp)#show active 
ip Access List bgp
 10 permit tcp host 1.2.3.4 any eq bgp
 20 permit tcp host 2.3.4.5 any eq bgp
 30 permit tcp host 3.4.5.6 any eq bgp
 40 permit tcp host 9.8.7.6 any eq bgp

Update:

Some changes are applied immediately on interface level, for example (no) shut, VLAN changes, … These are visible in show active as expected. Thanks for Nico for the clarification!

Anycasted services with Debian, bird, anycast-healthchecker and Cisco Nexus 7000

A while ago we started getting alerts, that one of our Kerberos KDCs had problem with the Kerberos database replication. A little digging revealed, that the problems are caused by load spikes on the KDC which were the result of a burst of legitimate queries fired by some systems we didn’t have much control over. Additionally we found that the MIT Kerberos implementation queries all KDCs provided in the configuration file in sequential order, so the first KDC get’s nearly all queries. While thinking about load balancing solutions, quickly anycast came to mind, so we decided to set it up. Anycast leverages the Equal Cost Multipath Routing (ECMP)  capability of common routers to distribute traffic to multiple next-hops for the same destination.

The solution consists of three corner stones:

  1. anycast-healtchecker as a means to check service availability
  2. bird as a BGP speaker on the KDCs and route reflectors
  3. Data center routers (Cisco Nexus 7010) speaking BGP to the route reflectors

The topology is as follows:

Topology KDCs

Continue reading Anycasted services with Debian, bird, anycast-healthchecker and Cisco Nexus 7000

Specifics and history of a Freifunk network

This post is part of the series Building your own Software Defined Network with Linux and Open Source Tools and covers the specifics of a Freifunk network as well as the history for the Freifunk Hochstift network which led to the latest re-design.

From a birds eye perspective, the Freifunk Hochstift infrastructure mainly consists of three building blocks:

  1. Distributed servers hosted in data centers in Paderborn as remotely in Germany providing infrastructure services
  2. Wireless backbones within the city centers of Paderborn, Warburg, etc.
  3. Freifunk nodes at homes, shops, enterprises, or elsewhere

This post will focus on the distributed servers as well as the wireless backbones and will only cover the around 1.000 client nodes from the perspective of connecting them to the backbone (“gateways“).

Continue reading Specifics and history of a Freifunk network

Beware of the details (and VLANs)

A friend today challenged me with this problem on an Ubuntu box:

auto eth2
iface eth2 inet static
    address 10.23.0.32
    netmask 31

auto eth2.210
iface eth2.210 inet static
    address 10.42.0.32
    netmask 31

root@box:~# ifup eth2.210
Cannot find device "eth2.210"
Failed to bring up eth2.210.

The first thing coming to mind is “package vlan missing?”, which it was. After installing it, it got more interesting:

root@box:~# ifup eth2.210
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
ifup: recursion detected for interface eth2 in parent-lock phase
ERROR: trying to add VLAN #210 to IF -:eth2:-  error: File exists
Cannot find device "eth2.210"
Failed to bring up eth2.210

A little poking around showed, that there’s no interface eth2.210 present in the system, but

root@box:~# cat /proc/net/vlan/config 
VLAN Dev name | VLAN ID
Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD
rename10       | 210  | eth2

root@box:~# ip l
[...]
10: rename10@eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether DE:AD:BE:EF:23:42 brd ff:ff:ff:ff:ff:ff
[...]

Deleting the renameNN interface an running ifup again, just created a renameNN+1 interface with kernel log entries like

[7366672.699018] rename14: renamed from eth2.210

I suspected some bugs in /etc/network/if-{,pre-}up.d/ but this even happend, when manually running

ip link add link eth2 name eth2.210 type vlan id 210

or

ip link add link eth2 name vlan210 type vlan id 210

Dafuq?

Continue reading Beware of the details (and VLANs)

Building your own Software Defined Network with Linux and Open Source Tools

Nearly two years ago, I started thinking about a next generation design for the Freifunk Hochstift backbone infrastructure, motivated by the limits and design choices made before (we were young and didn’t know better.. or didn’t listen..).

This post is the starting point of a series of posts about building a software defined wireless ISP network with Linux, a fistful of Open Source tools and low cost hardware.

Continue reading Building your own Software Defined Network with Linux and Open Source Tools

FFRL Routingdays – Learn to build the Internet

Over a year ago we (the Freifunk Rheinland Backbone Operators) organized the FFRL Routindays, a two day event where we gave a hands-on trainig on “How to build the Internet” or any internal network. As of periodic quesions about where to find the slides and videos here’s a summary of all media available of this event.

The event has entirely been in german language, as are all slides and lab guides. This is provided in the hope, that it still might be of some use even for non-german speakers.

The building blocks were

  1. Networking / IP basics
  2. Static routing + bird hands-on Lab
  3. OSPF
  4. eBGP
  5. eBGP -> OSPF redistribution
  6. iBGP
  7. Policy-based Routing

The complete slidedeck is available here. Thanks to the CCC VideoOperation Center, there are recordings of the event avilable on media.ccc.de, too.