Using QSFP-SFP+ adapters / break out cables with Cumulus Linux

As the two SFP+ ports of EdgeCore AS7726-32X aren’t supported by Cumulus Linux I had to use QSFP+-SFP+ adapters to make the 10G links work. As expected the port has be put in 10G mode to make this work, setting the link-speed isn’t enough though.

Configure the ports mode

Each port has it’s mode set in /etc/cumulus/ports.conf. This file looks like something like this:

# ports.conf --
# The Accton AS7726 has:
# 32 QSFP28 ports numbered 1-32
# These ports are configurable as 100G, 40G, or split into
# 4x25G, or 4x10G ports.

# QSFP28 ports
# <port label 1-32> = [4x10G|4x25G|40G|100G]


Using a QSFP+-SFP+ adapter now warrants 4x10G mode as this is the only mode breaking the port down in 10G ports although there is only one port.

To configure ports 29 and 30 accordingly the corresponding lines have to be set to


For these changes to take effect the swichtd service has to be restarted to program this into hardware. This is done by restarting the systemd service

systemctl restart switchd.service

Broken down interfaces will show up as swp<num>s<instance> (e.g. swp29s0) and should now be visible in the interface list

# ip -br l
swp29s0 UP   68:21:5f:39:19:b4 <BROADCAST,MULTICAST,UP,LOWER_UP> 
swp29s1 DOWN 68:21:5f:39:19:b5 <BROADCAST,MULTICAST> 
swp29s2 DOWN 68:21:5f:39:19:b6 <BROADCAST,MULTICAST> 
swp29s3 DOWN 68:21:5f:39:19:b7 <BROADCAST,MULTICAST> 
swp30s0 UP   68:21:5f:39:19:b8 <BROADCAST,MULTICAST,UP,LOWER_UP> 
swp30s1 DOWN 68:21:5f:39:19:b9 <BROADCAST,MULTICAST> 
swp30s2 DOWN 68:21:5f:39:19:ba <BROADCAST,MULTICAST> 
swp30s3 DOWN 68:21:5f:39:19:bb <BROADCAST,MULTICAST>

Configure the interface(s)

The break out interfaces no can be configured as usually by /etc/network/interfaces 

auto swp29s0
iface swp29s0
    address 2001:db8:2342:F000::1/126

auto swp30s0
iface swp30s0
    address 2001:db8:2342:F001::1/126

Those interface can be brought up with

# ifup swp29s0
# ifup swp30s0


# ifup -a


# ifreload -a

as you like 🙂

Initial setup for Cumulus Linux for networkers

I just had to set up Cumulus Linux on some EdgeCore AS7726-32X boxes and were a little underwhelmed by the available documentation. So this is my take at an initial setup guide.

Access to the switch

Assuming Cumulus is preinstalled on the box you have two ways to access the switch

by serial console

Fire up your minicom/screen/putty/whatever at 115200 and you will be greeted with a login prompt

by ssh via the mgmt interface

By default Cumulus Linux does dhcp on the mgmt interface(s), so you can ssh into the box by the IP given by DHCP.

The default credentials are

user: cumulus
pass: CumulusLinux!

Root access can be gained via sudo -i using the password from above.

Changing the password

cumulus@sw-01:~$ passwd
Changing password for cumulus.
(current) UNIX password: 
Enter new UNIX password: 
Retype new UNIX password:

Changing the hostname

As Cumulus Linux basically is a Debian Linux changing the hostname is easy and done in two steps:

# hostnamectl set-hostname <new-hostname>

Edit /etc/hosts and change the following line cumulus

to <fqdn> <hostname>

Setting up management in VRF

Edit /etc/network/interfaces with you favorite editor (vi and nano are present) and change the stanza for eth0 like the following

# The primary network interface
auto eth0
iface eth0
    vrf vrf_mgmt

auto vrf_mgmt
iface vrf_mgmt
    vrf-table auto

This will move interface eth0 into VRF vrf_mgmt which seperates routing from the main routing table. vrf-table auto let’s ifupdown2 decide on which Linux routing table it will use for the VRF. If you want to have a fixed table id you can specify that instead of auto. Note ifupdown2 by default requires VRF table ids to be over 1000.

ifup eth0 will bring up the new config on eth0 as well as the VRF. Another way to reload the interface configuration would be ifreload -a which would reload the configuration of all interfaces.

Setting up routed interfaces

Setting up IPs on interfaces happens in /etc/network/interfaces, too. The ports are numbered swp<num> with no leading zeros. Setting IPs on switchport 1 would look like this:

auto swp1
iface swp1
    address 2001:db8:2342:F000::1/126

Testinglos durch die Nacht…

Aus Gründen, die Euch nur noch mehr verunsichern würden, war es gerade alternativlos, diese Freifunkhymne zu bauen:

Wir ziehen durch die Knoten und die POPs dieser Stadt
Das ist unsre' Firmware, wie für uns alle gemacht
Oho Oho

Ich schließe meinen Browser, lösche jedes Cookie,
Gehe auf den Knot'n, so wie ich's im-mer tu'
Oho Oho

Was das zwischen uns auch ist, Links, die man nie vergisst
Und Dein Link hat mir gezeigt, das ist unser Mesh

Testinglos, durch die Nacht,
Bis ein neuer Build erwacht
Stablelos, einfach raus
Deine Knoten geh'n nicht aus!
Testinglos durch die Nacht
Spür was Gluon mit uns macht
Stablelos, signkeyfrei, großes Update für uns zwei
Wir sind heute ewig, tausend neue Knoten
Alles was ich hab, teil' ich mit Euch
Wir sind unzertrennlich, irgendwie verbunden
Komm nimm' meinen Link und mesh mit mir

Komm wir steigen auf das nächste Dach dieser Stadt
Schrauben einfach an was uns zusammen bringt
Oho oho

Bist Du richtig süchtig, Dish an Dish ganz verlinkt
Strahl in meine Schüssel und der Link geht an
Oho oho

Alles was ich will
Ist da, großer Uplink pur, ganz nah
Nein wir wollen hier nicht weg, alles ist perfekt

Testinglos durch die Nacht
Spür was Gluon mit uns macht
Stablelos, signkeyfrei, großes Update für uns zwei
Wir sind heute ewig, tausend neue Knoten
Alles was ich hab, teil' ich mit Euch
Wir sind unzertrennlich, irgendwie verbunden
Komm nimm' meinen Link und mesh mit mir

Funk pulsiert in meiner Hand
Testinglos durch die Nacht
Spür was Gluon mit uns macht
Stablelos, signkeyfrei, großes Update für uns zwei
Wir sind heute ewig, tausend neue Knoten
Alles was ich hab, teil' ich mit Euch
Wir sind unzertrennlich, irgendwie verbunden

OpenVPN and VRFs

Using VRFs on Linux enables a whole new set of network setups.

As described in the previous post about VRFs on Linux, VRFs allow to isolate different interfaces at layer 3. In the Freifunk Hochstift network we chose to consider the main or default VRF as the internal network and move any internet facing interfaces into an external VRF. Chosing this concept allows, to safely contain traffic within the internal network and only at  designated border routers leak eligible traffic into the internet.

In an distributed environment like the Freifunk Hochstift network, it is inevitable to connect different islands using VPN tunnels over the Internet. This could be done by the means of GRE tunnels as shown in the previous article about the border routers, or by means of encrypted VPNs like OpenVPN, IPsec or Wireguard. As the old infrastructure quite heavily relied on OpenVPN tunnels, and they worked quite well, the new setup should keep this building block in place.

Continue reading OpenVPN and VRFs

(Vlan-aware) Bridges on Linux

Lately I’ve had some conversations about how Linux sucks at bridging tagged VLANs into VMs, which just isn’t true anymore.

With recent Kernels Linux bridges have become vlan-aware and now allow configuring any bridge port like a port of any decent network switch with respect to 802.1q VLANS. A port can present a VLAN as untagged traffic as well as a number of VLANs in tagged mode. As can be expected, SVIs can be configured as vlan interfaces of a bridge, too.

The old brctl utility has been integrated within the iproute suite as part of of ip link. The commands map as follows:

brctl addbr br0
ip link add br0 type bridge
    [ forward_delay FORWARD_DELAY ]
    [ vlan_filtering VLAN_FILTERING ]
    [ vlan_default_pvid VLAN_D_PVID ]
    [ nf_call_iptables NF_CALL_IPT ]
    [ nf_call_ip6tables NF_CALL_IP6TABLES ]
    [ nf_call_arptables NF_CALL_ARPTABLES ]

brctl addif br0 eth0
ip link set eth0 master br0

Continue reading (Vlan-aware) Bridges on Linux

Seriously predictable interface names – An introduction to systemd .link files

Predictable interface names are a new thing. The most common argument made is that they are not really predictable though, depending on the point of view. How about making interface names predictable and meaningful in the same time?

Most admins will probably think of udev right now, which previously was heavily used to achieve exactly that. In times of systemd the new hotness are .link files which provide similar capabilities and allow even more options to be set for interfaces.

Continue reading Seriously predictable interface names – An introduction to systemd .link files

FrOSCon 13 Network Track – Videos and Slides

A month ago the 13th Free and Open Source Software Conference (FrOSCon) took place in St. Augustin, Germany.  At this years event I organized a two day Network Track designed for a broad audience of Linux folks, system administrators and developers to answer questions about networking topics they were afraid to ask or didn’t realize they wanted or had to know! As the lines between system engineering and network engineering keep on blurring it’s getting more and more important to broaden the focus in both worlds, keywords being things like SDN, IP-Fabric, Segment Routing etc. here.

The track started with a lecture about networking basics and over both days advanced to technically more sophisticated topics following a red line.

On Saturday the focus was on Layer 2 and Layer 3 fundamentals (Ethernet switching and routing), dynamic routing protocols as well as the Linux packet-filter. Sundays track started gently with VLANs, Bonding and Bridging and advanced to more sophisticated topics like policy-based routing, VRFs, Open vSwitch, Segment Routing and Software Defined Networking. The track concluded with an overview about Best Current Operational Practices and a Q&A sessions.

All these talks are – thanks to the nice folks at CCC-VOC – available on Video at (german audio) as are the slides (english):

Day 1

Day 2

  • Adv. topics in Layer2/3 – Light and dark magic with the Linux network stack  (Video) (Slides)
  • Segment Routing  (Video) (Slides)
  • Open vSwitch – The switch within your machine  (Video) (Slides)
  • Best Current Opertional Practices – Dos, Don’ts and lessons learned
    (Video) (Slides)
  • Building your own SDN – The penguin orchestrates the network  (Video) (Slides)
  • Grand Q&A  (Video)

Topology of a Freifunk network

This post is part of the series Building your own Software Defined Network with Linux and Open Source Tools and covers the re-designed topology of the distributed infrastructure.

From a birds eye perspective, the Freifunk Hochstift infrastructure mainly consists of three building blocks:

  1. Distributed servers hosted across Germany providing infrastructure services
  2. Wireless backbones within the city of Paderborn, Warburg, etc.
  3. Freifunk nodes at homes, shops, enterprises, or elsewhere

This post will focus on the distributed servers as well as the wireless backbones and will only cover the around 1.000 client nodes from the perspective of connecting them to the backbone (“gateways“).

With all the things mentions in Specifics and history of a Freifunk network in mind I got back to the drawing board and thought about a new design.

Continue reading Topology of a Freifunk network

GulaschProgrammierNacht 2018 – Awesome talks, fiber cuts and a lot of fun

As every year some weeks after the EasterHegg, the GulaschProgrammierNacht (GPN18) took place at  Hochschule für Gestaltung (HFG) and Zentrum für Kunst und Medien (ZKM) in Karlsruhe. It’s a four day event with a lot of lectures, workshops, Gulasch, a lounge and other amazing things; like any chaos event. As usual the C3VOC did an amazing job streaming and recording (nearly) all sessions!

CC-BY 4.0 by Flo Köhler


Awesome talks

The GPN had a huge programm with so many technical, cultural, social, … sessions. I would like to especially highlight Alles was ihr schon immer über Glasfasern wissen wolltet (de) by Marc & MomoModerne Kommandozeilen-Werkzeuge (de) Standards – Gut, dass so Viele zur Auswahl stehen (de) by Martin as well as One Brain, One Keyboard, One Editor (en) by Miro.

The network

At the NOC – as always – we had some fun with shiny hardware and running a 4,5 day conference ISP for fun and Tschunk. This year Juniper and Arista provided some nice boxes which made up the core of the GPN network. An Arista DCS-7280SR was the core and border router and connected the GPN18 to the world with 120Gbit/s, a Juniper QFX5100 distributed all that bandwidth within the building, which appeared to be a setup we might want to adopt for future GPNs. As it turns out Aristas Cisco-like CLI is not quite cisco, but better 🙂

As it happens, not everything went after plan and we have a major downtime in the middle of the night due to a fiber cut and a misconfiguration coming together – Murphy must have been at the GPN, too. We had built a 2x40Gb/s LAG with two redundant paths from core to distribution, but one interfaces was in 4x10G mode instead of 1x40G and therefor not part ofthe LAG (so check your ports kids!) and one fiber patch of the active path was taped to the wall and broken by accident. Luckily it was in the middle of the night.

Arista’s not Cisco – Nifty CLI features

At GPN18 we had an Arista DCS-7280SR as our core and border router. The Cisco-like CLI made it easy to configure the system as I know my way around IOS*.

While setting up the final BGP sessions to our upstreams at the GPN18 we by accident found out that Arista supports watch on their CLI which is quite awesome when you want to see if your peering are coming up.

router> watch sh ip bgp summary

The next thing Arista can do which Cisco can’t is show active. When you are in a config stanza like an interface, access-list or router bgp 13020, you can print the current configuration of this sub-tree; in configure mode! This is something every Cisco admin would love to have as it’s not possible to do a show running-config <thingy> for most parts of the config tree and you have to fiddle around with show running-config | section <something>.

As we grew fond of show active really fast we wanted to use it to verify our access-list changes etc. and were wondering why they didn’t show up in show active directly after attemping the chance.

core-hfg#sh ip access-lists bgp
IP Access List bgp
 10 permit tcp host any eq bgp
 20 permit tcp host any eq bgp
 30 permit tcp host any eq bgp

core-hfg#conf t
core-hfg(config)#ip access-list bgp
core-hfg(config-acl-bgp)#40 permit tcp host any eq bgp
core-hfg(config-acl-bgp)#show active 
ip Access List bgp
 10 permit tcp host any eq bgp
 20 permit tcp host any eq bgp
 30 permit tcp host any eq bgp

It turns out some config changes only get active when you leave the block you were editing, which is another subtle difference to Ciscos behavior!

core-hfg(config)#ip access-list bgp
core-hfg(config-acl-bgp)#show active 
ip Access List bgp
 10 permit tcp host any eq bgp
 20 permit tcp host any eq bgp
 30 permit tcp host any eq bgp
 40 permit tcp host any eq bgp


Some changes are applied immediately on interface level, for example (no) shut, VLAN changes, … These are visible in show active as expected. Thanks for Nico for the clarification!