This post is part of the series Building your own Software Defined Network with Linux and Open Source Tools and covers the hardware platforms used within the backbone network infrastructure.
In the early days into the project we didn’t have much funds but thankfully received quite some donations in terms of old hardware as well as money. As we were young and didn’t know what we know today, we went down quite some different roads, made lots of experiences along the way, eventually reaching the setup we have today. This posts lists most the platforms we used within the last years, basically only leaving out early wireless platforms and sponsored server machines.
As most Freifunk communities rely heavily on products from the portfolio of Ubiquiti Networks, quite some devices will be covered. In the following I will just call them ubnt.
Routers and Switches
Every network needs routers. Within the Freifunk backbone network we need routers for usual IPv4 + IPv6 traffic as well as routers for the B.A.T.M.A.N. advanced mesh network. We basically use(d) two categories of devices for these purposes:
Ubnt EdgeRouters
We started out with some EdgeRouter Lite and EdgeRouter POE boxes which at the time (2014/2015) were quite shiny little boxes with quite some bang for the buck. We even tried (and succeeded) to get B.A.T.M.A.N. adv. to run on one of them, but this turned out to be painful and not update-safe to we dropped the approach into the bin of history and the boxes went there with it.
When the EdgeRouter X and EdgeRouter X-SFP dropped they opened up a new incredibly cheap opportunity for simple routers and switches. We still use some of those in some small setups where we need a small switch with PoE and VLAN capabilities, with or without and the need for IP routing / CPE functionality. The fact that they can run OpenVPN with enough throughput to fill some not so broadband-ish DSL lines is the cherry on top.
PCengines APU Boards
The main platform we use as routers since 2015ish are APU boards by PCEngines. These small form factor, fanless and low power (6-8W) boards come with a regular amd64 CPU, 2GB or 4GB of RAM, 2-4 1GB NICs and can (and should) be fitted with a mS-ATA SSD.
They run any off the shelf Linux – Debian in our case – and therefore can run any daemon, tool, or infrastructure component you can imagine. We use bird for IP routing (with OSPF + BGP), VXLAN overlays to set up PTP L2 bridges and B.A.T.M.A.N. adv. on top, as described in more detail in a previous post. As of today our network consists of 35 backbone and edge routers based on this platforms and more to come.
For 19″ installations we use the 19″ version which includes the board + PSU.
TP-Link TL-SG3126 / TL-SG3424 / TL-SG5412F
Thanks to a gift we got our hands on a bunch of TP-Link switches from the 1Gb/s era, which can be managed, are VLAN and LACP capable, fanless, and have a very low power consumption.
We don’t use them anymore as they have been replaced by PoE capable devices in access POPs and by devices which can do 10Gb/s in the data centers, but still keep them around for ad hoc setups in refugee camps of other occasion. If you can get your hand on them somewhere and have a use case, they still are a good choice.
Netonix WISP Switches
For our access and backbone POPs which are connected via WiFi links we have mostly settled on Netonix WS-12-250-AC or WS-6-MINI. They support VLANs, LAGs, SNMP, remote logging, etc. and can be managed by a nice WebUI or CLI.
At the time we were looking for a new device for this role we had more actively been using Ubnt AF-5Xes which require 24VH PoE, which only few devices could provide at the time, one of them being the Netonix ones. They have become incredible expensive and hard to get in the last years so might need to look for an alternative although we don’t have any other reason to change to a new platform.
Cisco 3560 / 3750 / 4900-M / ME-3600X
Thanks to some other nice gifts we are the proud owners of a bunch of Cisco Catalyst 3560 & 3750 switches (with and without PoE) as well as a ME-3600X metro ethernet switch. They basically support everything one could hope for and a lot more. We use these as access and distribution switches in larger refugee installations and temporary setups.
Some years ago we got our hands on some Cisco Catalayst 4900-M switches which we are currently using as data center switches. With the right modules they provide a nice mix of 1GE copper and 1Gb/s + 10Gb/s module ports for fiber connections, and with dual PSUs provide power redundancy which previous DC switches did not provide. Given the space and power consumption they will likely be replaced shortly and another platform will be added to this post 🙂
Optical equipment
When we got our first metro dark fiber connection to connect two of our data centers we decided to go for CWDM on top of that link to build basic redundancy for failing transceivers. This way we can survive one transceiver decaying or going on vacation with any immediate impact to our network and have time to debug the issue and replace the broken transceiver.
As we are running this infrastructure in our spare time having more head room before any problem creates user noticeable impact is an important issue we factor into out design choices.
Wireless backbone
Most of our POPs are connected via Wireless links, as they are built on locations where the owners allow us to put stuff on the roof and maybe APs on the walls, but we don’t own any fiber nor would it be affordable to pull any. Today we only use 5GHz based wireless hardware from ubnt in our production network, but are evaluation 60GHz solutions too.
Ubnt AirFiber 5X
In 2015 we heard a lot of good things of AirFibers 5Xes and were able to get our hands on a bunch of those. Sadly the experience wasn’t that great as a lot of links went into RADAR detect on a regular basis rendering the links unreliable. Also the physical connectors on the LAN side seemed to have been flaky at least on the old models, so we started to migrate them out of our network rather soon.
Ubnt AirMax devices
Today most of our wireless backbone links are built using Ubnt Air Max devices operating in the 5GHz spectrum. This includes NanoBeams, PowerBeams, as well as LightBeams. Within the backbone we strictly build PTP connections as one would also do using wired connections.
For local connections in larger sites we sometimes use LiteBeam APs with a 120° sector antenna to fan out to multiple buildings.
Wireless access (points)
For client connectivity we we mainly use Ubnt access points. For indoor installations this usually are AC Lite or AC pro models, for outdoor installations Mesh and Mesh Pro are used in a lot of places. Given current lead times (of up to 10 months lately) for some of these devices we will be looking into alternatives, for example TP-Link Omada.
For a large installation in a refugee camp a set of Ruckus AC APs including controllers are used to provide WiFi access in a dense environment.
Servers
Obviously we also need some compute and storage to run basic infrastructure services. Most of these run on SuperMicro 1RU boxes sitting in some data centers. We also got some gifts of Dell, HP and Isilon hardware which we use for different purposes. Besides the latter they all or off the shelve servers with all of them running plain Debian Linux.