Nearly two years ago, I started thinking about a next generation design for the Freifunk Hochstift backbone infrastructure, motivated by the limits and design choices made before (we were young and didn’t know better.. or didn’t listen..).
This post is the starting point of a series of posts about building a software defined wireless ISP network with Linux, a fistful of Open Source tools and low cost hardware.
Back in 2015 we already had some automation built with Salt Stack for managing package installations and config files for the Debian Wheezy and Ubuntu based machines which helped a great deal managing some mid two digit number of boxes and VMs in our spare time. The problems arose from the then monolithic design of the Salt configuration and the need to split of parts of the network – including B.A.T.M.A.N. advanced based mesh networks – into smaller segments for scalability reasons. As B.A.T.M.A.N. adv is a layer-2 based mesh network, scalability is an issue. When the networks consists of nearly 1.000 nodes, BUM traffic (broadcasts and unknown multicast) is becoming a problem.
While redesigning the entire infrastructure we faced quite some challenges – some expected, some unexpected. Within this series I will dive into details about the building blocks and lessons learned while building a new Freifunk Hochstift infrastructure basically from scratch. The series will be roughly outlined as follows (articles written will be linked here):
- Network topology, IPAM considerations and automated IP configuration
- Specifics and history of a Freifunk network
- Topology, layer 2 and layer 3 design
- Hardware platforms
- Introduction to Salt Stack and the chosen metadata (pillar) layout
- Salt states, Python modules, and external pillar for NetBox data
- Building a loop-free and redundant internal backbone
- OSPF for loopback reachiblity and management networks
- iBGP using route reflection
- VRFs on Linux
- Containing connections to the external world in a VRF
OpenVPNWireguard tunnels between sites
- Connecting to the Internet through AS201701
- eBGP and traffic engineering
- A new approach with several B.A.T.M.A.N. adv. mesh networks
- Why tons of VLANs don’t scale – VXLAN overlays, management with ifupdown2/-ng and config generation
- MTU trouble
- Anycasted services
- To be continued
For those being impatient on further articles there is a talk (in German) about the Salt-Orchestrated Software Defined (Freifunk) Networks (slides) I gave at the GPN17 in Karlsruhe this year (recording). As the whole thing is up and running since spring of this year, the full configuration is already available on github and is still growing. Within future articles parts of the repository will be references and further explained. Stay tuned.
2019 Update:
In fall 2019 we organised a Freifunk Tag in Paderborn, where I talked about what we’ve learned in 5 years before (recording, talk in German).