Compare commits

...

2 commits

Author SHA1 Message Date
88354454ef
Merge branch 'feature-2'
All checks were successful
Tests / test (push) Successful in 4m4s
Release Pipeline / set-release-target (push) Successful in 2s
Release Pipeline / upload-assets (lab-host.sh, scripts/lab-host.sh) (push) Successful in 7s
Release Pipeline / build (amd64, linux) (push) Successful in 0s
Release Pipeline / build (arm64, darwin) (push) Successful in 0s
Release Pipeline / checksums (push) Successful in 6s
Release Pipeline / release (push) Successful in 25s
Release Pipeline / publish (push) Successful in 0s
Release Pipeline / build (push) Successful in 1m40s
2026-10-13 09:57:29 +02:00
e9a0c19b72
labctl: segments sur des bridges de l'hôte, routeurs et passerelle VRRP #2
All checks were successful
Tests / test (push) Successful in 3m46s
Signed-off-by: GnomeZworc <nicolas.boufidjeline@g3e.fr>
2026-10-13 09:57:09 +02:00
24 changed files with 1766 additions and 578 deletions

View file

@ -1,8 +1,8 @@
# labctl
Monte un **lab réseau virtuel** à partir d'une topologie YAML : des VM QEMU/KVM reliées par des
segments L2, des switchs qui font passerelle et NAT, FRR installé et configuré sur les nœuds qui
le demandent, et une provision libre (scripts, fichiers, services à vérifier) pour installer ce
segments L2 (des bridges de l'hôte), des routeurs qui font passerelle — en VRRP quand ils sont
plusieurs — et NAT, FRR installé et configuré sur les nœuds qui le demandent, et une provision libre (scripts, fichiers, services à vérifier) pour installer ce
que l'on veut tester. Le plan est **déterministe** : même fichier, mêmes adresses, MAC et ports.
`labctl` tourne sur tout hôte Linux avec KVM (virtualisation imbriquée si l'hôte est lui-même une
@ -32,36 +32,43 @@ labctl status | down | ssh <nœud> [commande] [-run rép]
```
Par défaut : `-run ~/lab-run`, `-cache ~/.cache/labctl`, `-timeout 20m`. `up` télécharge et vérifie
les images (`SHA512SUMS`), crée des disques neufs, démarre les nœuds (switchs d'abord), attend la
fin de cloud-init puis vérifie les services de chaque nœud. Un fichier invalide est refusé avec
les images (`SHA512SUMS`), crée des disques neufs, les bridges des segments et les taps des nœuds,
démarre les nœuds (routeurs d'abord), attend la fin de cloud-init puis vérifie les services de
chaque nœud ; `down` arrête les nœuds et supprime bridges et taps. `up` et `down` tournent en
root. Un fichier invalide est refusé avec
toutes ses erreurs et un code de sortie 1.
## Topologie
Voir `examples/evpn-2hv/` : un switch, un route reflector et deux hôtes en EVPN.
Voir `examples/evpn-2hv/` : deux routeurs en VRRP, un route reflector et deux hôtes en EVPN.
- `images` : `url` et `sums` (HTTPS).
- `segments` : `switch`, `cidr` IPv4 (/8 à /30), `mtu` (9000 par défaut). Nom ≤ 12 caractères.
- `nodes` : `role` (`switch`, `rr` ou `host`), `image`, `cpus`, `memory` (Mio), `segments`,
`addresses` (adresse fixée par segment). Champs facultatifs :
- `segments` : `cidr` IPv4 (/8 à /30), `mtu` (9000 par défaut), `routers` (facultatif, par ordre
de priorité : aucun = L2 pur, un = passerelle sur son interface, plusieurs = passerelle en
VRRP par FRR). Nom ≤ 12 caractères. Chaque segment est un bridge `lb-<segment>` de l'hôte.
- `nodes` : `role` (`router`, `rr` ou `host`), `image`, `cpus`, `memory` (Mio), `segments`,
`addresses` (adresse fixée par segment). Un routeur est un nœud comme les autres, attaché aux
segments dont il est routeur. Champs facultatifs :
- `secondary` — adresses supplémentaires hors du CIDR du segment ;
- `loopback` — adresse sur une interface `dummy` `lo1` ;
- `frr` — un `frr.conf` déposé tel quel ; FRR est installé au premier démarrage (`bgpd`, et
`bfdd` sur `switch` et `rr`) ;
`bfdd` sur `router` et `rr`, `vrrpd` sur les routeurs en VRRP) ;
- `provision` / `post_provision` — scripts exécutés **avant** / **après** FRR ;
- `files` — `{<nom>: <chemin>}`, fichiers **ou répertoires** gravés dans un ISO propre au nœud et
recopiés dans `/etc/lab/files/<nom>` avant les scripts (aucune limite de taille cloud-init) ;
- `services` — unités systemd que `up` vérifie actives.
Les chemins sont relatifs au fichier de topologie. Les scripts reçoivent `LAB_NODE`,
`LAB_UPLINK` (interface du premier segment, qui porte la route par défaut) et `LAB_FILES`.
Ordre au premier démarrage : script du switch ou de la loopback, `provision`, FRR,
`LAB_UPLINK` (interface qui porte la route par défaut : premier segment qui a une passerelle,
`mgmt0` pour un routeur)
et `LAB_FILES`. Ordre au premier démarrage : service du routeur ou de la loopback, `provision`, FRR,
`post_provision` — dans un seul script en `set -eu`, la première étape en échec met cloud-init en
erreur.
Déduit de façon déterministe : passerelle d'un segment = première adresse (portée par le switch) ;
adresses suivantes dans l'ordre de déclaration des nœuds ; MAC `02:4c:<nœud>:<nœud>:<segment>:<côté>` ;
SSH d'administration sur `127.0.0.1:<2200 + rang>`. **Réordonner les nœuds change les adresses.**
Déduit de façon déterministe : passerelle d'un segment = première adresse (s'il a un routeur) ;
adresses suivantes dans l'ordre de déclaration des nœuds ; MAC `02:4c:<nœud>:<nœud>:<segment>:00` ;
tap `lt<i>` par port ; SSH d'administration sur `127.0.0.1:<2200 + rang>`. **Réordonner les
nœuds change les adresses.** Migration depuis 0.1 (rôle `switch`) : voir le guide.
## Hôte loué chez Scaleway : `scripts/lab-host.sh`

View file

@ -26,8 +26,17 @@ const (
var (
execve = syscall.Exec
procDir = "/proc"
geteuid = os.Geteuid
)
func requireRoot(command string, stderr io.Writer) bool {
if geteuid() == 0 {
return true
}
fmt.Fprintf(stderr, "labctl: %s must run as root: it manages the bridges and taps of the lab on this host\n", command)
return false
}
func defaultDir(parts ...string) string {
home, err := os.UserHomeDir()
if err != nil {
@ -81,6 +90,9 @@ func upCmd(args []string, stdout, stderr io.Writer) int {
fs.Usage()
return 2
}
if !requireRoot("up", stderr) {
return 1
}
cache, err := filepath.Abs(*cacheDir)
if err != nil {
fmt.Fprintf(stderr, "labctl: %v\n", err)
@ -154,6 +166,9 @@ func downCmd(args []string, stdout, stderr io.Writer) int {
fs.Usage()
return 2
}
if !requireRoot("down", stderr) {
return 1
}
l, ok := current(*runDir, stdout, stderr)
if !ok {
return 1

View file

@ -17,6 +17,12 @@ func runLab(args ...string) (int, string, string) {
return code, stdout.String(), stderr.String()
}
func asRoot(t *testing.T) {
t.Helper()
geteuid = func() int { return 0 }
t.Cleanup(func() { geteuid = os.Geteuid })
}
func TestRun_UsageOnMissingArguments(t *testing.T) {
for _, args := range [][]string{nil, {"plan"}, {"deploy", "x.yml"}, {"plan", "a.yml", "b.yml"}, {"render", "a.yml", "dir"}, {"render", "-key", "k.pub", "a.yml"}, {"render", "-bogus"}} {
code, _, stderr := runLab(args...)
@ -32,9 +38,9 @@ func TestRun_PlanOfTheShippedExampleTopology(t *testing.T) {
t.Fatalf("code %d, stderr %s", code, stderr)
}
for _, want := range []string{
"lab evpn-2hv: nodes 4, segments 1, cables 3",
"gateway 192.168.14.1",
"hv2 underlay 192.168.14.12/24 02:4c:00:03:00:00 20004 <-> sw1 p2",
"lab evpn-2hv: nodes 5, segments 1, ports 5",
"segment underlay: 192.168.14.0/24, mtu 9000, bridge lb-underlay, gateway 192.168.14.1 by vrrp 1 on r1, r2",
"hv2 underlay 192.168.14.12/24 02:4c:00:04:00:00 lt4",
"rr1 lo1 10.255.255.1/32 underlay 169.254.0.3/28 rr1.conf",
} {
if !strings.Contains(stdout, want) {
@ -49,10 +55,10 @@ func TestRun_InvalidTopologyExitsWithErrors(t *testing.T) {
images:
deb: { url: https://example.invalid/a, sums: https://example.invalid/b }
segments:
under: { switch: sw, cidr: 10.0.0.0/31 }
under: { routers: [sw], cidr: 10.0.0.0/31 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
rr: { role: router, image: deb, cpus: 1, memory: 512, segments: [under] }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [under] }
rr: { role: spine, image: deb, cpus: 1, memory: 512, segments: [under] }
`
if err := os.WriteFile(path, []byte(doc), 0o600); err != nil {
t.Fatal(err)
@ -61,7 +67,7 @@ nodes:
if code != 1 || stdout != "" {
t.Fatalf("code %d, stdout %q", code, stdout)
}
for _, want := range []string{path, `role "router"`, "prefix length out of range"} {
for _, want := range []string{path, `role "spine"`, "prefix length out of range"} {
if !strings.Contains(stderr, want) {
t.Errorf("stderr does not mention %q:\n%s", want, stderr)
}
@ -86,7 +92,7 @@ func TestRun_RenderWritesEveryNodeFiles(t *testing.T) {
if code != 0 {
t.Fatalf("code %d, stderr %s", code, stderr)
}
for _, node := range []string{"sw1", "rr1", "hv1", "hv2"} {
for _, node := range []string{"r1", "r2", "rr1", "hv1", "hv2"} {
if !strings.Contains(stdout, filepath.Join(out, node)) {
t.Errorf("stdout does not list %s:\n%s", node, stdout)
}
@ -105,7 +111,7 @@ func TestRun_RenderWritesEveryNodeFiles(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(args), "\n-netdev\nuser,id=mgmt0,restrict=on,ipv6=off,hostfwd=tcp:127.0.0.1:2202-:22\n") {
if !strings.Contains(string(args), "\n-netdev\nuser,id=mgmt0,restrict=on,ipv6=off,hostfwd=tcp:127.0.0.1:2203-:22\n-device\nvirtio-net-pci,netdev=mgmt0,mac=02:4d:00:03:00:00,romfile=\n-netdev\ntap,id=underlay,ifname=lt3,script=no,downscript=no\n") {
t.Errorf("qemu.args is not one argument per line:\n%s", args)
}
userData, err := os.ReadFile(filepath.Join(out, "hv1", "user-data"))
@ -150,6 +156,7 @@ func TestRun_LifecycleUsage(t *testing.T) {
}
func TestRun_CommandsWithoutALabInTheRunDir(t *testing.T) {
asRoot(t)
run := t.TempDir()
for _, cmd := range []string{"status", "down", "ssh"} {
args := []string{cmd, "-run", run}
@ -166,10 +173,14 @@ func TestRun_CommandsWithoutALabInTheRunDir(t *testing.T) {
func TestRun_StatusOfAStoppedLab(t *testing.T) {
code, stdout, stderr := runLab("status", "-run", savedLab(t))
want := `node role state pid ssh
sw1 switch stopped - 127.0.0.1:2200
rr1 rr stopped - 127.0.0.1:2201
hv1 host stopped - 127.0.0.1:2202
hv2 host stopped - 127.0.0.1:2203
r1 router stopped - 127.0.0.1:2200
r2 router stopped - 127.0.0.1:2201
rr1 rr stopped - 127.0.0.1:2202
hv1 host stopped - 127.0.0.1:2203
hv2 host stopped - 127.0.0.1:2204
segment bridge state gateway
underlay lb-underlay absent 192.168.14.1
`
if code != 0 || stdout != want {
t.Errorf("code %d, stderr %q, stdout:\n%s", code, stderr, stdout)
@ -177,6 +188,7 @@ hv2 host stopped - 127.0.0.1:2203
}
func TestRun_DownOfAStoppedLab(t *testing.T) {
asRoot(t)
code, stdout, stderr := runLab("down", "-run", savedLab(t))
if code != 0 || stdout != "" || stderr != "" {
t.Errorf("code %d, stdout %q, stderr %q", code, stdout, stderr)
@ -208,7 +220,7 @@ func TestRun_SSHExecsSSHWithTheNodePort(t *testing.T) {
"-o", "StrictHostKeyChecking=no",
"-o", "UserKnownHostsFile=/dev/null",
"-o", "LogLevel=ERROR",
"-p", "2203",
"-p", "2204",
"debian@127.0.0.1",
"ip", "-br", "a",
}
@ -225,6 +237,7 @@ func TestRun_SSHUnknownNode(t *testing.T) {
}
func TestRun_UpRefusesAnInvalidTopologyAndKeepsTheSavedOne(t *testing.T) {
asRoot(t)
run := savedLab(t)
before, _ := os.ReadFile(filepath.Join(run, "topology.yml"))
bad := filepath.Join(t.TempDir(), "bad.yml")
@ -262,6 +275,7 @@ func TestIsTerminal_DevNullAndPipesAreNotTerminals(t *testing.T) {
}
func TestRun_UpRefusesToReplaceARunningLab(t *testing.T) {
asRoot(t)
run := savedLab(t)
before, _ := os.ReadFile(filepath.Join(run, "topology.yml"))
proc := t.TempDir()
@ -313,7 +327,8 @@ func TestRun_RenderShipsTheExampleFRRConfigs(t *testing.T) {
t.Fatalf("code %d, stderr %s", code, stderr)
}
for node, wants := range map[string][]string{
"sw1": {"router bgp 65100"},
"r1": {"router bgp 65100", "interface underlay\n", " vrrp 1 priority 200\n", " vrrp 1 ip 192.168.14.1\n"},
"r2": {"hostname r2", " vrrp 1 priority 190\n", " vrrp 1 ip 192.168.14.1\n"},
"rr1": {"bgp listen range 192.168.14.0/24 peer-group fabric"},
"hv1": {"hostname hv1", "neighbor 10.255.255.1 peer-group fabric"},
"hv2": {"hostname hv2", "neighbor 10.255.255.1 peer-group fabric"},
@ -359,9 +374,9 @@ func TestRun_RenderReportsAMissingFRRConfig(t *testing.T) {
images:
deb: { url: https://example.invalid/deb.qcow2, sums: https://example.invalid/SHA512SUMS }
segments:
underlay: { switch: sw1, cidr: 10.1.0.0/24 }
underlay: { routers: [sw1], cidr: 10.1.0.0/24 }
nodes:
sw1: { role: switch, image: deb, cpus: 1, memory: 512, frr: frr/absent.conf }
sw1: { role: router, image: deb, cpus: 1, memory: 512, segments: [underlay], frr: frr/absent.conf }
hv1: { role: host, image: deb, cpus: 1, memory: 512, segments: [underlay] }
`
if err := os.WriteFile(topo, []byte(doc), 0o600); err != nil {
@ -372,3 +387,34 @@ nodes:
t.Errorf("code %d, stderr %q", code, stderr)
}
}
func TestRun_UpAndDownRequireRoot(t *testing.T) {
geteuid = func() int { return 1000 }
t.Cleanup(func() { geteuid = os.Geteuid })
run := savedLab(t)
before, _ := os.ReadFile(filepath.Join(run, "topology.yml"))
other := filepath.Join(t.TempDir(), "other.yml")
if err := os.WriteFile(other, bytes.ReplaceAll(before, []byte("evpn-2hv"), []byte("other")), 0o600); err != nil {
t.Fatal(err)
}
for name, args := range map[string][]string{
"up": {"up", "-run", run, other},
"down": {"down", "-run", run},
} {
code, stdout, stderr := runLab(args...)
if code != 1 || stdout != "" || !strings.Contains(stderr, "labctl: "+name+" must run as root") {
t.Errorf("%s: code %d, stdout %q, stderr %q", name, code, stdout, stderr)
}
}
if after, _ := os.ReadFile(filepath.Join(run, "topology.yml")); string(after) != string(before) {
t.Error("the saved topology was replaced without root")
}
}
func TestRun_StatusAndSSHDoNotRequireRoot(t *testing.T) {
geteuid = func() int { return 1000 }
t.Cleanup(func() { geteuid = os.Geteuid })
if code, _, stderr := runLab("status", "-run", savedLab(t)); code != 0 || strings.Contains(stderr, "root") {
t.Errorf("status: code %d, stderr %q", code, stderr)
}
}

View file

@ -141,31 +141,37 @@ celui d'une release, vérifié) au lieu de compiler.
## Topologie
Un lab est décrit par un fichier YAML : des **nœuds** (les VM) et des **segments** (des réseaux L2
portés par un switch). Exemple livré : [`examples/evpn-2hv/topology.yml`](../examples/evpn-2hv/topology.yml).
Un lab est décrit par un fichier YAML : des **nœuds** (les VM) et des **segments** (des réseaux
L2). Exemple livré : [`examples/evpn-2hv/topology.yml`](../examples/evpn-2hv/topology.yml) — deux
routeurs en VRRP, un route reflector et deux hôtes en EVPN.
Chaque nœud non-switch est relié au switch de chacun de ses segments par un câble virtuel QEMU ;
le switch met ces câbles dans un bridge et porte la passerelle du segment.
Chaque segment est un **bridge Linux sur l'hôte du lab** (`lb-<segment>`), et chaque nœud qui y
est attaché y branche une carte par un tap (`lt<i>`). Le L2 ne dépend donc d'aucune VM : la
perte d'un routeur ne coupe pas le trafic direct entre les autres nœuds du segment. Un
**routeur** est un nœud comme les autres, qui porte en plus la passerelle et le NAT des segments
dont il est routeur.
Ce que le fichier déclare :
**`images`** — `url` de l'image qcow2 et `sums` du fichier de sommes à vérifier, tous deux en
`https://`.
**`segments`** — `switch` (un nœud de rôle `switch`), `cidr` IPv4 entre `/8` et `/30`, `mtu`
facultatif — 9000 par défaut, entre 1280 et 9000. Nom : 12 caractères au plus, minuscules et
chiffres, parce qu'il devient le nom d'interface dans les VM et, préfixé de `br-`, celui du
bridge (15 caractères au plus sous Linux).
**`segments`** — `cidr` IPv4 entre `/8` et `/30`, `mtu` facultatif — 9000 par défaut, entre 1280
et 9000 —, et `routers`, facultatif : les nœuds de rôle `router` qui portent la passerelle du
segment, **dans l'ordre de priorité**. Sans routeur, le segment est un L2 pur (lien entre VM,
lien inter-cluster) : pas de passerelle, pas de NAT. Nom : 12 caractères au plus, minuscules et
chiffres, parce qu'il devient le nom d'interface dans les VM et, préfixé de `lb-`, celui du bridge
sur l'hôte (15 caractères au plus sous Linux).
**`nodes`** — `role` (`switch`, `rr` ou `host`), `image`, `cpus`, `memory` en Mio (256 au
**`nodes`** — `role` (`router`, `rr` ou `host`), `image`, `cpus`, `memory` en Mio (256 au
moins), `segments` auxquels le nœud est relié, et `addresses` pour fixer l'adresse d'un nœud sur
un segment (`addresses: {underlay: 192.168.14.50}`). Un switch ne déclare ni `segments` ni
`addresses` : il porte ceux dont il est le `switch`. Champs facultatifs :
un segment (`addresses: {underlay: 192.168.14.50}`). Un routeur listé dans `segments.<s>.routers`
doit être attaché à `<s>`. Champs facultatifs :
- `secondary` — des adresses supplémentaires par segment, avec leur longueur de préfixe
(`secondary: {underlay: [169.254.0.3/28]}`), posées sur la même interface que l'adresse
principale : même L2, même MAC. Elles doivent être **hors** du CIDR du segment, pour ne
jamais croiser l'attribution automatique. Sur un switch, elles vont sur le bridge du segment ;
jamais croiser l'attribution automatique ;
- `loopback` — une adresse sur une interface `dummy` nommée `lo1` (`loopback: 10.255.255.1/32`) ;
- `frr` — le chemin d'un `frr.conf`, relatif au fichier de topologie : FRR est installé au
démarrage et la configuration déposée **telle quelle** (voir [FRR](#frr)) ;
@ -173,67 +179,112 @@ un segment (`addresses: {underlay: 192.168.14.50}`). Un switch ne déclare ni `s
démarrage **avant** FRR ;
- `post_provision` — un second script, exécuté **après** FRR ;
- `files` — des fichiers **ou répertoires** déposés dans `/etc/lab/files/<nom>` avant les
scripts, sous la forme `{<nom>: <chemin>}` (`files: {motd: motd}`). Ils voyagent
dans un `files.iso` propre au nœud (volume `LABFILES`), attaché comme second lecteur et
recopié au premier démarrage : la taille n'est pas limitée par cloud-init, et un binaire de
plusieurs dizaines de Mio passe ;
scripts, sous la forme `{<nom>: <chemin>}` (`files: {motd: motd}`). Ils voyagent dans un
`files.iso` propre au nœud (volume `LABFILES`), attaché comme second lecteur et recopié au
premier démarrage : la taille n'est pas limitée par cloud-init, et un binaire de plusieurs
dizaines de Mio passe ;
- `services` — les unités systemd que `labctl up` vérifie actives sur le nœud.
Les scripts reçoivent `LAB_NODE` (nom du nœud), `LAB_UPLINK` (interface de son premier segment,
qui porte la route par défaut) et `LAB_FILES` (`/etc/lab/files`). L'outil ne sait rien de ce
qu'ils installent.
Les scripts reçoivent `LAB_NODE` (nom du nœud), `LAB_UPLINK` (interface qui porte la route par
défaut : celle du premier segment du nœud qui a une passerelle, `mgmt0` pour un routeur) et
`LAB_FILES`
(`/etc/lab/files`). L'outil ne sait rien de ce qu'ils installent.
**Un seul script de provisionnement par nœud.** cloud-init exécute `runcmd` comme un script `sh`
sans `set -e` : seule la dernière commande compterait, et une provision en échec suivie d'un FRR
installé avec succès passerait pour un démarrage réussi. Chaque nœud reçoit donc
`/usr/local/sbin/lab-provision`, en `set -eu`, qui enchaîne ses étapes — script du switch ou de
`/usr/local/sbin/lab-provision`, en `set -eu`, qui enchaîne ses étapes — service du routeur ou de
la loopback, `provision`, FRR, `post_provision` ; `runcmd` n'appelle que lui, et la première
étape en échec met cloud-init en erreur.
`mgmt0` et `lo1` sont réservés : aucun segment ne peut porter ces noms.
Ce que l'outil en déduit, de façon déterministe — même fichier, même plan :
### Passerelle et routeurs
| Routeurs du segment | Passerelle |
|---|---|
| aucun | aucune : les nœuds n'y ont pas de route par défaut |
| un | la première adresse du CIDR, posée sur l'interface du routeur, en plus de la sienne |
| deux ou plus | la première adresse du CIDR, **virtuelle**, portée en VRRP v3 par `vrrpd` de FRR |
Avec plusieurs routeurs, `labctl` prépare tout sur chacun : interface `macvlan` `vr<segment>`
avec la MAC VRRP (`00:00:5e:00:01:<vrid>`) et la passerelle, créée **coupée** (`protodown on`) :
seul `vrrpd` la rétablit, quand il devient maître ; `arp_ignore=1` et `arp_announce=2`
sur l'interface du segment (sans quoi elle répondrait aux ARP de la passerelle avec sa propre
MAC), FRR installé avec `vrrpd`, et la configuration VRRP **ajoutée** à la fin de son `frr.conf`
(ou d'un `frr.conf` minimal s'il n'en déclare pas) :
```text
interface underlay
vrrp 1 version 3
vrrp 1 priority 200
no vrrp 1 preempt
vrrp 1 ip 192.168.14.1
exit
```
Le VRID est le rang du segment plus un (d'où au plus 255 segments à plusieurs routeurs, parmi les
premiers) ; la priorité vaut 200 pour le premier routeur de la liste, puis 10 de moins pour
chacun des suivants. **Pas de préemption** : un routeur qui revient reste en attente, plutôt que
de reprendre la passerelle avant d'avoir ses routes BGP — un prétest l'a montré, la préemption
coupait les sessions qui passaient par lui. Conséquence : le maître est le premier routeur dont
VRRP démarre, la priorité ne départageant que des démarrages simultanés ; un scénario qui arrête
« le maître » doit le demander (`vtysh -c 'show vrrp'`), pas le supposer. Bascule mesurée en
prétest : environ 2,6 s de perte (trois annonces manquées à une seconde d'intervalle).
Chaque routeur masque (NAT nftables) vers `mgmt0` les segments dont il est routeur, et sort par
le NAT de QEMU. Un nœud qui n'est pas routeur prend sa route par défaut et ses DNS (`1.1.1.1`,
`8.8.8.8`) sur le premier de ses segments qui a une passerelle.
### Plan déterministe
Ce que l'outil déduit du fichier — même fichier, même plan :
| | |
|---|---|
| Passerelle d'un segment | la première adresse du CIDR, portée par le switch sur `br-<segment>` |
| Adresse d'un nœud | les suivantes, **dans l'ordre de déclaration des nœuds** ; une adresse fixée par `addresses` est réservée d'abord et sautée par l'attribution automatique |
| Câbles | un par couple (segment, nœud), segments puis nœuds dans l'ordre de déclaration ; le câble *i* utilise les ports UDP `20000 + 2i` (côté nœud) et `20001 + 2i` (côté switch) |
| MAC | `02:4c:<nœud>:<nœud>:<segment>:<côté>` — préfixe localement administré, rang du nœud sur deux octets, rang du segment, `00` côté nœud et `01` côté switch |
| Interfaces | côté nœud, le nom du segment ; côté switch, `p<i>`, du rang du câble |
| Passerelle d'un segment | la première adresse du CIDR, si le segment a au moins un routeur |
| Adresse d'un nœud | les suivantes (dès la première sans routeur), **dans l'ordre de déclaration des nœuds** ; une adresse fixée par `addresses` est réservée d'abord et sautée par l'attribution automatique |
| Bridge | `lb-<segment>` sur l'hôte du lab |
| Ports | un par couple (segment, nœud), segments puis nœuds dans l'ordre de déclaration ; le port *i* est le tap `lt<i>` |
| MAC | `02:4c:<nœud>:<nœud>:<segment>:00` — préfixe localement administré, rang du nœud sur deux octets, rang du segment |
| Interfaces | dans la VM, le nom du segment |
| SSH d'administration | `127.0.0.1:<2200 + rang du nœud>` sur l'hôte du lab |
> [!WARNING]
> Réordonner les nœuds ou les segments dans le fichier **change les adresses, les MAC et les
> ports**. C'est assumé pour un lab ; `labctl plan` montre le résultat avant tout lancement.
> Réordonner les nœuds ou les segments dans le fichier **change les adresses, les MAC, les taps et
> les ports**. C'est assumé pour un lab ; `labctl plan` montre le résultat avant tout lancement.
Limites : 1000 nœuds, 256 segments, et autant de câbles que la plage UDP le permet (22 768).
Limites : 1000 nœuds, 256 segments.
`labctl plan` valide le fichier et affiche le plan, sans rien lancer :
```text
$ go run ./cmd/labctl plan examples/evpn-2hv/topology.yml
lab evpn-2hv: nodes 4, segments 1, cables 3
lab evpn-2hv: nodes 5, segments 1, ports 5
nodes
name role image cpus memory ssh
sw1 switch debian12 2 1024 MiB 127.0.0.1:2200
rr1 rr debian12 1 1024 MiB 127.0.0.1:2201
hv1 host debian12 2 2048 MiB 127.0.0.1:2202
hv2 host debian12 2 2048 MiB 127.0.0.1:2203
r1 router debian12 1 1024 MiB 127.0.0.1:2200
r2 router debian12 1 1024 MiB 127.0.0.1:2201
rr1 rr debian12 1 1024 MiB 127.0.0.1:2202
hv1 host debian12 2 2048 MiB 127.0.0.1:2203
hv2 host debian12 2 2048 MiB 127.0.0.1:2204
roles
name loopback secondary frr provision post_provision files services
sw1 - underlay 169.254.0.1/28 sw1.conf - - - -
r1 - underlay 169.254.0.1/28 r1.conf - - - -
r2 - underlay 169.254.0.2/28 r2.conf - - - -
rr1 lo1 10.255.255.1/32 underlay 169.254.0.3/28 rr1.conf - - - -
hv1 - - hv1.conf host.sh - motd -
hv2 - - hv2.conf host.sh - motd -
segment underlay: 192.168.14.0/24, mtu 9000, switch sw1, bridge br-underlay, gateway 192.168.14.1
node interface address mac udp switch port mac udp
rr1 underlay 192.168.14.2/24 02:4c:00:01:00:00 20000 <-> sw1 p0 02:4c:00:01:00:01 20001
hv1 underlay 192.168.14.11/24 02:4c:00:02:00:00 20002 <-> sw1 p1 02:4c:00:02:00:01 20003
hv2 underlay 192.168.14.12/24 02:4c:00:03:00:00 20004 <-> sw1 p2 02:4c:00:03:00:01 20005
segment underlay: 192.168.14.0/24, mtu 9000, bridge lb-underlay, gateway 192.168.14.1 by vrrp 1 on r1, r2
node interface address mac tap
r1 underlay 192.168.14.252/24 02:4c:00:00:00:00 lt0
r2 underlay 192.168.14.253/24 02:4c:00:01:00:00 lt1
rr1 underlay 192.168.14.2/24 02:4c:00:02:00:00 lt2
hv1 underlay 192.168.14.11/24 02:4c:00:03:00:00 lt3
hv2 underlay 192.168.14.12/24 02:4c:00:04:00:00 lt4
```
Un fichier invalide est refusé avec **toutes** ses erreurs à la fois, et un code de sortie 1. Les
@ -242,17 +293,31 @@ champs inconnus et les clés en double sont refusés aussi :
```text
$ labctl plan cassee.yml
labctl: cassee.yml:
segment underlay: rr1 is a rr, not a switch
segment underlay: rr1 is a rr, not a router
segment underlay: router r9 is not a declared node
segment underlay: cidr 10.250.0.0/31 prefix length out of range [/8, /30]
node sw1: switch carries no segment
node r1: must be attached to at least one segment
```
### Migration depuis 0.1
Le rôle `switch` et le champ `segments.<s>.switch` n'existent plus : `labctl` les refuse avec un
message qui dit quoi faire. Pour une topologie 0.1 :
1. `role: switch` devient `role: router` ;
2. le routeur déclare `segments: [<s>]` et, de préférence, une adresse fixée (`addresses`) — en
0.1, le switch n'avait que la passerelle ;
3. `segments.<s>.switch: <nom>` devient `segments.<s>.routers: [<nom>]` ;
4. ses `secondary` restent valables : elles vont désormais sur son interface du segment, et non
plus sur un bridge interne.
## FRR
Au premier démarrage, cloud-init installe FRR (`frr-stable` de `deb.frrouting.org`, sans les
paquets recommandés), active `bgpd` — et `bfdd` sur le switch et le route reflector —, puis
paquets recommandés), active `bgpd` — et `bfdd` sur les routeurs et les route reflectors,
`vrrpd` sur les routeurs d'un segment en VRRP —, puis
dépose le `frr.conf` du nœud et redémarre FRR. La mise à jour des index de paquets est réessayée
pendant cinq minutes : un nœud peut démarrer avant que le switch, par lequel il sort, n'ait posé
pendant cinq minutes : un nœud peut démarrer avant que le routeur, par lequel il sort, n'ait posé
son NAT.
La **clé du dépôt FRR** n'est pas téléchargée au démarrage : elle est enregistrée dans `labctl`
@ -275,7 +340,7 @@ voit pas — une provision qui arme un redémarrage, par exemple : la VM redéma
`runcmd`.
> [!NOTE]
> **Le switch du lab est un routeur Linux avec FRR, par choix.** Il joue le rôle générique de
> **Le routeur du lab est un Linux avec FRR, par choix.** Il joue le rôle générique de
> routeur de cluster : passerelle des nœuds et, dans l'exemple, session eBGP avec BFD vers le
> route reflector, dont il n'accepte que la loopback. L'équipement réel d'une infrastructure
> (MikroTik, Cisco, Juniper, Arista…) n'a besoin que de BGP et d'EVPN ; sa configuration propre
@ -300,17 +365,17 @@ Les chemins de la VM (`disk.qcow2`, `seed.iso`, `console.log`, `qmp.sock`, `qemu
du répertoire du nœud ; `-key` peut être répété, et accepte un fichier `authorized_keys` (lignes
vides et commentaires ignorés). Les fichiers sont créés en `0600`.
Ce que contiennent les arguments QEMU d'un nœud — extrait réel, côté réseau :
Ce que contiennent les arguments QEMU d'un nœud — côté réseau :
```text
-netdev
user,id=mgmt0,restrict=on,ipv6=off,hostfwd=tcp:127.0.0.1:2202-:22
user,id=mgmt0,restrict=on,ipv6=off,hostfwd=tcp:127.0.0.1:2203-:22
-device
virtio-net-pci,netdev=mgmt0,mac=02:4d:00:02:00:00,romfile=
virtio-net-pci,netdev=mgmt0,mac=02:4d:00:03:00:00,romfile=
-netdev
dgram,id=underlay,local.type=inet,local.host=127.0.0.1,local.port=20002,remote.type=inet,remote.host=127.0.0.1,remote.port=20003
tap,id=underlay,ifname=lt3,script=no,downscript=no
-device
virtio-net-pci,netdev=underlay,mac=02:4c:00:02:00:00,host_mtu=9000,romfile=
virtio-net-pci,netdev=underlay,mac=02:4c:00:03:00:00,host_mtu=9000,romfile=
```
Les choix qui s'y lisent :
@ -318,11 +383,11 @@ Les choix qui s'y lisent :
- **machine** `q35`, `-accel kvm -cpu host` — le KVM imbriqué des nœuds en dépend ;
`-nodefaults` pour qu'aucun périphérique implicite ne s'ajoute ;
- **administration** (`mgmt0`) : le NAT de QEMU, MAC `02:4d:<nœud>:<nœud>:00:00`, SSH redirigé
sur la boucle locale de l'hôte. `restrict=on` pour tous les nœuds **sauf le switch** : un
sur la boucle locale de l'hôte. `restrict=on` pour tous les nœuds **sauf les routeurs** : un
nœud isolé ne joint ni l'hôte ni l'extérieur par là, seule la redirection SSH passe.
`ipv6=off` partout (voir plus bas) ;
- **câbles** : `dgram` sur `127.0.0.1`, les deux extrémités d'un câble se répondent (port local
de l'une = port distant de l'autre), `host_mtu` annonce le MTU du segment au guest ;
- **segments** : un tap créé par `labctl up` (`script=no` : QEMU ne lance aucun script), branché
sur le bridge du segment ; `host_mtu` annonce le MTU du segment au guest ;
- `romfile=` vide sur toutes les cartes : pas de ROM de démarrage réseau, donc pas de repli sur
un démarrage PXE si le firmware ne trouve pas le disque. Pendant les premiers essais, une VM
restée bloquée sans rien écrire sur sa console, CPU au repos, avait toutes les apparences de
@ -330,18 +395,22 @@ Les choix qui s'y lisent :
Ce que fait cloud-init :
- **toutes les VM** : interfaces nommées d'après leur MAC (`mgmt0`, nom du segment, `p<i>`),
`dhcp4: false` partout, `mgmt0` en `10.0.2.15/24` **sans passerelle** ; connexion SSH par
clé seulement, utilisateur `debian`, `root` désactivé, mot de passe refusé ;
- **un nœud** : adresse sur chaque segment, MTU du segment, route par défaut et DNS
(`1.1.1.1`, `8.8.8.8`) sur son **premier** segment — la sortie Internet passe par le switch ;
- **le switch** : route par défaut par `mgmt0` ; un service `lab-switch` crée un bridge
`br-<segment>` par segment (STP désactivé, MTU du segment), y branche ses ports, porte la
passerelle, active le routage et masque (NAT nftables) les segments vers `mgmt0`. Le service
est rejoué à chaque démarrage.
- **toutes les VM** : interfaces nommées d'après leur MAC (`mgmt0`, nom du segment),
`dhcp4: false` partout, `mgmt0` en `10.0.2.15/24` ; connexion SSH par clé seulement,
utilisateur `debian`, `root` désactivé, mot de passe refusé ;
- **un nœud** : adresse et `secondary` sur chaque segment, MTU du segment, route par défaut et
DNS sur le premier de ses segments qui a une passerelle ; `mgmt0` **sans passerelle** ;
- **un routeur** : mêmes adresses, route par défaut par `mgmt0` ; un service `lab-router` active
le routage, pose les passerelles (adresse ou `macvlan` VRRP, voir
[Passerelle et routeurs](#passerelle-et-routeurs)) et le NAT nftables. Le service est rejoué à
chaque démarrage.
### Vérifié sur de vraies VM
Relevés faits avec le modèle 0.1 — le switch portait alors le bridge du segment, relié aux nœuds
par des câbles UDP ; ce qui touche au L2 et à la passerelle est à revérifier avec les bridges de
l'hôte.
Le switch et le route reflector n'ont pas besoin de KVM imbriqué : `sw1` et `rr1` ont été
démarrés **sur un Mac**, en émulation (TCG), avec Debian 12 `generic` et les fichiers produits
par `labctl render`.
@ -387,19 +456,45 @@ labctl ssh [-run dir] <nœud> [commande…]
3. génère une paire de clés SSH dans le répertoire du lab, si elle n'existe pas encore ;
4. pour chaque nœud : fichiers de `labctl render`, disque **neuf** en overlay qcow2 sur l'image
(`qemu-img create -b`, 20 Gio annoncés), image `cidata` et `files.iso` (`genisoimage`) ;
5. démarre les QEMU, **switchs d'abord**, détachés (`-daemonize`) : ils survivent à la session
5. crée les bridges des segments et les taps des nœuds (voir [Réseau de l'hôte](#réseau-de-lhôte)),
et refuse de continuer si l'un d'eux existe déjà ;
6. démarre les QEMU, **routeurs d'abord**, détachés (`-daemonize`) : ils survivent à la session
SSH qui les a lancés ;
6. attend sur chaque nœud la fin de cloud-init (`cloud-init status --wait` par SSH), jusqu'au
délai `-timeout`, puis vérifie ses services.
7. attend sur chaque nœud la fin de cloud-init (`cloud-init status --wait` par SSH), jusqu'au
délai `-timeout`, puis vérifie ses services (`frr` compris quand il est installé).
La topologie est copiée dans `<run>/topology.yml`. Un échec laisse les nœuds démarrés en place :
`labctl status`, puis `labctl down`.
**`labctl status`** — pour chaque nœud : rôle, état du processus QEMU, PID, port SSH sur la
boucle locale.
boucle locale ; pour chaque segment : bridge présent ou absent, passerelle.
**`labctl down`** — arrête chaque QEMU par `SIGTERM`, puis `SIGKILL` au bout de 30 s. Les disques
sont conservés jusqu'au prochain `up`, qui les recrée.
**`labctl down`** — arrête chaque QEMU par `SIGTERM`, puis `SIGKILL` au bout de 30 s, puis
supprime les taps et les bridges du lab. Les disques sont conservés jusqu'au prochain `up`, qui
les recrée.
`up` et `down` doivent tourner **en root** : ils créent et suppriment des interfaces de l'hôte.
### Réseau de l'hôte
Pour chaque segment, `labctl up` exécute :
```text
ip link add name lb-<segment> type bridge stp_state 0 mcast_snooping 0
ip link set dev lb-<segment> addrgenmode none
ip tuntap add dev lt<i> mode tap # pour chaque port du segment
ip link set dev lt<i> addrgenmode none
ip link set dev lt<i> mtu <mtu> master lb-<segment> up
ip link set dev lb-<segment> mtu <mtu> up
```
- **Aucune adresse côté hôte**, IPv4 ni IPv6 (`addrgenmode none` : pas même de lien local) :
l'hôte n'est pas joignable depuis un segment, et ne route rien entre eux.
- **Pas de STP ni de surveillance IGMP** : le bridge se comporte comme un switch simple, et les
annonces VRRP (`224.0.0.18`) circulent sans dépendre d'un querier.
- Si `br_netfilter` est chargé sur l'hôte, `bridge-nf-call-iptables` est mis à 0 : le pare-feu
de l'hôte ne filtre pas le trafic des segments.
- Un lab par hôte : les noms `lb-*` et `lt*` ne sont pas propres à un répertoire `-run`.
**`labctl ssh`** — ouvre un shell sur un nœud, ou y exécute une commande, avec la clé générée par
`up`. `labctl` cède la place à `ssh`, dont le code de retour est donc celui de la commande. Un
@ -433,7 +528,7 @@ PID réutilisé par un autre programme n'est donc jamais signalé.
### Vérifié sur le serveur de lab
Le 2026-10-04, sur un Xeon E5-2640 v3, Debian 12 et QEMU 7.2 sur le serveur, topologie à deux
Avec le modèle 0.1 (voir la remarque précédente). Le 2026-10-04, sur un Xeon E5-2640 v3, Debian 12 et QEMU 7.2 sur le serveur, topologie à deux
hôtes :
| Vérification | Résultat |
@ -519,6 +614,8 @@ au projet.
- **Clé SSH des VM** : générée par `labctl up` sur le serveur, c'est la seule clé autorisée dans
les VM. Elle ne quitte jamais le serveur, n'ouvre que les VM du lab — qui n'écoutent qu'en
boucle locale — et disparaît avec lui. La clé publique du poste n'est jamais envoyée aux VM.
- **Segments isolés de l'hôte** : bridges et taps sans adresse (voir
[Réseau de l'hôte](#réseau-de-lhôte)) ; seuls les routeurs sortent, par le NAT de QEMU.
- **Clés d'hôte des VM non vérifiées** par `labctl ssh` (`known_hosts` jetable) : elles changent
à chaque `up`. Acceptable uniquement parce que la connexion reste sur la boucle locale d'un
serveur auquel on s'est authentifié.

View file

@ -1,5 +1,5 @@
frr defaults traditional
hostname sw1
hostname r1
log syslog informational
!
ip prefix-list RR-LOOPBACK-IN seq 10 permit 10.255.255.1/32

View file

@ -0,0 +1,28 @@
frr defaults traditional
hostname r2
log syslog informational
!
ip prefix-list RR-LOOPBACK-IN seq 10 permit 10.255.255.1/32
!
route-map RR-IN permit 10
match ip address prefix-list RR-LOOPBACK-IN
exit
!
route-map NO-OUT deny 999
exit
!
router bgp 65100
no bgp default ipv4-unicast
bgp router-id 169.254.0.2
neighbor 169.254.0.3 remote-as 65000
neighbor 169.254.0.3 description rr1
neighbor 169.254.0.3 bfd
!
address-family ipv4 unicast
neighbor 169.254.0.3 activate
neighbor 169.254.0.3 route-map RR-IN in
neighbor 169.254.0.3 route-map NO-OUT out
exit-address-family
!
exit
!

View file

@ -7,14 +7,19 @@ images:
segments:
underlay:
switch: sw1
routers: [r1, r2]
cidr: 192.168.14.0/24
mtu: 9000
nodes:
sw1:
{ role: switch, image: debian12, cpus: 2, memory: 1024,
secondary: { underlay: [169.254.0.1/28] }, frr: frr/sw1.conf }
r1:
{ role: router, image: debian12, cpus: 1, memory: 1024, segments: [underlay],
addresses: { underlay: 192.168.14.252 }, secondary: { underlay: [169.254.0.1/28] },
frr: frr/r1.conf }
r2:
{ role: router, image: debian12, cpus: 1, memory: 1024, segments: [underlay],
addresses: { underlay: 192.168.14.253 }, secondary: { underlay: [169.254.0.2/28] },
frr: frr/r2.conf }
rr1:
{ role: rr, image: debian12, cpus: 1, memory: 1024, segments: [underlay],
addresses: { underlay: 192.168.14.2 }, secondary: { underlay: [169.254.0.3/28] },

View file

@ -107,7 +107,10 @@ func (l Lab) Up(ctx context.Context, fetcher provision.Fetcher, timeout time.Dur
if err != nil {
return err
}
for _, n := range switchesFirst(l.Plan, nodes) {
if err := l.NetworkUp(ctx); err != nil {
return err
}
for _, n := range routersFirst(l.Plan, nodes) {
if err := os.Remove(l.pidFile(n.Name)); err != nil && !errors.Is(err, os.ErrNotExist) {
return err
}
@ -134,10 +137,10 @@ func (l Lab) Up(ctx context.Context, fetcher provision.Fetcher, timeout time.Dur
return errors.Join(errs...)
}
func switchesFirst(p *topology.Plan, nodes []render.Node) []render.Node {
func routersFirst(p *topology.Plan, nodes []render.Node) []render.Node {
var first, rest []render.Node
for i, n := range nodes {
if p.Nodes[i].Role == topology.RoleSwitch {
if p.Nodes[i].Role == topology.RoleRouter {
first = append(first, n)
} else {
rest = append(rest, n)
@ -167,16 +170,16 @@ func (l Lab) waitReady(ctx context.Context, n topology.NodePlan) error {
}
}
func services(n topology.NodePlan) []string {
func services(p *topology.Plan, n topology.NodePlan) []string {
units := append([]string(nil), n.Services...)
if n.FRR != "" {
if render.NeedsFRR(p, n) {
units = append(units, "frr.service")
}
return units
}
func (l Lab) checkServices(ctx context.Context, n topology.NodePlan) error {
for _, unit := range services(n) {
for _, unit := range services(l.Plan, n) {
if err := l.Runner.Run(ctx, "ssh", l.sshArgs(n, true, []string{"systemctl", "is-active", "--quiet", unit})...); err != nil {
return fmt.Errorf("%s is not active: %w", unit, err)
}
@ -235,7 +238,10 @@ func (l Lab) Status(w io.Writer) error {
}
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t127.0.0.1:%d\n", n.Name, n.Role, state, shown, n.SSHPort)
}
return tw.Flush()
if err := tw.Flush(); err != nil {
return err
}
return l.networkStatus(w)
}
func (l Lab) Down(ctx context.Context) error {
@ -259,7 +265,10 @@ func (l Lab) Down(ctx context.Context) error {
}
fmt.Fprintf(l.Out, "%s: stopped\n", n.Name)
}
return errors.Join(errs...)
if len(errs) > 0 {
return errors.Join(errs...)
}
return l.NetworkDown(ctx)
}
func (l Lab) stop(ctx context.Context, name string, pid int) error {

View file

@ -38,6 +38,7 @@ type fakeRunner struct {
ssh map[string][]error
always map[string]error
inactive string
links map[string]bool
}
func (f *fakeRunner) Run(_ context.Context, name string, args ...string) error {
@ -54,6 +55,22 @@ func (f *fakeRunner) Run(_ context.Context, name string, args ...string) error {
return err
}
return os.WriteFile(private+".pub", []byte(labKey+"\n"), 0o644)
case "ip":
if f.links == nil {
f.links = map[string]bool{}
}
switch {
case len(args) == 4 && args[0] == "link" && args[1] == "show":
if !f.links[args[3]] {
return exitErr(1)
}
case args[0] == "link" && args[1] == "add" && args[2] == "name":
f.links[args[3]] = true
case args[0] == "tuntap" && args[1] == "add":
f.links[args[3]] = true
case args[0] == "link" && args[1] == "delete":
delete(f.links, args[3])
}
case "ssh":
port := args[indexOf(args, "-p")+1]
if args[len(args)-1] == f.inactive {
@ -81,6 +98,25 @@ func (f *fakeRunner) commands(name string) [][]string {
return out
}
func (f *fakeRunner) ipCalls() []string {
var out []string
for _, c := range f.commands("ip") {
out = append(out, strings.Join(c[1:], " "))
}
return out
}
func (f *fakeRunner) present(links ...string) {
f.mu.Lock()
defer f.mu.Unlock()
if f.links == nil {
f.links = map[string]bool{}
}
for _, l := range links {
f.links[l] = true
}
}
func indexOf(list []string, value string) int {
for i, v := range list {
if v == value {
@ -105,7 +141,10 @@ func mirror(t *testing.T) (*httptest.Server, string) {
return server, server.URL
}
func labPlan(t *testing.T, base, nodes string) *topology.Plan {
const underlay = ` underlay: { routers: [sw1], cidr: 10.250.0.0/24, mtu: 9000 }
`
func labPlan(t *testing.T, base, segments, nodes string) *topology.Plan {
t.Helper()
doc := `name: evpn-2hv
images:
@ -113,8 +152,7 @@ images:
url: ` + base + `/deb.qcow2
sums: ` + base + `/SHA512SUMS
segments:
underlay: { switch: sw1, cidr: 10.250.0.0/24, mtu: 9000 }
nodes:
` + segments + `nodes:
` + nodes
topo, err := topology.Parse([]byte(doc))
if err != nil {
@ -127,9 +165,18 @@ nodes:
return p
}
const switchLast = ` rr1: { role: rr, image: deb, cpus: 1, memory: 1024, segments: [underlay] }
const routerLast = ` rr1: { role: rr, image: deb, cpus: 1, memory: 1024, segments: [underlay] }
hv1: { role: host, image: deb, cpus: 4, memory: 16384, segments: [underlay], services: [agent.service] }
sw1: { role: switch, image: deb, cpus: 2, memory: 1024 }
sw1: { role: router, image: deb, cpus: 2, memory: 1024, segments: [underlay] }
`
const twoSegments = ` red: { routers: [r1, r2], cidr: 10.1.0.0/24 }
blue: { cidr: 10.2.0.0/24, mtu: 1500 }
`
const twoRouters = ` r1: { role: router, image: deb, cpus: 1, memory: 512, segments: [red] }
r2: { role: router, image: deb, cpus: 1, memory: 512, segments: [red] }
hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [red, blue] }
`
type fixture struct {
@ -140,6 +187,11 @@ type fixture struct {
}
func newFixture(t *testing.T, nodes string) *fixture {
t.Helper()
return newLab(t, underlay, nodes)
}
func newLab(t *testing.T, segments, nodes string) *fixture {
t.Helper()
server, base := mirror(t)
root := t.TempDir()
@ -147,7 +199,7 @@ func newFixture(t *testing.T, nodes string) *fixture {
out := &bytes.Buffer{}
return &fixture{
lab: Lab{
Plan: labPlan(t, base, nodes),
Plan: labPlan(t, base, segments, nodes),
RunDir: filepath.Join(root, "run"),
ProcDir: filepath.Join(root, "proc"),
Runner: runner,
@ -217,7 +269,7 @@ func waitDead(t *testing.T, pid int) {
}
func TestPID_NoPidFileMeansStopped(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
if pid, err := f.lab.PID("hv1"); pid != 0 || err != nil {
t.Errorf("PID = %d, %v", pid, err)
}
@ -225,7 +277,7 @@ func TestPID_NoPidFileMeansStopped(t *testing.T) {
func TestPID_RefusesACorruptedPidFile(t *testing.T) {
for _, content := range []string{"", "abc", "0", "-12"} {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
f.pidFile(t, "hv1", content)
if _, err := f.lab.PID("hv1"); err == nil || !strings.Contains(err.Error(), "not a pid") {
t.Errorf("pid file %q: error = %v", content, err)
@ -234,7 +286,7 @@ func TestPID_RefusesACorruptedPidFile(t *testing.T) {
}
func TestPID_RecognisesTheNodeProcess(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
want := f.process(t, "hv1", "sleep 30", qemu("hv1")...)
if pid, err := f.lab.PID("hv1"); pid != want || err != nil {
t.Errorf("PID = %d, %v, want %d", pid, err, want)
@ -249,7 +301,7 @@ func TestPID_IgnoresAPidReusedByAnotherProcess(t *testing.T) {
}
for name, cmdline := range cases {
t.Run(name, func(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
f.process(t, "hv1", "sleep 30", cmdline...)
if pid, err := f.lab.PID("hv1"); pid != 0 || err != nil {
t.Errorf("PID = %d, %v, want 0", pid, err)
@ -259,7 +311,7 @@ func TestPID_IgnoresAPidReusedByAnotherProcess(t *testing.T) {
}
func TestPID_IgnoresADeadProcess(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
pid := f.process(t, "hv1", "exit 0", qemu("hv1")...)
waitDead(t, pid)
if got, err := f.lab.PID("hv1"); got != 0 || err != nil {
@ -268,7 +320,7 @@ func TestPID_IgnoresADeadProcess(t *testing.T) {
}
func TestStatus(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
pid := strconv.Itoa(f.process(t, "hv1", "sleep 30", qemu("hv1")...))
var out bytes.Buffer
if err := f.lab.Status(&out); err != nil {
@ -281,14 +333,16 @@ func TestStatus(t *testing.T) {
want := fmt.Sprintf("node role state %-*s ssh\n", w, "pid") +
fmt.Sprintf("rr1 rr stopped %-*s 127.0.0.1:2200\n", w, "-") +
fmt.Sprintf("hv1 host running %-*s 127.0.0.1:2201\n", w, pid) +
fmt.Sprintf("sw1 switch stopped %-*s 127.0.0.1:2202\n", w, "-")
fmt.Sprintf("sw1 router stopped %-*s 127.0.0.1:2202\n", w, "-") +
"\nsegment bridge state gateway\n" +
"underlay lb-underlay absent 10.250.0.1\n"
if out.String() != want {
t.Errorf("status:\n%s\nwant:\n%s", out.String(), want)
}
}
func TestDown_StopsWithSIGTERM(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
pid := f.process(t, "hv1", "sleep 30", qemu("hv1")...)
if err := f.lab.Down(context.Background()); err != nil {
@ -305,7 +359,7 @@ func TestDown_StopsWithSIGTERM(t *testing.T) {
}
func TestDown_FallsBackToSIGKILL(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
f.lab.Stop = 100 * time.Millisecond
pid := f.process(t, "hv1", `trap "" TERM; while :; do sleep 0.05; done`, qemu("hv1")...)
time.Sleep(50 * time.Millisecond)
@ -322,7 +376,7 @@ func TestDown_FallsBackToSIGKILL(t *testing.T) {
}
func TestDown_NeverSignalsAProcessThatIsNotTheNode(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
pid := f.process(t, "hv1", "sleep 30", "/usr/sbin/sshd", "-D")
if err := f.lab.Down(context.Background()); err != nil {
@ -339,7 +393,7 @@ func TestDown_NeverSignalsAProcessThatIsNotTheNode(t *testing.T) {
}
func TestDown_ReportsACorruptedPidFileAndStopsTheOthers(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
f.pidFile(t, "rr1", "garbage")
pid := f.process(t, "hv1", "sleep 30", qemu("hv1")...)
@ -352,7 +406,7 @@ func TestDown_ReportsACorruptedPidFileAndStopsTheOthers(t *testing.T) {
}
func TestSSH_Arguments(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
key := filepath.Join(f.lab.RunDir, "lab_ed25519")
base := []string{
"-i", key,
@ -388,14 +442,14 @@ func TestSSH_Arguments(t *testing.T) {
}
func TestSSH_UnknownNode(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
if _, err := f.lab.SSH("hv9", true, nil); err == nil || !strings.Contains(err.Error(), `node "hv9" is not in lab evpn-2hv`) {
t.Errorf("error = %v", err)
}
}
func TestUp_StartsSwitchesFirstAndWaitsForEveryNode(t *testing.T) {
f := newFixture(t, switchLast)
func TestUp_StartsRoutersFirstAndWaitsForEveryNode(t *testing.T) {
f := newFixture(t, routerLast)
if err := f.lab.Up(context.Background(), f.fetcher, time.Second); err != nil {
t.Fatalf("Up: %v", err)
@ -409,7 +463,7 @@ func TestUp_StartsSwitchesFirstAndWaitsForEveryNode(t *testing.T) {
started = append(started, c[indexOf(c, "-name")+1])
}
if !reflect.DeepEqual(started, []string{"sw1", "rr1", "hv1"}) {
t.Errorf("start order = %v, want the switch first", started)
t.Errorf("start order = %v, want the router first", started)
}
ssh := f.runner.commands("ssh")
@ -438,7 +492,7 @@ func TestUp_StartsSwitchesFirstAndWaitsForEveryNode(t *testing.T) {
}
func TestUp_RetriesWhileSSHIsUnreachable(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
f.runner.ssh["2201"] = []error{exitErr(255), exitErr(255), nil}
if err := f.lab.Up(context.Background(), f.fetcher, time.Second); err != nil {
@ -460,7 +514,7 @@ func TestUp_ReadinessFailures(t *testing.T) {
}
for name, c := range cases {
t.Run(name, func(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
f.lab.Poll = 10 * time.Millisecond
f.runner.ssh["2201"] = c.answers
f.runner.always["2201"] = exitErr(255)
@ -478,7 +532,7 @@ func TestUp_ReadinessFailures(t *testing.T) {
}
func TestUp_RecoverableCloudInitErrorsAreReportedNotFatal(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
f.runner.ssh["2201"] = []error{exitErr(2)}
if err := f.lab.Up(context.Background(), f.fetcher, time.Second); err != nil {
@ -490,7 +544,7 @@ func TestUp_RecoverableCloudInitErrorsAreReportedNotFatal(t *testing.T) {
}
func TestUp_RefusesARunningLab(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
f.process(t, "hv1", "sleep 30", qemu("hv1")...)
err := f.lab.Up(context.Background(), f.fetcher, time.Second)
@ -504,7 +558,7 @@ func TestUp_RefusesARunningLab(t *testing.T) {
}
func TestUp_RemovesAStalePidFileBeforeStarting(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
pid := f.process(t, "hv1", "exit 0", qemu("hv1")...)
waitDead(t, pid)
@ -517,7 +571,7 @@ func TestUp_RemovesAStalePidFileBeforeStarting(t *testing.T) {
}
func TestUp_StopsAtTheFirstQEMUFailure(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
f.runner.fail = "qemu-system-x86_64"
err := f.lab.Up(context.Background(), f.fetcher, time.Second)
@ -534,7 +588,7 @@ func TestUp_StopsAtTheFirstQEMUFailure(t *testing.T) {
}
func TestStop_RefusesPidsThatTargetAGroup(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
f.lab.signal = func(pid int, sig syscall.Signal) error {
t.Fatalf("signal %v sent to pid %d", sig, pid)
return nil
@ -547,7 +601,7 @@ func TestStop_RefusesPidsThatTargetAGroup(t *testing.T) {
}
func TestUp_ChecksTheServicesOfEachRole(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
conf := filepath.Join(t.TempDir(), "sw1.conf")
if err := os.WriteFile(conf, []byte("hostname sw1\n"), 0o600); err != nil {
t.Fatal(err)
@ -574,7 +628,7 @@ func TestUp_ChecksTheServicesOfEachRole(t *testing.T) {
}
func TestUp_ReportsAnInactiveServiceAndWaitsForTheOthers(t *testing.T) {
f := newFixture(t, switchLast)
f := newFixture(t, routerLast)
f.runner.inactive = "agent.service"
err := f.lab.Up(context.Background(), f.fetcher, time.Second)
@ -586,3 +640,293 @@ func TestUp_ReportsAnInactiveServiceAndWaitsForTheOthers(t *testing.T) {
t.Errorf("output = %q", f.out.String())
}
}
func TestStatus_ShowsTheSegments(t *testing.T) {
f := newLab(t, twoSegments, twoRouters)
f.runner.present("lb-red")
var out bytes.Buffer
if err := f.lab.Status(&out); err != nil {
t.Fatal(err)
}
want := "\nsegment bridge state gateway\n" +
"red lb-red present 10.1.0.1\n" +
"blue lb-blue absent -\n"
if !strings.HasSuffix(out.String(), want) {
t.Errorf("status:\n%s\ndoes not end with:\n%s", out.String(), want)
}
}
func TestNetworkUp_CreatesBridgesAndTaps(t *testing.T) {
f := newLab(t, twoSegments, twoRouters)
if err := f.lab.NetworkUp(context.Background()); err != nil {
t.Fatalf("NetworkUp: %v", err)
}
want := []string{
"link show dev lb-red",
"link show dev lb-blue",
"link show dev lt0",
"link show dev lt1",
"link show dev lt2",
"link show dev lt3",
"link add name lb-red type bridge stp_state 0 mcast_snooping 0",
"link set dev lb-red addrgenmode none",
"tuntap add dev lt0 mode tap",
"link set dev lt0 addrgenmode none",
"link set dev lt0 mtu 9000 master lb-red up",
"tuntap add dev lt1 mode tap",
"link set dev lt1 addrgenmode none",
"link set dev lt1 mtu 9000 master lb-red up",
"tuntap add dev lt2 mode tap",
"link set dev lt2 addrgenmode none",
"link set dev lt2 mtu 9000 master lb-red up",
"link set dev lb-red mtu 9000 up",
"link add name lb-blue type bridge stp_state 0 mcast_snooping 0",
"link set dev lb-blue addrgenmode none",
"tuntap add dev lt3 mode tap",
"link set dev lt3 addrgenmode none",
"link set dev lt3 mtu 1500 master lb-blue up",
"link set dev lb-blue mtu 1500 up",
}
if got := f.runner.ipCalls(); !reflect.DeepEqual(got, want) {
t.Errorf("ip calls:\n%s\nwant:\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
}
if len(f.runner.calls) != len(want) {
t.Errorf("other commands ran: %q", f.runner.calls)
}
}
func TestNetworkUp_RefusesExistingLinks(t *testing.T) {
cases := map[string]struct {
link, want string
}{
"bridge": {"lb-blue", "segment blue: bridge lb-blue already exists on this host: 'labctl down' first, or remove it"},
"tap": {"lt2", "node hv: tap lt2 already exists on this host: 'labctl down' first, or remove it"},
}
for name, c := range cases {
t.Run(name, func(t *testing.T) {
f := newLab(t, twoSegments, twoRouters)
f.runner.present(c.link)
dir := filepath.Join(f.lab.ProcDir, "sys", "net", "bridge")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "bridge-nf-call-iptables"), []byte("1\n"), 0o644); err != nil {
t.Fatal(err)
}
err := f.lab.NetworkUp(context.Background())
if err == nil || err.Error() != c.want {
t.Errorf("error = %v, want %q", err, c.want)
}
for _, call := range f.runner.ipCalls() {
if !strings.HasPrefix(call, "link show dev ") {
t.Errorf("%q ran after the refusal", call)
}
}
if data, _ := os.ReadFile(filepath.Join(dir, "bridge-nf-call-iptables")); string(data) != "1\n" {
t.Errorf("bridge filter changed to %q", data)
}
})
}
}
func TestNetworkUp_DisablesTheBridgeFilterWhenPresent(t *testing.T) {
f := newLab(t, twoSegments, twoRouters)
dir := filepath.Join(f.lab.ProcDir, "sys", "net", "bridge")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
path := filepath.Join(dir, "bridge-nf-call-iptables")
if err := os.WriteFile(path, []byte("1\n"), 0o644); err != nil {
t.Fatal(err)
}
if err := f.lab.NetworkUp(context.Background()); err != nil {
t.Fatalf("NetworkUp: %v", err)
}
if data, err := os.ReadFile(path); err != nil || string(data) != "0\n" {
t.Errorf("bridge filter = %q, %v, want 0", data, err)
}
}
func TestNetworkUp_LeavesAnAbsentBridgeFilterAlone(t *testing.T) {
f := newLab(t, twoSegments, twoRouters)
if err := f.lab.NetworkUp(context.Background()); err != nil {
t.Fatalf("NetworkUp: %v", err)
}
if _, err := os.Stat(filepath.Join(f.lab.ProcDir, "sys")); !os.IsNotExist(err) {
t.Errorf("bridge filter created under the proc dir: %v", err)
}
}
func TestNetworkUp_ReportsTheFailingSegment(t *testing.T) {
f := newLab(t, twoSegments, twoRouters)
f.runner.fail = "ip"
err := f.lab.NetworkUp(context.Background())
if err == nil || err.Error() != "segment red: ip failed" {
t.Errorf("error = %v", err)
}
}
func TestNetworkDown_DeletesTapsThenBridgesThatExist(t *testing.T) {
f := newLab(t, twoSegments, twoRouters)
f.runner.present("lb-red", "lb-blue", "lt0", "lt2", "lt3")
if err := f.lab.NetworkDown(context.Background()); err != nil {
t.Fatalf("NetworkDown: %v", err)
}
var deleted []string
for _, call := range f.runner.ipCalls() {
if link, ok := strings.CutPrefix(call, "link delete dev "); ok {
deleted = append(deleted, link)
}
}
if want := []string{"lt0", "lt2", "lt3", "lb-red", "lb-blue"}; !reflect.DeepEqual(deleted, want) {
t.Errorf("deleted %v, want %v", deleted, want)
}
if len(f.runner.links) != 0 {
t.Errorf("links left: %v", f.runner.links)
}
}
func TestDown_RemovesTheNetworkAfterTheNodes(t *testing.T) {
f := newFixture(t, routerLast)
f.runner.present("lb-underlay", "lt0", "lt1", "lt2")
f.process(t, "hv1", "sleep 30", qemu("hv1")...)
if err := f.lab.Down(context.Background()); err != nil {
t.Fatalf("Down: %v", err)
}
var deleted []string
for _, call := range f.runner.ipCalls() {
if link, ok := strings.CutPrefix(call, "link delete dev "); ok {
deleted = append(deleted, link)
}
}
if want := []string{"lt0", "lt1", "lt2", "lb-underlay"}; !reflect.DeepEqual(deleted, want) {
t.Errorf("deleted %v, want %v", deleted, want)
}
if f.out.String() != "hv1: stopped\n" {
t.Errorf("output = %q", f.out.String())
}
}
func TestDown_KeepsTheNetworkWhenANodeCannotBeStopped(t *testing.T) {
f := newFixture(t, routerLast)
f.runner.present("lb-underlay", "lt0", "lt1", "lt2")
f.pidFile(t, "rr1", "garbage")
if err := f.lab.Down(context.Background()); err == nil {
t.Fatal("Down succeeded")
}
if calls := f.runner.ipCalls(); len(calls) != 0 {
t.Errorf("ip calls after a failed stop: %q", calls)
}
}
func TestUp_BuildsTheNetworkBeforeStartingQEMU(t *testing.T) {
f := newLab(t, twoSegments, twoRouters)
if err := f.lab.Up(context.Background(), f.fetcher, time.Second); err != nil {
t.Fatalf("Up: %v", err)
}
lastIP, firstQEMU := -1, -1
var started []string
for i, c := range f.runner.calls {
switch c[0] {
case "ip":
lastIP = i
case "qemu-system-x86_64":
if firstQEMU < 0 {
firstQEMU = i
}
started = append(started, c[indexOf(c, "-name")+1])
}
}
if lastIP < 0 || firstQEMU < 0 || lastIP > firstQEMU {
t.Errorf("last ip call at %d, first qemu start at %d", lastIP, firstQEMU)
}
if !reflect.DeepEqual(started, []string{"r1", "r2", "hv"}) {
t.Errorf("start order = %v", started)
}
for _, link := range []string{"lb-red", "lb-blue", "lt0", "lt1", "lt2", "lt3"} {
if !f.runner.links[link] {
t.Errorf("%s was not created", link)
}
}
}
func TestUp_RefusesAnExistingBridgeBeforeStartingQEMU(t *testing.T) {
f := newFixture(t, routerLast)
f.runner.present("lb-underlay")
err := f.lab.Up(context.Background(), f.fetcher, time.Second)
if err == nil || !strings.Contains(err.Error(), "bridge lb-underlay already exists") {
t.Errorf("error = %v", err)
}
if n := len(f.runner.commands("qemu-system-x86_64")); n != 0 {
t.Errorf("%d qemu starts", n)
}
}
func TestUp_StopsWhenTheNetworkFails(t *testing.T) {
f := newFixture(t, routerLast)
f.runner.fail = "ip"
err := f.lab.Up(context.Background(), f.fetcher, time.Second)
if err == nil || err.Error() != "segment underlay: ip failed" {
t.Errorf("error = %v", err)
}
if n := len(f.runner.commands("qemu-system-x86_64")); n != 0 {
t.Errorf("%d qemu starts", n)
}
}
func TestUp_ChecksFRROnVRRPRouters(t *testing.T) {
f := newLab(t, twoSegments, twoRouters)
if err := f.lab.Up(context.Background(), f.fetcher, time.Second); err != nil {
t.Fatalf("Up: %v", err)
}
var checks []string
for _, c := range f.runner.commands("ssh") {
if c[len(c)-4] == "systemctl" {
checks = append(checks, c[indexOf(c, "-p")+1]+" "+strings.Join(c[len(c)-4:], " "))
}
}
want := []string{
"2200 systemctl is-active --quiet frr.service",
"2201 systemctl is-active --quiet frr.service",
}
if !reflect.DeepEqual(checks, want) {
t.Errorf("checks = %q, want %q", checks, want)
}
}
func TestServices(t *testing.T) {
f := newLab(t, twoSegments+underlay, twoRouters+` sw1: { role: router, image: deb, cpus: 1, memory: 512, segments: [underlay], services: [x.service] }
`)
want := map[string][]string{
"r1": {"frr.service"},
"r2": {"frr.service"},
"hv": nil,
"sw1": {"x.service"},
}
for _, n := range f.lab.Plan.Nodes {
if got := services(f.lab.Plan, n); !reflect.DeepEqual(got, want[n.Name]) {
t.Errorf("%s services = %q, want %q", n.Name, got, want[n.Name])
}
}
}

122
internal/machine/network.go Normal file
View file

@ -0,0 +1,122 @@
package machine
import (
"context"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strconv"
"text/tabwriter"
"git.g3e.fr/syonad/labctl/internal/topology"
)
const bridgeFilter = "sys/net/bridge/bridge-nf-call-iptables"
func (l Lab) exists(ctx context.Context, link string) bool {
return l.Runner.Run(ctx, "ip", "link", "show", "dev", link) == nil
}
func (l Lab) NetworkUp(ctx context.Context) error {
for _, s := range l.Plan.Segments {
if l.exists(ctx, s.Bridge) {
return fmt.Errorf("segment %s: bridge %s already exists on this host: 'labctl down' first, or remove it", s.Name, s.Bridge)
}
}
for _, port := range l.Plan.Ports {
if l.exists(ctx, port.Tap) {
return fmt.Errorf("node %s: tap %s already exists on this host: 'labctl down' first, or remove it", port.Node, port.Tap)
}
}
if err := l.disableBridgeFilter(); err != nil {
return err
}
for _, s := range l.Plan.Segments {
if err := l.ip(ctx,
[]string{"link", "add", "name", s.Bridge, "type", "bridge", "stp_state", "0", "mcast_snooping", "0"},
[]string{"link", "set", "dev", s.Bridge, "addrgenmode", "none"},
); err != nil {
return fmt.Errorf("segment %s: %w", s.Name, err)
}
for _, port := range l.Plan.Ports {
if port.Segment != s.Name {
continue
}
mtu := strconv.Itoa(port.MTU)
if err := l.ip(ctx,
[]string{"tuntap", "add", "dev", port.Tap, "mode", "tap"},
[]string{"link", "set", "dev", port.Tap, "addrgenmode", "none"},
[]string{"link", "set", "dev", port.Tap, "mtu", mtu, "master", s.Bridge, "up"},
); err != nil {
return fmt.Errorf("node %s: %w", port.Node, err)
}
}
if err := l.ip(ctx, []string{"link", "set", "dev", s.Bridge, "mtu", strconv.Itoa(s.MTU), "up"}); err != nil {
return fmt.Errorf("segment %s: %w", s.Name, err)
}
}
return nil
}
func (l Lab) NetworkDown(ctx context.Context) error {
var errs []error
for _, port := range l.Plan.Ports {
if !l.exists(ctx, port.Tap) {
continue
}
if err := l.ip(ctx, []string{"link", "delete", "dev", port.Tap}); err != nil {
errs = append(errs, fmt.Errorf("node %s: %w", port.Node, err))
}
}
for _, s := range l.Plan.Segments {
if !l.exists(ctx, s.Bridge) {
continue
}
if err := l.ip(ctx, []string{"link", "delete", "dev", s.Bridge}); err != nil {
errs = append(errs, fmt.Errorf("segment %s: %w", s.Name, err))
}
}
return errors.Join(errs...)
}
func (l Lab) ip(ctx context.Context, commands ...[]string) error {
for _, args := range commands {
if err := l.Runner.Run(ctx, "ip", args...); err != nil {
return err
}
}
return nil
}
func (l Lab) disableBridgeFilter() error {
path := filepath.Join(l.ProcDir, bridgeFilter)
if _, err := os.Stat(path); errors.Is(err, os.ErrNotExist) {
return nil
}
if err := os.WriteFile(path, []byte("0\n"), 0o644); err != nil {
return fmt.Errorf("%s: %w", path, err)
}
return nil
}
func (l Lab) networkStatus(w io.Writer) error {
tw := tabwriter.NewWriter(w, 0, 0, 2, ' ', 0)
fmt.Fprintf(tw, "\nsegment\tbridge\tstate\tgateway\n")
for _, s := range l.Plan.Segments {
state := "absent"
if l.exists(context.Background(), s.Bridge) {
state = "present"
}
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\n", s.Name, s.Bridge, state, gatewayOf(s))
}
return tw.Flush()
}
func gatewayOf(s topology.SegmentPlan) string {
if !s.Gateway.IsValid() {
return "-"
}
return s.Gateway.String()
}

View file

@ -56,12 +56,12 @@ images:
url: ` + img.URL + `
sums: ` + img.Sums + `
segments:
underlay: { switch: sw1, cidr: 10.250.0.0/24, mtu: 9000 }
underlay: { routers: [sw1], cidr: 10.250.0.0/24, mtu: 9000 }
nodes:
sw1: { role: switch, image: debian12, cpus: 2, memory: 1024 }
rr1: { role: rr, image: debian12, cpus: 1, memory: 1024, segments: [underlay] }
hv1: { role: host, image: debian12, cpus: 4, memory: 16384, segments: [underlay] }
hv2: { role: host, image: debian12, cpus: 4, memory: 16384, segments: [underlay] }
sw1: { role: router, image: debian12, cpus: 2, memory: 1024, segments: [underlay] }
rr1: { role: rr, image: debian12, cpus: 1, memory: 1024, segments: [underlay] }
hv1: { role: host, image: debian12, cpus: 4, memory: 16384, segments: [underlay] }
hv2: { role: host, image: debian12, cpus: 4, memory: 16384, segments: [underlay] }
`
topo, err := topology.Parse([]byte(doc))
if err != nil {

View file

@ -31,7 +31,7 @@ func qemuArgs(p *topology.Plan, n topology.NodePlan, index int, dir string) []st
}
restrict := "on"
if n.Role == topology.RoleSwitch {
if n.Role == topology.RoleRouter {
restrict = "off"
}
args = append(args,
@ -39,19 +39,11 @@ func qemuArgs(p *topology.Plan, n topology.NodePlan, index int, dir string) []st
"-device", fmt.Sprintf("virtio-net-pci,netdev=%s,mac=%s,romfile=", AdminInterface, adminMAC(index)),
)
for _, c := range nodeCables(p, n.Name) {
args = append(args, cable(c.NodeInterface, c.NodePort, c.SwitchPort, c.NodeMAC.String(), c.MTU)...)
}
for _, c := range switchCables(p, n.Name) {
args = append(args, cable(c.SwitchInterface, c.SwitchPort, c.NodePort, c.SwitchMAC.String(), c.MTU)...)
for _, port := range nodePorts(p, n.Name) {
args = append(args,
"-netdev", fmt.Sprintf("tap,id=%s,ifname=%s,script=no,downscript=no", port.Interface, port.Tap),
"-device", fmt.Sprintf("virtio-net-pci,netdev=%s,mac=%s,host_mtu=%d,romfile=", port.Interface, port.MAC, port.MTU),
)
}
return args
}
func cable(id string, local, remote int, mac string, mtu int) []string {
return []string{
"-netdev", fmt.Sprintf("dgram,id=%s,local.type=inet,local.host=%s,local.port=%d,remote.type=inet,remote.host=%s,remote.port=%d",
id, loopback, local, loopback, remote),
"-device", fmt.Sprintf("virtio-net-pci,netdev=%s,mac=%s,host_mtu=%d,romfile=", id, mac, mtu),
}
}

View file

@ -119,32 +119,46 @@ func adminMAC(index int) net.HardwareAddr {
return net.HardwareAddr{0x02, 0x4d, byte(index >> 8), byte(index), 0x00, 0x00}
}
func nodeCables(p *topology.Plan, name string) []topology.Cable {
var out []topology.Cable
for _, c := range p.Cables {
if c.Node == name {
out = append(out, c)
func nodePorts(p *topology.Plan, name string) []topology.Port {
var out []topology.Port
for _, port := range p.Ports {
if port.Node == name {
out = append(out, port)
}
}
return out
}
func switchCables(p *topology.Plan, name string) []topology.Cable {
var out []topology.Cable
for _, c := range p.Cables {
if c.Switch == name {
out = append(out, c)
}
}
return out
}
func switchSegments(p *topology.Plan, name string) []topology.SegmentPlan {
func routedSegments(p *topology.Plan, name string) []topology.SegmentPlan {
var out []topology.SegmentPlan
for _, s := range p.Segments {
if s.Switch == name {
out = append(out, s)
for _, r := range s.Routers {
if r == name {
out = append(out, s)
}
}
}
return out
}
func segmentNamed(p *topology.Plan, name string) topology.SegmentPlan {
for _, s := range p.Segments {
if s.Name == name {
return s
}
}
return topology.SegmentPlan{}
}
func uplink(p *topology.Plan, name string) (topology.Port, bool) {
ports := nodePorts(p, name)
for _, port := range ports {
if segmentNamed(p, port.Segment).Gateway.IsValid() {
return port, true
}
}
if len(ports) > 0 {
return ports[0], false
}
return topology.Port{}, false
}

View file

@ -2,6 +2,7 @@ package render
import (
"reflect"
"strconv"
"strings"
"testing"
@ -18,12 +19,12 @@ images:
url: https://example.invalid/deb.qcow2
sums: https://example.invalid/SHA512SUMS
segments:
underlay: { switch: sw1, cidr: 10.250.0.0/24, mtu: 9000 }
underlay: { routers: [sw1], cidr: 10.250.0.0/24, mtu: 9000 }
nodes:
sw1: { role: switch, image: deb, cpus: 2, memory: 1024 }
rr1: { role: rr, image: deb, cpus: 1, memory: 1024, segments: [underlay] }
hv1: { role: host, image: deb, cpus: 4, memory: 16384, segments: [underlay] }
hv2: { role: host, image: deb, cpus: 4, memory: 16384, segments: [underlay] }
sw1: { role: router, image: deb, cpus: 2, memory: 1024, segments: [underlay] }
rr1: { role: rr, image: deb, cpus: 1, memory: 1024, segments: [underlay] }
hv1: { role: host, image: deb, cpus: 4, memory: 16384, segments: [underlay] }
hv2: { role: host, image: deb, cpus: 4, memory: 16384, segments: [underlay] }
`
const twoSegments = `name: two-seg
@ -32,13 +33,30 @@ images:
url: https://example.invalid/deb.qcow2
sums: https://example.invalid/SHA512SUMS
segments:
red: { switch: sw, cidr: 10.1.0.0/24 }
blue: { switch: sw, cidr: 10.2.0.0/24, mtu: 1500 }
red: { routers: [sw], cidr: 10.1.0.0/24 }
blue: { routers: [sw], cidr: 10.2.0.0/24, mtu: 1500 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [red, blue] }
hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [blue, red] }
`
const withVRRP = `name: vrrp
images:
deb:
url: https://example.invalid/deb.qcow2
sums: https://example.invalid/SHA512SUMS
segments:
red: { routers: [r1, r2], cidr: 10.1.0.0/24 }
blue: { routers: [r1], cidr: 10.2.0.0/24 }
pure: { cidr: 10.3.0.0/24 }
nodes:
r1: { role: router, image: deb, cpus: 1, memory: 512, segments: [red, blue] }
r2: { role: router, image: deb, cpus: 1, memory: 512, segments: [red, pure] }
r3: { role: router, image: deb, cpus: 1, memory: 512, segments: [pure] }
hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [pure, red] }
l2: { role: host, image: deb, cpus: 1, memory: 512, segments: [pure] }
`
func plan(t *testing.T, doc string) *topology.Plan {
t.Helper()
topo, err := topology.Parse([]byte(doc))
@ -182,7 +200,7 @@ func TestQEMU_HypervisorNetwork(t *testing.T) {
hv := nodeNamed(t, renderAll(t, twoHypervisors), "hv1")
wantNetdevs := []string{
"user,id=mgmt0,restrict=on,ipv6=off,hostfwd=tcp:127.0.0.1:2202-:22",
"dgram,id=underlay,local.type=inet,local.host=127.0.0.1,local.port=20002,remote.type=inet,remote.host=127.0.0.1,remote.port=20003",
"tap,id=underlay,ifname=lt2,script=no,downscript=no",
}
wantDevices := []string{
"virtio-net-pci,netdev=mgmt0,mac=02:4d:00:02:00:00,romfile=",
@ -196,27 +214,31 @@ func TestQEMU_HypervisorNetwork(t *testing.T) {
}
}
func TestQEMU_SwitchNetwork(t *testing.T) {
sw := nodeNamed(t, renderAll(t, twoHypervisors), "sw1")
func TestQEMU_RouterNetwork(t *testing.T) {
sw := nodeNamed(t, renderAll(t, twoSegments), "sw")
wantNetdevs := []string{
"user,id=mgmt0,restrict=off,ipv6=off,hostfwd=tcp:127.0.0.1:2200-:22",
"dgram,id=p0,local.type=inet,local.host=127.0.0.1,local.port=20001,remote.type=inet,remote.host=127.0.0.1,remote.port=20000",
"dgram,id=p1,local.type=inet,local.host=127.0.0.1,local.port=20003,remote.type=inet,remote.host=127.0.0.1,remote.port=20002",
"dgram,id=p2,local.type=inet,local.host=127.0.0.1,local.port=20005,remote.type=inet,remote.host=127.0.0.1,remote.port=20004",
"tap,id=red,ifname=lt0,script=no,downscript=no",
"tap,id=blue,ifname=lt2,script=no,downscript=no",
}
wantDevices := []string{
"virtio-net-pci,netdev=mgmt0,mac=02:4d:00:00:00:00,romfile=",
"virtio-net-pci,netdev=red,mac=02:4c:00:00:00:00,host_mtu=9000,romfile=",
"virtio-net-pci,netdev=blue,mac=02:4c:00:00:01:00,host_mtu=1500,romfile=",
}
if got := value(t, sw.QEMU, "-netdev"); !reflect.DeepEqual(got, wantNetdevs) {
t.Errorf("netdevs = %v\nwant %v", got, wantNetdevs)
}
if !hasValue(sw.QEMU, "-device", "virtio-net-pci,netdev=p2,mac=02:4c:00:03:00:01,host_mtu=9000,romfile=") {
t.Errorf("p2 device missing: %v", sw.QEMU)
if got := value(t, sw.QEMU, "-device"); !reflect.DeepEqual(got, wantDevices) {
t.Errorf("devices = %v\nwant %v", got, wantDevices)
}
}
func TestQEMU_OnlyTheSwitchReachesTheOutsideThroughAdministration(t *testing.T) {
for name, n := range renderAll(t, twoHypervisors) {
func TestQEMU_OnlyRoutersReachTheOutsideThroughAdministration(t *testing.T) {
for name, n := range renderAll(t, withVRRP) {
admin := value(t, n.QEMU, "-netdev")[0]
wantRestrict := "restrict=on"
if name == "sw1" {
if name == "r1" || name == "r2" || name == "r3" {
wantRestrict = "restrict=off"
}
if !strings.Contains(admin, ","+wantRestrict+",") {
@ -228,33 +250,28 @@ func TestQEMU_OnlyTheSwitchReachesTheOutsideThroughAdministration(t *testing.T)
}
}
func TestQEMU_EveryCableEndsMatch(t *testing.T) {
ends := map[string]int{}
for _, n := range renderAll(t, twoHypervisors) {
func TestQEMU_EveryTapOfThePlanIsOpenedOnce(t *testing.T) {
p := plan(t, withVRRP)
opened := map[string]int{}
for _, n := range renderAll(t, withVRRP) {
for _, nd := range value(t, n.QEMU, "-netdev") {
if !strings.HasPrefix(nd, "dgram,") {
continue
if strings.HasPrefix(nd, "dgram") || strings.HasPrefix(nd, "socket") {
t.Errorf("%s uses a %s netdev", n.Name, nd)
}
var local, remote string
for _, kv := range strings.Split(nd, ",") {
if v, ok := strings.CutPrefix(kv, "local.port="); ok {
local = v
}
if v, ok := strings.CutPrefix(kv, "remote.port="); ok {
remote = v
if tap, ok := strings.CutPrefix(kv, "ifname="); ok {
opened[tap]++
}
}
ends[local+">"+remote]++
}
}
for pair, count := range ends {
local, remote, _ := strings.Cut(pair, ">")
if count != 1 || ends[remote+">"+local] != 1 {
t.Errorf("cable end %s has no single matching end", pair)
}
if len(opened) != len(p.Ports) {
t.Errorf("%d taps opened, plan has %d ports", len(opened), len(p.Ports))
}
if len(ends) != 6 {
t.Errorf("%d cable ends, want 6", len(ends))
for _, port := range p.Ports {
if opened[port.Tap] != 1 {
t.Errorf("tap %s opened %d times", port.Tap, opened[port.Tap])
}
}
}
@ -270,7 +287,7 @@ func TestNetworkConfig_Hypervisor(t *testing.T) {
}
under := iface(t, doc, "underlay")
if under.Match.MACAddress != "02:4c:00:02:00:00" || under.SetName != "underlay" || under.MTU != 9000 ||
!reflect.DeepEqual(under.Addresses, []string{"10.250.0.3/24"}) {
!reflect.DeepEqual(under.Addresses, []string{"10.250.0.4/24"}) {
t.Errorf("underlay = %+v", under)
}
if !reflect.DeepEqual(under.Routes, []route{{To: "0.0.0.0/0", Via: "10.250.0.1"}}) {
@ -289,17 +306,27 @@ func TestNetworkConfig_DHCPIsExplicitlyOffEverywhere(t *testing.T) {
}
}
func TestNetworkConfig_Switch(t *testing.T) {
doc := network(t, nodeNamed(t, renderAll(t, twoHypervisors), "sw1"))
func TestNetworkConfig_Router(t *testing.T) {
doc := network(t, nodeNamed(t, renderAll(t, twoSegments), "sw"))
if len(doc.Ethernets) != 3 {
t.Fatalf("network-config = %+v", doc)
}
admin := iface(t, doc, "mgmt0")
if !reflect.DeepEqual(admin.Routes, []route{{To: "0.0.0.0/0", Via: "10.0.2.2"}}) ||
admin.Nameservers == nil || !reflect.DeepEqual(admin.Nameservers.Addresses, []string{"10.0.2.3"}) {
t.Errorf("switch mgmt0 = %+v", admin)
t.Errorf("router mgmt0 = %+v", admin)
}
for port, mac := range map[string]string{"p0": "02:4c:00:01:00:01", "p1": "02:4c:00:02:00:01", "p2": "02:4c:00:03:00:01"} {
e := iface(t, doc, port)
if e.Match.MACAddress != mac || e.SetName != port || e.MTU != 9000 || len(e.Addresses) != 0 || len(e.Routes) != 0 {
t.Errorf("%s = %+v", port, e)
for name, want := range map[string]struct {
mac, address string
mtu int
}{
"red": {"02:4c:00:00:00:00", "10.1.0.2/24", 9000},
"blue": {"02:4c:00:00:01:00", "10.2.0.2/24", 1500},
} {
e := iface(t, doc, name)
if e.Match.MACAddress != want.mac || e.SetName != name || e.MTU != want.mtu ||
!reflect.DeepEqual(e.Addresses, []string{want.address}) || len(e.Routes) != 0 || e.Nameservers != nil {
t.Errorf("%s = %+v", name, e)
}
}
}
@ -333,45 +360,71 @@ func TestUserData_PasswordLoginIsExplicitlyOff(t *testing.T) {
}
}
func TestUserData_SwitchBuildsBridgeGatewayAndNAT(t *testing.T) {
func TestNetworkConfig_DefaultRouteOnTheFirstSegmentWithAGateway(t *testing.T) {
doc := network(t, nodeNamed(t, renderAll(t, withVRRP), "hv"))
pure, red := iface(t, doc, "pure"), iface(t, doc, "red")
if len(pure.Routes) != 0 || pure.Nameservers != nil || !reflect.DeepEqual(pure.Addresses, []string{"10.3.0.3/24"}) {
t.Errorf("pure = %+v", pure)
}
if !reflect.DeepEqual(red.Routes, []route{{To: "0.0.0.0/0", Via: "10.1.0.1"}}) || red.Nameservers == nil ||
!reflect.DeepEqual(red.Addresses, []string{"10.1.0.4/24"}) {
t.Errorf("red = %+v", red)
}
}
func TestNetworkConfig_NoDefaultRouteWithoutGateway(t *testing.T) {
for name, e := range network(t, nodeNamed(t, renderAll(t, withVRRP), "l2")).Ethernets {
if len(e.Routes) != 0 || e.Nameservers != nil {
t.Errorf("%s = %+v, want no route", name, e)
}
}
}
func TestNetworkConfig_RouterHasNoDefaultRouteOnItsSegments(t *testing.T) {
doc := network(t, nodeNamed(t, renderAll(t, withVRRP), "r1"))
for _, name := range []string{"red", "blue"} {
if e := iface(t, doc, name); len(e.Routes) != 0 || e.Nameservers != nil {
t.Errorf("%s = %+v", name, e)
}
}
if admin := iface(t, doc, "mgmt0"); len(admin.Routes) != 1 {
t.Errorf("mgmt0 = %+v", admin)
}
}
func TestUserData_RouterBuildsGatewayAndNAT(t *testing.T) {
cfg := user(t, nodeNamed(t, renderAll(t, twoHypervisors), "sw1"))
if !reflect.DeepEqual(cfg.Packages, []string{"nftables"}) {
t.Errorf("packages = %v", cfg.Packages)
}
script := fileAt(t, cfg, "/usr/local/sbin/lab-switch")
script := fileAt(t, cfg, "/usr/local/sbin/lab-router")
if script.Permissions != "0755" {
t.Errorf("script permissions = %s", script.Permissions)
}
wantScript := `#!/bin/sh
set -eu
sysctl -qw net.ipv4.ip_forward=1
ip link add br-underlay type bridge stp_state 0 2>/dev/null || true
ip link set dev p0 master br-underlay
ip link set dev p0 up
ip link set dev p1 master br-underlay
ip link set dev p1 up
ip link set dev p2 master br-underlay
ip link set dev p2 up
ip link set dev br-underlay mtu 9000
ip addr replace 10.250.0.1/24 dev br-underlay
ip link set dev br-underlay up
nft -f /etc/lab-switch.nft
ip addr replace 10.250.0.1/24 dev underlay
nft -f /etc/lab-router.nft
`
if script.Content != wantScript {
t.Errorf("script:\n%s\nwant:\n%s", script.Content, wantScript)
}
nft := fileAt(t, cfg, "/etc/lab-switch.nft").Content
nftFile := fileAt(t, cfg, "/etc/lab-router.nft")
if nftFile.Permissions != "0644" {
t.Errorf("nft permissions = %s", nftFile.Permissions)
}
for _, want := range []string{
"add table ip lab_nat\ndelete table ip lab_nat\n",
"type nat hook postrouting priority srcnat;",
`ip saddr { 10.250.0.0/24 } oifname "mgmt0" masquerade`,
} {
if !strings.Contains(nft, want) {
t.Errorf("nft rules miss %q:\n%s", want, nft)
if !strings.Contains(nftFile.Content, want) {
t.Errorf("nft rules miss %q:\n%s", want, nftFile.Content)
}
}
unit := fileAt(t, cfg, "/etc/systemd/system/lab-switch.service").Content
for _, want := range []string{"Type=oneshot", "RemainAfterExit=yes", "ExecStart=/usr/local/sbin/lab-switch", "After=network-online.target", "WantedBy=multi-user.target"} {
unit := fileAt(t, cfg, "/etc/systemd/system/lab-router.service").Content
for _, want := range []string{"Type=oneshot", "RemainAfterExit=yes", "ExecStart=/usr/local/sbin/lab-router", "After=network-online.target", "WantedBy=multi-user.target"} {
if !strings.Contains(unit, want) {
t.Errorf("unit misses %q", want)
}
@ -379,29 +432,82 @@ nft -f /etc/lab-switch.nft
if !reflect.DeepEqual(cfg.Runcmd, [][]string{{"/usr/local/sbin/lab-provision"}}) {
t.Errorf("runcmd = %v", cfg.Runcmd)
}
if got := fileAt(t, cfg, "/usr/local/sbin/lab-provision").Content; got != "#!/bin/sh\nset -eu\nsystemctl daemon-reload\nsystemctl enable --now lab-switch.service\n" {
if got := fileAt(t, cfg, "/usr/local/sbin/lab-provision").Content; got != "#!/bin/sh\nset -eu\nsystemctl daemon-reload\nsystemctl enable --now lab-router.service\n" {
t.Errorf("lab-provision:\n%s", got)
}
}
func TestUserData_SwitchWithTwoSegments(t *testing.T) {
func TestUserData_RouterWithTwoSegments(t *testing.T) {
cfg := user(t, nodeNamed(t, renderAll(t, twoSegments), "sw"))
script := fileAt(t, cfg, "/usr/local/sbin/lab-switch").Content
for _, want := range []string{
"ip link set dev p0 master br-red\n",
"ip link set dev p1 master br-blue\n",
"ip link set dev br-blue mtu 1500\n",
"ip addr replace 10.2.0.1/24 dev br-blue\n",
script := fileAt(t, cfg, "/usr/local/sbin/lab-router").Content
want := "sysctl -qw net.ipv4.ip_forward=1\nip addr replace 10.1.0.1/24 dev red\nip addr replace 10.2.0.1/24 dev blue\nnft -f /etc/lab-router.nft\n"
if !strings.HasSuffix(script, want) {
t.Errorf("script:\n%s\ndoes not end with:\n%s", script, want)
}
if nft := fileAt(t, cfg, "/etc/lab-router.nft").Content; !strings.Contains(nft, "ip saddr { 10.1.0.0/24, 10.2.0.0/24 }") {
t.Errorf("nft:\n%s", nft)
}
}
func TestUserData_RouterScriptWithVRRPAndSingleSegments(t *testing.T) {
script := fileAt(t, user(t, nodeNamed(t, renderAll(t, withVRRP), "r1")), "/usr/local/sbin/lab-router").Content
want := `#!/bin/sh
set -eu
sysctl -qw net.ipv4.ip_forward=1
sysctl -qw net.ipv4.conf.red.arp_ignore=1 net.ipv4.conf.red.arp_announce=2
ip link show dev vrred >/dev/null 2>&1 || { ip link add vrred link red type macvlan mode bridge && ip link set dev vrred protodown on; }
ip link set dev vrred addrgenmode random address 00:00:5e:00:01:01
ip addr replace 10.1.0.1/24 dev vrred
ip link set dev vrred up
ip addr replace 10.2.0.1/24 dev blue
nft -f /etc/lab-router.nft
`
if script != want {
t.Errorf("script:\n%s\nwant:\n%s", script, want)
}
r2 := fileAt(t, user(t, nodeNamed(t, renderAll(t, withVRRP), "r2")), "/usr/local/sbin/lab-router").Content
if !strings.Contains(r2, "ip addr replace 10.1.0.1/24 dev vrred\n") || strings.Contains(r2, "dev pure") {
t.Errorf("r2 script:\n%s", r2)
}
}
func TestUserData_RouterNATCoversTheSegmentsItRoutes(t *testing.T) {
nodes := renderAll(t, withVRRP)
for name, want := range map[string]string{
"r1": "ip saddr { 10.1.0.0/24, 10.2.0.0/24 }",
"r2": "ip saddr { 10.1.0.0/24 }",
} {
if !strings.Contains(script, want) {
t.Errorf("script misses %q:\n%s", want, script)
if nft := fileAt(t, user(t, nodeNamed(t, nodes, name)), "/etc/lab-router.nft").Content; !strings.Contains(nft, want) {
t.Errorf("%s nft:\n%s\nmisses %s", name, nft, want)
}
}
if strings.Contains(script, "p1 master br-red") || strings.Contains(script, "p0 master br-blue") {
t.Errorf("port bridged on the wrong segment:\n%s", script)
cfg := user(t, nodeNamed(t, nodes, "r3"))
if nft := fileAt(t, cfg, "/etc/lab-router.nft").Content; nft != "add table ip lab_nat\ndelete table ip lab_nat\n" {
t.Errorf("r3 nft:\n%s", nft)
}
if nft := fileAt(t, cfg, "/etc/lab-switch.nft").Content; !strings.Contains(nft, "ip saddr { 10.1.0.0/24, 10.2.0.0/24 }") {
t.Errorf("nft:\n%s", nft)
if script := fileAt(t, cfg, "/usr/local/sbin/lab-router").Content; script != "#!/bin/sh\nset -eu\nsysctl -qw net.ipv4.ip_forward=1\nnft -f /etc/lab-router.nft\n" {
t.Errorf("r3 script:\n%s", script)
}
}
func TestVRRPMAC(t *testing.T) {
for vrid, want := range map[int]string{1: "00:00:5e:00:01:01", 16: "00:00:5e:00:01:10", 255: "00:00:5e:00:01:ff"} {
if got := vrrpMAC(vrid); got != want {
t.Errorf("vrrpMAC(%d) = %s, want %s", vrid, got, want)
}
}
}
func TestVRRPPriorityFollowsTheRoutersOrder(t *testing.T) {
var routers []string
for i := 0; i < 22; i++ {
routers = append(routers, "r"+strconv.Itoa(i))
}
s := topology.SegmentPlan{Routers: routers}
for name, want := range map[string]int{"r0": 200, "r1": 190, "r19": 10, "r20": 1, "r21": 1} {
if got := vrrpPriority(s, name); got != want {
t.Errorf("priority of %s = %d, want %d", name, got, want)
}
}
}
@ -415,7 +521,7 @@ func TestMetaData(t *testing.T) {
}
}
func TestRender_AdminMacsAreUniqueAndApartFromCableMacs(t *testing.T) {
func TestRender_AdminMacsAreUniqueAndApartFromPortMacs(t *testing.T) {
seen := map[string]string{}
for name, n := range renderAll(t, twoHypervisors) {
for _, d := range value(t, n.QEMU, "-device") {
@ -429,8 +535,8 @@ func TestRender_AdminMacsAreUniqueAndApartFromCableMacs(t *testing.T) {
}
}
}
if len(seen) != 10 {
t.Errorf("%d distinct macs, want 10", len(seen))
if len(seen) != 8 {
t.Errorf("%d distinct macs, want 8", len(seen))
}
}

View file

@ -15,9 +15,9 @@ images:
url: https://example.invalid/deb.qcow2
sums: https://example.invalid/SHA512SUMS
segments:
underlay: { switch: sw1, cidr: 192.168.14.0/24 }
underlay: { routers: [sw1], cidr: 192.168.14.0/24 }
nodes:
sw1: { role: switch, image: deb, cpus: 2, memory: 1024, secondary: { underlay: [169.254.0.1/28] }, frr: sw1.conf }
sw1: { role: router, image: deb, cpus: 2, memory: 1024, segments: [underlay], secondary: { underlay: [169.254.0.1/28] }, frr: sw1.conf }
rr1: { role: rr, image: deb, cpus: 1, memory: 1024, segments: [underlay], secondary: { underlay: [169.254.0.3/28] }, loopback: 10.255.255.1/32, frr: rr1.conf }
hv1: { role: host, image: deb, cpus: 4, memory: 16384, segments: [underlay], frr: hv1.conf, provision: hv.sh, post_provision: hv-post.sh, files: { agent.yml: two.yml }, services: [agent.service] }
hv2: { role: host, image: deb, cpus: 4, memory: 16384, segments: [underlay] }
@ -79,16 +79,20 @@ sed -i 's/^bgpd=no/bgpd=yes/' /etc/frr/daemons
func TestRoles_SecondaryAddressFollowsThePrimaryOnTheNode(t *testing.T) {
doc := network(t, nodeNamed(t, renderRoles(t), "rr1"))
got := iface(t, doc, "underlay").Addresses
if !reflect.DeepEqual(got, []string{"192.168.14.2/24", "169.254.0.3/28"}) {
if !reflect.DeepEqual(got, []string{"192.168.14.3/24", "169.254.0.3/28"}) {
t.Errorf("rr1 underlay addresses = %v", got)
}
}
func TestRoles_SwitchCarriesItsSecondaryOnTheBridge(t *testing.T) {
script := fileAt(t, user(t, nodeNamed(t, renderRoles(t), "sw1")), "/usr/local/sbin/lab-switch").Content
want := "ip addr replace 192.168.14.1/24 dev br-underlay\nip addr replace 169.254.0.1/28 dev br-underlay\nip link set dev br-underlay up\n"
if !strings.Contains(script, want) {
t.Errorf("lab-switch:\n%s\ndoes not contain:\n%s", script, want)
func TestRoles_RouterCarriesItsSecondaryOnItsSegmentInterface(t *testing.T) {
sw1 := nodeNamed(t, renderRoles(t), "sw1")
under := iface(t, network(t, sw1), "underlay")
if !reflect.DeepEqual(under.Addresses, []string{"192.168.14.2/24", "169.254.0.1/28"}) || len(under.Routes) != 0 {
t.Errorf("sw1 underlay = %+v", under)
}
script := fileAt(t, user(t, sw1), "/usr/local/sbin/lab-router").Content
if !strings.Contains(script, "ip addr replace 192.168.14.1/24 dev underlay\n") || strings.Contains(script, "169.254.0.1") {
t.Errorf("lab-router:\n%s", script)
}
}
@ -156,7 +160,7 @@ func TestRoles_ProvisioningIsOneScriptThatStopsAtTheFirstFailure(t *testing.T) {
env := "LAB_NODE=hv1 LAB_UPLINK=underlay LAB_FILES=/etc/lab/files "
files := "mkdir -p /etc/lab/files /run/lab-files\nmount -o ro LABEL=LABFILES /run/lab-files\ncp -R /run/lab-files/. /etc/lab/files/\numount /run/lab-files\n"
for name, steps := range map[string]string{
"sw1": "systemctl daemon-reload\nsystemctl enable --now lab-switch.service\n/usr/local/sbin/lab-frr\n",
"sw1": "systemctl daemon-reload\nsystemctl enable --now lab-router.service\n/usr/local/sbin/lab-frr\n",
"hv1": files + env + "/usr/local/sbin/lab-node-provision\n/usr/local/sbin/lab-frr\n" + env + "/usr/local/sbin/lab-node-post-provision\n",
} {
cfg := user(t, nodeNamed(t, nodes, name))
@ -238,6 +242,40 @@ func TestRoles_UplinkIsTheInterfaceCarryingTheDefaultRoute(t *testing.T) {
}
}
func TestRoles_UplinkSkipsSegmentsWithoutGateway(t *testing.T) {
p := plan(t, withVRRP)
want := map[string]string{
"hv": "LAB_NODE=hv LAB_UPLINK=red LAB_FILES=/etc/lab/files ",
"l2": "LAB_NODE=l2 LAB_UPLINK=pure LAB_FILES=/etc/lab/files ",
"r1": "LAB_NODE=r1 LAB_UPLINK=mgmt0 LAB_FILES=/etc/lab/files ",
"r3": "LAB_NODE=r3 LAB_UPLINK=mgmt0 LAB_FILES=/etc/lab/files ",
}
for _, n := range p.Nodes {
if w, ok := want[n.Name]; ok {
if got := provisionEnv(p, n); got != w {
t.Errorf("%s provision environment = %q, want %q", n.Name, got, w)
}
}
}
}
func TestRoles_UplinkIsGivenToTheProvisionScript(t *testing.T) {
doc := strings.Replace(withVRRP, "segments: [pure, red] }", "segments: [pure, red], provision: hv.sh }", 1)
nodes, err := Render(plan(t, doc), Options{RunDir: "/srv/lab", AuthorizedKeys: []string{labKey}, Provision: map[string]string{"hv": "#!/bin/sh\n"}})
if err != nil {
t.Fatalf("Render: %v", err)
}
for _, n := range nodes {
if n.Name != "hv" {
continue
}
got := fileAt(t, user(t, n), "/usr/local/sbin/lab-provision").Content
if got != "#!/bin/sh\nset -eu\nLAB_NODE=hv LAB_UPLINK=red LAB_FILES=/etc/lab/files /usr/local/sbin/lab-node-provision\n" {
t.Errorf("lab-provision:\n%s", got)
}
}
}
func TestRoles_NodesWithoutProvisionReceiveNoProvisionFiles(t *testing.T) {
nodes := renderRoles(t)
for _, name := range []string{"sw1", "rr1", "hv2"} {
@ -253,8 +291,67 @@ func TestRoles_FRRDaemons(t *testing.T) {
p := plan(t, withRoles)
want := map[string][]string{"sw1": {"bgpd", "bfdd"}, "rr1": {"bgpd", "bfdd"}, "hv1": {"bgpd"}}
for _, n := range p.Nodes {
if w, ok := want[n.Name]; ok && !reflect.DeepEqual(frrDaemons(n), w) {
t.Errorf("%s daemons = %v, want %v", n.Name, frrDaemons(n), w)
if w, ok := want[n.Name]; ok && !reflect.DeepEqual(frrDaemons(p, n), w) {
t.Errorf("%s daemons = %v, want %v", n.Name, frrDaemons(p, n), w)
}
}
}
func TestRoles_VRRPAddsVrrpdOnlyToTheRoutersThatRunIt(t *testing.T) {
p := plan(t, withVRRP)
want := map[string][]string{"r1": {"bgpd", "bfdd", "vrrpd"}, "r2": {"bgpd", "bfdd", "vrrpd"}, "r3": {"bgpd", "bfdd"}, "hv": {"bgpd"}}
needs := map[string]bool{"r1": true, "r2": true, "r3": false, "hv": false, "l2": false}
for _, n := range p.Nodes {
if w, ok := want[n.Name]; ok && !reflect.DeepEqual(frrDaemons(p, n), w) {
t.Errorf("%s daemons = %v, want %v", n.Name, frrDaemons(p, n), w)
}
if got := NeedsFRR(p, n); got != needs[n.Name] {
t.Errorf("NeedsFRR(%s) = %v, want %v", n.Name, got, needs[n.Name])
}
}
}
func TestRoles_VRRPRouterWithoutFRRFileGetsAGeneratedConfig(t *testing.T) {
nodes := renderAll(t, withVRRP)
cfg := user(t, nodeNamed(t, nodes, "r1"))
want := "frr defaults traditional\nhostname r1\n!\ninterface red\n vrrp 1 version 3\n vrrp 1 priority 200\n no vrrp 1 preempt\n vrrp 1 ip 10.1.0.1\nexit\n!\n"
if f := fileAt(t, cfg, "/etc/lab/frr.conf"); f.Content != want || f.Permissions != "0640" {
t.Errorf("r1 frr.conf (%s):\n%s\nwant:\n%s", f.Permissions, f.Content, want)
}
vrrpd := "sed -i 's/^vrrpd=no/vrrpd=yes/' /etc/frr/daemons\n"
bfd := "sed -i 's/^bfdd=no/bfdd=yes/' /etc/frr/daemons\n"
tail := "install -o frr -g frr -m 0640 /etc/lab/frr.conf /etc/frr/frr.conf\nsystemctl restart frr\n"
if f := fileAt(t, cfg, "/usr/local/sbin/lab-frr"); f.Content != frrScriptHead+bfd+vrrpd+tail {
t.Errorf("r1 lab-frr:\n%s", f.Content)
}
fileAt(t, cfg, "/usr/share/keyrings/frrouting.gpg")
want = "#!/bin/sh\nset -eu\nsystemctl daemon-reload\nsystemctl enable --now lab-router.service\n/usr/local/sbin/lab-frr\n"
if got := fileAt(t, cfg, "/usr/local/sbin/lab-provision").Content; got != want {
t.Errorf("r1 lab-provision:\n%s\nwant:\n%s", got, want)
}
for _, name := range []string{"r3", "hv", "l2"} {
for _, f := range user(t, nodeNamed(t, nodes, name)).WriteFiles {
if strings.Contains(f.Path, "frr") {
t.Errorf("%s receives %s", name, f.Path)
}
}
}
}
func TestRoles_VRRPStanzaFollowsTheUserFRRConfig(t *testing.T) {
doc := strings.Replace(withVRRP, "segments: [red, pure] }", "segments: [red, pure], frr: r2.conf }", 1)
o := Options{RunDir: "/srv/lab", AuthorizedKeys: []string{labKey}, FRR: map[string]string{"r2": "hostname r2\nrouter bgp 65001\n\n\n"}}
nodes, err := Render(plan(t, doc), o)
if err != nil {
t.Fatalf("Render: %v", err)
}
for _, n := range nodes {
if n.Name != "r2" {
continue
}
want := "hostname r2\nrouter bgp 65001\ninterface red\n vrrp 1 version 3\n vrrp 1 priority 190\n no vrrp 1 preempt\n vrrp 1 ip 10.1.0.1\nexit\n!\n"
if got := fileAt(t, user(t, n), "/etc/lab/frr.conf").Content; got != want {
t.Errorf("r2 frr.conf:\n%s\nwant:\n%s", got, want)
}
}
}
@ -275,7 +372,7 @@ func TestRoles_RefusesAnUnreadFRRConfig(t *testing.T) {
}
}
func TestRoles_SwitchLoopbackIsCreatedByTheSwitchScript(t *testing.T) {
func TestRoles_RouterLoopbackIsCreatedByTheRouterScript(t *testing.T) {
doc := strings.Replace(withRoles, "secondary: { underlay: [169.254.0.1/28] }, frr: sw1.conf", "secondary: { underlay: [169.254.0.1/28] }, loopback: 10.255.254.1/32, frr: sw1.conf", 1)
nodes, err := Render(plan(t, doc), roleOptions())
if err != nil {
@ -287,10 +384,16 @@ func TestRoles_SwitchLoopbackIsCreatedByTheSwitchScript(t *testing.T) {
sw1 = n
}
}
script := fileAt(t, user(t, sw1), "/usr/local/sbin/lab-switch").Content
want := "ip link set dev br-underlay up\nip link add lo1 type dummy 2>/dev/null || true\nip addr replace 10.255.254.1/32 dev lo1\nip link set dev lo1 up\nnft -f /etc/lab-switch.nft\n"
cfg := user(t, sw1)
script := fileAt(t, cfg, "/usr/local/sbin/lab-router").Content
want := "ip addr replace 192.168.14.1/24 dev underlay\nip link add lo1 type dummy 2>/dev/null || true\nip addr replace 10.255.254.1/32 dev lo1\nip link set dev lo1 up\nnft -f /etc/lab-router.nft\n"
if !strings.HasSuffix(script, want) {
t.Errorf("lab-switch:\n%s\ndoes not end with:\n%s", script, want)
t.Errorf("lab-router:\n%s\ndoes not end with:\n%s", script, want)
}
for _, f := range cfg.WriteFiles {
if strings.HasPrefix(f.Path, "/usr/local/sbin/lab-node") || strings.Contains(f.Path, "lab-node.service") {
t.Errorf("sw1 receives %s", f.Path)
}
}
}

View file

@ -12,9 +12,10 @@ import (
)
const (
SwitchScript = "/usr/local/sbin/lab-switch"
SwitchNFT = "/etc/lab-switch.nft"
SwitchUnit = "/etc/systemd/system/lab-switch.service"
RouterScript = "/usr/local/sbin/lab-router"
RouterNFT = "/etc/lab-router.nft"
RouterUnit = "/etc/systemd/system/lab-router.service"
VRRPPrefix = "vr"
NodeScript = "/usr/local/sbin/lab-node"
NodeUnit = "/etc/systemd/system/lab-node.service"
@ -99,14 +100,14 @@ func userData(p *topology.Plan, n topology.NodePlan, o Options) ([]byte, error)
}
var steps []string
switch {
case n.Role == topology.RoleSwitch:
case n.Role == topology.RoleRouter:
cfg.Packages = []string{"nftables"}
cfg.WriteFiles = []writeFile{
{Path: SwitchScript, Permissions: "0755", Content: switchScript(p, n)},
{Path: SwitchNFT, Permissions: "0644", Content: switchNFT(p, n.Name)},
{Path: SwitchUnit, Permissions: "0644", Content: unit("Lab switch: bridges, gateways and NAT", SwitchScript)},
{Path: RouterScript, Permissions: "0755", Content: routerScript(p, n)},
{Path: RouterNFT, Permissions: "0644", Content: routerNFT(p, n.Name)},
{Path: RouterUnit, Permissions: "0644", Content: unit("Lab router: gateways and NAT", RouterScript)},
}
steps = append(steps, "systemctl daemon-reload", "systemctl enable --now lab-switch.service")
steps = append(steps, "systemctl daemon-reload", "systemctl enable --now lab-router.service")
case n.Loopback.IsValid():
cfg.WriteFiles = []writeFile{
{Path: NodeScript, Permissions: "0755", Content: "#!/bin/sh\nset -eu\n" + loopbackLines(n)},
@ -126,11 +127,11 @@ func userData(p *topology.Plan, n topology.NodePlan, o Options) ([]byte, error)
cfg.WriteFiles = append(cfg.WriteFiles, writeFile{Path: NodeProvision, Permissions: "0755", Encoding: "b64", Content: base64.StdEncoding.EncodeToString([]byte(o.Provision[n.Name]))})
steps = append(steps, provisionEnv(p, n)+NodeProvision)
}
if n.FRR != "" {
if NeedsFRR(p, n) {
cfg.WriteFiles = append(cfg.WriteFiles,
writeFile{Path: FRRKey, Permissions: "0644", Encoding: "b64", Content: base64.StdEncoding.EncodeToString(frrKey)},
writeFile{Path: FRRConfig, Permissions: "0640", Content: o.FRR[n.Name]},
writeFile{Path: FRRScript, Permissions: "0755", Content: frrScript(n)},
writeFile{Path: FRRConfig, Permissions: "0640", Content: frrConfig(p, n, o.FRR[n.Name])},
writeFile{Path: FRRScript, Permissions: "0755", Content: frrScript(p, n)},
)
steps = append(steps, FRRScript)
}
@ -156,27 +157,77 @@ func retry(command string) string {
}
func provisionEnv(p *topology.Plan, n topology.NodePlan) string {
uplink := ""
if cables := nodeCables(p, n.Name); len(cables) > 0 {
uplink = cables[0].NodeInterface
port, _ := uplink(p, n.Name)
up := port.Interface
if n.Role == topology.RoleRouter {
up = AdminInterface
}
return fmt.Sprintf("LAB_NODE=%s LAB_UPLINK=%s LAB_FILES=%s ", n.Name, uplink, FilesDir)
return fmt.Sprintf("LAB_NODE=%s LAB_UPLINK=%s LAB_FILES=%s ", n.Name, up, FilesDir)
}
func frrDaemons(n topology.NodePlan) []string {
if n.Role == topology.RoleHost {
return []string{"bgpd"}
func vrrpSegments(p *topology.Plan, name string) []topology.SegmentPlan {
var out []topology.SegmentPlan
for _, s := range routedSegments(p, name) {
if s.VRID != 0 {
out = append(out, s)
}
}
return []string{"bgpd", "bfdd"}
return out
}
func frrScript(n topology.NodePlan) string {
func NeedsFRR(p *topology.Plan, n topology.NodePlan) bool {
return n.FRR != "" || len(vrrpSegments(p, n.Name)) > 0
}
func frrDaemons(p *topology.Plan, n topology.NodePlan) []string {
daemons := []string{"bgpd"}
if n.Role != topology.RoleHost {
daemons = append(daemons, "bfdd")
}
if len(vrrpSegments(p, n.Name)) > 0 {
daemons = append(daemons, "vrrpd")
}
return daemons
}
func frrConfig(p *topology.Plan, n topology.NodePlan, given string) string {
segments := vrrpSegments(p, n.Name)
if len(segments) == 0 {
return given
}
var b strings.Builder
if given == "" {
fmt.Fprintf(&b, "frr defaults traditional\nhostname %s\n!\n", n.Name)
} else {
b.WriteString(strings.TrimRight(given, "\n") + "\n")
}
for _, s := range segments {
fmt.Fprintf(&b, "interface %s\n vrrp %d version 3\n vrrp %d priority %d\n no vrrp %d preempt\n vrrp %d ip %s\nexit\n!\n",
s.Name, s.VRID, s.VRID, vrrpPriority(s, n.Name), s.VRID, s.VRID, s.Gateway)
}
return b.String()
}
func vrrpPriority(s topology.SegmentPlan, name string) int {
for i, r := range s.Routers {
if r == name {
return max(200-10*i, 1)
}
}
return 1
}
func vrrpMAC(vrid int) string {
return fmt.Sprintf("00:00:5e:00:01:%02x", vrid)
}
func frrScript(p *topology.Plan, n topology.NodePlan) string {
var b strings.Builder
b.WriteString("#!/bin/sh\nset -eu\nexport DEBIAN_FRONTEND=noninteractive\n. /etc/os-release\n")
fmt.Fprintf(&b, "echo \"deb [signed-by=%s] %s ${VERSION_CODENAME} %s\" > /etc/apt/sources.list.d/frr.list\n", FRRKey, FRRRepo, FRRSuite)
b.WriteString(retry("apt-get update -qq --error-on=any"))
fmt.Fprintf(&b, "apt-get install -y -qq --no-install-recommends %s\n", FRRPackages)
for _, d := range frrDaemons(n) {
for _, d := range frrDaemons(p, n) {
fmt.Fprintf(&b, "sed -i 's/^%s=no/%s=yes/' /etc/frr/daemons\n", d, d)
}
fmt.Fprintf(&b, "install -o frr -g frr -m 0640 %s /etc/frr/frr.conf\n", FRRConfig)
@ -199,77 +250,62 @@ func networkConfig(p *topology.Plan, n topology.NodePlan, index int) ([]byte, er
SetName: AdminInterface,
Addresses: []string{AdminAddress},
}
if n.Role == topology.RoleSwitch {
router := n.Role == topology.RoleRouter
if router {
admin.Routes = []route{{To: "0.0.0.0/0", Via: AdminGateway}}
admin.Nameservers = &nameservers{Addresses: []string{AdminDNS}}
for _, c := range switchCables(p, n.Name) {
doc.Ethernets[c.SwitchInterface] = ethernet{
Match: match{MACAddress: c.SwitchMAC.String()},
SetName: c.SwitchInterface,
MTU: c.MTU,
}
}
}
doc.Ethernets[AdminInterface] = admin
for i, c := range nodeCables(p, n.Name) {
up, routed := uplink(p, n.Name)
for _, port := range nodePorts(p, n.Name) {
e := ethernet{
Match: match{MACAddress: c.NodeMAC.String()},
SetName: c.NodeInterface,
MTU: c.MTU,
Addresses: []string{c.NodeAddress.String()},
Match: match{MACAddress: port.MAC.String()},
SetName: port.Interface,
MTU: port.MTU,
Addresses: []string{port.Address.String()},
}
for _, prefix := range n.Secondary[c.Segment] {
for _, prefix := range n.Secondary[port.Segment] {
e.Addresses = append(e.Addresses, prefix.String())
}
if i == 0 {
e.Routes = []route{{To: "0.0.0.0/0", Via: gatewayOf(p, c.Segment)}}
if !router && routed && port.Segment == up.Segment {
e.Routes = []route{{To: "0.0.0.0/0", Via: segmentNamed(p, port.Segment).Gateway.String()}}
e.Nameservers = &nameservers{Addresses: Nameservers}
}
doc.Ethernets[c.NodeInterface] = e
doc.Ethernets[port.Interface] = e
}
return yaml.Marshal(doc)
}
func gatewayOf(p *topology.Plan, segment string) string {
for _, s := range p.Segments {
if s.Name == segment {
return s.Gateway.String()
}
}
return ""
}
func switchScript(p *topology.Plan, n topology.NodePlan) string {
name := n.Name
func routerScript(p *topology.Plan, n topology.NodePlan) string {
var b strings.Builder
b.WriteString("#!/bin/sh\nset -eu\nsysctl -qw net.ipv4.ip_forward=1\n")
for _, s := range switchSegments(p, name) {
fmt.Fprintf(&b, "ip link add %s type bridge stp_state 0 2>/dev/null || true\n", s.Bridge)
for _, c := range switchCables(p, name) {
if c.Segment != s.Name {
continue
}
fmt.Fprintf(&b, "ip link set dev %s master %s\n", c.SwitchInterface, s.Bridge)
fmt.Fprintf(&b, "ip link set dev %s up\n", c.SwitchInterface)
for _, s := range routedSegments(p, n.Name) {
gateway := fmt.Sprintf("%s/%d", s.Gateway, s.Network.Bits())
if s.VRID == 0 {
fmt.Fprintf(&b, "ip addr replace %s dev %s\n", gateway, s.Name)
continue
}
fmt.Fprintf(&b, "ip link set dev %s mtu %d\n", s.Bridge, s.MTU)
fmt.Fprintf(&b, "ip addr replace %s/%d dev %s\n", s.Gateway, s.Network.Bits(), s.Bridge)
for _, prefix := range n.Secondary[s.Name] {
fmt.Fprintf(&b, "ip addr replace %s dev %s\n", prefix, s.Bridge)
}
fmt.Fprintf(&b, "ip link set dev %s up\n", s.Bridge)
vrrp := VRRPPrefix + s.Name
fmt.Fprintf(&b, "sysctl -qw net.ipv4.conf.%s.arp_ignore=1 net.ipv4.conf.%s.arp_announce=2\n", s.Name, s.Name)
fmt.Fprintf(&b, "ip link show dev %s >/dev/null 2>&1 || { ip link add %s link %s type macvlan mode bridge && ip link set dev %s protodown on; }\n", vrrp, vrrp, s.Name, vrrp)
fmt.Fprintf(&b, "ip link set dev %s addrgenmode random address %s\n", vrrp, vrrpMAC(s.VRID))
fmt.Fprintf(&b, "ip addr replace %s dev %s\n", gateway, vrrp)
fmt.Fprintf(&b, "ip link set dev %s up\n", vrrp)
}
b.WriteString(loopbackLines(n))
fmt.Fprintf(&b, "nft -f %s\n", SwitchNFT)
fmt.Fprintf(&b, "nft -f %s\n", RouterNFT)
return b.String()
}
func switchNFT(p *topology.Plan, name string) string {
func routerNFT(p *topology.Plan, name string) string {
var networks []string
for _, s := range switchSegments(p, name) {
for _, s := range routedSegments(p, name) {
networks = append(networks, s.Network.String())
}
if len(networks) == 0 {
return "add table ip lab_nat\ndelete table ip lab_nat\n"
}
return fmt.Sprintf(`add table ip lab_nat
delete table ip lab_nat
table ip lab_nat {

View file

@ -12,7 +12,7 @@ import (
func (p *Plan) Write(w io.Writer) error {
tw := tabwriter.NewWriter(w, 0, 0, 2, ' ', 0)
fmt.Fprintf(tw, "lab %s: nodes %d, segments %d, cables %d\n", p.Name, len(p.Nodes), len(p.Segments), len(p.Cables))
fmt.Fprintf(tw, "lab %s: nodes %d, segments %d, ports %d\n", p.Name, len(p.Nodes), len(p.Segments), len(p.Ports))
fmt.Fprintf(tw, "\nnodes\n")
fmt.Fprintf(tw, " name\trole\timage\tcpus\tmemory\tssh\n")
@ -37,21 +37,29 @@ func (p *Plan) Write(w io.Writer) error {
}
for _, s := range p.Segments {
fmt.Fprintf(tw, "\nsegment %s: %s, mtu %d, switch %s, bridge %s, gateway %s\n",
s.Name, s.Network, s.MTU, s.Switch, s.Bridge, s.Gateway)
fmt.Fprintf(tw, " node\tinterface\taddress\tmac\tudp\t\tswitch port\tmac\tudp\n")
for _, c := range p.Cables {
if c.Segment != s.Name {
fmt.Fprintf(tw, "\nsegment %s: %s, mtu %d, bridge %s, %s\n", s.Name, s.Network, s.MTU, s.Bridge, gateway(s))
fmt.Fprintf(tw, " node\tinterface\taddress\tmac\ttap\n")
for _, port := range p.Ports {
if port.Segment != s.Name {
continue
}
fmt.Fprintf(tw, " %s\t%s\t%s\t%s\t%d\t<->\t%s %s\t%s\t%d\n",
c.Node, c.NodeInterface, c.NodeAddress, c.NodeMAC, c.NodePort,
c.Switch, c.SwitchInterface, c.SwitchMAC, c.SwitchPort)
fmt.Fprintf(tw, " %s\t%s\t%s\t%s\t%s\n", port.Node, port.Interface, port.Address, port.MAC, port.Tap)
}
}
return tw.Flush()
}
func gateway(s SegmentPlan) string {
switch {
case len(s.Routers) == 0:
return "no gateway"
case s.VRID == 0:
return fmt.Sprintf("gateway %s on %s", s.Gateway, s.Routers[0])
default:
return fmt.Sprintf("gateway %s by vrrp %d on %s", s.Gateway, s.VRID, strings.Join(s.Routers, ", "))
}
}
func loopback(n NodePlan) string {
if !n.Loopback.IsValid() {
return ""

View file

@ -9,8 +9,10 @@ import (
)
const (
SSHBasePort = 2200
CableBasePort = 20000
SSHBasePort = 2200
BridgePrefix = "lb-"
TapPrefix = "lt"
maxNodes = 1000
maxSegments = 256
@ -21,15 +23,16 @@ type Plan struct {
Images []Image
Segments []SegmentPlan
Nodes []NodePlan
Cables []Cable
Ports []Port
}
type SegmentPlan struct {
Name string
Switch string
Bridge string
Network netip.Prefix
Gateway netip.Addr
Routers []string
VRID int
MTU int
}
@ -49,20 +52,14 @@ type NodePlan struct {
Services []string
}
type Cable struct {
Segment string
MTU int
Node string
NodeInterface string
NodeMAC net.HardwareAddr
NodeAddress netip.Prefix
NodePort int
Switch string
SwitchInterface string
SwitchMAC net.HardwareAddr
SwitchPort int
type Port struct {
Segment string
MTU int
Node string
Interface string
MAC net.HardwareAddr
Address netip.Prefix
Tap string
}
func Compute(t *Topology) (*Plan, error) {
@ -102,17 +99,22 @@ func Compute(t *Topology) (*Plan, error) {
var errs []error
for segIndex, s := range t.Segments {
network := netip.MustParsePrefix(s.CIDR)
gateway := network.Addr().Next()
p.Segments = append(p.Segments, SegmentPlan{
segment := SegmentPlan{
Name: s.Name,
Switch: s.Switch,
Bridge: "br-" + s.Name,
Bridge: BridgePrefix + s.Name,
Network: network,
Gateway: gateway,
Routers: append([]string(nil), s.Routers...),
MTU: s.MTU,
})
}
if len(s.Routers) > 0 {
segment.Gateway = network.Addr().Next()
}
if len(s.Routers) > 1 {
segment.VRID = segIndex + 1
}
p.Segments = append(p.Segments, segment)
addresses, err := allocate(t, s, network, gateway)
addresses, err := allocate(t, s, network, segment.Gateway)
if err != nil {
errs = append(errs, err)
continue
@ -123,26 +125,18 @@ func Compute(t *Topology) (*Plan, error) {
if !ok {
continue
}
cable := len(p.Cables)
p.Cables = append(p.Cables, Cable{
Segment: s.Name,
MTU: s.MTU,
Node: n.Name,
NodeInterface: s.Name,
NodeMAC: mac(nodeIndex, segIndex, 0),
NodeAddress: netip.PrefixFrom(address, network.Bits()),
NodePort: CableBasePort + 2*cable,
Switch: s.Switch,
SwitchInterface: "p" + strconv.Itoa(cable),
SwitchMAC: mac(nodeIndex, segIndex, 1),
SwitchPort: CableBasePort + 2*cable + 1,
p.Ports = append(p.Ports, Port{
Segment: s.Name,
MTU: s.MTU,
Node: n.Name,
Interface: s.Name,
MAC: mac(nodeIndex, segIndex),
Address: netip.PrefixFrom(address, network.Bits()),
Tap: TapPrefix + strconv.Itoa(len(p.Ports)),
})
}
}
if last := CableBasePort + 2*len(p.Cables) - 1; last > 65535 {
errs = append(errs, fmt.Errorf("cables: %d cables need udp ports up to %d, beyond 65535", len(p.Cables), last))
}
if len(errs) > 0 {
return nil, errors.Join(errs...)
}
@ -151,7 +145,12 @@ func Compute(t *Topology) (*Plan, error) {
func allocate(t *Topology, s Segment, network netip.Prefix, gateway netip.Addr) (map[string]netip.Addr, error) {
broadcast := lastAddr(network)
used := map[netip.Addr]string{gateway: s.Switch + " (gateway)"}
used := map[netip.Addr]string{}
first := network.Addr().Next()
if gateway.IsValid() {
used[gateway] = "the gateway"
first = gateway.Next()
}
result := map[string]netip.Addr{}
var errs []error
@ -179,7 +178,7 @@ func allocate(t *Topology, s Segment, network netip.Prefix, gateway netip.Addr)
}
}
next := gateway.Next()
next := first
for _, name := range auto {
for next != broadcast && used[next] != "" {
next = next.Next()
@ -206,8 +205,8 @@ func lastAddr(network netip.Prefix) netip.Addr {
return netip.AddrFrom4([4]byte{byte(v >> 24), byte(v >> 16), byte(v >> 8), byte(v)})
}
func mac(node, segment, side int) net.HardwareAddr {
return net.HardwareAddr{0x02, 0x4c, byte(node >> 8), byte(node), byte(segment), byte(side)}
func mac(node, segment int) net.HardwareAddr {
return net.HardwareAddr{0x02, 0x4c, byte(node >> 8), byte(node), byte(segment), 0x00}
}
func contains(list []string, value string) bool {

View file

@ -27,15 +27,26 @@ func computeError(t *testing.T, doc string) string {
return err.Error()
}
func cableOf(t *testing.T, p *Plan, node, segment string) Cable {
func portOf(t *testing.T, p *Plan, node, segment string) Port {
t.Helper()
for _, c := range p.Cables {
if c.Node == node && c.Segment == segment {
return c
for _, port := range p.Ports {
if port.Node == node && port.Segment == segment {
return port
}
}
t.Fatalf("no cable for %s on %s", node, segment)
return Cable{}
t.Fatalf("no port for %s on %s", node, segment)
return Port{}
}
func segmentOf(t *testing.T, p *Plan, name string) SegmentPlan {
t.Helper()
for _, s := range p.Segments {
if s.Name == name {
return s
}
}
t.Fatalf("no segment %s", name)
return SegmentPlan{}
}
func nodeOf(t *testing.T, p *Plan, name string) NodePlan {
@ -51,12 +62,12 @@ func nodeOf(t *testing.T, p *Plan, name string) NodePlan {
const twoHypervisors = header + `
segments:
underlay: { switch: sw1, cidr: 10.250.0.0/24, mtu: 9000 }
underlay: { routers: [sw1], cidr: 10.250.0.0/24, mtu: 9000 }
nodes:
sw1: { role: switch, image: deb, cpus: 2, memory: 1024 }
rr1: { role: rr, image: deb, cpus: 1, memory: 1024, segments: [underlay] }
hv1: { role: host, image: deb, cpus: 4, memory: 16384, segments: [underlay] }
hv2: { role: host, image: deb, cpus: 4, memory: 16384, segments: [underlay] }
sw1: { role: router, image: deb, cpus: 2, memory: 1024, segments: [underlay] }
rr1: { role: rr, image: deb, cpus: 1, memory: 1024, segments: [underlay] }
hv1: { role: host, image: deb, cpus: 4, memory: 16384, segments: [underlay] }
hv2: { role: host, image: deb, cpus: 4, memory: 16384, segments: [underlay] }
`
func TestCompute_TwoHypervisorsPlan(t *testing.T) {
@ -66,34 +77,34 @@ func TestCompute_TwoHypervisorsPlan(t *testing.T) {
t.Fatalf("%d segments, want 1", len(p.Segments))
}
s := p.Segments[0]
if s.Bridge != "br-underlay" || s.Gateway != netip.MustParseAddr("10.250.0.1") || s.MTU != 9000 || s.Switch != "sw1" {
if s.Bridge != "lb-underlay" || s.Gateway != netip.MustParseAddr("10.250.0.1") || s.MTU != 9000 || !reflect.DeepEqual(s.Routers, []string{"sw1"}) || s.VRID != 0 {
t.Errorf("segment = %+v", s)
}
want := []struct {
node, address, nodeMAC, switchMAC, switchIface string
nodePort, switchPort int
node, address, mac, tap string
}{
{"rr1", "10.250.0.2/24", "02:4c:00:01:00:00", "02:4c:00:01:00:01", "p0", 20000, 20001},
{"hv1", "10.250.0.3/24", "02:4c:00:02:00:00", "02:4c:00:02:00:01", "p1", 20002, 20003},
{"hv2", "10.250.0.4/24", "02:4c:00:03:00:00", "02:4c:00:03:00:01", "p2", 20004, 20005},
{"sw1", "10.250.0.2/24", "02:4c:00:00:00:00", "lt0"},
{"rr1", "10.250.0.3/24", "02:4c:00:01:00:00", "lt1"},
{"hv1", "10.250.0.4/24", "02:4c:00:02:00:00", "lt2"},
{"hv2", "10.250.0.5/24", "02:4c:00:03:00:00", "lt3"},
}
if len(p.Cables) != len(want) {
t.Fatalf("%d cables, want %d", len(p.Cables), len(want))
if len(p.Ports) != len(want) {
t.Fatalf("%d ports, want %d", len(p.Ports), len(want))
}
for _, w := range want {
c := cableOf(t, p, w.node, "underlay")
if c.NodeAddress.String() != w.address {
t.Errorf("%s address = %s, want %s", w.node, c.NodeAddress, w.address)
port := portOf(t, p, w.node, "underlay")
if port.Address.String() != w.address {
t.Errorf("%s address = %s, want %s", w.node, port.Address, w.address)
}
if c.NodeMAC.String() != w.nodeMAC || c.SwitchMAC.String() != w.switchMAC {
t.Errorf("%s macs = %s / %s, want %s / %s", w.node, c.NodeMAC, c.SwitchMAC, w.nodeMAC, w.switchMAC)
if port.MAC.String() != w.mac {
t.Errorf("%s mac = %s, want %s", w.node, port.MAC, w.mac)
}
if c.NodePort != w.nodePort || c.SwitchPort != w.switchPort {
t.Errorf("%s ports = %d / %d, want %d / %d", w.node, c.NodePort, c.SwitchPort, w.nodePort, w.switchPort)
if port.Tap != w.tap {
t.Errorf("%s tap = %s, want %s", w.node, port.Tap, w.tap)
}
if c.NodeInterface != "underlay" || c.SwitchInterface != w.switchIface || c.Switch != "sw1" || c.MTU != 9000 {
t.Errorf("%s cable = %+v", w.node, c)
if port.Interface != "underlay" || port.MTU != 9000 {
t.Errorf("%s port = %+v", w.node, port)
}
}
@ -127,104 +138,206 @@ func TestCompute_IsStableAcrossRuns(t *testing.T) {
func TestCompute_AddressesFollowDeclarationOrderNotNames(t *testing.T) {
p := compute(t, header+`
segments:
under: { switch: sw, cidr: 10.0.0.0/24 }
under: { routers: [sw], cidr: 10.0.0.0/24 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
zeta: { role: rr, image: deb, cpus: 1, memory: 512, segments: [under] }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [under] }
alfa: { role: rr, image: deb, cpus: 1, memory: 512, segments: [under] }
`)
if got := cableOf(t, p, "zeta", "under").NodeAddress.String(); got != "10.0.0.2/24" {
t.Errorf("zeta = %s, want 10.0.0.2/24", got)
}
if got := cableOf(t, p, "alfa", "under").NodeAddress.String(); got != "10.0.0.3/24" {
t.Errorf("alfa = %s, want 10.0.0.3/24", got)
for node, want := range map[string]string{"zeta": "10.0.0.2/24", "sw": "10.0.0.3/24", "alfa": "10.0.0.4/24"} {
if got := portOf(t, p, node, "under").Address.String(); got != want {
t.Errorf("%s = %s, want %s", node, got, want)
}
}
}
func TestCompute_ExplicitAddressIsKeptAndSkippedByAllocation(t *testing.T) {
p := compute(t, header+`
segments:
under: { switch: sw, cidr: 10.0.0.0/24 }
under: { routers: [sw], cidr: 10.0.0.0/24 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [under], addresses: {under: 10.0.0.254} }
a: { role: rr, image: deb, cpus: 1, memory: 512, segments: [under] }
b: { role: rr, image: deb, cpus: 1, memory: 512, segments: [under], addresses: {under: 10.0.0.3} }
c: { role: rr, image: deb, cpus: 1, memory: 512, segments: [under] }
`)
for node, want := range map[string]string{"a": "10.0.0.2/24", "b": "10.0.0.3/24", "c": "10.0.0.4/24"} {
if got := cableOf(t, p, node, "under").NodeAddress.String(); got != want {
for node, want := range map[string]string{"sw": "10.0.0.254/24", "a": "10.0.0.2/24", "b": "10.0.0.3/24", "c": "10.0.0.4/24"} {
if got := portOf(t, p, node, "under").Address.String(); got != want {
t.Errorf("%s = %s, want %s", node, got, want)
}
}
}
func TestCompute_TwoSegmentsCableOrderMacsAndPorts(t *testing.T) {
func TestCompute_SegmentWithoutRouterHasNoGateway(t *testing.T) {
p := compute(t, header+`
segments:
red: { switch: sw, cidr: 10.1.0.0/24 }
blue: { switch: sw, cidr: 10.2.0.0/24, mtu: 1500 }
l2: { cidr: 10.3.0.0/24 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [blue, red] }
rr: { role: rr, image: deb, cpus: 1, memory: 512, segments: [red] }
a: { role: host, image: deb, cpus: 1, memory: 512, segments: [l2] }
b: { role: host, image: deb, cpus: 1, memory: 512, segments: [l2] }
`)
got := make([]string, 0, len(p.Cables))
for _, c := range p.Cables {
got = append(got, c.Segment+"/"+c.Node+"/"+c.SwitchInterface)
s := segmentOf(t, p, "l2")
if s.Gateway.IsValid() || s.VRID != 0 || len(s.Routers) != 0 || s.Bridge != "lb-l2" {
t.Errorf("segment = %+v, want no gateway", s)
}
if strings.Join(got, ",") != "red/hv/p0,red/rr/p1,blue/hv/p2" {
t.Errorf("cable order = %v", got)
}
blue := cableOf(t, p, "hv", "blue")
if blue.NodeMAC.String() != "02:4c:00:01:01:00" || blue.NodePort != 20004 || blue.MTU != 1500 || blue.NodeAddress.String() != "10.2.0.2/24" {
t.Errorf("hv on blue = %+v", blue)
}
if red := cableOf(t, p, "hv", "red"); red.NodeMAC.String() != "02:4c:00:01:00:00" {
t.Errorf("hv on red mac = %s", red.NodeMAC)
for node, want := range map[string]string{"a": "10.3.0.1/24", "b": "10.3.0.2/24"} {
if got := portOf(t, p, node, "l2").Address.String(); got != want {
t.Errorf("%s = %s, want %s", node, got, want)
}
}
}
func TestCompute_EveryMacAndPortIsUnique(t *testing.T) {
func TestCompute_SegmentWithoutRouterAcceptsTheFirstAddress(t *testing.T) {
p := compute(t, header+`
segments:
a: { switch: sw, cidr: 10.1.0.0/24 }
b: { switch: sw, cidr: 10.2.0.0/24 }
l2: { cidr: 10.3.0.0/24 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
a: { role: host, image: deb, cpus: 1, memory: 512, segments: [l2] }
b: { role: host, image: deb, cpus: 1, memory: 512, segments: [l2], addresses: {l2: 10.3.0.1} }
`)
for node, want := range map[string]string{"a": "10.3.0.2/24", "b": "10.3.0.1/24"} {
if got := portOf(t, p, node, "l2").Address.String(); got != want {
t.Errorf("%s = %s, want %s", node, got, want)
}
}
}
func TestCompute_SegmentWithOneRouter(t *testing.T) {
p := compute(t, header+`
segments:
red: { routers: [r1], cidr: 10.1.0.0/24 }
nodes:
hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [red] }
r1: { role: router, image: deb, cpus: 1, memory: 512, segments: [red] }
`)
s := segmentOf(t, p, "red")
if s.Gateway != netip.MustParseAddr("10.1.0.1") || s.VRID != 0 || !reflect.DeepEqual(s.Routers, []string{"r1"}) {
t.Errorf("segment = %+v", s)
}
for node, want := range map[string]string{"hv": "10.1.0.2/24", "r1": "10.1.0.3/24"} {
if got := portOf(t, p, node, "red").Address.String(); got != want {
t.Errorf("%s = %s, want %s", node, got, want)
}
}
}
func TestCompute_SegmentWithTwoRoutersUsesVRRP(t *testing.T) {
p := compute(t, header+`
segments:
red: { routers: [r1], cidr: 10.1.0.0/24 }
blue: { routers: [r2, r1], cidr: 10.2.0.0/24 }
pure: { cidr: 10.3.0.0/24 }
nodes:
r1: { role: router, image: deb, cpus: 1, memory: 512, segments: [red, blue] }
r2: { role: router, image: deb, cpus: 1, memory: 512, segments: [blue], addresses: {blue: 10.2.0.250} }
hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [red, blue, pure] }
`)
blue := segmentOf(t, p, "blue")
if blue.VRID != 2 || blue.Gateway != netip.MustParseAddr("10.2.0.1") || !reflect.DeepEqual(blue.Routers, []string{"r2", "r1"}) {
t.Errorf("blue = %+v", blue)
}
if red := segmentOf(t, p, "red"); red.VRID != 0 {
t.Errorf("red vrid = %d, want 0", red.VRID)
}
if pure := segmentOf(t, p, "pure"); pure.VRID != 0 || pure.Gateway.IsValid() {
t.Errorf("pure = %+v", pure)
}
for node, want := range map[string]string{"r1": "10.2.0.2/24", "r2": "10.2.0.250/24", "hv": "10.2.0.3/24"} {
if got := portOf(t, p, node, "blue").Address.String(); got != want {
t.Errorf("%s = %s, want %s", node, got, want)
}
}
}
func TestCompute_TwoSegmentsPortOrderMacsAndTaps(t *testing.T) {
p := compute(t, header+`
segments:
red: { routers: [sw], cidr: 10.1.0.0/24 }
blue: { cidr: 10.2.0.0/24, mtu: 1500 }
nodes:
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [red] }
hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [blue, red] }
rr: { role: rr, image: deb, cpus: 1, memory: 512, segments: [red, blue] }
`)
got := make([]string, 0, len(p.Ports))
for _, port := range p.Ports {
got = append(got, port.Segment+"/"+port.Node+"/"+port.Tap)
}
if strings.Join(got, ",") != "red/sw/lt0,red/hv/lt1,red/rr/lt2,blue/hv/lt3,blue/rr/lt4" {
t.Errorf("port order = %v", got)
}
blue := portOf(t, p, "hv", "blue")
if blue.MAC.String() != "02:4c:00:01:01:00" || blue.Tap != "lt3" || blue.MTU != 1500 || blue.Interface != "blue" || blue.Address.String() != "10.2.0.1/24" {
t.Errorf("hv on blue = %+v", blue)
}
if red := portOf(t, p, "hv", "red"); red.MAC.String() != "02:4c:00:01:00:00" || red.Address.String() != "10.1.0.3/24" {
t.Errorf("hv on red = %+v", red)
}
if rr := portOf(t, p, "rr", "blue"); rr.MAC.String() != "02:4c:00:02:01:00" || rr.Address.String() != "10.2.0.2/24" {
t.Errorf("rr on blue = %+v", rr)
}
}
func TestCompute_MacEncodesNodeAndSegmentRanks(t *testing.T) {
p := compute(t, generated(3, 300))
if got := portOf(t, p, "n299", "s2").MAC.String(); got != "02:4c:01:2b:02:00" {
t.Errorf("n299 on s2 mac = %s, want 02:4c:01:2b:02:00", got)
}
}
func TestCompute_EveryMacTapAndPortIsUnique(t *testing.T) {
p := compute(t, header+`
segments:
a: { routers: [sw], cidr: 10.1.0.0/24 }
b: { routers: [sw], cidr: 10.2.0.0/24 }
nodes:
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [a, b] }
n1: { role: rr, image: deb, cpus: 1, memory: 512, segments: [a, b] }
n2: { role: rr, image: deb, cpus: 1, memory: 512, segments: [a, b] }
n3: { role: host, image: deb, cpus: 1, memory: 512, segments: [b, a] }
`)
macs := map[string]bool{}
taps := map[string]bool{}
ports := map[int]bool{}
for _, n := range p.Nodes {
if ports[n.SSHPort] {
t.Errorf("ssh port %d used twice", n.SSHPort)
}
ports[n.SSHPort] = true
}
for _, c := range p.Cables {
for _, m := range []string{c.NodeMAC.String(), c.SwitchMAC.String()} {
if macs[m] {
t.Errorf("mac %s used twice", m)
}
macs[m] = true
for _, port := range p.Ports {
if macs[port.MAC.String()] {
t.Errorf("mac %s used twice", port.MAC)
}
for _, port := range []int{c.NodePort, c.SwitchPort} {
if ports[port] {
t.Errorf("port %d used twice", port)
}
ports[port] = true
macs[port.MAC.String()] = true
if taps[port.Tap] {
t.Errorf("tap %s used twice", port.Tap)
}
taps[port.Tap] = true
}
if len(macs) != 12 {
t.Errorf("%d distinct macs, want 12", len(macs))
if len(macs) != 8 || len(taps) != 8 {
t.Errorf("%d distinct macs and %d distinct taps, want 8", len(macs), len(taps))
}
}
func TestCompute_MacsAreLocallyAdministeredUnicast(t *testing.T) {
for _, c := range compute(t, twoHypervisors).Cables {
for _, m := range []string{c.NodeMAC.String(), c.SwitchMAC.String()} {
if !strings.HasPrefix(m, "02:") {
t.Errorf("mac %s is not locally administered unicast", m)
}
for _, port := range compute(t, twoHypervisors).Ports {
if m := port.MAC.String(); !strings.HasPrefix(m, "02:") {
t.Errorf("mac %s is not locally administered unicast", m)
}
}
}
func TestCompute_InterfaceAndTapNamesFitInIFNAMSIZ(t *testing.T) {
p := compute(t, generated(23, 1000))
for _, port := range p.Ports {
if len(port.Tap) > 15 || len(port.Interface) > 15 {
t.Fatalf("port %+v has a name longer than 15", port)
}
}
for _, s := range p.Segments {
if len(s.Bridge) > 15 {
t.Fatalf("bridge %s is longer than 15", s.Bridge)
}
}
}
@ -233,9 +346,9 @@ func TestCompute_AddressRejections(t *testing.T) {
base := func(addresses string) string {
return header + `
segments:
under: { switch: sw, cidr: 10.0.0.0/29 }
under: { routers: [sw], cidr: 10.0.0.0/29 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [under] }
a: { role: rr, image: deb, cpus: 1, memory: 512, segments: [under]` + addresses + ` }
b: { role: rr, image: deb, cpus: 1, memory: 512, segments: [under], addresses: {under: 10.0.0.5} }
`
@ -244,7 +357,7 @@ nodes:
{"outside", ", addresses: {under: 10.0.1.2}", "node a: address 10.0.1.2 is outside segment under"},
{"network", ", addresses: {under: 10.0.0.0}", "node a: address 10.0.0.0 is the network or broadcast"},
{"broadcast", ", addresses: {under: 10.0.0.7}", "node a: address 10.0.0.7 is the network or broadcast"},
{"gateway", ", addresses: {under: 10.0.0.1}", "node a: address 10.0.0.1 on segment under is already taken by sw (gateway)"},
{"gateway", ", addresses: {under: 10.0.0.1}", "node a: address 10.0.0.1 on segment under is already taken by the gateway"},
{"duplicate", ", addresses: {under: 10.0.0.5}", "node b: address 10.0.0.5 on segment under is already taken by a"},
}
for _, c := range cases {
@ -254,37 +367,58 @@ nodes:
}
}
func TestCompute_RouterAddressCannotBeTheGateway(t *testing.T) {
got := computeError(t, header+`
segments:
under: { routers: [sw], cidr: 10.0.0.0/24 }
nodes:
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [under], addresses: {under: 10.0.0.1} }
`)
requireContains(t, got, "node sw: address 10.0.0.1 on segment under is already taken by the gateway")
}
func TestCompute_ReportsExhaustedSegment(t *testing.T) {
got := computeError(t, header+`
segments:
tiny: { switch: sw, cidr: 10.0.0.0/30 }
tiny: { routers: [sw], cidr: 10.0.0.0/30 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [tiny] }
a: { role: rr, image: deb, cpus: 1, memory: 512, segments: [tiny] }
b: { role: rr, image: deb, cpus: 1, memory: 512, segments: [tiny] }
`)
requireContains(t, got, "segment tiny: no address left in 10.0.0.0/30 for node b")
requireContains(t, got, "segment tiny: no address left in 10.0.0.0/30 for node a")
}
func TestCompute_FillsSegmentExactly(t *testing.T) {
p := compute(t, header+`
segments:
tiny: { switch: sw, cidr: 10.0.0.0/29 }
tiny: { routers: [sw], cidr: 10.0.0.0/29 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [tiny] }
a: { role: rr, image: deb, cpus: 1, memory: 512, segments: [tiny] }
b: { role: rr, image: deb, cpus: 1, memory: 512, segments: [tiny] }
c: { role: rr, image: deb, cpus: 1, memory: 512, segments: [tiny] }
d: { role: rr, image: deb, cpus: 1, memory: 512, segments: [tiny] }
e: { role: rr, image: deb, cpus: 1, memory: 512, segments: [tiny] }
`)
if got := cableOf(t, p, "e", "tiny").NodeAddress.String(); got != "10.0.0.6/29" {
if got := portOf(t, p, "d", "tiny").Address.String(); got != "10.0.0.6/29" {
t.Errorf("last node = %s, want 10.0.0.6/29", got)
}
}
func TestCompute_FillsSegmentWithoutRouterExactly(t *testing.T) {
p := compute(t, header+`
segments:
tiny: { cidr: 10.0.0.0/30 }
nodes:
a: { role: rr, image: deb, cpus: 1, memory: 512, segments: [tiny] }
b: { role: rr, image: deb, cpus: 1, memory: 512, segments: [tiny] }
`)
if got := portOf(t, p, "b", "tiny").Address.String(); got != "10.0.0.2/30" {
t.Errorf("last node = %s, want 10.0.0.2/30", got)
}
}
func TestCompute_RefusesInvalidTopology(t *testing.T) {
requireContains(t, computeError(t, strings.Replace(valid, "role: rr", "role: router", 1)), `role "router"`)
requireContains(t, computeError(t, strings.Replace(valid, "role: rr", "role: spine", 1)), `role "spine"`)
}
func TestLastAddr(t *testing.T) {
@ -305,41 +439,86 @@ func TestWrite_TwoHypervisorsPlan(t *testing.T) {
if err := compute(t, twoHypervisors).Write(&buf); err != nil {
t.Fatalf("Write: %v", err)
}
want := `lab lab-test: nodes 4, segments 1, cables 3
want := `lab lab-test: nodes 4, segments 1, ports 4
nodes
name role image cpus memory ssh
sw1 switch deb 2 1024 MiB 127.0.0.1:2200
sw1 router deb 2 1024 MiB 127.0.0.1:2200
rr1 rr deb 1 1024 MiB 127.0.0.1:2201
hv1 host deb 4 16384 MiB 127.0.0.1:2202
hv2 host deb 4 16384 MiB 127.0.0.1:2203
segment underlay: 10.250.0.0/24, mtu 9000, switch sw1, bridge br-underlay, gateway 10.250.0.1
node interface address mac udp switch port mac udp
rr1 underlay 10.250.0.2/24 02:4c:00:01:00:00 20000 <-> sw1 p0 02:4c:00:01:00:01 20001
hv1 underlay 10.250.0.3/24 02:4c:00:02:00:00 20002 <-> sw1 p1 02:4c:00:02:00:01 20003
hv2 underlay 10.250.0.4/24 02:4c:00:03:00:00 20004 <-> sw1 p2 02:4c:00:03:00:01 20005
segment underlay: 10.250.0.0/24, mtu 9000, bridge lb-underlay, gateway 10.250.0.1 on sw1
node interface address mac tap
sw1 underlay 10.250.0.2/24 02:4c:00:00:00:00 lt0
rr1 underlay 10.250.0.3/24 02:4c:00:01:00:00 lt1
hv1 underlay 10.250.0.4/24 02:4c:00:02:00:00 lt2
hv2 underlay 10.250.0.5/24 02:4c:00:03:00:00 lt3
`
if buf.String() != want {
t.Errorf("output:\n%s\nwant:\n%s", buf.String(), want)
}
}
func TestWrite_GatewayLines(t *testing.T) {
var buf bytes.Buffer
if err := compute(t, header+`
segments:
red: { routers: [r1], cidr: 10.1.0.0/24 }
blue: { routers: [r2, r1], cidr: 10.2.0.0/24, mtu: 1500 }
pure: { cidr: 10.3.0.0/24 }
nodes:
r1: { role: router, image: deb, cpus: 1, memory: 512, segments: [red, blue] }
r2: { role: router, image: deb, cpus: 1, memory: 512, segments: [blue] }
hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [blue, pure] }
`).Write(&buf); err != nil {
t.Fatalf("Write: %v", err)
}
for _, want := range []string{
"lab lab-test: nodes 3, segments 3, ports 5\n",
"\nsegment red: 10.1.0.0/24, mtu 9000, bridge lb-red, gateway 10.1.0.1 on r1\n",
"\nsegment blue: 10.2.0.0/24, mtu 1500, bridge lb-blue, gateway 10.2.0.1 by vrrp 2 on r2, r1\n",
"\nsegment pure: 10.3.0.0/24, mtu 9000, bridge lb-pure, no gateway\n",
" hv pure 10.3.0.1/24 02:4c:00:02:02:00 lt4\n",
} {
if !strings.Contains(buf.String(), want) {
t.Errorf("plan:\n%s\ndoes not contain %q", buf.String(), want)
}
}
}
func generated(segments, nodes int) string {
var b strings.Builder
b.WriteString(header + "segments:\n")
names := make([]string, segments)
for i := range names {
names[i] = fmt.Sprintf("s%d", i)
fmt.Fprintf(&b, " %s: { switch: sw, cidr: 10.%d.0.0/16 }\n", names[i], i%250)
fmt.Fprintf(&b, " %s: { routers: [sw], cidr: 10.%d.0.0/16 }\n", names[i], i%250)
}
b.WriteString("nodes:\n sw: { role: switch, image: deb, cpus: 1, memory: 512 }\n")
fmt.Fprintf(&b, "nodes:\n sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [%s] }\n", strings.Join(names, ", "))
for i := 1; i < nodes; i++ {
fmt.Fprintf(&b, " n%d: { role: rr, image: deb, cpus: 1, memory: 512, segments: [%s] }\n", i, strings.Join(names, ", "))
}
return b.String()
}
func withVRRPAt(segments, rank int) string {
var b strings.Builder
b.WriteString(header + "segments:\n")
names := make([]string, segments)
for i := range names {
names[i] = fmt.Sprintf("s%d", i)
routers := "[r1]"
if i == rank {
routers = "[r1, r2]"
}
fmt.Fprintf(&b, " %s: { routers: %s, cidr: 10.%d.0.0/16 }\n", names[i], routers, i%250)
}
fmt.Fprintf(&b, "nodes:\n r1: { role: router, image: deb, cpus: 1, memory: 512, segments: [%s] }\n", strings.Join(names, ", "))
fmt.Fprintf(&b, " r2: { role: router, image: deb, cpus: 1, memory: 512, segments: [%s] }\n", names[rank])
return b.String()
}
func TestValidate_RejectsTooManyNodes(t *testing.T) {
requireContains(t, validationError(t, generated(1, 1001)), "nodes: 1001 declared, at most 1000")
}
@ -354,6 +533,24 @@ func TestValidate_RejectsTooManySegments(t *testing.T) {
requireContains(t, validationError(t, generated(257, 2)), "segments: 257 declared, at most 256")
}
func TestCompute_RejectsCablesBeyondUDPPortRange(t *testing.T) {
requireContains(t, computeError(t, generated(23, 1000)), "cables: 22977 cables need udp ports up to 65953, beyond 65535")
func TestValidate_RejectsVRRPBeyondTheVRIDRange(t *testing.T) {
msg := validationError(t, withVRRPAt(256, 255))
requireContains(t, msg, "segment s255: a segment with several routers must be among the first 255")
}
func TestCompute_AcceptsVRRPOnTheLastVRID(t *testing.T) {
p := compute(t, withVRRPAt(256, 254))
if got := segmentOf(t, p, "s254").VRID; got != 255 {
t.Errorf("vrid = %d, want 255", got)
}
}
func TestCompute_ManyPortsNeedNoPortRange(t *testing.T) {
p := compute(t, generated(23, 1000))
if len(p.Ports) != 23000 {
t.Fatalf("%d ports, want 23000", len(p.Ports))
}
if last := p.Ports[len(p.Ports)-1].Tap; last != "lt22999" {
t.Errorf("last tap = %s, want lt22999", last)
}
}

View file

@ -11,9 +11,9 @@ import (
const withRoles = header + `
segments:
underlay: { switch: sw1, cidr: 192.168.14.0/24 }
underlay: { routers: [sw1], cidr: 192.168.14.0/24 }
nodes:
sw1: { role: switch, image: deb, cpus: 2, memory: 1024, secondary: { underlay: [169.254.0.1/28] }, frr: frr/sw1.conf }
sw1: { role: router, image: deb, cpus: 2, memory: 1024, segments: [underlay], secondary: { underlay: [169.254.0.1/28] }, frr: frr/sw1.conf }
rr1: { role: rr, image: deb, cpus: 1, memory: 1024, segments: [underlay], secondary: { underlay: [169.254.0.3/28] }, loopback: 10.255.255.1/32, frr: frr/rr1.conf }
hv1: { role: host, image: deb, cpus: 4, memory: 16384, segments: [underlay], frr: /etc/lab/hv1.conf, provision: provision/hv.sh, post_provision: /opt/lab/post.sh, files: { agent.yml: agent/two.yml, deploy.sh: /opt/deploy.sh }, services: [agent.service] }
`
@ -95,10 +95,10 @@ func TestValidate_RoleFieldRejections(t *testing.T) {
t.Run(name, func(t *testing.T) {
doc := header + `
segments:
red: { switch: sw, cidr: 10.1.0.0/24 }
blue: { switch: sw, cidr: 10.2.0.0/24 }
red: { routers: [sw], cidr: 10.1.0.0/24 }
blue: { routers: [sw], cidr: 10.2.0.0/24 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [red, blue] }
hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [red, blue] }
` + c.node + `
`
@ -107,29 +107,29 @@ nodes:
}
}
func TestValidate_SwitchSecondaryOnlyOnItsOwnSegments(t *testing.T) {
func TestValidate_RouterSecondaryOnlyOnItsOwnSegments(t *testing.T) {
doc := header + `
segments:
red: { switch: sw, cidr: 10.1.0.0/24 }
blue: { switch: other, cidr: 10.2.0.0/24 }
red: { routers: [sw], cidr: 10.1.0.0/24 }
blue: { routers: [other], cidr: 10.2.0.0/24 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512, secondary: { red: [169.254.0.1/28], blue: [169.254.1.1/28] } }
other: { role: switch, image: deb, cpus: 1, memory: 512 }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [red], secondary: { red: [169.254.0.1/28], blue: [169.254.1.1/28] } }
other: { role: router, image: deb, cpus: 1, memory: 512, segments: [blue] }
hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [red, blue] }
`
msg := validationError(t, doc)
requireContains(t, msg, "node sw: secondary address given for segment blue it is not attached to")
if bytes.Contains([]byte(msg), []byte("segment red")) {
t.Errorf("the switch's own segment was refused:\n%s", msg)
t.Errorf("the router's own segment was refused:\n%s", msg)
}
}
func TestValidate_LoopbackInterfaceNameIsReserved(t *testing.T) {
doc := header + `
segments:
lo1: { switch: sw, cidr: 10.1.0.0/24 }
lo1: { routers: [sw], cidr: 10.1.0.0/24 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [lo1] }
hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [lo1] }
`
requireContains(t, validationError(t, doc), "segment lo1: name is reserved for the loopback interface")
@ -156,9 +156,9 @@ func TestWrite_NoRolesSectionWithoutRoleFields(t *testing.T) {
var buf bytes.Buffer
doc := header + `
segments:
underlay: { switch: sw1, cidr: 10.250.0.0/24 }
underlay: { routers: [sw1], cidr: 10.250.0.0/24 }
nodes:
sw1: { role: switch, image: deb, cpus: 1, memory: 512 }
sw1: { role: router, image: deb, cpus: 1, memory: 512, segments: [underlay] }
rr1: { role: rr, image: deb, cpus: 1, memory: 512, segments: [underlay] }
`
if err := compute(t, doc).Write(&buf); err != nil {
@ -176,7 +176,7 @@ func TestValidate_ProvisionRejections(t *testing.T) {
}{
"the hypervisor role is gone": {
`hv: { role: hypervisor, image: deb, cpus: 1, memory: 512, segments: [red] }`,
`node hv: role "hypervisor" must be one of switch, rr, host`,
`node hv: role "hypervisor" must be one of router, rr, host`,
},
"file name leaving the files directory": {
`hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [red], files: { ../etc/passwd: a.yml } }`,
@ -203,9 +203,9 @@ func TestValidate_ProvisionRejections(t *testing.T) {
t.Run(name, func(t *testing.T) {
doc := header + `
segments:
red: { switch: sw, cidr: 10.1.0.0/24 }
red: { routers: [sw], cidr: 10.1.0.0/24 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [red] }
` + c.node + `
`
requireContains(t, validationError(t, doc), c.want)
@ -216,9 +216,9 @@ nodes:
func TestValidate_ProvisionIsAllowedOnEveryRole(t *testing.T) {
doc := header + `
segments:
red: { switch: sw, cidr: 10.1.0.0/24 }
red: { routers: [sw], cidr: 10.1.0.0/24 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512, provision: sw.sh, files: { a.conf: a.conf }, services: [lab-switch.service] }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [red], provision: sw.sh, files: { a.conf: a.conf }, services: [lab-router.service] }
rr: { role: rr, image: deb, cpus: 1, memory: 512, segments: [red], provision: rr.sh, post_provision: rr-post.sh }
hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [red], files: { agent.yml: agent.yml }, services: [agent.service] }
`

View file

@ -11,7 +11,7 @@ import (
)
const (
RoleSwitch = "switch"
RoleRouter = "router"
RoleRR = "rr"
RoleHost = "host"
@ -32,10 +32,12 @@ type Image struct {
}
type Segment struct {
Name string
Switch string
CIDR string
MTU int
Name string
Routers []string
CIDR string
MTU int
legacySwitch string
}
type Node struct {
@ -61,9 +63,10 @@ type fileImage struct {
}
type fileSegment struct {
Switch string `yaml:"switch"`
CIDR string `yaml:"cidr"`
MTU int `yaml:"mtu"`
Routers []string `yaml:"routers"`
CIDR string `yaml:"cidr"`
MTU int `yaml:"mtu"`
Switch string `yaml:"switch"`
}
type fileNode struct {
@ -145,7 +148,7 @@ func Parse(data []byte) (*Topology, error) {
if mtu == 0 {
mtu = DefaultMTU
}
t.Segments = append(t.Segments, Segment{Name: name, Switch: s.Switch, CIDR: s.CIDR, MTU: mtu})
t.Segments = append(t.Segments, Segment{Name: name, Routers: s.Routers, CIDR: s.CIDR, MTU: mtu, legacySwitch: s.Switch})
}
for _, name := range order["nodes"] {
n := f.Nodes[name]

View file

@ -40,10 +40,10 @@ func requireContains(t *testing.T, got, want string) {
func TestParse_KeepsDeclarationOrderOfNodesAndSegments(t *testing.T) {
topo := parse(t, header+`
segments:
zulu: { switch: sw, cidr: 10.0.1.0/24 }
alpha: { switch: sw, cidr: 10.0.2.0/24 }
zulu: { routers: [sw], cidr: 10.0.1.0/24 }
alpha: { routers: [sw], cidr: 10.0.2.0/24 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [zulu, alpha] }
zeta: { role: rr, image: deb, cpus: 1, memory: 512, segments: [zulu, alpha] }
beta: { role: rr, image: deb, cpus: 1, memory: 512, segments: [zulu] }
`)
@ -65,9 +65,9 @@ nodes:
func TestParse_DefaultsMTUTo9000(t *testing.T) {
topo := parse(t, header+`
segments:
under: { switch: sw, cidr: 10.0.0.0/24 }
under: { cidr: 10.0.0.0/24 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [under] }
`)
if topo.Segments[0].MTU != 9000 {
t.Errorf("MTU = %d, want 9000", topo.Segments[0].MTU)
@ -77,7 +77,7 @@ nodes:
func TestParse_RejectsUnknownField(t *testing.T) {
_, err := Parse([]byte(header + `
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512, ram: 4 }
sw: { role: router, image: deb, cpus: 1, memory: 512, ram: 4 }
`))
if err == nil || !strings.Contains(err.Error(), "ram") {
t.Fatalf("unknown field not rejected: %v", err)
@ -87,7 +87,7 @@ nodes:
func TestParse_RejectsDuplicateNode(t *testing.T) {
_, err := Parse([]byte(header + `
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
sw: { role: router, image: deb, cpus: 1, memory: 512 }
sw: { role: rr, image: deb, cpus: 1, memory: 512 }
`))
if err == nil || !strings.Contains(err.Error(), "already defined") {
@ -109,9 +109,9 @@ func TestLoad_ReportsPathOnError(t *testing.T) {
const valid = header + `
segments:
under: { switch: sw, cidr: 10.0.0.0/24 }
under: { routers: [sw], cidr: 10.0.0.0/24 }
nodes:
sw: { role: switch, image: deb, cpus: 1, memory: 512 }
sw: { role: router, image: deb, cpus: 1, memory: 512, segments: [under] }
rr: { role: rr, image: deb, cpus: 1, memory: 512, segments: [under] }
`
@ -129,9 +129,11 @@ func TestValidate_Rejections(t *testing.T) {
{"image url not https", strings.Replace(valid, "url: https://", "url: http://", 1), "image deb: url"},
{"image sums not https", strings.Replace(valid, "sums: https://", "sums: ftp://", 1), "image deb: sums"},
{"no node", header + "\nsegments: {}\nnodes: {}\n", "at least one node"},
{"segment switch missing", strings.Replace(valid, "switch: sw, cidr", "cidr", 1), "segment under: switch is required"},
{"segment switch unknown", strings.Replace(valid, "switch: sw, cidr", "switch: ghost, cidr", 1), "switch ghost is not a declared node"},
{"segment switch not a switch", strings.Replace(valid, "switch: sw, cidr", "switch: rr, cidr", 1), "rr is a rr, not a switch"},
{"legacy segment switch", strings.Replace(valid, "routers: [sw], cidr", "switch: sw, cidr", 1), "segment under: switch was replaced by routers"},
{"legacy role switch", strings.Replace(valid, "role: router", "role: switch", 1), "node sw: role switch was renamed router"},
{"segment router unknown", strings.Replace(valid, "routers: [sw]", "routers: [ghost]", 1), "segment under: router ghost is not a declared node"},
{"segment router not a router", strings.Replace(valid, "routers: [sw]", "routers: [rr]", 1), "segment under: rr is a rr, not a router"},
{"segment router listed twice", strings.Replace(valid, "routers: [sw]", "routers: [sw, sw]", 1), "segment under: router sw listed twice"},
{"segment name too long", strings.ReplaceAll(valid, "under", "underlayunder"), "segment underlayunder: name must match"},
{"segment named like the admin interface", strings.ReplaceAll(valid, "under", "mgmt0"), "segment mgmt0: name is reserved"},
{"segment name with dash", strings.ReplaceAll(valid, "under", "un-der"), "segment un-der: name must match"},
@ -143,19 +145,17 @@ func TestValidate_Rejections(t *testing.T) {
{"cidr too small", strings.Replace(valid, "10.0.0.0/24", "10.0.0.0/31", 1), "prefix length out of range"},
{"cidr too large", strings.Replace(valid, "10.0.0.0/24", "10.0.0.0/7", 1), "prefix length out of range"},
{"bad node name", strings.Replace(valid, " rr: {", " RR: {", 1), "node RR: name must match"},
{"bad role", strings.Replace(valid, "role: rr", "role: router", 1), `role "router"`},
{"bad role", strings.Replace(valid, "role: rr", "role: spine", 1), `role "spine"`},
{"unknown image", strings.Replace(valid, "role: rr, image: deb", "role: rr, image: ubuntu", 1), `image "ubuntu" is not declared`},
{"no cpu", strings.Replace(valid, "role: rr, image: deb, cpus: 1", "role: rr, image: deb, cpus: 0", 1), "node rr: cpus must be at least 1"},
{"memory too low", strings.Replace(valid, "cpus: 1, memory: 512, segments", "cpus: 1, memory: 255, segments", 1), "node rr: memory must be at least 256"},
{"switch with segments", strings.Replace(valid, "role: switch, image: deb, cpus: 1, memory: 512 }", "role: switch, image: deb, cpus: 1, memory: 512, segments: [under] }", 1), "node sw: a switch carries its segments"},
{"switch with addresses", strings.Replace(valid, "role: switch, image: deb, cpus: 1, memory: 512 }", "role: switch, image: deb, cpus: 1, memory: 512, addresses: {under: 10.0.0.9} }", 1), "node sw: a switch carries its segments"},
{"switch without segment", valid + " sw2: { role: switch, image: deb, cpus: 1, memory: 512 }\n", "node sw2: switch carries no segment"},
{"memory too low", strings.Replace(valid, "role: rr, image: deb, cpus: 1, memory: 512", "role: rr, image: deb, cpus: 1, memory: 255", 1), "node rr: memory must be at least 256"},
{"router without segment", valid + " r2: { role: router, image: deb, cpus: 1, memory: 512 }\n", "node r2: must be attached to at least one segment"},
{"node without segment", valid + " rr2: { role: rr, image: deb, cpus: 1, memory: 512 }\n", "node rr2: must be attached to at least one segment"},
{"node on unknown segment", strings.Replace(valid, "segments: [under]", "segments: [over]", 1), "node rr: segment over is not declared"},
{"node on segment twice", strings.Replace(valid, "segments: [under]", "segments: [under, under]", 1), "node rr: segment under listed twice"},
{"segment without node", strings.Replace(valid, " under: { switch: sw, cidr: 10.0.0.0/24 }", " under: { switch: sw, cidr: 10.0.0.0/24 }\n empty: { switch: sw, cidr: 10.0.9.0/24 }", 1), "segment empty: no node is attached"},
{"address on foreign segment", strings.Replace(valid, "segments: [under]", "segments: [under], addresses: {over: 10.0.0.9}", 1), "address given for segment over"},
{"address unparsable", strings.Replace(valid, "segments: [under]", "segments: [under], addresses: {under: 10.0.0}", 1), `address "10.0.0" on under`},
{"node on unknown segment", strings.Replace(valid, "role: rr, image: deb, cpus: 1, memory: 512, segments: [under]", "role: rr, image: deb, cpus: 1, memory: 512, segments: [over]", 1), "node rr: segment over is not declared"},
{"node on segment twice", strings.Replace(valid, "role: rr, image: deb, cpus: 1, memory: 512, segments: [under]", "role: rr, image: deb, cpus: 1, memory: 512, segments: [under, under]", 1), "node rr: segment under listed twice"},
{"segment without node", strings.Replace(valid, " under: { routers: [sw], cidr: 10.0.0.0/24 }", " under: { routers: [sw], cidr: 10.0.0.0/24 }\n empty: { cidr: 10.0.9.0/24 }", 1), "segment empty: no node is attached"},
{"address on foreign segment", strings.Replace(valid, "role: rr, image: deb, cpus: 1, memory: 512, segments: [under]", "role: rr, image: deb, cpus: 1, memory: 512, segments: [under], addresses: {over: 10.0.0.9}", 1), "node rr: address given for segment over"},
{"address unparsable", strings.Replace(valid, "role: rr, image: deb, cpus: 1, memory: 512, segments: [under]", "role: rr, image: deb, cpus: 1, memory: 512, segments: [under], addresses: {under: 10.0.0}", 1), `node rr: address "10.0.0" on under`},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
@ -164,10 +164,53 @@ func TestValidate_Rejections(t *testing.T) {
}
}
func TestValidate_RejectsRouterNotAttachedToItsSegment(t *testing.T) {
doc := header + `
segments:
red: { routers: [r], cidr: 10.1.0.0/24 }
blue: { routers: [r], cidr: 10.2.0.0/24 }
nodes:
r: { role: router, image: deb, cpus: 1, memory: 512, segments: [blue] }
hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [red, blue] }
`
msg := validationError(t, doc)
requireContains(t, msg, "segment red: router r is not attached to it")
if strings.Contains(msg, "segment blue") {
t.Errorf("the router's own segment was refused:\n%s", msg)
}
}
func TestValidate_AcceptsSegmentWithoutRouter(t *testing.T) {
doc := header + `
segments:
l2: { cidr: 10.3.0.0/24 }
nodes:
a: { role: host, image: deb, cpus: 1, memory: 512, segments: [l2] }
b: { role: host, image: deb, cpus: 1, memory: 512, segments: [l2] }
`
if err := parse(t, doc).Validate(); err != nil {
t.Fatalf("Validate: %v", err)
}
}
func TestValidate_AcceptsTwoRoutersOnASegment(t *testing.T) {
doc := header + `
segments:
under: { routers: [r1, r2], cidr: 10.0.0.0/24 }
nodes:
r1: { role: router, image: deb, cpus: 1, memory: 512, segments: [under] }
r2: { role: router, image: deb, cpus: 1, memory: 512, segments: [under] }
hv: { role: host, image: deb, cpus: 1, memory: 512, segments: [under] }
`
if err := parse(t, doc).Validate(); err != nil {
t.Fatalf("Validate: %v", err)
}
}
func TestValidate_ReportsEveryErrorAtOnce(t *testing.T) {
doc := strings.Replace(valid, "role: rr", "role: router", 1)
doc := strings.Replace(valid, "role: rr", "role: spine", 1)
doc = strings.Replace(doc, "10.0.0.0/24", "10.0.0.0/31", 1)
got := validationError(t, doc)
requireContains(t, got, `role "router"`)
requireContains(t, got, `role "spine"`)
requireContains(t, got, "prefix length out of range")
}

View file

@ -18,6 +18,8 @@ const (
ReservedInterface = "mgmt0"
LoopbackInterface = "lo1"
MaxVRID = 255
)
var (
@ -74,14 +76,23 @@ func (t *Topology) Validate() error {
if s.Name == LoopbackInterface {
add("segment %s: name is reserved for the loopback interface", s.Name)
}
sw, ok := nodes[s.Switch]
switch {
case s.Switch == "":
add("segment %s: switch is required", s.Name)
case !ok:
add("segment %s: switch %s is not a declared node", s.Name, s.Switch)
case sw.Role != RoleSwitch:
add("segment %s: %s is a %s, not a switch", s.Name, s.Switch, sw.Role)
if s.legacySwitch != "" {
add("segment %s: switch was replaced by routers: declare %s with role %s, attach it to %s through its segments, and list it in segments.%s.routers", s.Name, s.legacySwitch, RoleRouter, s.Name, s.Name)
}
seenRouters := map[string]bool{}
for _, name := range s.Routers {
r, ok := nodes[name]
switch {
case seenRouters[name]:
add("segment %s: router %s listed twice", s.Name, name)
case !ok:
add("segment %s: router %s is not a declared node", s.Name, name)
case r.Role != RoleRouter:
add("segment %s: %s is a %s, not a %s", s.Name, name, r.Role, RoleRouter)
case !contains(r.Segments, s.Name):
add("segment %s: router %s is not attached to it", s.Name, name)
}
seenRouters[name] = true
}
if s.MTU < MinMTU || s.MTU > MaxMTU {
add("segment %s: mtu %d out of range [%d, %d]", s.Name, s.MTU, MinMTU, MaxMTU)
@ -99,19 +110,25 @@ func (t *Topology) Validate() error {
}
}
attached := map[string]int{}
hosting := map[string]int{}
for _, s := range t.Segments {
hosting[s.Switch]++
for i, s := range t.Segments {
if len(s.Routers) > 1 {
if i+1 > MaxVRID {
add("segment %s: a segment with several routers must be among the first %d, its rank sets the vrrp id", s.Name, MaxVRID)
}
}
}
attached := map[string]int{}
for _, n := range t.Nodes {
if !namePattern.MatchString(n.Name) {
add("node %s: name must match %s", n.Name, namePattern)
}
switch n.Role {
case RoleSwitch, RoleRR, RoleHost:
case RoleRouter, RoleRR, RoleHost:
case "switch":
add("node %s: role switch was renamed %s: attach it to its segments through segments and addresses, and list it in segments.<name>.routers", n.Name, RoleRouter)
default:
add("node %s: role %q must be one of %s, %s, %s", n.Name, n.Role, RoleSwitch, RoleRR, RoleHost)
add("node %s: role %q must be one of %s, %s, %s", n.Name, n.Role, RoleRouter, RoleRR, RoleHost)
}
if !images[n.Image] {
add("node %s: image %q is not declared", n.Name, n.Image)
@ -123,15 +140,6 @@ func (t *Topology) Validate() error {
add("node %s: memory must be at least %d MiB", n.Name, MinMemory)
}
validateExtras(n, segments, add)
if n.Role == RoleSwitch {
if len(n.Segments) > 0 || len(n.Addresses) > 0 {
add("node %s: a switch carries its segments through segments.<name>.switch, not through segments or addresses", n.Name)
}
if hosting[n.Name] == 0 {
add("node %s: switch carries no segment", n.Name)
}
continue
}
if len(n.Segments) == 0 {
add("node %s: must be attached to at least one segment", n.Name)
}
@ -174,16 +182,8 @@ func (t *Topology) Validate() error {
func validateExtras(n Node, segments map[string]Segment, add func(string, ...any)) {
carried := map[string]bool{}
if n.Role == RoleSwitch {
for name, s := range segments {
if s.Switch == n.Name {
carried[name] = true
}
}
} else {
for _, name := range n.Segments {
carried[name] = true
}
for _, name := range n.Segments {
carried[name] = true
}
names := make([]string, 0, len(n.Secondary))
for name := range n.Secondary {

14
release_notes/0.2.0.md Normal file
View file

@ -0,0 +1,14 @@
# Segments sur des bridges de l'hôte, routeurs et passerelle VRRP
**Changement incompatible** : une topologie 0.1 doit être migrée (voir « Migration depuis 0.1 »
dans `docs/guide.md`).
- Chaque segment est un bridge Linux de l'hôte (`lb-<segment>`), chaque interface de VM un tap
(`lt<i>`) ; plus de sockets UDP. `labctl up` et `labctl down` exigent root, `down` retire les
bridges et les taps, `status` affiche les segments.
- Le rôle `switch` devient `router`, et `segments.<s>.switch` devient `segments.<s>.routers`,
une liste ordonnée. Un segment sans routeur est un L2 pur, sans passerelle.
- Un routeur fait du forwarding et masque vers `mgmt0` les segments qu'il route (service
`lab-router`). Avec deux routeurs ou plus, la passerelle est portée en VRRP v3 par `vrrpd` de
FRR, sans préemption ; bascule d'environ 2,6 s à l'arrêt brutal du maître.
- Exemple `examples/evpn-2hv` : deux routeurs `r1` et `r2` en VRRP.