Labs / Automation / Network as code

Lab 1 of 5 · Network as code

Config backups: every router's config in git

Back up router configs into git, find a bad change, restore it.

Start this lab 5 steps

The idea

Ruth looks after three library routers. Keep every router's config in git each night, use a diff to find the two lines that broke a branch over the weekend, and learn why pasting the old config back doesn't fix it.

The config is text, and only what somebody set
The config is text, and only what somebody set
Two runs, one commit
Two runs, one commit
The diff prints only what differs
The diff prints only what differs
A merge adds lines and removes none
A merge adds lines and removes none
Replace removes the line only the router has
Replace removes the line only the router has

Steps

  1. A router's config is a page of text

    Fernhill Libraries has three branches, Elm Street, Oak Lane and Ash Road, with a router each: elm, oak and ash. The catalogue server sits at Elm Street, and the self-checkout kiosk at Oak Lane looks books up on it through oak and elm.

    Last spring a router died, and Ruth set up the spare from memory. It took an afternoon, and she missed a route. Next time she wants a copy of each router's setup, kept off the router.

    What would that copy hold? These routers run FRR, routing software for Linux with a Cisco-style command line.

    On them, as on Cisco's, every setting is written as the command that sets it, and the router keeps the list it runs on: its running config. show running-config prints it.

    Ruth's machine, archive, logs in to each router over SSH with a key, and a login lands in the router's own command line. So ssh netops@elm 'show running-config' runs that one command on elm and prints the answer on archive.

    archive:~# ssh netops@elm 'show running-config'
  2. Every router, into git

    One copy is a start. Ruth wants a fresh copy of all three every night, and a way to see what changed between nights. Files in a folder hold the copies, and git remembers every version of them.

    /backups on archive is an empty git repository: a folder whose history git keeps. /root/backup.sh fills it:

    for router in elm oak ash; do
      ssh -o BatchMode=yes netops@$router 'show running-config' \
        | grep -v -e '^Building configuration' -e '^Current configuration' -e '^$' > $router.cfg
    done
    
    git add *.cfg
    if git diff --cached --quiet; then
      echo "No router changed since the last backup: nothing new to save."
    else
      git commit -q -m "Backup $(date '+%a %d %b %H:%M')"
      git show --stat --format='Saved %h: %s' HEAD
    fi
    

    The loop writes elm.cfg, oak.cfg and ash.cfg, minus the two header lines and the blank ones. -o BatchMode=yes makes ssh fail instead of waiting at a password prompt nobody will answer. git add picks the files to keep, git diff --cached --quiet asks whether any of them differs from the last saved version, and git commit saves them together as one version, a commit, with a short id.

    Run it, then run it again straight away.

    archive:~# /root/backup.sh
    archive:~# /root/backup.sh
  3. What changed over the weekend?

    Monday morning, Oak Lane calls: the kiosk can't look up any books. Nobody admits to changing anything, but Leo, a contractor, tidied up the routers on Saturday. Say the backup you took a moment ago was Friday night's.

    Check the kiosk first. Then take a fresh backup and ask git what changed between the last two:

    git -C /backups diff HEAD~1 HEAD
    

    -C /backups runs git in that repository. HEAD is the newest commit, the backup you're about to take, and HEAD~1 is the one before it, Friday's.

    git diff prints the lines that differ: - in front of a line the older version had, + in front of one the newer version has, and a few unchanged lines around each change. The diff --git, ---, +++ and @@ lines at the top only say which file, and where in it.

    kiosk:~# curl -sS -m 3 http://10.0.11.20/
    archive:~# /root/backup.sh
    archive:~# git -C /backups diff HEAD~1 HEAD
  4. Put the old lines back

    The obvious fix: give oak its Friday config again. /root/restore.sh does that. restore.sh oak HEAD~1 takes oak.cfg as it was in commit HEAD~1 and types its lines into oak's configuration mode (configure terminal, where typed lines change the setup) over SSH:

    MODE=merge
    ...
    wanted=$(git show "$commit:$router.cfg" | settings) || exit 1
    running=$(ssh -o BatchMode=yes netops@$router 'show running-config' | settings)
    
    send=$wanted
    if [ "$MODE" = replace ]; then
      ...
    fi
    

    git show HEAD~1:oak.cfg prints a file as it was in that commit, and settings keeps the lines that set something, sorted. In MODE=merge, the script sends every one of those lines, as if you'd pasted them in. A line the router already has changes nothing, and a missing one comes back.

    archive:~# /root/restore.sh oak HEAD~1
    kiosk:~# curl -sS -m 3 http://10.0.11.20/
  5. Replace: make the router match the backup

    A replace makes the router's config exactly the backup: lines the backup doesn't have come out, and missing lines go in. Cisco calls it configure replace, Juniper load override.

    restore.sh has a replace mode:

    if [ "$MODE" = replace ]; then
      # Each line only the router has, turned around: 'X' becomes 'no X', and 'no X' becomes 'X'.
      extra=$(comm -23 <(echo "$running") <(echo "$wanted") | sed -e 's/^no //;t' -e 's/^/no /')
      # Removals go first, so a setting the backup sets again (a new hostname) isn't cleared after it.
      [ -n "$extra" ] && send="$extra
    $send"
    fi
    

    comm -23 compares two sorted lists and keeps the lines found only in the first: what oak runs that the backup lacks. On these routers, as on Cisco's, no in front of a setting takes it away, so the sed puts no in front of each extra line, or takes it off a line that already starts with one.

    Change MODE=merge to MODE=replace. Saving runs restore.sh oak HEAD~1 again.

    kiosk:~# curl -sS -m 3 http://10.0.11.20/
    archive:~# /root/restore.sh oak HEAD~1

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab