Labs / Automation / Network as code

Lab 3 of 5 ยท Network as code

Python for networks: one script, many routers

Network automation in Python: one script drives four routers over SSH.

Start this lab 6 steps

The idea

Ines runs the network of Atlas Tea, with a Router in each of four branches. Tonight every branch needs a route to the new backup vault. Write it once in Python, run it against all four, and handle the router that doesn't answer.

The script starts the same ssh you typed
The script starts the same ssh you typed
A longer list, the same loop
A longer list, the same loop
A path of names down to the next hop
A path of names down to the next hop
One line to all four, two changes
One line to all four, two changes
An uncaught error ends the whole run
An uncaught error ends the whole run
Four waits, end to end or all at once
Four waits, end to end or all at once

Steps

  1. One router, by hand and from Python

    Atlas Tea imports tea and sells it from four branch shops: Oslo, Kyiv, Lima and Pune. Each branch has a router, and Ines looks after all four from her machine, ops.

    A new backup vault opened at head office on 10.0.50.0/24, and each branch router needs a route to it. Ines added it on oslo and lima by hand. Two routers to go, and Atlas Tea is opening a fifth branch next month.

    Each router lets ops log in over SSH with a key, no password typed, and a login lands in the router's own command line, not a Linux shell. So ssh netops@oslo 'show ip route 10.0.50.0/24' runs that one router command and comes back.

    Python can run the same command. Here's /root/check.py:

    for router in routers:
        answer = subprocess.run(
            ["ssh", "-o", "BatchMode=yes", f"netops@{router}", "show ip route 10.0.50.0/24"],
            capture_output=True, text=True, timeout=20,
        )
        print(f"== {router}")
        print(answer.stdout)
    

    subprocess.run starts a program from a list: ssh, then each word you'd type after it, with the quoted router command as one item. f"netops@{router}" is an f-string: Python puts the value of router where the braces are.

    BatchMode=yes tells ssh never to wait for a password, since a script has nobody to type one. capture_output=True keeps what ssh prints, and text=True hands it back as a string, answer.stdout.

    routers comes from /root/routers.txt, which lists only oslo for now.

    ops:~# ssh netops@oslo 'show ip route 10.0.50.0/24'
    ops:~# python3 /root/check.py
  2. The inventory: every router on the list

    check.py doesn't name any router. It reads them from /root/routers.txt, one per line:

    routers = open("/root/routers.txt").read().split()
    

    .read() gives the whole file as one string, and .split() cuts it at every space and line break into a list: ["oslo"]. Then for router in routers: runs the indented lines once for each name.

    A list of the devices a script works on is called an inventory. Keeping it apart from the code means adding the fifth branch next month is one new line in a text file, and no script changes. Every automation tool works this way, Ansible included.

    Add the other three routers to the inventory: kyiv, lima and pune, one per line. Saving runs check.py again.

  3. JSON instead of screen text

    To find the next hop in * 10.0.1.2, via mgmt, weight 1, a script would have to cut the line apart: skip the star, stop at the comma. That breaks the day a new router version adds a word. Text like this is made for people.

    Most router commands can answer in JSON instead, with every value under a name. Add json to the end of the command and look. Spread out, with most fields left out, the answer is shaped like this:

    { "10.0.50.0/24": [
        { "protocol": "static",
          "nexthops": [ { "ip": "10.0.1.2", "interfaceName": "mgmt" } ] }
    ] }
    

    The network holds a list of routes, since one network can have several, and each route a list of next-hop addresses. /root/audit.py reads it, with WANTED set to "10.0.50.0/24":

    table = json.loads(answer.stdout)
    if WANTED in table:
        hop = table[WANTED][0]["nexthops"][0]["ip"]
        print(f"{router}: {WANTED} via {hop}")
    

    json.loads turns JSON text into Python: objects become dictionaries and lists become lists. WANTED in table asks whether the dictionary has that name, and [0] picks a list's first item. So the path reads: the network, its first route, that route's first next hop, its "ip".

    Then run audit.py for all four routers.

    ops:~# ssh netops@oslo 'show ip route 10.0.50.0/24 json'
    ops:~# python3 /root/audit.py
  4. Push the route to every router

    Time to fix kyiv and pune. Adding a route by hand takes three router commands: configure terminal opens config mode, where changes are typed, then the route, then end to leave. /root/push.py sends those three to every router on the inventory:

    ROUTE = "ip route 10.0.50.0/24 10.0.1.2"
    
    for router in routers:
        commands = f"configure terminal\n{ROUTE}\nend"
        subprocess.run(
            ["ssh", "-o", "BatchMode=yes", f"netops@{router}", commands],
            check=True, timeout=20,
        )
        print(f"{router}: sent {ROUTE}")
    

    \n is a line break, so the three commands arrive one per line, and this router's command line runs them in turn, as if typed. Many vendors' routers take only one command per SSH request, which is why libraries like Netmiko keep a session open and type into it. check=True makes Python stop with an error if ssh reports a failure, so a refused command can't pass as done.

    It sends the route to all four routers, oslo and lima included, which have it already. Run it, then run audit.py again.

    ops:~# python3 /root/push.py
    ops:~# python3 /root/audit.py
  5. When a router doesn't answer

    Networks fail in the middle of things. Kyiv's line to head office has gone down, and kyiv can't be reached.

    Run audit.py again and watch what one silent router does to the others. The inventory lists oslo, kyiv, lima, pune, in that order.

    Then make audit.py carry on past a router that doesn't answer. Python's try and except catch an error so the program can deal with it instead of stopping:

        try:
            answer = subprocess.run(
                ["ssh", "-o", "BatchMode=yes", f"netops@{router}", f"show ip route {WANTED} json"],
                capture_output=True, text=True, timeout=20, check=True,
            )
        except subprocess.CalledProcessError as failure:
            print(f"{router}: no answer: {failure.stderr.strip()}")
            continue
    

    CalledProcessError is the error check=True raises when ssh fails. failure.stderr holds what ssh said, and continue jumps to the next router. Put the subprocess.run(...) call inside a try like this one.

    ops:~# python3 /root/audit.py
  6. Ask all four at once

    Kyiv's line is back. Run /root/fast.py: the same audit, timed. It prints how long the four routers took.

    Each SSH login spends most of its time waiting: for the network to carry each message to a branch far away and back, and for the router to answer. One router at a time, the waits add up.

    fast.py uses a pool of threads. A thread is one line of work inside a program, and several threads can wait at the same time:

    WORKERS = 1
    
    with ThreadPoolExecutor(max_workers=WORKERS) as pool:
        for line in pool.map(ask, routers):
            print(line)
    

    ask(router) asks one router and returns one line. pool.map(ask, routers) hands each router to ask on one of the pool's threads, and gives back the results in inventory order. With max_workers=1 there's one thread, so it's the plain loop again.

    Change WORKERS to 4, so each router gets a thread of its own. Saving runs fast.py again.

    ops:~# python3 /root/fast.py

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab