Peterlog

Occasionally I have to do scripting work outside of our config management tool of choice, Juju ¹ - eg. for bootstrapping or one-off jobs. I had used Fabric version 1 previously (as well as Plumbum) for those, and was looking at Fabric 2 (respectively it's sidekick Invoke) now.

So whats Fabric and Invoke

Invoke is a high-level lib for running shell commands. It also offers simple command-line parsing functionality. Fabric offers remote command execution via ssh and builds on Invoke.

Their predecessor, Fabric v1, used to do both local and remote task execution. Invoke was split off as the ssh interfacing functionality carries some heavy dependencies (Paramiko and the underlying crypto libs), placing an unnecessary burden on users who only need local task exec.

Invoke Tasks

Installation

Install it into a virtual env via pip: pip install invoke. Or via your package manager, e.g. on Ubuntu with sudo apt install python3-invoke

Hello world

Invoke tasks look something like the below.

from invoke import task

@task
def hello(ctxt):
    print("Hi there!")

Tasks are functions that are decorated with the task decorator. They take at least one arg, the context (more on that below). Typically they're put in a file tasks.py.

If put in a tasks.py they can be run with the invoke program, for instance:

$ invoke hello
Hi there!

Note you don't have to use the invoke program - tasks can be called from your regular Python code just as well.

Running shell commands

The context comes into play when running shell code. For example, calling the hostname program:

from invoke import task

@task
def host(ctxt):
    ctxt.run("hostname")

The above task will run the "hostname" program via the contexts' run() method. The context object serves as an interface to your system and also holds configuration information.

Results object and hiding stdout/stderr

The results of ctxt.run() are passed back in a results object. It contains stdout, stderr and the exit code of the command passed in.

By default, run() will also copy stdout and stderr from the subordinate shell to the controlling terminal. For instance, the below task will echo 3 lines.

from invoke import task

@task
def echo3(ctxt):
    ctxt.run("for i in {1..3}; do echo line $i ; done")

If you're going to process the output it's often useful to prevent copying stdout/stderr. This can be done with the hide flag to run(), for instance:

from invoke import task

@task
def mountdev(ctxt):
    res = ctxt.run("mount | grep /dev/", hide='both')
    print("Exit code:", res.exited)
    print("Stdout:", res.stdout)

Full docs for the results object are here

Warn instead of abort

If a shell command exits with a non-zero exit code, Invoke by default will bail out with an UnexpectedExit exception. This is a safe default for many jobs, but sometimes a non-zero exit is expected or can safely be ignored. The run() method takes a warn kwarg (default False) to control this

Echo

For debugging it can be useful to have the shell command to be run echoed back to you; do this with run(echo=True)

Updateing the env

The run() method also takes an env kwarg. These values are passed in in addition to the inherited environment. Eg., setting python path:

from invoke import task

@task
def mypath(ctxt):
    ctxt.run('someprogram', env={'PYTHONPATH': 'my/libs/'})

More run params

Some more options for the run() method are documented in the Runners.run API docs for replacing env, pty control, input and output stream manipulation.

Sudo

There's also some syntactic sugar for running shell commands via sudo via the sudo() method exposed in the context object. All args of the run() method are applicable here as well. In addition you can pass in args user and password. The latter can (and probably should) be passed in via configuration as well - more on the configuration system below

Note that for entering passwords by hand the sudo() wrapper should be avoided, and run("sudo somecommand") be used instead.

More on running invoke

Params

The invoke program supports passing in args from the command line. Arguments to task functions become commandline args. By specifying keyword args it's also possible to initiate type casting. Type info is also reflected in the help output (more below on help).

First a task that takes an arg 'name', secondly a kwarg 'num' with a default value of 0. The latter is interpreted as a type hint.

from invoke import task

@task
def heya(ctxt, name):
    print("Heya,", name)

@task
def integertask(ctxt, num=0):
    print("Type is", type(num))

Command line invocation of tasks:

$ invoke heya --name Peter      
Heya, Peter

$ invoke integertask --num 1
Type is <class 'int'>

Listing tasks and help

When running invoke with the --list flag it will output a list of defined tasks:

$ invoke --list
Available tasks:

    hello

Tasks can also be annotated with documentation, which will be used to output help/usage info. For example, given a task definition:

@task(help={'num': "Number to operate on "})
def mult3(ctxt, num=0):
    """Multiply a number by 3
    """
    return num * 3

Then, requesting help would display something like the below:

$ invoke --help mult3
Usage: inv[oke] [--core-opts] mult3 [--options] [other tasks here ...]

Docstring:
  Multiply a number by 3


Options:
  -n INT, --num=INT   Number to operate on

Note that 'num' can be given either with -n or --num. Also note that since we gave 'num' an integer default value help prints the appropriate type

Configuration system

Invoke has a nifty configuration system which can be used to both steer invokes behaviour, but is also available for task configuration. Sources for configuration values are Python code, environment variables, system-wide config files (ie. /etc/invoke.yaml), per-user configuration files (ie. ~/.invoke.yaml), or per-project config files (ie. invoke.yaml alongside a tasks.py), or passed in explicitly as invoke -f myconf.yaml.

The resulting configuration is put in dict-like object (possibly nested).

Basic example

Let's say your ~/.invoke.yaml has entries:

foo:
  bar: 123
  quux:
    - a
    - b
    - c

Then your context objects will have a ctxt.foo dictionary-like object:

@task
def footask(ctxt):
    print(ctxt.foo)

$ invoke footask  
<DataProxy: {'bar': 123, 'quux': ['a', 'b', 'c']}>

Note, config files can be given in json and yaml format. For details on this and the lookup rules and formats refer to the docs.

Changing task file name

By default, invoke will load tasks from a tasks module, for instance a file tasks.py in the current directory. If you'd rather use another file name, pass in the --collection <module> arg. Eg., given a file footasks.py.

invoke --collection footasks ...

If this file is not in the current directory, pass in a directory arg with --search-root /some/dir

Modules given with the --collection arg are first searched for on sys.path, then the current working dir, and from there upwards in the file system. It's also possible to give the --collection= arg multiple times - then all of the modules are searched for.

Remote execution

The Fabric library and executable are used for remote execution of tasks, just like the invoke lib are used for local tasks.

Installation

Install into a venv via pip: pip install fabric, or via your package manager, e.g. on Ubuntu: sudo apt install python3-fabric

Connecting and running

Fabric uses invoke to execute commands remotely via ssh. One of the principal interfaces for this is the Connection object. For instance, to run a command on a remote host 'myhost':

from fabric import Connection
con = Connection('myhost')
result = con.run('uname -a')

The con.run() method works much like Invokes context objects, and in fact Connection inherits from the Context class.

Authentication is done much like a regular ssh command line invocation, public key auth and ssh agents are supported.

Connections also can be opened with explicit user and port: Connection(user='peter', host='myhost', port=22).run('uname')

Configuration

Fabrics configuration works similarly as Invokes' does, only the default config files are named fabric instead of invoke (eg. fabric.yaml instead of invoke.yaml).

Sudo

Similarly to the sudo method of invoke described above, fabric offers a sudo() method to conveniently run privileged commands. For non-passwordless sudo, it's possible to set passwords in configuration files (e.g. fabric.yaml):

sudo:
  password: sikkritpass

Alternatively, you can have fab prompt you for sudo passwords by giving the --prompt-for-sudo-password flag upon invocation.

Multiple servers

Server groups

Fabric has a notion of server groups that allows to run the same operation on several remote machines. This comes in two flavors: SerialGroup objects serialize access to every server, while the ThreadingGroup executes in parallel but otherwise has the same API.

In the example below a function upload_and_unpack() is executed on a group of servers. The function will get a connection object for each server. Also note the c.put() method of transferring files, see below for more on file transfer.

from fabric import SerialGroup as Group
# Use for parallel execution: from fabric import ThreadingGroup as Group

def upload_and_unpack(c):
    if c.run('test -f /opt/mydata/myfile', warn=True).failed:
        c.put('myfiles.tgz', '/opt/mydata')
        c.run('tar -C /opt/mydata -xzvf /opt/mydata/myfiles.tgz')

for connection in Group('web1', 'web2', 'web3'):
    upload_and_unpack(connection)

Connection configuration

For configuring connection SSH parameters, Connections take a connect_kwargs argument which is passed to Paramiko. Parameters therefore are Paramikos, e.g. configuring a specific key is done with:

con = Connection(
    'myhost',
    connect_kwargs={
        "key_filename": os.path.expanduser("~/.ssh/id_rsa_special")
        },)

Gateways

Many installations are not directly reachable via SSH but can only be reached via a bastion or gateway host. Fabric supports this quite neatly via the gateway parameter:

con = Connection('myhost.internal', gateway=Connection('bastion.example.com'))

The above will bounce the connection to myhost.internal via the bastion.example.com connection. The connection can be configured just like any other with custom ssh parameters.

File transfer

Often, you'll want to transfer files to and from remote servers for remote execution, be it data files or scripts you want to run remotely. Fabric provides the .put() and .get() methods on connections for this. For example, upload a Python script and run it:

from fabric import Connection
con = Connection('myhost')
con.put('myscript.py', remote='/home/ubuntu')
con.run('python3 /home/ubuntu/myscript.py')

References

Footnotes

We're using Juju for our configuration and application modelling↩︎

sabaini gmbh

Fabric Automation

So whats Fabric and Invoke

Invoke Tasks

Installation

Hello world

Running shell commands

Results object and hiding stdout/stderr

Warn instead of abort

Echo

Updateing the env

More run params

Sudo

More on running invoke

Params

Listing tasks and help

Configuration system

Basic example

Changing task file name

Remote execution

Installation

Connecting and running

Configuration

Sudo

Multiple servers

Server groups

Connection configuration

Gateways

File transfer

References

Footnotes