Speed up Ansible with Mitogen!

→ Russian version

Ansible is one of most popular Configuration Management Systems nowadays. After it was acquired by Red Hat in 2015 Ansible has reached numbers of thousands of contributors and became maybe one of most used deployment and orchestration tools. Its use-cases are quite impressive.

Ansible works by SSH connections to remote hosts. It opens SSH session, logs in to the shell, copy python code via network and create a temporary file on remote hosts with this code. In the next step, it executes the current file with python interpreter. All this workflow is pretty heavy and there are multiple ways to make it faster and lighter.

One of these ways is using SSH pipelines which reuses one SSH session for copying python code of multiple tasks and prevent opening multiple sessions, which saves a lot of time. (Just don’t forget to disable requiretty settings for sudo on the remote side in /etc/sudoers)

The new way to speed up Ansible is a great python library called Mitogen. If somebody like me was not familiar with it — this library allows fast execution of python code on a remote host and Ansible is only one of its cases. Mitogen uses UNIX pipes on remote machines while passing "pickled" python code compressed with zlib. This allows to run it fast and without much traffic. If you're interested you can read details about how it works in its "How it works" page. But we'll focus today on Ansible related part of it.

Mitogen in specific circumstances can speed up your Ansible in a few times and significantly lower your bandwidth. Let's check the most popular use cases and figure out if it's helpful for us.

The most popular use cases for me to run Ansible are: creating configuration files on a remote host, packages installation, downloading and uploading files from and to a remote host. Maybe you want to check other use cases, please leave comments to this article.

Let's start rolling!

Configuring Mitogen for Ansible is pretty simple:
Install the Mitogen module:

pip install mitogen

Then either configure environment variables or set configuration options in ansible.cfg file, both options are fine:
Let's assume /usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy is your path to installed Mitogen library.

export ANSIBLE_STRATEGY_PLUGINS=/usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy
export ANSIBLE_STRATEGY=mitogen_linear

or

[defaults]
strategy = mitogen_linear
strategy_plugins = /usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy

Prepare Ansible in virtualenv, with and without Mitogen enabled:

virtualenv mitogen_ansible
./mitogen_ansible/bin/pip install ansible==2.7.10 mitogen
virtualenv pure_ansible
./pure_ansible/bin/pip install ansible==2.7.10

Please pay attention that Mitogen 0.2.7 doesn't work with Ansible 2.8 (for May 2019)

Create aliases:

alias pure-ansible-playbook='$(pwd)/pure_ansible/bin/ansible-playbook'
alias mitogen-ansible-playbook='ANSIBLE_STRATEGY_PLUGINS=/usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy:$(pwd)/mitogen_ansible/lib/python3.7/site-packages/ansible_mitogen/plugins/strategy ANSIBLE_STRATEGY=mitogen_linear $(pwd)/mitogen_ansible/bin/ansible-playbook'

Now let's try the playbook that creates file on remote:

---
- hosts: all
  gather_facts: false
  tasks:
    - name: Create files with copy content module
      copy:
        content: |
          test file {{ item }}
        dest: ~/file_{{item}}
      with_sequence: start=1 end={{ n }}

And run it with Mitogen and without while creating 10 files:

time mitogen-ansible-playbook file_creation.yml -i hosts -e n=10 &>/dev/null

real    0m2.603s
user    0m1.152s
sys     0m0.096s

time pure-ansible-playbook file_creation.yml -i hosts -e n=10 &>/dev/null

real    0m5.908s
user    0m1.745s
sys     0m0.643s

Right now we see improvement in x2 times. Let's check it for 20, 30, ..., 100 files:

time pure-ansible-playbook file_creation.yml -i hosts -e n=100 &>/dev/null

real    0m51.775s
user    0m8.039s
sys     0m6.305s

time mitogen-ansible-playbook file_creation.yml -i hosts -e n=100 &>/dev/null

real    0m4.331s
user    0m1.903s
sys     0m0.197s

Eventually, we improved execution time in more than 10 times!

Now let's try different scenarios and see how it improves:

Scenario of uploading files from the local host to remote (with copy module):
Scenario of creating files on the remote host with copy module:
Scenario with fetching files from the remote host to local:

Let's try the last scenario on a few (3) remote hosts, for example uploading files scenario:
Uploading files to multiple hosts

As we can see the Mitogen saves us both time and bandwidth in these scenarios. But if the bottleneck is not Ansible, but for example I/O of disk or network, or somewhere else, then it's hard to expect from Mitogen to help of course.

Let's run for example packages installation with yum/dnf and python modules installation with pip.
Packages were pre-cached to avoid dependencies on network glitches:

---
- hosts: all
  gather_facts: false
  tasks:
    - name: Install packages
      become: true
      package:
        name:
          - samba
          - httpd
          - nano
          - ruby
        state: present

    - name: Install pip modules
      become: true
      pip:
        name:
          - pytest-split-tests
          - bottle
          - pep8
          - flask
        state: present

With Mitogen it takes 12 seconds, as well as with pure Ansible.

In Mitogen for Ansible page you can see additional benchmarks and measurements. As the page declares:

Mitogen cannot improve a module once it is executing, it can only ensure the module executes as quickly as possible

That's why it's important to find where your bottlenecks are and if they are related to Ansible operations, Mitogen will help you to solve it and speed your playbooks up significantly.