Run ansible-playbook commands directly on Ansible Tower server

Why?

You might want to run Ansible commands directly from the Ansible Tower server itself to debug a specific issue. While it’s generally not recommended to have manual projects that are not at all tied to a version control system like Git, there are rare moments you might need it. I certainly did!

I hit an issue with a specific AWS Ansible module and nothing seemed to get it working. From my laptop directly to AWS, the module worked great. So something was clearly different.

You might think… “Don’t I just run ansible-playbook from the server and that’s it?”. Yes, this works for the basic modules that don’t require special Python modules. For those that do, such as AWS modules require the boto3 Python module, you will need a little magic.

So let’s get into it!

The Problem

As previously stated, running ansible-playbook command should work but sadly it doesn’t include the extra Python modules necessary for Ansible modules like those for AWS where it needs the boto3 Python module.

Let’s login directly to our Ansible Tower server and go to the default projects folder where we can create a folder for our playbook.

ssh root@ansible-tower.example.com
cd /var/lib/awx/projects

As an example, let’s use this simple playbook that gets all AWS regions. You can of course use an existing playbook that you created or want to test.

---
- name: Cleanup all AWS instances, snapshots, etc.
  hosts: localhost
  connection: local
  gather_facts: false

  tasks:
    - aws_region_facts:

And notice the error we get when we run it - boto3 is missing and required.

# ansible-playbook -i inventory/inventory_local aws.yml

PLAY [Cleanup all AWS instances, snapshots, etc.] ******************************************

TASK [aws_region_facts] ********************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "boto3 required for this module"}
to retry, use: --limit @/var/lib/awx/projects/test/aws.retry

PLAY RECAP *********************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1

Let’s check which Python interpreter ansible-playbook is using.

$ cat /usr/bin/ansible-playbook | grep python
#!/usr/bin/python2

So let’s now see if python2 has the boto3 module. We run the same Python interpreter and check by importing the boto3 module.

# python2
Python 2.7.5 (default, Feb 20 2018, 09:19:12)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named boto3

However, when we run this same playbook within Ansible Tower it succeeds without any errors. What is Ansible Tower doing that is different?

Do not install pip and install the boto3 module yourself! It is not a good idea to make changes like this to the Ansible Tower server. Read below for the right solution.

Let’s keep going…

Let’s search for the boto3 module on the server.

$ cd /; find . -name "boto3"
./var/lib/awx/venv/awx/lib/python2.7/site-packages/boto3
./var/lib/awx/venv/ansible/lib/python2.7/site-packages/boto3

So now we have found the location where Ansible Tower holds these extra Python modules. Now what?

Well, we could tell Python where to find these modules by using the environment variable PYTHONPATH and setting it to both of the site-packages paths as shown below, but this is not the best solution!

export PYTHONPATH=/var/lib/awx/venv/awx/lib/python2.7/site-packages/;/var/lib/awx/venv/ansible/lib/python2.7/site-packages/

The Solution

So how does it work within Ansible Tower?

Ansible Tower 3.0 and later uses virtualenv. Virtualenv creates isolated Python environments to avoid problems caused by conflicting dependencies and differing versions.

Virtualenv works by simply creating a folder which contains all of the necessary executables and dependencies for a specific version of Python. (In our case, the boto3 module!)

Ansible Tower creates two virtualenvs during installation – one is used to run Tower, while the other is used to run Ansible. This allows Tower to run in a stable environment, while allowing you to add or update modules to your Ansible Python environment as necessary to run your playbooks. For more information on virtualenv, see the Python Guide to Virtual Environments.

So how do we access this virtualenv?

On the Ansible Tower server, run the following command as root:

. /var/lib/awx/venv/ansible/bin/activate

You will then get a new prompt looking similar to this:

(ansible) [root@ansible-tower.example.com project] $

Now when we run the ansible-playbook command the Python interpreter in the virtual environment will be invoked with all the necessary modules and dependencies that come included with Ansible Tower.

# ansible-playbook -i inventory/inventory_local aws.yml

PLAY [Cleanup all AWS instances, snapshots, etc.] ******************************************

TASK [aws_region_facts] ********************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "The aws_region_facts module requires a region and none was found in configuration, environment variables or module parameters"}
to retry, use: --limit @/var/lib/awx/projects/test/aws.retry

PLAY RECAP *********************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1

This is exactly the error I saw in Ansible Tower and now I have finally reproduced the problem I originally saw in Ansible Tower. Now I can properly debug it and figure out what is wrong. In fact, this was how I eventually logged the following Ansible issue.

I hope this has helped you get to know Ansible Tower a bit better.

Advanced

As a bonus, you can additionally use this virtual environment to properly add new Python modules that you might need that did not come included with Ansible Tower. To do so, start in the virtual environment and then run the pip command to install the required module.

(ansible) [root@ansible-tower.example.com project] $ pip install mypackagename

Be careful! Note the modules you install because you will need to make sure they are installed on each Ansible Tower node! Additionally it is good to document this somewhere so you can install Ansible Tower again on another environment one day. Otherwise you will forget you ever installed that module.

Good luck!