Category Archives: DevOps

Checking OS Version Across Multiple Hosts with Ansible

Often when you are maintaining a large number of servers, it is useful to be able to query those systems all at once to find out information like IP address, configured hostnames, and even the OS version.  

In this post we’re going to focus on pulling the OS version from multiple systems at once with a single Ansible playbook.  

First, lets get an idea of how our directory structure should look in the end, then we’ll break things down:

Ansible playbook and file-system layout:

[~/sources/ansible]
$ tree .
.
├── get-os-version.yaml
├── group_vars
│   └── linux
├── hosts
│   └── home
│   ├── archive.jbldata.com
│   ├── jbllnxwks.jbldata.com
│   └── yoga2.jbldata.com
└── host_vars

4 directories, 5 files

Now that we know how things should look in the end, lets setup our host configurations.

Host configurations:

[~/sources/ansible]
$ tree hosts/
hosts/
└── home
 ├── archive.jbldata.com
 ├── jbllnxwks.jbldata.com
 └── yoga2.jbldata.com

1 directory, 3 files

[~/sources/ansible]
$ cat hosts/home/archive.jbldata.com 
[linux]
archive.jbldata.com

As you can see, the host configurations can be fairly simple and straight-forward to start, following the directory structure outlined above. Now lets setup our group_vars.

Group configuration:

[~/sources/ansible]
$ cat group_vars/linux 
---
ansible_ssh_private_key_file: /home/jbl/.ssh/jbldata_id_rsa

In this case, each of the servers I’m dealing with are secured by password-protected SSH keys, so I’m setting up my group vars to reference the correct SSH private key to use when connecting to these servers.  Pretty simple so far?  Great, now lets look at the playbook.

The Ansible playbook:

[~/sources/ansible]
$ cat get-os-version.yaml 
---
- name: Check OS Version via /etc/issue
 hosts: linux
 tasks:
 - name: cat /etc/issue
 shell: cat /etc/issue
 register: etc_issue
 - debug: msg="{{etc_issue.stdout_lines}}"

This playbook is very simple, but does exactly what we need.   Here we are specifying the use of the ‘shell’ module in order to execute the cat command on our remote servers.

We use the ‘register’ keyword to save the resulting output of the command in a variable called ‘etc_issue’.  We then use the ‘debug’ module to print the contents of that variable via ‘etc_issue’.  

When executing a command via the ‘shell’ module, there are several return values that we have access to, which are also now captured in the ‘etc_issue’ variable. In order to access the specific return value we are interested in, we use ‘debug’ to dump the STDOUT return value specifically via ‘etc_issue.stdout_lines’.

Now we have an Ansible playbook and associated configuration that allows us to quickly query multiple servers for their OS version.

It’s important to note that since I’m using password-protected SSH keys, that I’m using SSH Agent before I execute the playbook.  This only has to be done once for repeated runs of the same playbook within your current terminal session, for example:

[~/sources/ansible]
$ ssh-agent 
SSH_AUTH_SOCK=/tmp/ssh-82DKhToCuPUu/agent.4994; export SSH_AUTH_SOCK;
SSH_AGENT_PID=4995; export SSH_AGENT_PID;
echo Agent pid 4995;


[~/sources/ansible]
$ ssh-add ~/.ssh/jbldata_id_rsa
Enter passphrase for /home/jbl/.ssh/jbldata_id_rsa: 
Identity added: /home/jbl/.ssh/jbldata_id_rsa (/home/jbl/.ssh/jbldata_id_rsa)

Now, we’re ready to execute the ansible playbook.  Here’s the resulting output:

[~/sources/ansible]
$ ansible-playbook -i hosts/home get-os-version.yaml 

PLAY [Check OS Version via /etc/issue] *****************************************

TASK [setup] *******************************************************************
ok: [jbllnxwks.jbldata.com]
ok: [yoga2.jbldata.com]
ok: [archive.jbldata.com]

TASK [cat /etc/issue] **********************************************************
changed: [yoga2.jbldata.com]
changed: [jbllnxwks.jbldata.com]
changed: [archive.jbldata.com]

TASK [debug] *******************************************************************
ok: [archive.jbldata.com] => {
 "msg": [
 "Ubuntu 14.04.3 LTS \\n \\l"
 ]
}
ok: [yoga2.jbldata.com] => {
 "msg": [
 "Ubuntu 14.04.5 LTS \\n \\l"
 ]
}
ok: [jbllnxwks.jbldata.com] => {
 "msg": [
 "Debian GNU/Linux 5.0 \\n \\l"
 ]
}

PLAY RECAP *********************************************************************
archive.jbldata.com : ok=3 changed=1 unreachable=0 failed=0 
jbllnxwks.jbldata.com : ok=3 changed=1 unreachable=0 failed=0 
yoga2.jbldata.com : ok=3 changed=1 unreachable=0 failed=0

And that’s pretty much it!  Now we just have to add more hosts under our hosts/ configuration, and we can query as many servers as we want from a single command.  Happy orchestrating!

Ansible Playbooks – Externalization and Deduplication

Image result for ansible

Externalization and Deduplication

Developers who understand the concepts of modularity and deduplication should immediately recognize the power behind being able to include settings and commands from external files.   It is seriously counter-productive to maintain multiple scripts or playbooks that have large blocks of code or settings that are exactly the same.   This is an anti-pattern.

Ansible is a wonderful tool, however it can often be implemented in counter-productive ways.  Lets take variables for example.

Instead of maintaining a list of the same variables across multiple playbooks, it is better to use Variable File Separation.

The Ansible documentation provides an excellent example of how to do this.  However I feel that the reasoning behind why you would want to do it falls short in describing the most common use-case, deduplication.

The documentation discusses the possible needs around security or information sensitivity.  I also believe that deduplication should be added to that list.  Productivity around how playbooks are managed can be significantly increased if implemented in a modular fashion using Variable File Separation, or vars_files.   This by the way also goes for use of the includes_vars module.

Here are a list of reasons why you should immediately consider a deduplication project around your Ansible playbook variables:

Save Time Updating Multiple Files

This may seem like a no-brainer, but depending on the skills and experience of the person writing the playbook, this can become a significant hindrance to productivity.   Because of Ansible’s agent-less and decentralized manner, playbooks can be written by anyone who wants to get started with systems automation.  Often, these can be folks without significant proficiencies in programmer-oriented text editors such as Vim, Emacs, or Eclipse – or with bash scripting experience around command-line tools like awk, sed, and grep.

It is easy to imagine a Java developer without significant Linux command-line experience opening up one playbook at a time, and modifying the value for the same variable, over and over… and over again.

The best way for folks without ninja text-editing skills to stay productive is to deduplicate, and store common variables and tasks in external files that are referenced by multiple playbooks.

Prevent Bugs and Inconsistent Naming Conventions

In a perfect world, everyone would understand what a naming convention was.  All our variables would be small enough to type quickly, clear enough to understand its purpose, and simple enough that there would never be a mis-spelling or type-o.  This is rarely the case.

If left un-checked, SERVER_1_IP can also be SERVER1_IP, Server_1_IP, and server_1_ip.  All different variable names across multiple files, referencing the same value for the exact same purpose.

This mess can be avoided by externalizing this information in a shared file.

Delegate Maintenance and Updates to Variables That Change Frequently

In some environments, there may be playbook variables that need to change frequently.  If these variables are part of some large all-encompassing playbook that only some key administrators have access to be able to modify, your teams could be left waiting for your administrator to have free cycles available just to make a simple change.  Again, deduplication and externalization to the rescue!  Have these often-changing variables externalized so that users who need these changes immediately can go ahead and commit these changes to very specific, isolated files within your version control system that they have special rights to modify.

Cleaner Version Control History (and therefore Audit History)

If you have the same variables referenced by multiple files, and you make changes to each of those files before you commit them to version control, then your version control history can become a complete mess.  Your version control history will show a change to a single value affecting multiple files.  If you come from a software development background, and are familiar with the concept of code reviews, then you can appreciate being able to look at a simple change to a hard-coded value (or a constant), and see that it only affects one or two files.

I hope the reasons above convince some of you to start browsing your playbook repositories for possible candidates for deduplication.  I really believe that such refactoring projects can boost productivity and execution speed for individuals and teams looking to push changes faster while minimizing obstacles around configurations shared by multiple systems.  Send me a note if this inspires you to start your own deduplication project!