Category Archives: Professional

For IT and business related material

Checking OS Version Across Multiple Hosts with Ansible

Often when you are maintaining a large number of servers, it is useful to be able to query those systems all at once to find out information like IP address, configured hostnames, and even the OS version.  

In this post we’re going to focus on pulling the OS version from multiple systems at once with a single Ansible playbook.  

First, lets get an idea of how our directory structure should look in the end, then we’ll break things down:

Ansible playbook and file-system layout:

[~/sources/ansible]
$ tree .
.
├── get-os-version.yaml
├── group_vars
│   └── linux
├── hosts
│   └── home
│   ├── archive.jbldata.com
│   ├── jbllnxwks.jbldata.com
│   └── yoga2.jbldata.com
└── host_vars

4 directories, 5 files

Now that we know how things should look in the end, lets setup our host configurations.

Host configurations:

[~/sources/ansible]
$ tree hosts/
hosts/
└── home
 ├── archive.jbldata.com
 ├── jbllnxwks.jbldata.com
 └── yoga2.jbldata.com

1 directory, 3 files

[~/sources/ansible]
$ cat hosts/home/archive.jbldata.com 
[linux]
archive.jbldata.com

As you can see, the host configurations can be fairly simple and straight-forward to start, following the directory structure outlined above. Now lets setup our group_vars.

Group configuration:

[~/sources/ansible]
$ cat group_vars/linux 
---
ansible_ssh_private_key_file: /home/jbl/.ssh/jbldata_id_rsa

In this case, each of the servers I’m dealing with are secured by password-protected SSH keys, so I’m setting up my group vars to reference the correct SSH private key to use when connecting to these servers.  Pretty simple so far?  Great, now lets look at the playbook.

The Ansible playbook:

[~/sources/ansible]
$ cat get-os-version.yaml 
---
- name: Check OS Version via /etc/issue
 hosts: linux
 tasks:
 - name: cat /etc/issue
 shell: cat /etc/issue
 register: etc_issue
 - debug: msg="{{etc_issue.stdout_lines}}"

This playbook is very simple, but does exactly what we need.   Here we are specifying the use of the ‘shell’ module in order to execute the cat command on our remote servers.

We use the ‘register’ keyword to save the resulting output of the command in a variable called ‘etc_issue’.  We then use the ‘debug’ module to print the contents of that variable via ‘etc_issue’.  

When executing a command via the ‘shell’ module, there are several return values that we have access to, which are also now captured in the ‘etc_issue’ variable. In order to access the specific return value we are interested in, we use ‘debug’ to dump the STDOUT return value specifically via ‘etc_issue.stdout_lines’.

Now we have an Ansible playbook and associated configuration that allows us to quickly query multiple servers for their OS version.

It’s important to note that since I’m using password-protected SSH keys, that I’m using SSH Agent before I execute the playbook.  This only has to be done once for repeated runs of the same playbook within your current terminal session, for example:

[~/sources/ansible]
$ ssh-agent 
SSH_AUTH_SOCK=/tmp/ssh-82DKhToCuPUu/agent.4994; export SSH_AUTH_SOCK;
SSH_AGENT_PID=4995; export SSH_AGENT_PID;
echo Agent pid 4995;


[~/sources/ansible]
$ ssh-add ~/.ssh/jbldata_id_rsa
Enter passphrase for /home/jbl/.ssh/jbldata_id_rsa: 
Identity added: /home/jbl/.ssh/jbldata_id_rsa (/home/jbl/.ssh/jbldata_id_rsa)

Now, we’re ready to execute the ansible playbook.  Here’s the resulting output:

[~/sources/ansible]
$ ansible-playbook -i hosts/home get-os-version.yaml 

PLAY [Check OS Version via /etc/issue] *****************************************

TASK [setup] *******************************************************************
ok: [jbllnxwks.jbldata.com]
ok: [yoga2.jbldata.com]
ok: [archive.jbldata.com]

TASK [cat /etc/issue] **********************************************************
changed: [yoga2.jbldata.com]
changed: [jbllnxwks.jbldata.com]
changed: [archive.jbldata.com]

TASK [debug] *******************************************************************
ok: [archive.jbldata.com] => {
 "msg": [
 "Ubuntu 14.04.3 LTS \\n \\l"
 ]
}
ok: [yoga2.jbldata.com] => {
 "msg": [
 "Ubuntu 14.04.5 LTS \\n \\l"
 ]
}
ok: [jbllnxwks.jbldata.com] => {
 "msg": [
 "Debian GNU/Linux 5.0 \\n \\l"
 ]
}

PLAY RECAP *********************************************************************
archive.jbldata.com : ok=3 changed=1 unreachable=0 failed=0 
jbllnxwks.jbldata.com : ok=3 changed=1 unreachable=0 failed=0 
yoga2.jbldata.com : ok=3 changed=1 unreachable=0 failed=0

And that’s pretty much it!  Now we just have to add more hosts under our hosts/ configuration, and we can query as many servers as we want from a single command.  Happy orchestrating!

Ansible Playbooks – Externalization and Deduplication

Image result for ansible

Externalization and Deduplication

Developers who understand the concepts of modularity and deduplication should immediately recognize the power behind being able to include settings and commands from external files.   It is seriously counter-productive to maintain multiple scripts or playbooks that have large blocks of code or settings that are exactly the same.   This is an anti-pattern.

Ansible is a wonderful tool, however it can often be implemented in counter-productive ways.  Lets take variables for example.

Instead of maintaining a list of the same variables across multiple playbooks, it is better to use Variable File Separation.

The Ansible documentation provides an excellent example of how to do this.  However I feel that the reasoning behind why you would want to do it falls short in describing the most common use-case, deduplication.

The documentation discusses the possible needs around security or information sensitivity.  I also believe that deduplication should be added to that list.  Productivity around how playbooks are managed can be significantly increased if implemented in a modular fashion using Variable File Separation, or vars_files.   This by the way also goes for use of the includes_vars module.

Here are a list of reasons why you should immediately consider a deduplication project around your Ansible playbook variables:

Save Time Updating Multiple Files

This may seem like a no-brainer, but depending on the skills and experience of the person writing the playbook, this can become a significant hindrance to productivity.   Because of Ansible’s agent-less and decentralized manner, playbooks can be written by anyone who wants to get started with systems automation.  Often, these can be folks without significant proficiencies in programmer-oriented text editors such as Vim, Emacs, or Eclipse – or with bash scripting experience around command-line tools like awk, sed, and grep.

It is easy to imagine a Java developer without significant Linux command-line experience opening up one playbook at a time, and modifying the value for the same variable, over and over… and over again.

The best way for folks without ninja text-editing skills to stay productive is to deduplicate, and store common variables and tasks in external files that are referenced by multiple playbooks.

Prevent Bugs and Inconsistent Naming Conventions

In a perfect world, everyone would understand what a naming convention was.  All our variables would be small enough to type quickly, clear enough to understand its purpose, and simple enough that there would never be a mis-spelling or type-o.  This is rarely the case.

If left un-checked, SERVER_1_IP can also be SERVER1_IP, Server_1_IP, and server_1_ip.  All different variable names across multiple files, referencing the same value for the exact same purpose.

This mess can be avoided by externalizing this information in a shared file.

Delegate Maintenance and Updates to Variables That Change Frequently

In some environments, there may be playbook variables that need to change frequently.  If these variables are part of some large all-encompassing playbook that only some key administrators have access to be able to modify, your teams could be left waiting for your administrator to have free cycles available just to make a simple change.  Again, deduplication and externalization to the rescue!  Have these often-changing variables externalized so that users who need these changes immediately can go ahead and commit these changes to very specific, isolated files within your version control system that they have special rights to modify.

Cleaner Version Control History (and therefore Audit History)

If you have the same variables referenced by multiple files, and you make changes to each of those files before you commit them to version control, then your version control history can become a complete mess.  Your version control history will show a change to a single value affecting multiple files.  If you come from a software development background, and are familiar with the concept of code reviews, then you can appreciate being able to look at a simple change to a hard-coded value (or a constant), and see that it only affects one or two files.

I hope the reasons above convince some of you to start browsing your playbook repositories for possible candidates for deduplication.  I really believe that such refactoring projects can boost productivity and execution speed for individuals and teams looking to push changes faster while minimizing obstacles around configurations shared by multiple systems.  Send me a note if this inspires you to start your own deduplication project!

Managing Your SMTP Relay With Postfix – Correctly Rejecting Mail for Non-local Users

Image result for SMTP postfix

I manage a few personal mail relays that I use for relaying my own mail and for experimentation purposes (mail logs are a great source of unique and continuously flowing data that can you use to try out different ideas in GUI, database, or parser development).  One of them was acting up recently.  I got a message from my upstream mail-queue host saying that they’ve queued up quite a bit of mail for me over the last few weeks, and that I should investigate, as they do want to avoid purging the queue of valid mail.

Clearly I wanted to avoid queuing up mail on a remote server that is intended for my domain, and so I set out about understanding the problem.

What I found was that there was a setting in my /etc/postfix/main.cf that, although it was technically a valid setting, was incorrect for the role that mail-server was playing.  Specifically the mail server was supposed to be rejecting email completely for non-local users, instead of just deferring it with a “try again later” message.

In this case, I’m using Postfix v2.5.5. The settings that control this configuration in /etc/postfix/main.cf are as follows:

  • unknown_local_recipient_reject_code
  • local_recipient_maps

local_recipient_maps

local_receipient_maps defines the accounts that this mail server will accept and relay mail for. All other accounts would be “rejected” by the mail server.

However, how rejected mail is treated by Postfix depends on how it is configured, and this was the problem with this particular server.

For Postfix, it is possible to mark a message as “rejected”, but actually have it mean “rejected right now, but maybe not permanently, so try again later”. This “try again later” will cause the e-mail message to be queued on the upstream server, until it reaches some kind of retry time-out and delivery is once again attempted. Of course this will fail again, and again.

This kind of configuration is great for testing purposes, because it allows you to test the same messages over and over again without losing them, or to queue them up so that they can be reviewed to ensure they are indeed invalid e-mail messages. However this is not the state you want your mail server to be in permanently. At some point once things are ready for long-term (production) use, you want your mail server to actually reject messages permanently.

unknown_local_recipient_reject_code

That is where unknown_local_recipient_reject_code comes in. This configuration property controls what the server means when it “rejects” a message. Does it mean right now, or permanently?

The SMTP server response code to reject mail permanently is 550, and the code to reject mail only temporarily is 450.

Here is how you would configure Postfix to reject mail only temporarily:

unknown_local_recipient_reject_code = 450

And here is how you set Postfix to reject mail permanently:

unknown_local_recipient_reject_code = 550

In my case, changing the unknown_local_recipient_reject_code from 450 to 550 is what solved the problem.

In summary, if you ever run into an issue with your Postfix mail server where you believe mail is set to be REJECTED but it still seems to be queuing up on your up-stream mail relay, double-check the unknown_local_recipient_reject_code.

# Local recipients defined by local unix accounts and aliases only
local_recipient_maps = proxy:unix:passwd.byname $alias_maps

# 450 (try again later), 550 (reject mail)
unknown_local_recipient_reject_code = 550

References
http://www.postfix.org/LOCAL_RECIPIENT_README.html
http://www.postfix.org/postconf.5.html#unknown_local_recipient_reject_code

Bad advice on “free advice”

Cross-post from LinkedIn, in response to How Seeking ‘Free’ Works Against Our Career Success:

I cannot completely agree here. There are many who offer free advice that also happens to be good advice. Alternatively, it is important for advice seekers to learn how to distinguish between good and bad advice by learning to think critically about the information they are receiving – by asking deeper, probing questions. Every answer received should lead to further questions. While I do agree that it is important to learn how to be independent and make your own way in this world (as in the example of parents encouraging children to pay for their own education), I do not see how this directly relates to giving or receiving free advice, or how free advice (as suggested in this article) can be considered to be bad advice without further inquiry. With regard to the job seeker asking for his/her resume to be reviewed, that was simply a lazy request. You cannot help those who are not willing to put in the effort to help themselves, regardless of whether or not your advice is free.