The blog of Leonid Mirsky

Creating hadoop test environment with ansible and vagrant

2013-11-19T18:31:00Z

I love Puppet!

I always hated shell scripting.

When I discovered Puppet a year ago, I finally found a good programmable way to describe the infrastructure I am working with. A lot changed since my first encounter with Puppet, especially, since plenty of new and interesting tools appeared which compete with Puppet for the heart of a sysadmin.

One of these shiny new configuration management tools is Ansible. I decided to give it a try, and to configure a local Hadoop development environment using it and vagrant.

Prerequisites

In this tutorial I will use vagrant version 1.3.5, and VirtualBox version 4.1.12 on Ubuntu. Ansible must be installed on the same system as vagrant because the vagrant Ansible provisioner relies on that to perform it’s magic. I used ansible version 1.3.3 since it supports the roles functionality which we will use in this tutorial.

Vagrant configuration

Let’s start by downloading Ansible example modules into your working directory. We will use the Hadoop playbook to do most of the heavy lifting for us.

[leonid@laptop ~]$ wget https://github.com/ansible/ansible-examples/archive/master.zip
[leonid@laptop ~]$ unzip master.zip ansible-examples-master/hadoop/*

For the Vagrant configuration, I used the following Vagrantfile.

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "centos_6_3_x86_64"
  config.vm.box_url = "http://files.vagrantup.com/precise64.box"

  config.vm.synced_folder ".", "/vagrant"

  config.vm.provider "virtualbox" do |v|
    v.customize ["modifyvm", :id, "--cpus", "1", "--memory", "512"]
  end

  config.vm.define "hadoop_master" do |hadoop_master|
    hadoop_master.vm.network "private_network", ip: "192.168.50.4"
    hadoop_master.vm.hostname = "master.internal"
  end

  config.vm.define "hadoop_slave1" do |hadoop_slave1|
    hadoop_slave1.vm.network "private_network", ip: "192.168.50.5"
    hadoop_slave1.vm.hostname = "slave1.internal"
  end

  config.vm.define "hadoop_slave2" do |hadoop_slave2|
    hadoop_slave2.vm.network "private_network", ip: "192.168.50.6"
    hadoop_slave2.vm.hostname = "slave2.internal"
    
    hadoop_slave2.vm.provision :ansible do |ansible|
      ansible.inventory_path = "inventory"
      ansible.verbose = "v"
      ansible.sudo = true
      ansible.playbook = "ansible-examples-master/hadoop/site.yml"
    end
  end
end

We are going to provision a simple Hadoop cluster, without HA configuration, which will consist of one master node and 2 slave nodes.

I limited the VM resources to 1 cpu and 512MB ram per each VM, since I didn’t want this environment to interfere with other stuff I am running on my laptop:

config.vm.provider "virtualbox" do |v|
  v.customize ["modifyvm", :id, "--cpus", "1", "--memory", "512"]
end

Also, I configured 3 VM instances which will be provisioned and managed by Vagrant. I assigned ip addresses and hostnames to each VM, which will be very handy later.

config.vm.define "hadoop_master" do |hadoop_master|
  hadoop_master.vm.network "private_network", ip: "192.168.50.4"
  hadoop_master.vm.hostname = "master.internal"
end

config.vm.define "hadoop_slave1" do |hadoop_slave1|
  hadoop_slave1.vm.network "private_network", ip: "192.168.50.5"
  hadoop_slave1.vm.hostname = "slave1.internal"
end

config.vm.define "hadoop_slave2" do |hadoop_slave2|
  hadoop_slave2.vm.network "private_network", ip: "192.168.50.6"
  hadoop_slave2.vm.hostname = "slave2.internal"

I assigned the Ansible provisioner to the last VM not by chance, this step is actually very important. In Vagrant, in a multi-vm configuration, VM machines will be started according to their declaration order. I used that knowledge to my advantage, in order to make sure that the provisioning process will be executed only after all the VMs are up and running.

Why it is important? because otherwise Ansible will fail, since it will try to connect to some not-yet-created machines. This technique (others will call it a hack) is described in more detail in this github issue

hadoop_slave2.vm.provision :ansible do |ansible|
  ansible.inventory_path = "inventory"
  ansible.verbose = "v"
  ansible.sudo = true
  ansible.playbook = "ansible-examples-master/hadoop/site.yml"
end

Ansible

I am not willing to dive deep into Ansible, but it is important to note it’s 2 main configuration files: Ansibles inventory and playbook.yml files.

The inventory file defines which machines will be managed by Ansible and which groups these machines are assigned to (and by group assignment you actually telling which roles should be assigned to these nodes). I think that it is a very powerful concept, which as everything else in Ansible, is implemented radically simpler than in Puppet. You can find more information about the inventory file here

Here is the content of the inventory file I used:

[hadoop_all:children]
hadoop_masters
hadoop_slaves

[hadoop_master_primary]
master.internal ansible_ssh_host=192.168.50.4

[hadoop_master_secondary]

[hadoop_masters:children]
hadoop_master_primary
hadoop_master_secondary

[hadoop_slaves]
slave1.internal ansible_ssh_host=192.168.50.5
slave2.internal ansible_ssh_host=192.168.50.6

Another important file is the playbook yml file. I used the default site.yml (ansible-examples-master/hadoop/site.yml) which comes with the example repository. It is easier to view this file as the starting point of the whole provisioning process, and Ansible’s execution process starts from there.

Adjusting Hadoop playbooks

I think I never had a chance to download a community playbook/module/cookbook that worked from the first run, and Ansible’s hadoop playbooks aren’t an exception. In order to make them work with our setup, we need to adjust a bit the default configuration.

First of all, we will needed to install the libselinux-python as a prerequisite for the selinux module to work. To do that add the following line to the ansible-examples-master/hadoop/roles/common/tasks/common.yml file:

- name: Install SElinux ansible module dependencies
  yum: name=libselinux-python state=installed
- name: Disable SELinux in conf file
  selinux: state=disabled

Secondly, by default, the example playbooks create the /etc/hosts file on all the VM’s with the internal ip. In order that all our VM’s will be able to communicate with each other, we will change the default parameters to use the extermal ip instead. To do so, change the following line in the ansible-examples-master/hadoop/group_vars/all file:

iface: eth1

And as our last change for today, we will needed to explicitly set the permissions of hadoop configuration to 644 in order to make sure we won’t fail on permissions during the provisioning. Let’s do so, by adding the mode=644 line to ansible-examples-master/hadoop/roles/hadoop_primary/tasks/hadoop_master_no_ha.yml file:

- name: Copy the hadoop configuration files for no ha
  template: src=roles/common/templates/hadoop_conf/.j2 dest=/etc/hadoop/conf/ mode=644
  with_items: 
   - core-site.xml
   - hadoop-metrics.properties
   - hadoop-metrics2.properties

Run!

Finally everything is ready for launch! Now just hit vagrant up at the command line, and in about 10 minutes you will have a working hadoop environment.

Let’s go another extra mile, and check that our hadoop is working. Review the jobtracker webpage by accessing http:/192.168.50.4:50030/jobtracker.jsp in your browser. It should show up some basic jobtracker stats.

For a more elaborate use of our environment, you will need to adjust the playbooks even more. For example, you will probably need to create the mapred directories. Since Ansible is so easy to start with, I am sure that you will be able to catch up easily, and to complete these extra steps without a hassle.

Summary

My first impression of Ansible is quite positive, I can tell that it is much easier to start using this tool than puppet, and actually it seems quite powerful. I will need to test it in a production scenario to see if it will work for a more complex scenarios.

Getting started with MCollective

2013-10-08T18:31:00Z

Recently I was working on a simple continuous deployment process for puppet code. The process steps are as follows:

A new puppet code is pushed to a github repository
Codeship.io service picks the changes, and runs the build
Using Capistrano (or any other SSH framework), the new code is pushed to puppet servers
All puppet agents are run immediately to pick up the changes, and re-provision their hosts accordingly

Although the last step looks trivial, but actually, because of puppet’s communication module, it isn’t so easy to do. By default, puppet master can’t initiate a direct connection to managed agents, so in order to achieve that I decided to give MCollective a try.

MCollective components

Mcollective is a framework for building service orchestration on top of puppet regular infrastructure.

Without going into too much details, MCollective consists of 3 main components:

Server - despite it’s name, it is the agent you are going to install on all your puppet managed nodes. The server daemon will listen to a specified queue for actions, and respond accordingly
Client - using this component you can fire off your orchestration jobs to all/subset of subscribed nodes (which have the server component configured correctly)
Middleware - Since MCollective architecture is pub/sub , middleware component is generally a queue, to which you will send jobs using the client, and to which the server will subscribe to listen for new actions.

Let’s play

An easy way to start playing around with MCollective is by using a vagrant ready repository

[vagrant@middleware ~]$ git clone https://github.com/ripienaar/mcollective-vagrant.git
[vagrant@middleware ~]$ cd mcollective-vagrant

Edit the Vagrantfile (pay attention to the amount of instances which will be previsioned in addition to the middleware. By default it is 5, and it can be a little bit intensive on some laptops)

[vagrant@middleware ~]$ vi Vagrantfile
[vagrant@middleware ~]$ vagrant up

After all the VM’s are up (and it can take a while), connect to the middleware server (since by default the client is installed there), and check that all the nodes are configured correctly

[vagrant@middleware ~]$ mco ping
node0.example.net                        time=85.38 ms
middleware.example.net                   time=90.15 ms

To get all the available information on a specific node run:

[vagrant@middleware ~]$ mco inventory node0.example.net
Inventory for node0.example.net:
 
Agents:
  discovery       filemgr         integration    
  nettest         nrpe            package        
  process         puppet          rpcutil        
  service         urltest                        
 
Data Plugins:
  agent           fstat           nettest        
  nrpe            process         puppet         
  resource        service                        
 
Configuration Management Classes:
  default                        mcollective                   
  mcollective::agent::filemgr    mcollective::agent::integration
  mcollective::agent::nettest    mcollective::agent::nrpe      
  mcollective::agent::package    mcollective::agent::process   
  mcollective::agent::puppet     mcollective::agent::service   
  mcollective::agent::urltest    mcollective::config           
  mcollective::install           mcollective::service          
  motd                           nagios                        
  nrpe                           nrpe::params                  
  puppet                         repos                         
  roles::node                    settings                      
 
Facts:
  architecture => x86_64
  augeasversion => 0.9.0
  caller_module_name => mcollective
  clientcert => node0.example.net
  clientversion => 2.7.17

The output here is actually very helpful:

You can see all the classes applied to the node (Configuration Management Classes)
All the node facts, and their values
The MCollective data plugins and available MCollective agents (for a matter of simplicity, consider them all as MCollective functionality you can request from that node. The original terminology is a bit confusing)

Filters

Till now, we were addressing all the managed nodes or a specific server. However usually you would like to communicate with a subset of your environment. For that reason MCollective provides an easy way to define command filters.

For example, we can see a summary of all the node’s rubysitedir fact

[vagrant@middleware ~]$ mco facts rubysitedir
Report for fact: rubysitedir
 
        /usr/lib/ruby/site_ruby/1.8             found 2 times
        /usr/lib/ruby/site_ruby/1.9.3           found 1 times
 
Finished processing 3 / 3 hosts in 60.00 ms

… and then we can easily filter the nodes with ruby version 1.8

[vagrant@middleware ~]$ mco ping -F rubysitedir=/usr/lib/ruby/site_ruby/1.8
node0.example.net                        time=53.12 ms
middleware.example.net                   time=54.14 ms

… or we can query by applied classes (which nodes has nagios class applied to them?)

[vagrant@middleware ~]$ mco ping -C nagios
node0.example.net                        time=49.30 ms
middleware.example.net                   time=52.98 ms

… also we can combine the filters for a more complex query expression

[vagrant@middleware ~]$ mco ping -S "operatingsystem=CentOS and roles::node"
node0.example.net                        time=38.22 ms

Plugins

Since MCollective is a framework, by performing the default installation you actually will get very limit set of commands. The vagrant repository we were working with till now, already installed some plugins for you, but remember that in production you will need to take care of the plugins installation in addition to the basic MCollective components configuration. You can check out the available community plugins here.

For the continuous deployment process I mentioned earlier, I needed the puppet plugin, which provides an easy way to enable, disable and run puppet daemons. For example, to rerun all puppet agents, you can just execute the following command

[vagrant@middleware ~]$ mco puppet runonce

Conclusion

MCollective isn’t so intuitive for a beginner, probably because of all the different components terminology and a over complicated documentation. I hope this post will ease a bit the first steps, and make this solid tool more available for a wider audience.

For advanced MCollective topics, you should review the following sources:

Great presentation by @repienaar with lots of useful examples
MCollective architecture review by puppetlabs
Additional CLI tricks and examples