When you need to configure multiple servers it’s always good to look at some configuration management software like Puppet of Chef. But there is a new kid in town and it looks promissing. I’ve played around with Ansible a bit to setup a complex network configration on 5 blade servers running a CentOS minimal. I mentioned this last weekend to Fabian Arrotin last weekend at FOSDEM and he told me to put that info online. So here it is!
So I have 5 blade servers and one NAS server. All servers have 2 network cards. They are connected to two network switches. Every server needs a bond of the two network cards and then some vlans for connecting to the internet and to each other.
I have setup the network ports connected to the server on both switches to untagged vlan 99 and tagged vlan 100, 1001 and 1002. vlan 99 is connected to a DHCP and TFTP running on the NAS. This DHCP / TFTP server supplies the blade servers with a PXE boot into a CentOS Minimal installation with a anaconda ks.cfg file. So when a blade server boots from PXE, it will automatically be (re-)installed. Every server is has two empty SATA disks, and in the BIOS I configured all servers to first try to boot from harddisk and if that’s not working, boot from the network through PXE. So after I turned on every server it was automatically installing CentOS on all servers, and rebooted. In the ks.cfg I configured that every server should do a DHCP on eth0 and eth1. So the next step is to setup the complex networking through ansible.
I first gathered all MAC addresses from all network cards and I installed ansible on my notebook. Then I created a skeleton directory for the ansible configuration
cd ~
mkdir ansible-config
cd ansible-config
mkdir acme adhoc files handlers playbooks tasks templates vars
First create the main plabook file:
cat > acme/setup.yml << EOF --- - include: ../playbooks/init_new_hosts.yml - include: ../playbooks/reboot_new_hosts.yml EOF
Then create the playbook file to setup networking:
cat > playbooks/init_new_hosts.yml << EOF
---
- hosts: newservers
user: root
vars_files:
- ../vars/netconfig.yml
- ../vars/macs/\${ansible_default_ipv4.macaddress}.yml
tasks:
- include: ../tasks/networking.yml
EOF
And the playbook file to reboot the hosts:
cat > playbooks/reboot_new_hosts.yml << EOF --- - hosts: newservers user: root gather_facts: false tasks: - include: ../tasks/reboot.yml EOF
And now the networking task:
cat > tasks/networking.yml << EOF
---
- name: Setup networking (general)
action: template owner=root group=root mode=644 src=../templates/etc/sysconfig/network dest=/etc/sysconfig/network
- name: Setup networking (interfaces)
action: template owner=root group=root mode=644 src=../templates/etc/sysconfig/network-scripts/ifcfg-interface dest=/etc/sysconfig/network-scripts/ifcfg-\${item}
with_items: $net_interfaces
- name: Setup networking (routes)
action: template owner=root group=root mode=644 src=../templates/etc/sysconfig/network-scripts/route-interface dest=/etc/sysconfig/network-scripts/route-\${net_privateif}
EOF
And the reboot task:
cat > tasks/reboot.yml << EOF --- - name: Reboot action: command /sbin/shutdown -r +5 "Reboot is triggered by Ansible" EOF
We now need some template files. Starting with the file for /etc/sysconfig/network:
mkdir -p templates/etc/sysconfig/network-scripts
cat > templates/etc/sysconfig/network << EOF
NETWORKING=yes
HOSTNAME={{ net_hostname }}
GATEWAY={{ networks[net_config[net_defaultif].X_network].netrange }}.{{ networks[net_config[net_defaultif].X_network].gateway }}
DNS1=8.8.8.8
DNS2=208.67.222.222
EOF
The template file for the /etc/sysconfig/network-scripts/ifcfg config file:
cat > templates/etc/sysconfig/network-scripts/ifcfg-interface << EOF
DEVICE="{{ item }}"
{%- if net_config[item].X_network is defined %}
IPADDR="{{ networks[net_config[item].X_network].netrange }}.{{ net_config[item].X_hostipaddr }}"
PREFIX="{{ networks[net_config[item].X_network].prefix }}"
{%- endif %}
{%- for key, value in net_config[item].items() %}
{%- if (key[:2] != 'X_') %}
{{ key|upper }}="{{ value }}"
{%- endif %}
{%- endfor %}
EOF
And last the template for /etc/sysconfig/network-scripts/route to make all private IPv4 ranges always be routed to a private gateway and not through the internet:
cat > templates/etc/sysconfig/network-scripts/route-interface << EOF
10.0.0.0/8 via {{ networks[net_config[net_privateif].X_network].netrange }}.{{ networks[net_config[net_privateif].X_network].gateway }}
172.16.0.0/12 via {{ networks[net_config[net_privateif].X_network].netrange }}.{{ networks[net_config[net_privateif].X_network].gateway }}
192.168.0.0/16 via {{ networks[net_config[net_privateif].X_network].netrange }}.{{ networks[net_config[net_privateif].X_network].gateway }}
EOF
So, that were all the preperations. It’s now time to configure the variables. First we need to define, which networks there are in vars/netconfig.yml:
cat > vars/netconfig.yml << EOF
---
networks:
inet: { netrange: '198.51.100', prefix: '26', gateway: '1' }
srv1: { netrange: '10.1.1', prefix: '24', 'gateway': '254' }
cloud: { netrange: '10.1.2', prefix: '24', 'gateway': '254' }
EOF
For every server, we can now define the network configuration in a seperate yml file. For example I could use the next config for blade1:
mkdir -p vars/netconfig
cat > vars/netconfig/blade1.exa.nl << EOF
---
net_hostname: 'blade1.exa.nl'
net_defaultif: 'bond0.100'
net_privateif: 'bond0.1001'
net_interfaces: [ 'eth0', 'eth1', 'bond0', 'bond0.100', 'bond0.1001', 'bond0.1002' ]
net_config:
'eth0': { hwaddr: '00:12:34:56:ab:11', bootproto: 'none', onboot: 'yes', master: 'bond0', slave: 'yes' }
'eth1': { hwaddr: '00:12:34:56:ab:12', bootproto: 'none', onboot: 'yes', master: 'bond0', slave: 'yes' }
'bond0': { bootproto: 'none', onboot: 'yes', bonding_opts: 'mode=5 miimon=80', userctl: 'no' }
'bond0.100': { bootproto: 'static', onboot: 'yes', vlan: 'yes', X_network: 'inet', X_hostipaddr: '21' }
'bond0.1001': { bootproto: 'static', onboot: 'yes', vlan: 'yes', X_network: 'srv1', X_hostipaddr: '101' }
'bond0.1002': { bootproto: 'static', onboot: 'yes', vlan: 'yes', X_network: 'cloud', X_hostipaddr: '101' }
EOF
In this file specially for blade1, I first defined a hostname. Followed by the default interface. That’s the interface where we will setup the default route. The private interface is something I use, to setup all private IP ranges (192.168., 172.16., and 10.) so any traffic for those ip-address will be forwarded to a internet gateway and not through the internet. Then I define a list of all interfaces (needed for Ansible to loop over). Then a configuration for every interface is next. Offcourse you must change the hwaddr to the correct hwaddr of your network cards. All options defined for the interface are put in uppercase in the ifcfg config file, except for the ones starting with X_. I use those internally, to look up the network range. The X_hostipaddr is the last part of the ip-address and must be unique for every server.
We can do the same for the other blade servers, so here’s an example for blade 2 and blade 3:
cat > vars/netconfig/blade2.exa.nl << EOF
---
net_hostname: 'blade2.exa.nl'
net_defaultif: 'bond0.100'
net_privateif: 'bond0.1001'
net_interfaces: [ 'eth0', 'eth1', 'bond0', 'bond0.100', 'bond0.1001', 'bond0.1002' ]
net_config:
'eth0': { hwaddr: '00:12:34:56:ab:21', bootproto: 'none', onboot: 'yes', master: 'bond0', slave: 'yes' }
'eth1': { hwaddr: '00:12:34:56:ab:22', bootproto: 'none', onboot: 'yes', master: 'bond0', slave: 'yes' }
'bond0': { bootproto: 'none', onboot: 'yes', bonding_opts: 'mode=5 miimon=80', userctl: 'no' }
'bond0.100': { bootproto: 'static', onboot: 'yes', vlan: 'yes', X_network: 'inet', X_hostipaddr: '22' }
'bond0.1001': { bootproto: 'static', onboot: 'yes', vlan: 'yes', X_network: 'srv1', X_hostipaddr: '102' }
'bond0.1002': { bootproto: 'static', onboot: 'yes', vlan: 'yes', X_network: 'cloud', X_hostipaddr: '102' }
EOF
cat > vars/netconfig/blade3.exa.nl << EOF
---
net_hostname: 'blade3.exa.nl'
net_defaultif: 'bond0.100'
net_privateif: 'bond0.1001'
net_interfaces: [ 'eth0', 'eth1', 'bond0', 'bond0.100', 'bond0.1001', 'bond0.1002' ]
net_config:
'eth0': { hwaddr: '00:12:34:56:ab:31', bootproto: 'none', onboot: 'yes', master: 'bond0', slave: 'yes' }
'eth1': { hwaddr: '00:12:34:56:ab:32', bootproto: 'none', onboot: 'yes', master: 'bond0', slave: 'yes' }
'bond0': { bootproto: 'none', onboot: 'yes', bonding_opts: 'mode=5 miimon=80', userctl: 'no' }
'bond0.100': { bootproto: 'static', onboot: 'yes', vlan: 'yes', X_network: 'inet', X_hostipaddr: '23' }
'bond0.1001': { bootproto: 'static', onboot: 'yes', vlan: 'yes', X_network: 'srv1', X_hostipaddr: '103' }
'bond0.1002': { bootproto: 'static', onboot: 'yes', vlan: 'yes', X_network: 'cloud', X_hostipaddr: '103' }
EOF
I assume, you can imagine what the contents of the config files for blade4 and blade5 will be. So We now have all configuration ready, we only have to link the configuration. I use the mac addresses of the default interface for that.
mkdir -p vars/macs ln -s ../netconfig/blade1.exa.nl vars/macs/00:12:34:56:ab:11 ln -s ../netconfig/blade1.exa.nl vars/macs/00:12:34:56:ab:12 ln -s ../netconfig/blade2.exa.nl vars/macs/00:12:34:56:ab:21 ln -s ../netconfig/blade2.exa.nl vars/macs/00:12:34:56:ab:22 ln -s ../netconfig/blade3.exa.nl vars/macs/00:12:34:56:ab:31 ln -s ../netconfig/blade3.exa.nl vars/macs/00:12:34:56:ab:32 ln -s ../netconfig/blade4.exa.nl vars/macs/00:12:34:56:ab:41 ln -s ../netconfig/blade4.exa.nl vars/macs/00:12:34:56:ab:42 ln -s ../netconfig/blade5.exa.nl vars/macs/00:12:34:56:ab:51 ln -s ../netconfig/blade5.exa.nl vars/macs/00:12:34:56:ab:52
The last part is to enter the current (temporary) dhcp ip addresses of the servers in our /etc/ansible/hosts file like this:
[newservers]
192.168.99.51
192.168.99.52
192.168.99.53
192.168.99.54
192.168.99.55
And now we are ready to setup the networking for those servers with ansible!
Just run:
ansible-playbook acme/setup.yml
Maybe you nee to add -k to ask for the root password, when you don’t have a SSH key setup yet.
Ansible will log on to the server through their current dhcp ip address. Based on the mac address of the default ipv4 interface it will include a netconfig file.
In this file the networking information for that server is defined and will be setup by Ansible. When everything goed alright, ansible will start a reboot of the server in 5 minutes.
If you think something is wrong, you can always cancel the reboot with the following command
ansible newservers -m command -a /sbin/shutdown -c “Reboot is cancelled by Ansible”
I don’t know it this is the prettiest or best way to do this, but it was fun to setup this kind of configuration management with Ansible. There are probably a few ways to enhance or improve my setup. but to my opinion it does show that you can use Ansible for your configuration management right after installing a standard minimal CentOS without any other requirements.
If you are going to copy these scripts for you own use, please check if the network config files are exactly what you expect, or else you will not be able to access the server after the reboot.
I also placed the files in a git repo at googlecode, if you want to try it out yourself: http://code.google.com/p/exarv-ansible-example-complexnetwork/
Have fun!
Edit (2013-03-13): Just updated to ansible 1.0 and the scripts were not working anymore. Updated the blog and added the changes to the git repo
