bubble.marmotte.net
From Glee
Contents
Overview
Bubble is an OVH kemsirve (aka kimsufi) server. It's full of Bursts, which are its KVM virtual servers.
(Re)Installation
Installed with CentOS 6 64bit in English. Steps below are mostly raw notes, yet easily understandable.
Post-Install Manual Steps
vi /etc/sysconfig/network # HOSTNAME=bubble.marmotte.net # IPV6FORWARDING=yes vi /etc/sysconfig/selinux # SELINUX=enforcing touch /.autorelabel vi /etc/crontab # remove the last line for rtm vi /etc/rc.local # remove everything vi /etc/fstab # remove /home and swap umount /home mdadm --manage --stop /dev/md2 vi /etc/mdadm.conf # remove md2 line swapoff -a fdisk /dev/sda # delete 3 & 2 # n p 2 : first 1306 last 120557 # n p 3 : first 120558 last enter # t 2 fd # t 3 fd fdisk /dev/sdb # same changes partprobe # reboot if the kernel can't re-read all partition tables properly mdadm --create /dev/md2 --metadata=0.90 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2 mdadm --create /dev/md3 --metadata=0.90 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3 mkswap -f /dev/md3 vi /etc/fstab # add : /dev/md3 none swap defaults 0 0 swapon -a free
Upgrade to RHEL6 somehow. Ugly stuff :
rpm -e --nodeps centos-release yum localinstall --nogpgcheck redhat-release-server-* vi /etc/yum.repos.d/el.repo vi /etc/yum.conf # remove "bugtracker_url", "distroverpkg" rm -rf /var/cache/yum/* yum update yum list extras yum downgrade <names> yum remove yum-plugin-fastestmirror.noarch
Main /srv Mount Point & Repositories
pvcreate /dev/md2 vgcreate vg0 /dev/md2 lvcreate -L512G -nsrv vg0 vi /etc/mdadm.conf # add md2 and md3 with UUID # get with mdadm --detail /dev/md2 mkfs.ext4 -j -m 1 -L srv /dev/vg0/srv tune2fs -c 0 -i 0 /dev/vg0/srv vi /etc/fstab # add : /dev/vg0/srv /srv ext4 noatime,nodiratime,commit=120 0 0 # !!! Add ,commit=120 for all ext* filesystems! mount /srv
Tweaks
yum remove bind bind-chroot vi /etc/resolv.conf # remove 127.0.0.1 yum install ntp chkconfig --level 345 ntpd on service ntpd start
KVM Hypervisor
yum update yum install kvm python-virtinst libvirt.x86_64 virt-top rm -f /etc/libvirt/qemu/networks/autostart/default.xml service libvirtd start chkconfig --level 345 ksm on chkconfig --level 345 ksmtuned on chkconfig rpcbind off chkconfig rpcgssd off chkconfig rpcidmapd off
Network
- /etc/sysconfig/network-scripts/ifcfg-eth0 :
DEVICE=eth0 BRIDGE=br0 ONBOOT=yes
- /etc/sysconfig/network-scripts/ifcfg-br0 :
# Main public Ethernet DEVICE=br0 TYPE=Bridge BOOTPROTO=static IPADDR=94.23.230.197 NETMASK=255.255.255.0 GATEWAY=94.23.230.254 ONBOOT=yes # IPv6 IPV6INIT=yes IPV6ADDR=2001:41d0:2:60a6::1/64 IPV6_DEFAULTGW=fe80::5:73ff:fea0:0%br0
- /etc/sysconfig/network-scripts/ifcfg-br1 :
# Private Ethernet (IPv4 only) DEVICE=br1 TYPE=Bridge BOOTPROTO=static IPADDR=192.168.19.254 NETMASK=255.255.255.0 ONBOOT=yes # IPv6 IPV6INIT=yes IPV6ADDR=2001:41d0:2:60a6:20::ffff/80
Give IPv4 access to the guests being installed :
vi /etc/sysctl.conf # net.ipv4.ip_forward = 1 sysctl -p iptables -t nat -A POSTROUTING -o br0 -s 192.168.0.0/16 -j MASQUERADE service iptables save
Kernel
OVH has that annoying tendency to force using custom-built non package-managed kernels. To switch back to official RHEL kernels :
# install a non-latest kernel yum install kernel-2.6.32-279.el6 vi /boot/grub/grub.conf # copy over content from an existing RHEL server # fix kernel line, example : kernel /boot/vmlinuz-2.6.32-279.el6.x86_64 ro root=/dev/md1 reboot yum install kernel # no more grubby error reboot
Network
Addresses
- IPv4 Addresses :
- Main : 94.23.221.166
- Failover :
- 178.33.164.41 : DNS, Mail, Git (ssh), Puppet (restricted)
- 178.33.164.42 : Main Web, XMPP, Mailman, Tracker
- 178.33.164.43 : Directly on burst11 for sip.marmotte.net
- IPv6 Addresses :
- Main : 2001:41d0:2:60a6::1/64
- Internal Bridge : 2001:41d0:2:60a6:20::ffff/80
IPv6 Neighbor Discovery
To work around OVH blocking traffic based on MAC addresses, npd6 is running with IPv6 forwarding enabled.
Bursts
Burst List
-
burst01 : EL6 DNS -
burst02 : EL4 Legacy Apache/MySQL - burst03 : EL5 Mail
- burst04 : EL6 Puppet
- burst05 : EL6 MySQL
-
burst06 : EL5 Apache (thias only) -
burst07 : EL6 Apache (outsiders, suexec) -
burst08 : EL6 Mailman -
burst09 : EL6 Git -
burst10 : EL6 OpenERP -
burst11 : EL6 FreeSWITCH -
burst12 : EL6 Yubikey
New Burst Install
Example :
virt-install --autostart --name=burst11 \ --ram=512 --vcpus=2 --os-variant=rhel6 --hvm --accelerate \ --location=http://dl.marmotte.net/private/redhat/el6Server/x86_64/install \ --disk=path=/dev/vg1/burst11 --network=bridge=br1 --mac=RANDOM --nographics --keymap=fr \ --extra-args="console=ttyS0,9600 ks=http://dl.marmotte.net/private/ks/marmotte/burst11.ks"
Then enable puppet :
yum -y install puppet grep -q puppet.marmotte.ici /etc/hosts || echo "178.33.164.41 puppet.marmotte.ici" >> /etc/hosts puppet agent --onetime --no-daemonize --server=puppet.marmotte.ici --verbose --logdest=console --logdest=/var/log/puppet/puppet.log