bubble.marmotte.net

From Glee
Jump to: navigation, search

Overview

Bubble is an OVH kemsirve (aka kimsufi) server. It's full of Bursts, which are its KVM virtual servers.

(Re)Installation

Installed with CentOS 6 64bit in English. Steps below are mostly raw notes, yet easily understandable.

Post-Install Manual Steps

vi /etc/sysconfig/network
# HOSTNAME=bubble.marmotte.net
# IPV6FORWARDING=yes
vi /etc/sysconfig/selinux
# SELINUX=enforcing
touch /.autorelabel
vi /etc/crontab # remove the last line for rtm
vi /etc/rc.local # remove everything
vi /etc/fstab
# remove /home and swap
umount /home
mdadm --manage --stop /dev/md2
vi /etc/mdadm.conf # remove md2 line
swapoff -a
fdisk /dev/sda
# delete 3 & 2
# n p 2 : first 1306 last 120557
# n p 3 : first 120558 last enter
# t 2 fd
# t 3 fd
fdisk /dev/sdb # same changes
partprobe
# reboot if the kernel can't re-read all partition tables properly
mdadm --create /dev/md2 --metadata=0.90 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2
mdadm --create /dev/md3 --metadata=0.90 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
mkswap -f /dev/md3
vi /etc/fstab # add : /dev/md3        none    swap    defaults        0       0
swapon -a
free

Upgrade to RHEL6 somehow. Ugly stuff :

rpm -e --nodeps centos-release
yum localinstall --nogpgcheck redhat-release-server-*
vi /etc/yum.repos.d/el.repo
vi /etc/yum.conf
# remove "bugtracker_url", "distroverpkg"
rm -rf /var/cache/yum/*
yum update
yum list extras
yum downgrade <names>
yum remove yum-plugin-fastestmirror.noarch

Main /srv Mount Point & Repositories

pvcreate /dev/md2
vgcreate vg0 /dev/md2
lvcreate -L512G -nsrv vg0
vi /etc/mdadm.conf
# add md2 and md3 with UUID
# get with mdadm --detail /dev/md2
mkfs.ext4 -j -m 1 -L srv /dev/vg0/srv
tune2fs -c 0 -i 0 /dev/vg0/srv
vi /etc/fstab # add : /dev/vg0/srv    /srv    ext4    noatime,nodiratime,commit=120      0       0
# !!! Add ,commit=120 for all ext* filesystems!
mount /srv

Tweaks

yum remove bind bind-chroot
vi /etc/resolv.conf # remove 127.0.0.1
yum install ntp
chkconfig --level 345 ntpd on
service ntpd start

KVM Hypervisor

yum update
yum install kvm python-virtinst libvirt.x86_64 virt-top
rm -f /etc/libvirt/qemu/networks/autostart/default.xml
service libvirtd start
chkconfig --level 345 ksm on
chkconfig --level 345 ksmtuned on
chkconfig rpcbind off
chkconfig rpcgssd off
chkconfig rpcidmapd off

Network

  • /etc/sysconfig/network-scripts/ifcfg-eth0 :
DEVICE=eth0
BRIDGE=br0
ONBOOT=yes
  • /etc/sysconfig/network-scripts/ifcfg-br0 :
# Main public Ethernet
DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
IPADDR=94.23.230.197
NETMASK=255.255.255.0
GATEWAY=94.23.230.254
ONBOOT=yes
# IPv6
IPV6INIT=yes
IPV6ADDR=2001:41d0:2:60a6::1/64
IPV6_DEFAULTGW=fe80::5:73ff:fea0:0%br0
  • /etc/sysconfig/network-scripts/ifcfg-br1 :
# Private Ethernet (IPv4 only)
DEVICE=br1
TYPE=Bridge
BOOTPROTO=static
IPADDR=192.168.19.254
NETMASK=255.255.255.0
ONBOOT=yes
# IPv6
IPV6INIT=yes
IPV6ADDR=2001:41d0:2:60a6:20::ffff/80

Give IPv4 access to the guests being installed :

vi /etc/sysctl.conf
# net.ipv4.ip_forward = 1
sysctl -p
iptables -t nat -A POSTROUTING -o br0 -s 192.168.0.0/16 -j MASQUERADE
service iptables save

Kernel

OVH has that annoying tendency to force using custom-built non package-managed kernels. To switch back to official RHEL kernels :

# install a non-latest kernel
yum install kernel-2.6.32-279.el6
vi /boot/grub/grub.conf  # copy over content from an existing RHEL server
# fix kernel line, example : kernel /boot/vmlinuz-2.6.32-279.el6.x86_64 ro root=/dev/md1
reboot
yum install kernel
# no more grubby error
reboot

Network

Addresses

  • IPv4 Addresses :
    • Main : 94.23.221.166
    • Failover :
      • 178.33.164.41 : DNS, Mail, Git (ssh), Puppet (restricted)
      • 178.33.164.42 : Main Web, XMPP, Mailman, Tracker
      • 178.33.164.43 : Directly on burst11 for sip.marmotte.net
  • IPv6 Addresses :
    • Main : 2001:41d0:2:60a6::1/64
    • Internal Bridge : 2001:41d0:2:60a6:20::ffff/80

IPv6 Neighbor Discovery

To work around OVH blocking traffic based on MAC addresses, npd6 is running with IPv6 forwarding enabled.

Bursts

Burst List

  • burst01 : EL6 DNS
  • burst02 : EL4 Legacy Apache/MySQL
  • burst03 : EL5 Mail
  • burst04 : EL6 Puppet
  • burst05 : EL6 MySQL
  • burst06 : EL5 Apache (thias only)
  • burst07 : EL6 Apache (outsiders, suexec)
  • burst08 : EL6 Mailman
  • burst09 : EL6 Git
  • burst10 : EL6 OpenERP
  • burst11 : EL6 FreeSWITCH
  • burst12 : EL6 Yubikey

New Burst Install

Example :

virt-install --autostart --name=burst11 \
  --ram=512 --vcpus=2 --os-variant=rhel6 --hvm --accelerate \
  --location=http://dl.marmotte.net/private/redhat/el6Server/x86_64/install \
  --disk=path=/dev/vg1/burst11 --network=bridge=br1 --mac=RANDOM --nographics --keymap=fr \
  --extra-args="console=ttyS0,9600 ks=http://dl.marmotte.net/private/ks/marmotte/burst11.ks"

Then enable puppet :

yum -y install puppet
grep -q puppet.marmotte.ici /etc/hosts || echo "178.33.164.41 puppet.marmotte.ici" >> /etc/hosts
puppet agent --onetime --no-daemonize --server=puppet.marmotte.ici --verbose --logdest=console --logdest=/var/log/puppet/puppet.log