Installing unxsVZ
Update: This install guide is getting old very fast. Please use LVM and setup with free 1Gb to be able to use vzdump for clones and zero down time template creation from running containers.
Now easy yum install is available (but still being tested -ok for quick evaluation)! Please email us if you encounter bugs or problems.
For production use evaluation, and for all the latest features and improvements, you should export the latest source code via svn as explained at the bottom of the unxsVZ home page. Please call Gary, Hugo or Dilva at +1 310-356-6265 -or email supportgrp @ unixservice . com- for free help getting a stable system installed and working. We only test and support free installs for CentOS 5.2+ and Firefox.
This install article assumes that you are starting from empty servers. If you already have an in production OpenVZ container system, you still can install unxsVZ as long as you add the existing containers, and then run some simple mysql CLI operations. Please contact us for free help on importing existing system data.
Overview
Now you can quickly install on a single node for evaluation just using yum (or rpm's directly.) We only will cover CentOS 5 for simplicity (and to allow us to limit our testing to a single OS for now, while at the same time simplifying our rpm provisioning.)
To setup an actual production system you will have to (in most cases) compile unxsVZ from source code to customize for your mySQL server system (we recommend using a multi-master replication cluster running on 3 different hardware nodes and accessed via a 2 mysql-proxy server front end.)
Installing unxsVZ and then running unxsVZ controlled networks of geographically distant servers is not hard, but it is a job for current or future Linux IT professionals.
First of all you will install everything on a single hardware node (server) and once it is fully tested and loaded with all the software you will need. You will clone/replicate this base servers hard disk to all others. All the drives will be bootable for fast remote datacenter hands to yank, swap and power up for you with no confusing phone call instructions or system admin skills required.
Here we will cover an example installation and configuration of a 7 node unxsVZ datacenter. We will use CentOS 5.2 (32 bit) on the hardware nodes (a popular enterprise class Linux distribution that has proven very stable.) And the yum package manager as much as possible, for easy and reliable system updates. Hints and tips from the experience gathered from many ISP installs will be provided.
Recommended reading and intended audience
You should read the OpenVZ Intro (pdf), skim through the OpenVZ Users Guide (pdf) and read at least the quick install section of OPenVZ Wiki.
You should be excited by the definite advantages that modern virtualization will bring, to your in-house system or to the public internet services you will or are responsible in providing. You should have some experience with routine Linux system administration for internet services like mail, web hosting and DNS. Experience with at least one major internet service in depth is a good idea (for example DNS or Email.) Being able to read BASH scripts, simple SQL statements, and understand simple C programs may help, but it is in no way necessary.
Hardware setup
Lets say we have 7 brand new IBM x3250 servers with 4Gb of RAM and two hot swap 250Gb drives each. Let's not waste time installing CentOS 5 on each server. We will install everything including OpenVZ and unxsVZ on a single server and then clone the rest of the drives with dd (partimage will speed things up if you are not very familiar with fdisk and dd.) We recommend that you keep everything as simple as possible and avoid any so called NG file systems. You probably don't like being up at 4am with broken critical services, crossing your fingers that your backup is fine (not to mention that the servers are in Germany, you are in Tuscon, and the backup is in Chicago.) Use what you have experience with. If you have little experience be conservative: ext2, noatime, no labels in /etc/fstab or /boot/grub/grub/conf. Instead of expensive RAID, get servers with dual auto-fail-over power supplies and make sure your datacenter has decent UPSs that you can monitor for orderly shutdowns. In other words go Google not over-clocker-gamer-hobbyist.
Have a local backup always available. What has worked for us: RAID 1 hardware mirroring (burned in and fully tested via multiple power failures and brownouts) or simpler and with an added level of security: Your second drive should only be mounted by your server backup script and should be an identical bootable image kept in sync every hour (or more depending on your application and security alert system) with a good rsync script. If all your layers of security fail, based on history, it seems that crackers will not search for other non-mounted drives to also root kit etc. Test, test and test again. Do not use the server in production until you can swap drives and reboot perfectly even after yanking the power with the server running. Do not use it until the backup system works perfectly even when the lights go out in the middle of your rsync. Go nuts and try swapping your OS running drives while the server is running, see what happens. Make sure your /etc/fstab does not use labels or you will get funny results on most servers. Make sure that you (or better your server supplier runs) stress on your drives and system for at least 8 hours for a correct burn-in. And please...!DO NOT USE YOUR PRODUCTION DRIVES MORE THAN 3 YEARS EVER! Rotate your drives, add new ones, use your hot swap and live migration capabilities to keep your servers always running and do it after lunch, with no worries.
A good starting point for file system layout is 20Gb for your / root partition (keep /boot here also.) And the rest of your drive /vz. As fdisk and dd experts will note you will not have to dd your whole drive and waste time copying blank space. Just the / root partition and then mke2fs /dev/sdb3 (/vz) for example, while cloning 13 drives. Here is an example of an in production hardware node disk layout:
[root@serverx1 ~]# sfdisk -l Disk /dev/sda: 30401 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sda1 * 0+ 2549 2550- 20482843+ 83 Linux /dev/sda2 2550 3569 1020 8193150 82 Linux swap / Solaris /dev/sda3 3570 30400 26831 215520007+ 83 Linux /dev/sda4 0 - 0 0 0 Empty Disk /dev/sdb: 30401 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdb1 * 0+ 2549 2550- 20482843+ 83 Linux /dev/sdb2 2550 3569 1020 8193150 82 Linux swap / Solaris /dev/sdb3 3570 30400 26831 215520007+ 83 Linux /dev/sdb4 0 - 0 0 0 Empty
Network setup
Your servers should have 2 1000Mb/s NICS. For example the main NIC, eth0 should be on your public IP subnet, your eth1 NIC should be on your private unxsVZ datacenter wide subnet. All your unxsVZ managed servers in the same datacenter should be able to communicate very quickly via eth1 with each other.
We like to use 10.0.0.0/24 for the private inter hardware node subnet -less typing. The 10.0.0.0/24 subnet can also be used for certain types of containers, and finally we also use 192.168.0.0/24 for private to node subnet for certain types of service containers that will become clearer later.
Thankfully unxsVZ.tIP keeps track of all your free and used IPs for you.
OpenVZ installation
Once your base server is up and fully tested and has been updated(1) follow these instructions (using the yum method only) OpenVZ quick installation and then download the OpenVZ provided CentOS5 precreated template here. You can use 64 bit CentOS on the hardware node (ct0 in VZ lingo) but your containers should be 32 bit, since there is nothing really to gain, unless you have very special requirements and do not mind the lib64 complications when building or getting software.
Place your downloaded template in /vz/template/cache/ and using the /etc/vz/conf/ sample conf file provided by OpenVZ experiment with creating, stoping, starting, destroying containers from the command line. Test vzlist, vzctl <veid> enter and as many other vz tools as you like. The idea is to understand what you will automate later with unxsVZ.
(1) Using yum, we will only use yum for this install guide. Get it and learn it if applicable to you.
Installing unxsVZ
Follow these list of steps for installing the unxsVZ backend (CentOS only)
Preparing Your Environment
- Make sure your server has an Apache server running. If not install with:
# yum install httpd # /etc/init.d/httpd start
- Make sure your server has a MySQL server running. If not install with:
# yum install mysql-server # /etc/init.d/mysqld start
- Make sure your MySQL server has a root password set, if not use the command below. Please replace newpassword with the MySQL root password you want to use:
# mysqladmin -u root password "newpassword"
- Prepare your yum configuration for using our repository
# wget http://unixservice.com/rpm/i386/unxsyum-1.0-1.i386.rpm # rpm -i unxsyum-1.0-1.i386.rpm
- If you will be using our source to build unxsVZ install required libraries and programs:
# yum install ucidr unxstemplate mysql-devel gcc make
- Prepare your yum repo for rpmforge
# wget http://dag.wieers.com/rpm/packages/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.i386.rpm # rpm -i rpmforge-release-0.3.6-1.el5.rf.i386.rpm
- Install rrdtool from rpmforge, if you like cool per container traffic graphics!
# yum install rrdtool
- Don't forget to setup the services you've installed to start on the next boot!
# chkconfig --level 3 mysqld on # chkconfig --level 3 httpd on
Installing The Software
- Install unxsVZ and unxsUBC:
# yum install unxsvz
- If you get an error run yum clean packages since we are currently correcting the rpm without updating the rpm release number.
- Then add the following crontab entries: (root crontab only.)
# crontab -e (Add these lines) # #unxsVZ entries */5 * * * * sleep 10; nice /usr/sbin/unxsUBC > /dev/null 2>&1; * * * * * nice /var/www/cgi-bin/unxsVZ.cgi ProcessJobQueue >> /var/log/unxsVZ.ProcessJobQueue.log 2>&1;
Post Install Basic Configuration
Install initial unxsVZ data
- Run unxsVZ from the command line
# export ISMROOT=/usr/local/share # /var/www/cgi-bin/unxsVZ.cgi Initialize <the mysql root passwd you set above>
- If that fails you will have to drop the unxsvz database via the mysql command line and check file permissions etc. and do again.
Login and change unxsVZ Root user password ASAP
- Log into unxsVZ backend with the default Root / wsxedc login.
- For example in your browser enter the location of your new unxsVZ backend web interface
https://192.168.22.130/cgi-bin/unxsVZ.cgi
- You should be at the Root user dashboard and see something like:
- Click on the [Main] tab.
- Click on the tAuthorize link.
- You'll see the Root tAuthorize record loaded.
- Press [Modify].
- Update the cPasswd field with the password you want to use.
- Press [Confirm Modify].
- Click any button or link and you will be logged out of the application.
- Re-login with your new Root password.
Start setting up
- Create your first datacenter: Click on the tDatacenter tab.
- Press the [New] button.
- Enter your datacenter name and then press the [Create New Record] button.
- Click on the new Datacenter0 Property Panel Contact and enter contact information (while learning by doing, about tProperty.)
- Create a tNode record for the HN: Click on the tNode tab.
- Press [New], enter your HN full hostname, as displayed by the command below, at the cLabel field. Then finsh with the [Create New Record] button as usual.
# hostname -f
- Note: Don't forget to select at the uDatacenter field the datacenter you've created in the previous step. Then press [Confirm New]
- Make sure your mysql server is not accessible from the network. Edit the /etc/my.cnf.
[mysqld] datadir=/var/lib/mysql # #Add this next line until your firewall is configured or if you really need networking. # #Notes: # unxsVZ rpm binary is compiled to only use the mysql.sock. # You could use a mysqlproxy on this socket to connect to remote mysql servers as # a workaround if you do not want to compile unxsVZ with a modified local.h file. # skip-networking socket=/var/lib/mysql/mysql.sock user=mysql # Default to using old password format for compatibility with mysql 3.x # clients (those using the mysqlclient10 compatibility package). old_passwords=1 [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid
Deploying Containers
For deploying a new container, you have to:
- Login into the unxsVZ backend.
- Add an IP or IPs via tIP.
- Add an OpenVZ nameserver space separated list (Ex. 192.168.0.2 192.168.10.2) via the tNameserver link under the Main tab.
- Add an OpenVZ searchdomain (Ex. mycompany.com) via the tSearchdomain link under the Main tab.
- Click con the tContainer tab.
- Press the [New] button.
- Enter container cLabel (Ex. hosting0) and cHostname (Ex. hosting0.container) at the right panel.
- Select an IP address from the cIP drop-down (if none are available add one or a CIDR range at the tIP tab.)
- Select an OS template (Ex. centos-5-x86-minimal.)
- Select an /etc/vz/conf/ configuration template via uConfig (Ex. minimal.)
- Select the nameserver.
- Select the searchdomain.
- Select the node.
- Select the datacenter.
- Press [Create New Record]. Then you should see something like this
- Press [Deploy [cLabel]] at the left panel and in a few minutes the container will be deployed.
Installing per container traffic graphs
- Make sure rrdtool is installed.
# yum install rrdtool
- Login to the unxsVZ cgi backend, and click on the Main tab. Then click on the tConfiguration link.
- Create a new record cName=cTrafficDirURL and cValue=/traffic with the now familiar [New], [Create New Record] buttons. Leave the node and container selects blank. Only set the datacenter of this configuration record (you should be able to understand why soon.)
- There is a bug in the 1.0-2 rpm that installs the vz_traffic_log.sh with the wrong permissions, fix like so:
# chmod 700 /usr/local/sbin/vz_traffic_log.sh
- There is another bug in both the 1.0-2 and 1.0-3 rpms that must be fixed like so:
# mkdir /var/lib/rrd/
- Edit your /usr/local/sbin/vz_traffic_log.sh scp bottom section. Probably commenting it out until your datacenter set of nodes is operational. That line should be replaced by a allnodescp.sh script that uses root ssh key based passwordless copying of files to all nodes.
- Edit your root crontab. The sleep 20 seconds is to reduce exact minute crontab activity on heavily loaded hardware nodes.
# crontab -e (add the following lines) #unxsVZ per container graphs */5 * * * * sleep 20; nice /usr/local/sbin/vz_traffic_log.sh >> /var/log/unxsVZ.ProcessJobQueue.log 2>&1;
Clone the drives now and get all your servers running and tested
Now that all the needed software has been installed and tested you can start duplicating your drives from the one perfect master drive of your base server. You can use partimage (running from a partimage live CD for example, saving the images on /vz for example) or just use fdisk, mke2fs and dd and cp -ia of /vz content. If very remote and you can't afford or trust your datacenter's remote hands you can remotely (ssh) do the dd thing from your mounted but mostly quiescent base master hard drive to you /dev/sdb for example empty target drive. We recommend that you physically label and use hdparm -i to create a db of all your drives serial numbers for future reference. If you use RAID 1 you only need to duplicate 6 drives, if using the alternative high security rsync mirror drive system mentioned then it will be 13 for our example datacenter system.
Once all your hardware nodes are running and you have tested the swaping of drives and your rsync or RAID mirror backup system. You can continue with more configuration and actual real service container and node setup.
Configuring inter-node passwordless scp (and ssh) sessions
By using standard root ssh-keygen setup. Make sure all nodes can scp from and to each other with no dialogue required. You will then be able to copy configurations and allow unxsVZ to migrate containers among many other things. Hint for the junior system admins among us: An allnodescp.sh script should be installed on all nodes to allow for example allnodescp.sh /etc/mail/sendmail.cf to install that file on all nodes. As well a allnodecmd.sh script can be very helpful, for example allnodecmd.sh "/etc/init.d/sendmail restart" like operations may be very time saving. For added security your node firewalls should only allow this root passwordless communication on your 10.0.0.0/24 datacenter subnet. You can also use scripts like this via non root sudo accounts with the all node ssh key system so as to avoid allnodecmd.sh "rm -rf /" type actions ;).
Configuring node per container iptables rules via unxsVZ.tTemplates
- After you create a container but before deploying (or you can stop the container with red stop button at the bottom of the properties panel) you can setup migratable iptables rules (or anything else that can be scripted and then loaded in tTemplate.) These are the OpenVZ mount/umount scripts that run on the node on container start and stop (see node /etc/vz/conf/VEID.mount and VEID.umount files.)
- To do so you must know the name of the template you will be using and must make sure that the node has everything setup (and setup on reboot) for your mount/umount container scripts to work. If you need the scripts to be work on any node extra care must be taken that they will work on any of your unxsVZ managed hardware nodes.
- As an example we will use the default /etc/sysconfig/iptables rules that come usually with a CentOS 5 server. Here we make sure they are setup and working to use with the openPubVE.mount and openPubVE.umount templates that come with the unxsVZ rpm and source code distributions.
[root@node2vm ~]# /etc/init.d/iptables start Flushing firewall rules: [ OK ] Setting chains to policy ACCEPT: mangle filter [ OK ] Unloading iptables modules: [ OK ] Applying iptables firewall rules: [ OK ] Loading additional iptables modules: ip_conntrack_netbios_n[ OK ] [root@node2vm ~]# iptables -L -n Chain INPUT (policy ACCEPT) target prot opt source destination RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0 Chain FORWARD (policy ACCEPT) target prot opt source destination RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0 Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain RH-Firewall-1-INPUT (2 references) target prot opt source destination ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmp type 255 ACCEPT esp -- 0.0.0.0/0 0.0.0.0/0 ACCEPT ah -- 0.0.0.0/0 0.0.0.0/0 ACCEPT udp -- 0.0.0.0/0 224.0.0.251 udp dpt:5353 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:631 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:631 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited [root@node2vm ~]# iptables -t nat -L -n Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@node2vm ~]# chkconfig --level 3 iptables on [root@node2vm ~]# chkconfig --list cpuspeed 0:off 1:on 2:on 3:on 4:on 5:on 6:off crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off httpd 0:off 1:off 2:off 3:on 4:off 5:off 6:off iptables 0:off 1:off 2:off 3:on 4:off 5:off 6:off irqbalance 0:off 1:off 2:on 3:on 4:on 5:on 6:off lvm2-monitor 0:off 1:on 2:on 3:on 4:on 5:on 6:off messagebus 0:off 1:off 2:off 3:on 4:on 5:on 6:off microcode_ctl 0:off 1:off 2:on 3:on 4:on 5:on 6:off mysqld 0:off 1:off 2:off 3:on 4:off 5:off 6:off network 0:off 1:off 2:on 3:on 4:on 5:on 6:off readahead_early 0:off 1:off 2:on 3:on 4:on 5:on 6:off sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off syslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off vz 0:off 1:off 2:on 3:on 4:on 5:on 6:off
- We have created a test container on the same subnet as the hardware node (we refer to these IPs as public even if they are RFC non routables IPs) As shown here (larger image):
- The image shows that we have already clicked on the Name property and then hit the [New] button at tProperty and created first the mount and then the umount entries (order is not important.)
- Now we are ready to deploy the container. We wait a minute or so for the job queue to run and then we check iptables and vzlist to illustrate what actually happened on the node. We only show part of the standard RH-Firewall-1-INPUT chain that the mount script added for us. Other important things to know about are also shown. Deployed container: (larger image)
[root@node2vm ~]# vzlist CTID NPROC STATUS IP_ADDR HOSTNAME 101 4 running 192.168.22.14 hosting0.container [root@node2vm ~]# iptables -L -n Chain INPUT (policy ACCEPT) target prot opt source destination RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0 Chain FORWARD (policy ACCEPT) target prot opt source destination RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0 Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain RH-Firewall-1-INPUT (2 references) target prot opt source destination ACCEPT all -- 0.0.0.0/0 192.168.22.14 ... [root@node2vm ~]# ls -l /etc/vz/conf/101.* -rw-r--r-- 1 root root 1801 May 11 10:21 /etc/vz/conf/101.conf -rwx------ 1 root root 651 May 11 10:20 /etc/vz/conf/101.mount* -rwx------ 1 root root 649 May 11 10:20 /etc/vz/conf/101.umount*
