Amazon's EC2 provides a great platform for your applications, it can allow you to scale your application seemingly infinitely. This is great, however there are people like myself who like to have more control over the servers we run. Amazon provides a decent set of prebuilt systems, mostly ready to use, there are also a vast selection of community images, however I don't much like these, call me paranoid but one just doesn't know whats installed. I feel that the best option is to build your own image, the process, although not for a novice, is fairly straight forward. I'm going to attempt to detail what I feel is the best process, this will be based on Debian, and the process will require an existing Debian system.
What you'll need:
- Working Debian System - You could use any other distro, but you'll have to do some research to replace some of these commands
- Amazon AWS Account (Create one free @ http://aws.amazon.com/)
- Amazon Account Number (#1)
- Access Key ID and Secret Access Key (#2 and 3)
- x.509 Cert, you'll need both files pk-xxxx and cert-xxxx (one the tab labeled #4)
- About 2-3 hours depending on your internet connection.
First thing is to install some required packages.
apt-get install debootstrap ruby sun-java6-bin libopenssl-ruby curl zip
mkdir /opt/ami_tools mkdir /opt/api_tools mkdir ~/ami_tools cd ~/ami_tools wget http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip unzip -d ./ ec2-ami-tools.zip cp -R ./ec2-ami-tools-188.8.131.52/* /opt/ami_tools/ #note that your version number may be different than this. mkdir ~/api_tools cd ~/api_tools wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip unzip -d ./ ec2-api-tools.zip cp -R ./ec2-api-tools-184.108.40.206/* /opt/api_tools/ #note that your version number may be different than this. #add the ami-tools and api-tools to the $PATH var PATH=$PATH";/opt/ami_tools/bin;/opt/api_tools/bin" #this bit here is not necessary but I like to clean up cd ~ rm -Rf ami_tools/ rm -Rf api_tools/
Next we need to create a disk image for our AMI, you can use most any size you want from ~350MB (roughly the smallest Debian/Apache server I've been able to build) to sever hundreds of GB (I don't know where the roof is). I'm using a VM to build the AMI it has a second disk attached to it which is where I store the images it is mounted at /ami we'll assign this to variable so that you can copy and past most of my commands. First lets assign the name of our image to a variable so you can copy and past, this also helps reduce typos.
export AMI_Name=debian_lamp export AMI_Path=/ami/$AMI_Name #change this to fit your environment.Now lets create what will become our virtual disk.
dd if=/dev/zero of=$AMI_Path count=10240 bs=1MThe output should be something like
10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 36.9953 s, 290 MB/sThis will create a file of all zeros whose size which 10240 blocks of 1 MB (~10GB), you can adjust this to be however large you'd like. This will take a few minutes, when it's done let create a file system on the new file.
mkfs.ext3 -F $AMI_Paththe output should be something like
mke2fs 1.41.3 (12-Oct-2008) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 655360 inodes, 2621440 blocks 131072 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=2684354560 80 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 31 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override.If you don't want to use ext3 you can choose most any other option. Now that we've got a filesystem its time to mount this disk and get the operating system setup.
mkdir /chroot mount -o loop $AMI_Path /chrootVerify that it is mounted and that everything looks good:
mount /ami/debian_lamp on /chroot type ext3 (rw,loop=/dev/loop1) df -H /ami/debian_lamp 11G 158M 9.9G 2% /chrootNow we can bootstrap Debian unto this disk
debootstrap --arch i386 lenny /chroot/ http://ftp.debian.orgThis one is also going to take a while, it will download and install all of the packages needed for a base system to run, except for the kernel. Once that is finished we'll enter the disk with a `chroot /chroot`. Now everything we do here effects only the virtual disk, which will become our AMI, not our host system. Now that we've created an environment, we can start preparing that environment for Amazon and we can install some of our software now.
chroot /chrootNow, until we leave the chroot everything we do only affects only that environment. Lets first setup some of the 'devices' that are both required, and nice to have.
mount /proc cd /dev MAKEDEV console MAKEDEV std #You'll need these to mount Amazon EBS Volumes. mknod /dev/sdf b 8 80 mknod /dev/sdg b 8 80 mknod /dev/sdh b 8 80 mknod /dev/sdi b 8 80 mknod /dev/sdj b 8 80 mknod /dev/sdk b 8 80 mknod /dev/sdl b 8 80 mknod /dev/sdm b 8 80 mknod /dev/sdn b 8 80 mknod /dev/sdo b 8 80 mknod /dev/sdp b 8 80Now lets set a root password
passwdSetup both networking, and fstab
echo -e 'auto lo\niface lo inet loopback\nauto eth0\niface eth0 inet dhcp' >> /etc/network/interfaces echo -e '/dev/sda1 / ext3 defaults 0 1\n/dev/sda2 swap swap defaults 0 0' > /etc/fstabThere are a few packages which will make things easier and at least one which is required (ssh).
apt-get install ssh locales #locales & tzconfig make it easier to install other packages dpkg-reconfigure locales dpkg-reconfigure tzconfigAt this point you can install any packages you need, if this is going to be a LAMP system:
apt-get install apache2 php5 php5-mysql libapache2-mod-php5 mysql-common mysql-serverYou may also choose to add any code that you would like to have. If you are running a website you can upload it. When you've installed all of the packages that you want, leave the chroot, and unmount it.
exit umount -l /chrootYou'll need to upload both the certificate files that you downloaded from Amazon (pk-xxx & cert-xxx) and we'll assign some variables to them (I keep mine in ~/.ec2)
export EC2_PRIVATE_KEY=~/.ec2/pk-xxxx.pem export EC2_CERT=~/.ec2/cert-xxxx.pemSet the Java environment variables.
export JAVA_HOME=/usr/If your not sure where java is run
Were almost done, I promise ;).
So now were going to bundle the image into an AMI and upload it to Amazon. To do this we'll set a few variables to make things a bit easier.
#These are fake I promise, although if you really want to try feel free. export EC2_ACCOUNT_NUM=12345678901 export EC2_ACCESS_KEY=qwerty123 export EC2_SECRET_KEY=321ytrewq export EC2_HOME=/opt/ami_toolsLets create a variable to store the name of the S3 bucket were going to upload to.
export S3_BUCKET=ami_bucketNow lets create the bundle, this is done by the Amazon AMI tools package we downloaded earlier. This process is going to take a while and is going to consume some disk space in /tmp/.
/opt/ami_tools/bin/ec2-bundle-image -i $AMI_Name --cert $EC2_CERT --privatekey $EC2_PRIVATE_KEY -u $EC2_ACCOUNT_NUMNow we'll begin the upload process, this is also done by the Amazon AMI tools package we downloaded. This is also going to take a while depending on the size of your Image and the speed of your internet.
/opt/ami_tools/bin/ec2-upload-bundle -b $S3_BUCKET-m /tmp/$AMI_Name.manifest.xml -a $EC2_ACCESS_KEY -s $EC2_SECRET_KEYNow we can use the EC2 API to start an instance of our new image so that we can test it.
export EC2_HOME=/opt/api_toolsWe can register the AMI which will allow us to create an instance of it.
/opt/api_tools/bin/ec2-register $S3_BUCKET/$AMI_Name.manifest.xmlThe result from that command will include the AMI Id of the new AMI, it will look like 'ami-s35123'. Now lets run an instance of this image.
#Be sure to use the AMI ID you got from the ec2-register command /opt/api_tools/bin/ec2-run-instances ami-s35123This command will also output some important information, including the instance id in the format 'i-chd3382'. We can use this id to check the status of the instance.
#Again use the id from above. /opt/api_tools/bin/ec2-describe-instances i-chd3382If this isn't your first AMI your going to need to open port 22 so that you can connect to ssh on the new server.
/opt/api_tools/bin/ec2-authorize default -p 22You should now be able to connect to the server, you can get the ip from the 'ec2-describe-instances' command and with the same command you'll see when it's online and ready to use (it's status will be 'running').