Resize a Vagrant VMDK Drive

Using Vagrant as a development environment is great. But you have to watch the available disk space when doing some funky database benchmarking. Last Friday i wrote a script to populate a test database with over 60 million records. And left it running over the weekend. With the idea it would be finished when i returned to the office on Monday. Well it actually was still running. But the Vagrant instance was out of disk space when i logged in. Damn. Should have made sure i had a big disk attached to this instance.

O well. My first idea was to just drop the database that i wanted to use for testing. But due to no space available on the system. I could not drop it. I really couldn’t do anything on the system. Even writing config files failed. So where did all the disk space go?

$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/debian-605-root 10G 9.8G 0.2G 99% /

That’s really not that much space left. And /var/mysql/ibdata1 for the InnoDB tables was taking up 7 GB. And since i have a couple of InnoDB tables i can’t just delete this file. So need to come up with something else. And after a bit of Googling. I came across this post. Which seemed promising. But the most important part is left out. And instead linked to some horrible garbled page. That is completely unreadable. So i had to do some digging to figure this out on my own. For cloning the disks i followed the steps in the article however. The Virtual Box files are located in the user folder in the following path

$ cd ~/VirtualBox VMs/development_*

The first thing i did was convert the VMDK image to a VDI image so we can actually do some partition resizing.

$ VBoxManage clonehd box-disk1.vmdk box-disk1.vdi –format VDI

So now we have a new VDI image. And we need to resize it so it has sufficient space for what we are trying to do. In my case 20 GB is sufficient.

$ VBoxManage modifyhd box-disk1.vdi –resize 20000

Now that the new disk is created it needs to be attached to the vagrant virtual machine. This is easily achieved by doing right click on the VM in VirtualBox and select Settings

Storage > SATA Controller > Add hard disk (select existing and point to the new VDI image)

Now boot the Vagrant image in VirtualBox so we have a screen to work with. When it has booted up. Login.

First create a new Primary partition in the unused space of type Linux LVM (8e) with cfdisk.

$ cfdisk /dev/sda

When the partition table has been written. It’s time to create a new physical volume on the new partition with the help of pvcreate

$ pvcreate /dev/sda3
successfully created

For the next steps we will be needing the Volume Group name. We will get this by running pvdisplay

$ pvdisplay | grep “VG Name”
VG Name debian-605

So now we have all the bits ready. And its time to extend the root partition with the newly created partition.

$ vgextend debian-605 /dev/sda3
Volume group “debian-605” successfully extended

Awesome! next step extending the logical partition

$ lvextend /dev/debian-605/root /dev/sda3
Extending logical volume root to 18.84 GiB
Logical volume root successfully resized

Almost there. The last thing left to do is resize the filesystem to make all space available.

$ resize2fs /dev/debian-605/root
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/debian-605/root is mounted on /; on-line resizing required old desc_blocks = 1, new_desc_blocks = 2
Performing an on-line resize of /dev/debian-605/root to 4937728 (4k) blocks.
The filesystem on /dev/debian-605/root is now 4937728 blocks long.

It’s that simple really. So let’s check.

$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/debian-605-root 19G 8.6G 9.1G 49% /

After this i did not bother to convert the VDI image back to a VDMK image. But if you have to for some reason. You could do that with the commands below. And some manual editing of the VirtualBox xml config.

$ VBoxManage internalcommands sethduuid box-disk1.vdi
$ VBoxManage clonehd box-disk1.vdi box-disk1.vmdk –format VMDK

So that’s it really for resizing the Vagrant hard drive. The only thing left for me was to cleanup the large ibdata1 file. And setup Mysql to use separate ibdata files for InnoDB table.

$ vi /etc/mysql/my.cnf
[mysqld]
innodb_file_per_table

Restart Mysql and dump all needed databases except the system tables like (mysql, information_schema).

$ /etc/init.d/mysql restart
$ echo ‘show databases;’ | mysql -uroot -p | grep -v ^Database$ | grep -v ^information_schema$ | grep -v ^mysql$ | xargs mysqldump -uroot -p –databases > dump.sql

When that’s done. Drop all InnoDB database tables. Remove the ibdata and in_log files from /var/lib/mysql.

$ /etc/init.d/mysql stop
$ rm /var/mysql/ibdata1* /var/mysql/ib_log*

Now we restart Mysql. And reimport the databases.

$ /etc/init.d/mysql start
$ mysql -uroot -p < dump.sql

GIT: Coming Up With a Good Workflow

A new environment to work in always brings new challenges. And this one is no exception. And i will be working with git from now on. I tried switching before. But always went back to subversion. Because of my job. And for somebody who worked with subversion solely for the past… say 5 to 7 years it’s kind of a new concept and takes some time to get a grasp on. Ok ok. It’s just another version control system. But it takes some getting used to.

And two days ago i kinda got stuck in rebase hell. And had to take some time to figure this all out without doing any damage. Which resulted in another problem. There simply is no revert command like in subversion. But git is pretty smart about this kinda things. So getting the branch full of failed merges back to it’s original state can easily be achieved with two commands and some concentration.

First check the reflog for a point where to start from

$ git reflog

From the output pick the point where to revert back to. Which looks something like HEAD{2}. And run

$ git reset –hard HEAD{2}

So back to square one. My main problem was that i had not kept my branch up to date with the remote repository. Which you can do with the pull command while on the master branch. Or with the rebase command while working on a local branch. Rebase is git’s alternative way of bringing changes together. So the idea behind it is that when your local branch is up to date. The final merge will become as straight forward as possible.

I finally managed to get my changes merged in to master. But did not feel very confident about the whole process. So I need to come up with a steady work flow. And stick to it. To not waste time and run into these rebase disasters.

This is what i have so far. And seems to be a solid routine. Start with a nice new checkout of the project at hand. And branch it for the changes to be done.

$ git clone git://some.repository.url:repos.git
$ git branch dev-branch

After working on the newly created branch for a while. Some changes are bound to be ready to be pulled into master. Keeping an eye on the build logs or commit messages should work fine. When this happens it’s time to bring those changes into the dev-branch. After pulling them into master.

$ git checkout master
$ git pull
$ git checkout dev-branch

If there is a need to rebase the branch with for instance master. This is the moment. This might cause non-fast-forward issues. But i have not completely figured out why i run into these issues every now and then. A way to solve this might be to use the –force switch when pushing to remote. But i am not a big fan of forcing things like this.

$ git rebase

If everything is fine some more work can be done on the dev branch. Until a new change in master is ready to be pulled in. In which case it’s rinse and repeat the previous part. When the changeset is ready to be merged back into master. Switch to master bring it up up o date with a pull. And merge the changes from the dev-branch.

$ git checkout master
$ git pull
$ git push dev-branch

Again when all is fine all changes will be available in master. If something goes wrong during git-rebase and / or merging. Git will kindly state so. And some manual merging with git’s mergetool will have to be done. But that’s something for another day.

Workaround for Running WoW With Patch 5.0.4 on Ubuntu

After a fresh install i couldn’t get WoW to work. I’m not that big on playing games lately. But this bugged me. Good thing we have the interwebs.

Starting the battle.net agent manually before firing up the WoW launcher solved the launcher crashing.

$ wine ~/.wine/drive_c/users/Public/Application\ Data/Battle.net/Agent/Agent.1267/Agent.exe –nohttpauth &
$ wine ~/.wine/drive_c/Program\ Files\ \(x86\)/World\ of\ Warcraft/Launcher.exe &

And installing vcrun2008 with winetricks solved the problem with the updater hanging.

Setup a Basic PHP Development Environment With Vagrant and Chef-solo

So there are plenty of posts about this out there already. And this is nothing new i guess. But i am going to post it anyway. This weekend i spend some time playing with vagrant. And every setup guide seemed to be missing some steps the make the whole process a bit of a hassle. So i compiled my own steps in the post below. The goal was to create a basic Debian box for my PHP development. From which i could serve my local development files.

So to start of we need some packages.

Virtualbox

To create the virtual development image

NFS packages

To make sharing files between host and guest easy as pie

Vagrant

To package the virtual image and manage newly created box

$ sudo apt-get install virtualbox vagrant nfs-common nfs-kernel-server

Now is a good moment to download any Linux distro you want. I chose Debian. As it´s the most common in my day to day work.

$ wget http://cdimage.debian.org/debian-cd/6.0.5/i386/iso-cd/debian-6.0.5-i386-netinst.iso

Now launch virtualbox and create a new virtual machine. Just do what you normally do when you create a VM. But make sure to choose VMDK `Virtual Machine Disk´ as the format. The rest should be ok. When the VM is created start it up and choose the freshly downloaded Linux ISO to start the installation.

I only installed the base system + the SSH server. And when that’s all done. And the VM is running. Login as root and install some packages vagrant needs.

$ apt-get install sudo ruby ruby-dev libopenssl-ruby rdoc ri irb build-essential ssl-cert curl rubygems puppet

I won’t be setting up a Chef server to manage the vagrant images. But i will be using Chef-Solo to manage installing packages. So let’s install Chef.

install chef

Setup the opscode package repositories for APT

$ echo “deb http://apt.opscode.com/ `lsb_release -cs`-0.10 main” | tee /etc/apt/sources.list.d/opscode.list
$ gpg –keyserver keys.gnupg.net –recv-keys 83EF826A
$ gpg –export packages@opscode.com | tee /etc/apt/trusted.gpg.d/opscode-keyring.gpg > /dev/null

And finally install the opscode keymanager and and of course Chef.

$ apt-get update
$ apt-get install opscode-keyring
$ apt-get install chef

The installer will ask for the chef URL. I used none because i will be using chef solo

Now we need to add a user for vagrant to connect with.

$ adduser vagrant
passwrd : vagrant
$ groupadd admin
$ usermod -a -G admin vagrant

Make sure the admin users can sudo without a password.

$ visudo
%admin ALL=(ALL) NOPASSWD: ALL

Disable DNS for the SSH server. This should be a performance gain.

$ vi /etc/etc/ssh/sshd_config
UseDNS no

Change the MOTD to something nice

$ bash -c “echo ‘Sweeet! A Vagrant box cooked by Chef!!’ > /etc/motd”
$ chmod 0777 /etc/motd

Switch to the vagrant user to setup the insecure vagrant public SSH key.

$ su vagrant
$ mkdir -p ~/.ssh
$ chmod 0700 ~/.ssh
$ curl -o ~/.ssh/authorized_keys https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub
$ chmod 0600 ~/.ssh/authorized_keys

That’s it for the VM. Now it’s time for vagrant to do some magic. First we need to create a vagrant package from the newly created VM.

$ vagrant package –base devbox
$ mv package.box devbox.box

Done! Let’s test it.

$ vagrant box add devbox devbox.box
$ mkdir test && cd test
$ vagrant init devbox
$ vagrant up
$ vagrant ssh

Voila ssh’d into the VM

$ exit
$ vagrant halt
$ vagrant destroy

So that’s all fine and dandy. But a base system with SSH is not that useful. So that’s where Chef comes in. Like i mentioned earlier i won’t be using a Chef-Server. I will be using Chef-Solo. And will be doing a very basic setup.

A Chef install script is called a cookbook. And they can be found all over the place. But the main repository is over here (find more here https://github.com/opscode/cookbooks). So in our vagrant folder we create a folder called cookbooks.

$ mkdir cookbooks
$ cd cookbooks

And we install some basic cookbooks. Plus a very simple one i hacked together to install the latest dotdeb package repositories.

https://github.com/tlenss/misc/tree/master/chef/cookbooks/dotdeb

$ git clone https://github.com/opscode/cookbooks

This will be enough for now. And will allow for some basic PHP development. To enable these package we have to edit the Vagrantfile file.

$ vi Vagrantfile

Add these lines for apache forwarding and NFS sharing

config.vm.share_folder “www”, “/var/www”, “/dev/location”

uncomment the :hostonly line so we can acces apache from our local box. And setup a hosts file mapping

config.vm.network :hostonly, “192.168.164.123”
$ sudo echo “192.168.164.123 dev.box” >> /etc/hosts

Configure chef’s cookbook path and add some recipes from the cloned cookbooks.

config.vm.provision :chef_solo do |chef|
  chef.json = {
    "mysql" => {
      "server_root_password" => "somepassword"
      "bind_address" => "127.0.0.1"
    }
  }

  chef.cookbooks_path = "cookbooks"
  chef.add_recipe("dotdeb")
  chef.add_recipe("dotdeb::php54")
  chef.add_recipe("openssl")
  chef.add_recipe("apache2")
  chef.add_recipe("apache2::mod_php5")
  chef.add_recipe("apache2::mod_rewrite")
  chef.add_recipe("mysql")
  chef.add_recipe("mysql::server")
  chef.add_recipe("memcached")
  chef.add_recipe("vim")
  chef.add_recipe("php")
  chef.add_recipe("php::module_curl")
  chef.add_recipe("php::module_mysql")
  chef.add_recipe("php::module_memcache")
  chef.add_recipe("php::module_sqlite3")
end

Time to call vagrant again. And this time the startup takes a bit longer. All the selected packages get installed. And once that’s done. And a fresh development box is waiting.

$ vagrant up
$ vagrant ssh

Let’s check PHP

$ php -v
PHP 5.4.6-1~dotdeb.0 (cli) (built: Aug 19 2012 08:45:58)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2012 Zend Technologies

What about apache?

$ ps -aux | grep apache
root 790 0.1 0.2 5352 2580 ? Ss 23:40 0:00 /usr/sbin/apache2 -k start
www-data 793 0.0 0.1 5124 1764 ? S 23:40 0:00 /usr/sbin/apache2 -k start
www-data 797 0.0 0.2 546408 2412 ? Sl 23:40 0:00 /usr/sbin/apache2 -k start
www-data 798 0.0 0.2 546408 2404 ? Sl 23:40 0:00 /usr/sbin/apache2 -k start
www-data 811 0.0 0.2 546408 2424 ? Sl 23:40 0:00 /usr/sbin/apache2 -k start

Nice! And Pointing the browser to http://localhost:8080 loads the NFS shared folder up through Apache. That’s it for now. Time to sleep…

Adding Colors to PHP CLI Script Output

I always loved writing CLI scripts. And in PHP this is no different. I write a lot of CLI importers, scrapers, reporters, etc. And sometimes i want a bit more funk in my output then the standard black and white. When output is important i will add some colors to make things more clear. And this is not hard at all.

First of it’s probably a good thing to gather some of the available colors. A good resource is available here. Some of the colors and their corresponding codes can be found below.

Black 0;30
Blue 0;34
Green 0;32
Cyan 0;36
Red 0;31
Purple 0;35
Brown 0;33
Light Gray 0;37
Dark Gray 1;30
Light Blue 1;34
Light Green 1;32
Light Cyan 1;36
Light Red 1;31
Light Purple 1;35
Yellow 1;33
White 1;37

So how do i output colors you ask? Well that’s just plain easy. It’s a bit like setting up colors in your linux PS1 env variable.

Display a line in red for instance can be simply done by ‘echoing’ from the script with some extra formatting for bash to pickup. I will try to explain everything in the line below a bit later.

    echo "\033[31m some colored text \033[0m some white text \n";
    echo "\033[32m some colored text \033[0m some white text \n";

So what’s going on here. First we use an escape character so we can actually define a output color. This is done with \033 (\e). Then we open the color statement with [31m. Red in this case. The “some colored text” will be the text outputted in a different color. And after that we have to close the color statement with \033[0m.

Easy as cake… that’s all!

Abbywinters Is Looking for New Developers

If you’re looking for a new challenging and exiting Senior Webdeveloper position. Don’t look any further. If you already think you have the job of your dreams. Think again!

abbywinters.com (NSFW) is one of the largest and most ethical, highly rated, well designed, and successful erotic websites in the world today. abbywinters.com is the WINNER of the AVN 2011 Awards for Best Membership site!

And we are looking to hire a new talented webdeveloper to expand our small team. What would you think about joining our small Agile team of highly qualified professionals?

You will be creating sexy, exiting and game changing experiences for the web, work for one of the industry leaders. And just be part of an awesome company. Some of the jobs key elements are:

  • Implementing development projects

  • Leading informal mentoring during day-to-day work

  • Contribute to design of development projects

  • Track, reduce, and prevent technical debt in Web Development projects

Motivated by principles of social responsibility, we deliver provocative media by embracing imagination, creativity and emerging technologies. Our models, customers and business partners are inspired by our fervid passion.

Our experienced staff use state-of-the-art content production facilities to produce 10 shoots a week from concept to finished art, utilizing the most advanced digital capture, post production and delivery systems in the world.

You will be working directly with our Web Dev Manager, Lead developer and colleagues in the web dev team. We need each individual to contribute for us to continue as a pioneer in our industry.

If you posses a “Can do” attitude. Would like to work in the center of Amsterdam. And are able to identify your self in the criteria below. You might want to head over to our career portal for a more detailed description.

Technical competencies – Required

  • High level of skill with PHP 5

  • High level of skill with Object Oriented Programming

  • High level of skill with HTML/CSS

  • High level of skill with JavaScript

  • High level of skill with Internet Applications

  • Moderate level of skill with Unit Testing and Test Driven Design

  • Moderate level of skill with MySQL

  • Moderate level of skill with Windows XP operating system

  • Experience with the GNU/Linux operating system

  • Competent with Revision Control systems (Subversion)

  • Bachelor of Science in Computer Science, or equivalent experience

  • Zend Certified Engineer, or equivalent experience

  • At least 5 years experience in Web Application Development

Technical competencies – Desired

  • Moderate level of skill with the Apache HTTP server

  • Good understanding of the Model-View-Controller pattern

  • Good understanding of the ActiveRecord Object-Relational-Mapping pattern

  • Familiarity with Agile software development practices (Scrum)

  • E-commerce

  • Agile development experience

Regular Expression Name Based Matches

Normally when i write regular expressions with for instance PHP´s preg_match. I will use the standard $matches array to catch the result of pattern matches. This array has a normal numeric index for each match found. And looks something like

Normal preg_match
$matches = array();
preg_match("/^Get(.+)Repository$/", "GetFooBarRepository", $matches);
var_dump($matches);
array(2) {
  [0] => string(19) "GetFooBarRepository"
  [1] => string(6) "FooBar"
}

And there is nothing wrong with that. Except for the fact that numeric indexes are not always easy to work with. And it does not look all that clean in the code itself. SO last week my LD pointed out the fact that the likes using name based indexes for the matched patterns. And this is pretty sweet. I have seen it before but never bothered to adopt it myself. And the result looks like

Name based index preg_match
$matches = array();
preg_match("/^Get(?<repositoryname>.+)Repository$/", "GetFooBarRepository", $matches);
var_dump($matches);
array(3) {
  [0] => string(19) "GetFooBarRepository"
  'repositoryName' => string(6) "FooBar"
  [1] => string(6) "FooBar"
}

And i have to say. It looks a lot cleaner. So i added this to my bag of tricks. And will be using it from now on.

PHP: Only Variables Should Be Passed by Reference

Last week i got this error while doing some coding with a new library. And at first i didn’t quite get what was causing this. The line in question looked like

$url = reset($file->getPaths());

So how can that throw an error like “Only variables should be passed by reference” you might ask? Well as it turns out to be. PHP functions don’t like arguments returned from another function. As with this case. The ->getPaths() method holds a reference to the returned array. Which it shouldn’t but that’s another thing.

So i guess from now i will doing the thing below. Or check for references beforehand!

$paths = $file->getPaths();
$url = reset($paths);

Using SSH Key Authentication With EGit in ZendStudio

For the past few months i have been switching some projects over to git from Subversion. And have been trying to get the hang of all the command line options available. And i will be doing that for a while longer until i get completely comfortable. And for communication to the remote git server i have been using SSH key authentication. Which works smoothly just like it did with Subversion.

But i wanted to check out the GIT support available in Zend Studio 9. And hit a problem pretty quickly. But i will describe that below. First i will create a local clone of my git project.

$ git clone ssh://[somehost]/~/git/project.git

To test if everything is working i do a test commit. If that succeeds if push it out to the remote master.

$ cd project.git
$ touch TEST
$ git commit
$ git push origin master

So that works fine. Now time to see how Zen Studio handles this. To create a project i use the Create from Git option. And select the local checkout i just created. This will read the whole repository configuration. And you are basically done from here. But as i mentioned earlier, i had some difficulties getting things running smoothly. I discovered that when it was time to push changes to the master repository.

When i used the Push to Upstream option. I was greeted by a login panel that seemed to have selected the correct SSH key and user to perform the login. But when i typed the password, it just kept asking for the password. Again and again. Hmm. That sucks! The password was correct. I tried with a newly created key. No luck either. The last thing i tried was updating to a nightly build of Egit found here. But this offered no solution either.

After reading a couple of complaints i found this bug report for the EGit eclipse plugin. The thread contains a solution for the login issue i was having. Gotta love Google!

Apparently the problem has to do with the encryption algorithm used to create the SSH keys. In this case the EGit plugin (which uses Jsch to do the SSH communication) was having problems with AES encrypted keys. And to solve the problem the Jsch library should be replaced with a newer version to make things work again.

So lets download this JSch library and update it manually. The library (JSch v 0.1.46) can be found here.

$ cd ZendStudio9
$ find . -name ‘*jsch*’ -type f

Found it plugins/com.jcraft.jsch_0.1.41.v201101211617.jar. So let’s try to update that.

$ cp plugins/com.jcraft.jsch_0.1.41.v201101211617.jar plugins/com.jcraft.jsch_0.1.41.v201101211617.jar.backup
$ wget http://sourceforge.net/projects/jsch/files/jsch.jar/0.1.46/jsch-0.1.46.jar/download
$ mv jsch-0.1.46.jar plugins/com.jcraft.jsch_0.1.41.v201101211617.jar

After restarting ZS everything worked fine again. Another problem solved! Thanks to the guys who posted in the EGit bug thread. Some good community Karma here! Time for other things. Hope it helps!

Recruiter Rant

While doing our routine sprint retrospective today. We got interrupted by our office manager Wendy. An important phone call for me. Hmm… Should have known.

I picked up the phone. And the guy (Amoria Bond) on the other end immediately started his sales pitch. O shit another one of those nasty recruiters. So after listening to him for a few seconds i quickly interrupted him. Told him i was not interested in a new job at all. And that he was a jerk and extremely unprofessional for calling me at the office. Completely unaware (or maybe intentionally?) what impact this might have if for instance my manager picks up the phone. Besides it’s just plain rude.

Now don’t get me wrong. It’s not a rant for nothing. I don’t hate recruiters. And have worked with some professional ones in the past. I’ve always enjoyed communicating with linda-lotte while she was still working for Recruit4it. And the guys at Starapple are OK as well. But this guy today really pissed me off with his aggressive unmannered approach. It’s a shame i didn’t catch his name while i was in rant mode. But please don’t ever call me again.

Update:

It seems to be some form of new tactic to call developers in the office where they work. This happened a couple of times more after this post. So this is for the next recruiter that calls me in the office. I’ll personally come over and kick your ass!!

Copyright © 2015 - Thijs Lensselink - Powered by Octopress