Addendums

The invisible Programmer Antagonist who lives in my head, who is an expert in all technologies including VirtualBox and knows all Best Practices under the sun, asks me “Why do you want to rename a virtual hard drive? Have you considered that the renaming of hard drive is not a feature of VirtualBox for a reason?” The Programmer Antagonist inside my head doesn’t actually know the reason because he’s self-constructed and I personally don’t know why it’s a bad idea, but I give developers the benefit of the doubt so “surely there is one”; feel free to provide your ideas. Needless to say, though, this inner critic has made me drag my feet about making this, my last of the “VirtualBox on the Command Line” posts (for now - this covers the scope of the stuff I’ve messed around with). I feel nervous about sharing because it’s a hack, but at the same time, hacks can often save the day, especially if it’s as a result of a feature that is not built in but you still need to use (I’ve also had to develop similar hacks for my company’s version of redmine for features the devs have probably left out of the UI for good reason - like removing watchers that are not myself from tickets and deleting write-lock comments that won’t get emailed to stakeholders as a result of the write-lock.)

The truth is, a few months back I ran into a pretty good use case for renaming hard drives. I cracked into a pre-configured VirtualBox system and found a set of machines with names akin to these:

stageweb_a
stageweb-b
stageweb-c.some_subnet
stageweb_d

When I listed the hard drives, I’d get something like this:

stageweb_a
stageweb_2
stageweb-c
stageweb-d.some_subnet

Really, I’ve just made up some names here and introduced some arbitrary inconsistencies, but take a look at these and you get an idea that there was supposed to be a consistent naming scheme to these machines and hard drives and that the current names clearly don’t follow it.

Consistency is pretty easy to add to the machines themselves because VM renaming is easy. Not so with renaming the hard drives. And the other problem with renaming the hard drives isn’t just for consistency’s sake - when I dug deeper, I found configurations between machine and hard drive that were doomed to confuse any person who tried to administer the system in the future. By this I mean that a machine like stageweb_a would be pointing to the hard drive named stageweb-d.some_subnet instead of the logical stageweb_a hard drive. So things were bonkers.

Anyhow, let’s say we want to rename a hard drive from “horrible_name.vdi” to “excellent_name.vdi”.  Its parent machine (the VM) has already been renamed to “excellent_machine”.

Start by shutting down the parent machine and detaching the hard drive:


VBoxManage -q controlvm excellent_name poweroff


VBoxManage -q storageattach excellent_name --storagectl "IDE Controller" --port 0 --device 0 --type hdd --medium none


Now, clone the hard drive and give the cloned hard drive the new name. This is the “hack” for renaming.


VBoxManage clonehd horrible_name.vdi excellent_name.vdi --format VDI --remember


(The –remember option “opens” the hard drive and otherwise registers it with the VirtualBox system on the host machine.)

Now, mount the new hard drive to the host machine, and restart the host machine:


VBoxManage -q storageattach excellent_name --storagectl "IDE Controller" --port 0 --device 0 --type hdd --medium excellent_name.vdi
VBoxHeadless -s excellent_name &


Confirm that you can access the machine with the cloned hard drive attached. If this is the case, you can remove the old hard drive:


VBoxManage -q closemedium disk horrible_name.vdi
rm /home/addie/.VirtualBox/HardDrives/horrible_name.vdi


(Obviously, you’d be in your own home directory here and not mine.)

This is clearly not perfect, but given everything I’ve documented so far, it’s actually pretty repetitive of stuff that’s already been done. Still, it’s worth it to document the actual process just to show how the renaming of a virtual hard drive is possible - albeit by cloning, which preserves the content but not unique identifiers like the UUID. For me, the content is more than enough, so this has worked for me in cleaning up my VirtualBox systems.

Thanks for reading and hope these commands have been of some help!

Virtualization is really useful for webservers, which are pretty low-resource but are good to have in numbers for the sake of redundancy. So you can make much better use of a resource-rich system by virtualizing it and putting several webservers on it - if that’s your particular task of the moment.

That’s one perk for the marriage of virtualization and webservers. The one I like the best, though, is easy duplication. See, setting up a webserver on FreeBSD takes a LOT of time. I know there are some great advantages to being able to directly compile all your software via ports (more sophisticated configurations, etc.), but we simply aren’t using any of those advantages with regards to our dependencies. It also strikes me as pretty crazy that basic programs like bash, vim, and screen don’t come pre-installed. You can literally take a nap while waiting for vim to install.

As a result, getting a single webserver set up can take up to a full workday - between time-consuming port installations and myriad CPAN dependencies (our web app has a Perl foundation). And it hurts the programmer’s aesthetic of “don’t repeat yourself” to do this same webserver setup across several machines - although this is exactly what I’ve had to do in the past. Luckily, we can almost entirely skip that hefty setup and install with VirtualBox.

There are two ways to go about this - by exporting and then importing a VM under a new name, or by cloning a virtual hard drive and attaching the clone to a newly created VM. I actually prefer the latter, because the configuration of the VM itself isn’t necessarily a step I want to skip - but the installation of all the software on the hard drive is. I suspect that most people would prefer the import / export option, but I just have more of a mostly-inexplicable attachment to hd cloning (much like how I have always preferred while loops over for loops despite their individual strengths / weaknesses). Anyhow, I’ll show how both are done in this entry.

To start out with: make sure the vm you are either exporting or whose hard drive you are cloning is powered off. That said, you won’t need to detach the hard drive you are cloning from the VM, if you decide to clone.

Here’s how I’d export my vm “addievm”. Note that you have to specify the name of the file you’re going to output to, and .ovf is the common extension.


VBoxManage export addievm --output addievm.ovf


Exporting isn’t instantaneous, but you’ll see a little text prompt iterating from 0% to 100% to give you an idea of your progress. If you’ve done any configuration on your machine since setting it up (like installing the operating system and some software), this can take a few minutes, and longer for systems with very large hard drives.

The export (for a single machine - you can actually export multiple machines at once, but I’m not particularly interested in it) will give you three files - in this case, they’ll be addievm.ovf, addievm.mf, and addievm.vmdk. The .ovf has the VM configuration, and the .vmdk is the virtual hard drive (a format that is compatible with virtualization platforms like VMWare, more info here). The .mf file contains the UUIDs of the virtual machine and the virtual hard drive. You’ll want to keep all three in the same place (i.e. at the same directory level) so VirtualBox can locate the additional two files (.mf and .vmdk) during an import of the .ovf. Note also that the .vmdk takes the name of the virtual hard drive itself, not the name you have given to the .ovf file - so if I had a hard drive named addiehd.vdi instead of addievm.hdi and ran the same command as above, my resulting files would be addievm.ovf, addievm.mf, and addiehd.vmdk.

To import - in this use case, to import as a copy on the same machine as the original - you can run the VBoxManage import command in the same location that you exported to. You’ll want to start with this command:


VBoxManage import -n addievm.ovf


This command doesn’t actually perform the import. Instead, because of the use of the -n option (–dry-run also works), you’ll get a description of what the import will look like with its defaults set, and the options you can use to change those defaults. Here’s what mine looks like for addievm:


Disks: vmdisk1 52428800000 -1 http://www.vmware.com/specifications/vmdk.html#sparse addievm.vmdk 1261356544 -1
Virtual system 0:
0: Suggested OS type: "FreeBSD_64"
(change with "--vsys 0 --ostype "; use "list ostypes" to list all possible values)
1: Suggested VM name "addievm_1"
(change with "--vsys 0 --vmname ")
2: Number of CPUs: 1
(change with "--vsys 0 --cpus ")
3: Guest memory: 1024 MB
(change with "--vsys 0 --memory ")
4: Network adapter: orig Bridged, config 3, extra type=Bridged
5: CD-ROM
(disable with "--vsys 0 --unit 5 --ignore")
6: IDE controller, type PIIX4
(disable with "--vsys 0 --unit 6 --ignore")
7: Hard disk image: source image=addievm.vmdk, target path=/home/addie/.VirtualBox/HardDisks/addievm.vmdk, controller=6;channel=0
(change controller with "--vsys 0 --unit 7 --controller ";
disable with "--vsys 0 --unit 7 --ignore")


The only one I really want to change in my typical use case is the name of the vm. So my command to actually import the copy will look like:


VBoxManage import addievm.ovf --vsys 0 --vmname addievm_copy


You’ll get similar output as you did with the “dummy run”, but this time it will be followed up with the text “progress bar” as with the export, incrementing from 0% to 100%. Expect a similar wait on import as you did on export.

You can also import as many times as you’d like. You’ll see from the “Hard disk image” information that the hard disk created on the import is renamed like addievm_1.vmdk, addievm_2.vmdk, etc. You aren’t able to change how this is named from the import. I really don’t like having so little control over the naming of hard disks with regards to keeping a collection of virtual machines and virtual hardware organized, but I have found a renaming workaround hack that I’ll be documenting in another post soon.

As for taking the second approach, start by creating a new VM and configuring it as you see fit. The settings don’t have to be the same for a VM with a cloned hard drive, with the exception of HD-related settings (like the operating system). Then, go ahead and clone the disk:


VBoxManage clonehd addievm.vdi addie-clone.vdi --format VDI --remember


The options for this command are very similar to those for creating a hard disk from scratch. The “–remember” option will automatically register the cloned hd with your system of virtual machines and hardware, so if you use the command


VBoxManage list hdds


the cloned hard drive will show up.

Then it’s just a matter of attaching the cloned hard drive to the new VM you’ve created. You’ll need to start up the cloned machine and change things like hostname, networking configuration, etc. so there aren’t any conflicts when both machines are running at once on the same network, but such changes are independent to the actual systems that are being cloned and are out of scope for this documentation.

As I mentioned before, one of the issues you might discover here is that, especially with import and export, you don’t have the full control over naming of your virtual hardware that you’d like. Although there is support in VBoxManage for renaming a machine, there isn’t support for renaming a piece of hardware. I’ve found ways to get around this that I’ll document in my next post; it’s a bit hacky but leaves me with a system that I feel like I actually have full control over.

The last few entries have explained, command-by-command, how to create virtual machines and prepare them for OS installation, with some details about how to unwind this work (i.e. delete everything cleanly). In the process of writing the entries, I’ve learned new details about how to attach / remove storage devices (like hard drives and DVDs) in VirtualBox 3.1.2, which requires an extra layer of abstraction.

I thought I’d stick all the commands together in sequence for easy reference. Of course, I’m doing the same thing as a shell script right now to create some real machines of my own. As a result, I’m using the “-q” option in all commands to suppress the standard headers that appear after the typical VBoxManage command is run.

Creating a VM, attaching a hard drive and OS ISO, enabling networking and VRDP
This particular VM has 50 GB hard drive, 1024 MB RAM, FreeBSD 8.0 for AMD 64 as its OS, main bridged interface named eth0, and VRDP port 3390.


VBoxManage -q createhd --filename addievm.vdi --size 50000 --format VDI --variant Standard

VBoxManage -q createvm --name addievm --ostype FreeBSD_64 --register

VBoxManage -q storageattach addievm --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium /home/addie/8.0-RELEASE-amd64-disc1.iso


After running all of these commands, you should have a VM that is ready to be started up for OS install.

If something went horribly wrong, you may want to remove all your work and start again (I’ve had to do this more times than I’d care to admit). So here are the commands for unwinding all of the work you’ve just done without leaving extra traces lying around (the one thing left registered after this process will be the DVD used for install; I don’t remove it because I end up re-using it soon enough).

Removing the VM I just created - and its attachments


VBoxManage -q storageattach addievm --storagectl "IDE Controller" --port 0 --device 0 --type hdd --medium none

VBoxManage -q storageattach addievm --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium emptydrive

VBoxManage -q unregistervm addievm --delete


That’s all for the summary - now on to new VirtualBox tasks!

This is at once one of the simpler and one of the most crucial of these posts. Your virtual machine is going to be pretty useless unless you have an operating system installed on it, and specifying the OS type when creating the machine doesn’t take care of the OS install for you.

The two things that deserve documentation in this process are the act of mounting the install ISO onto the VM (and unmounting it after install), and accessing the VM remotely to be able to view the interface for the OS install. The actual details of any specific OS install will be left out here because they’re outside of the scope of this documentation.

The caveats introduced in the Tricky Bits post apply here considerably, so if I’m seeming too brief on some details, refer to that post for clarification.

To get started with mounting an ISO, download and copy over the relevant file to your user’s home directory. In the case of my “addievm”, I’m installing a copy of FreeBSD 8.0 for a 64-bit processor, and that’s the filename I’ll be using in these examples.

In older versions of VirtualBox, here is how I would create the virtual representation of the DVD and then attach it to the machine:


VBoxManage registerimage dvd 8.0-RELEASE-amd64-bootonly.iso
VBoxManage list dvds
VBoxManage modifyvm addievm --dvd {UUID for ISO}


I found that I always had to use the UUID to get a successful result with this; there isn’t a good “name” to reference like with a VM or a harddrive. You can get the UUID really quickly with the “list” command that comes second in this series.

In newer versions of VirtualBox, where you are attaching storage media to a controller versus directly to the virtual machine, the command is slightly different. This command assumes that the controller has been created and the hard drive has been attached to port 0; essentially the commands listed in Tricky Bits.


VBoxManage storageattach addievm --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium /home/addie/8.0-RELEASE-amd64-bootonly.iso


Now that the DVD is mounted, we can start the machine using the commands listed in VBoxManage and VBoxHeadless. But there’s no way to access the interface using a headless server. This is where VRDP comes into play.

In Creating and deleting VMs, I enabled VRDP and assigned it a port number. This port is important for accessing the VM remotely.

I use Ubuntu Desktop for my development workstation and found that the default remote desktop application which comes with the system doesn’t work for RDP support. So I installed the program rdesktop from Synaptic, and it does the job. To launch, type into your terminal:


rdesktop {host ip}:{guest port}


Presuming my host is on a local network with static ip 192.168.0.50, and the vrdp port for my VM is 3390, I’d run:


rdesktop 192.168.0.50:3390


If all goes well, a UI for your interface will pop up and you can work through the setup accordingly.

Once installation is complete you’ll want to unmount the DVD so the system stops booting into the installation. In older versions of VirtualBox:


VBoxManage modifyvm addievm --dvd none


In newer versions:


VBoxManage storageattach addievm --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium emptydrive


If you’d like to suspend VRDP support at any time (maybe you want to move back to the OSE version), make sure you have networking and SSH configured on your virtual machine so you can access it without needing a graphical interface.

Next entry will jump into more sophisticated work: cloning virtual hard drives in order to save time on configuration of near-identical systems.

I am creating some new VMs on a fresh install of Ubuntu 9.10 (Karmic/Kreepy Koala), so now is a great time to extract and organize each step used to create a new VM on the command line, and double-check all my documentation in real time. Here we go.

As with my last post, the machine I’m creating in these comments is called “addie_vm” but can be replaced with the machine name of your choice.

I typically start by creating the virtual hard disk for my VM. To keep things from getting confusing, I always name the hard disk with the same name as my VM. Note that renaming virtual hard disks is MUCH harder than renaming a VM, so name carefully!

VBoxManage createhd --filename addie_vm.vdi --size 100000 --format VDI --variant Standard

Some notes: VDI is a standard virtualbox format and what virtual hard disks default to in the GUI. Size is specified in MB, so what I am creating here is a 100 GB hard disk. The “Standard” variant means that the actual space used on your physical hard drive will grow to a cap of 100 GB versus taking up those 100 GB from the get-go. If you’d like your virtual hard drive to take up its capacity on the physical hard drive, use the “Fixed” variant instead (This is a useful option if you’re planning very carefully about how the entire host system’s resources are being used).

Next, create the VM with some basic features configured. The VMs I am creating will have FreeBSD installed on them, so I run:

VBoxManage createvm --name addie_vm --ostype FreeBSD_64 --register

The –register arg allows you to skip a step; you won’t have to run VBoxManage registervm to get the system to recognize your new machine. If you’re using another OS, you’ll need to include a specific string specifying your OS type after the –ostype arg. You can see your options by running:

VBoxManage list ostypes

Now, glue two pieces together, as it were - attach your hard drive to the VM you just created.*

VBoxManage modifyvm addie_vm --hda addie_vm.vdi

At this point the “basics” are in place, but here are a couple of options I set before I go about installing the OS on the new VM (and I will explore some of these in greater detail later on):


VBoxManage modifyvm addie_vm --memory 1024


This command sets the RAM for your machine - the number following –memory should be in megabytes. The VM will be created with a default of 128 MB RAM, which is a bit on the small side, so adjusting it before you get to tinkering around with your machine is recommended. As with hard disk space, you’ll want to know the total RAM of your host system, and the total RAM of your other VMs, and plan accordingly. If too much host RAM is allocated, and the host system is being used for anything other than running VMs, you’ll run into some serious performance problems on your host machine.

Enabling networking is also important. Networking is one of my weakest areas knowledge-wise, and past versions of VirtualBox have made getting internet working on VMs pretty complicated, especially if you are using only the command line. If you’re using an older copy of VirtualBox (2.0), you’ll probably want to look elsewhere for documentation, starting with help.ubuntu.com’s documentation.

Luckily, things are easier in newer versions of VirtualBox. I use a bridged network connection, so the VM uses the same hardware as my host machine. To see the available bridged interfaces, run:

VBoxManage list bridgedifs

If you’re going with system defaults with regards to networking, you’ll probably be using “eth0″. Interface name in hand, you’ll then run:


VBoxManage modifyvm addie_vm --nic1 bridged
VBoxManage modifyvm addie_vm --bridgeadapater1 eth0


The first command configures your machine to use a bridged adapter. The second command points the adapter to the appropriate interface. The “1″ in –nic1 and –bridgeadapter1 specifies that this is the first adapter; you can configure multiple adapters if needed.

Finally, and I will revisit this later, I like enabling vrdp - the VirtualBox Remote Desktop Protocol - and assigning a port for the machine in question (the port is optional - VBoxHeadless will assign one for you if you don’t have one already).  Note that these options are only available in the closed source version of VirtualBox (unfortunately).


VBoxManage modifyvm addie_vm --vrdp on
VBoxManage modifyvm addie_vm --vrdpport 3390


To close, let’s trash my creation. Getting rid of a VM is actually somewhat tricky from the command line, because you must first detach the VM from its hard drive, and to make sure you aren’t cluttering up the VirtualBox configuration and leaving around garbage, you’ll want to unregister both VM and hard disk. Here’s the series of steps. Start with your VM powered off:

VBoxManage modifyvm addie_vm --hda1 none

This command disconnects the virtual hard drive from your virtual machine.

You can go ahead and unregister the hard drive and VM now:


VBoxManage unregistervm addie_vm --delete
VBoxManage closemedium disk addie_vm.vdi


You’re done with the VM now - thanks to the –delete flag, your VM is gone. The hard drive is still lingering though, and if you want to save disk space, you’re now in the position to safely delete it. Assuming your VirtualBox configuration lives in .VirtualBox under your home directory, you can remove it this way:


rm ~/.VirtualBox/HardDisks/addie_vm.vdi


… and that’s all for this entry. Next up: cloning hard drives so you don’t have to go through system setup on several machines in a row.

* If you’re using VirtualBox CSE 3.1.x, you may run into issues here - I have.   I’ll try to write about resolving them in a future post; if I do I’ll link to it in this aside.

This is the first of many entries I’m hoping to write on using VirtualBox on the command line (i.e. with the server edition of your favorite unix-based OS; in my case Ubuntu) after having about a year’s worth of experience using the GUI version of the software. I’m no expert with VirtualBox, but I’ve found that it’s been a tremendous aid with my work, second only to git’s branching in allowing me to switch rapidly between several projects at once with minimal overhead.

Documenting usage of the software on the command line is great territory for some blog posts too. I keep pulling out the same note-covered, four-page printout of the VBoxManage usage string, because extenuating circumstances keep requiring machine administration. Instead of adding to the mess note-wise, I figured I’d do a little cleanup of my documentation and summarize the commands I’m using the most often.

The two command-line programs I’m using to administer VirtualBox are VBoxManage and VBoxHeadless. VBoxManage is the be-all, end-all program for creating, deleting, and configuring VMs and all their components. VBoxHeadless is the program used to run VMs on a headless server (such as Ubuntu Server Edition, like I’m using). It’s the server-edition equivalent of using the program VirtualBox on the command line, which is the “standard”, GUI version (and what you’ll see in your process list if you’re running a vm on the GUI version). (If you try running VirtualBox on the command line with a headless machine, like I sometimes do when I’m feeling especially tired and not thinking straight, you’ll get X11-related complaints.)

I inherited some pre-configured vms with this server, so the first thing I needed to do was learn how to start and stop my machine. Starting a virtual machine is possible with VBoxHeadless or with VBoxManage; most of the documentation I read recommended VBoxHeadless for my use case.

Here’s how I start a virtual machine called “addie_vm” on the command line:

VBoxHeadless -s addie_vm &

-s is a shortened version of the option –startvm, which is hopefully self-explanatory. You follow it up with either the vm name or its UUID (a unique identifier for the VM that I use in other circumstances but is much less human-readable).

You will want to run the process in the background using the ampersand, or you’ll lose access to your terminal.

Here’s the rough equivalent with VBoxManage (it appears to start up a few additional processes along with VBoxHeadless):

VBoxManage startvm addie_vm --type headless

Running in the background isn’t needed in this case.

I typically power off my virtual machine using the command

VBoxManage controlvm addies_vm poweroff

This is obviously preferrable to killing the VM process, which is the equivalent of manually powering off or unplugging your physical machine. Which is not to say I haven’t had to do either on occasion.

Next entry I will document how I’ve been creating and removing virtual machines and their hard disks, something I’ve had to do a lot of in recent days.

Share in the comments if there’s anything about this documentation that is missing to the detriment of my system; there are so many options within a program like VBoxManage that it’s easy to miss details (or glaze over them when reading documentation).