VirtualBox does have a command line interface. You can start up headless images and such pretty easily. You could even start images with a VNC server, and control the machine from your browser. One less visual tool.
Oh, I should clarify, I don't use it as a server or anything like that. I have VirtualBox and a Windows XP image for any University work that requires Windows software. Right now that's, reluctantly, Microsoft Visual Studio.