About

Ian Ozsvald picture

This is Ian Ozsvald's blog, I'm an entrepreneurial geek, a Data Science/ML/NLP/AI consultant, founder of the Annotate.io social media mining API, author of O'Reilly's High Performance Python book, co-organiser of PyDataLondon, co-founder of the SocialTies App, author of the A.I.Cookbook, author of The Screencasting Handbook, a Pythonista, co-founder of ShowMeDo and FivePoundApps and also a Londoner. Here's a little more about me.

High Performance Python book with O'Reilly View Ian Ozsvald's profile on LinkedIn Visit Ian Ozsvald's data science consulting business Protecting your bits. Open Rights Group

26 July 2012 - 17:32Using ZeroFree to shrink a VirtualBox Linux Image

My development Ubuntu image inside VirtualBox was using too much space to store empty but non-zero disk blocks on its virtual drive. This sucked space from my laptop’s SSD (which is already not big enough!). Shrinking it by zeroing the blocks took a little bit of effort.

Inside VirtualBox if I boot my Ubuntu 11.04 instance ‘df -h’ reports 14GB used of a 27GB virtual drive (where the .vdi image is 22GB). Back on the host laptop I see that the .vdi file is 22GB which suggests that 8GB could be freed up. The problem is that the unused blocks have been used and then deleted but not reset to 0, so VirtualBox keeps them allocated.

The solution is simple using these instructions:

  • Delete any snapshots in VirtualBox so you’re just dealing with 1 .vdi file (not sure if this helps but it seemed sensible)
  • Get the PartImage ISO (about 300MB), it includes zerofree (unlike my usual favourite SysRescueCD)
  • Mount the PartImage ISO as a CD drive in VirtualBox, boot the virtual machine, it’ll start PartImage
  • Run “$ fsck /dev/sda1″ to confirm that the Ubuntu hd (/dev/sda1) is clean
  • Make a temporary mount point “$ mkdir /mnt/ss”
  • Mount the drive read-only and without modifying /etc as our host image is read-only “$ mount -n -o ro -t ext4 /dev/sda1 /mnt/ss”
  • Check the space that’s used “$ du -h -s /mnt/ss” – for me this reported 14GB of a 27GB drive
  • Let zerofree reset all the unused blocks “$ zerofree /dev/sda1″ – took about 5 minutes for a 22GB .vdi on my SSD
  • Close the virtual machine
  • Back on the host laptop ask vboxmanage to compact all the zero’d blocks “$ vboxmanage modifyhd –compact /home/ian/<path>…/myimage.vdi” and remember to use a fully-qualified path (not ./myimage.vdi) else it reports an error
  • Wait 2 minutes
  • Check the .vdi image size – mine shrank from 22GB to 17GB
  • Reboot the virtual image to confirm that the disk is unaffected (in the virtual machine it still reports 14GB used on a 27GB partition)

This solution looks easier but requires the dynamically allocated virtual image to expand to its fullest size which won’t work for me – right now my laptop has less free space than the virtual image is allowed to use!


Ian applies Data Science as an AI/Data Scientist for companies in ModelInsight, sign-up for Data Science tutorials in London. Historically Ian ran Mor Consulting. He also founded the image and text annotation API Annotate.io, co-authored SocialTies, programs Python, authored The Screencasting Handbook, lives in London and is a consumer of fine coffees.

1 Comment | Tags: Life, Ubuntu

2 August 2011 - 11:57Dell E6420 with Ubuntu 10.10 (Maverick Meerkat) 32 bit

Having hacked away with Natty Narwahl for a few weeks I’m regressing to the 10.10 distribution provided by Dell here. Installation took 20 minutes, it allowed me to use the previous ext4 partition (I had to edit it using the advanced configuration and set the ext4 partition’s mount point from blank to ‘/’). I formatted the partition too for good measure. I made sure to reload the package list (via Synaptics) and let it fetch updates.

Running ‘uname -a’ reports that this is 32 bit: “Linux ian-Latitude-E6420 2.6.35-30-generic-pae #54-Ubuntu SMP Tue Jun 7 20:28:33 UTC 2011 i686 GNU/Linux”

Next I followed the instructions here to get access to sound and the touchpad (on the fresh install the ‘pad worked but had no side-scroll, now it has side-scroll). I used my previous instructions to get the edgers version of the NVIDIA drivers (not the ones on the Dell site), Optimus was already disabled and the NVIDIA drivers ‘just worked’. I had to install the Dell sound driver but then it also ‘just worked’. Flash with sound seems to have worked out of the box too.

Wifi was a pain – the Dell links didn’t work but downloading this (in Synaptic – the pae version) via this for my Broadcom BCM 5800 (ID: 0a5c:5800) gave me wifi on a reboot. I’ve also upgraded Firefox 3 to 5 via this.

Suspend and hibernate seem to be stable (unlike before with the 11.04 install – it randomly got stuck and lost my desktop). Rather pleasingly although I was getting a gig of Dropbox over Wifi and compiling new sources the battery tool reported 6 hours of battery life (which seemed true-ish, maybe 4 hours would have  been right, though I did have the screen on darkest as it was very late in the night). This beats the max 2 hours I got before with 11.04.

Overall regressing to the 10.10 build from Dell seems to be the right move. Update two weeks later – using the Dell image is definitely the right thing to do, everything ‘just works’ like it is supposed to. I get 4-6 hours battery life using the NVIDIA graphics card as my primary display.

Update – I’ve uploaded a modified script that disables the touchpad for a fraction of a second when you’re typing. This is necessary as the ALPS touchpad identifies itself as a PS/2 mouse rather than a trackpad due to proprietary drivers. The script is in my github repo as Dell_E6420_Touchpad_AutoDisabler. It contains minor fixes from Philip Aston’s excellent version here.

Update (Nov 2011) – Having used 10.10 for 2 months I’ve got some problems that I’ll list.

  1. About 1 in 20 lid closes do not cause the suspend behaviour to start. The result is that the laptop stays ‘on’ with the lid shut. After an hour it tries to go into (I guess) hibernate, for some reason it gets stuck. Next it gets hot, the fans run on full and after a while it is cooking at 80 degrees in my laptop bag, merrily eating the battery. If I get it in time I can open the screen – the backlight is on but nothing responds and I have to force power-off (holding the power button for 5 seconds). If I don’t get it in time it just kills the battery. Upon a reboot it boots a fresh session and everything is fine, sans all the previous session info (this hasn’t yet led to corruption)
  2. About once a month the machine freezes during use. It has happened just after a clean boot (after logging in, before doing anything). It has happened after days of use and many suspends. The behaviour is a total system lock, the screen doesn’t update, no mouse etc. A force power off is required.
  3. The in-built camera normally works with Skype, sometimes it fails to start and a reboot is required. The picture is grainy and doesn’t cope with low lighting conditions (I haven’t tried this on Windows). Using an older Logitech QuickCam Pro 9000 I get a bright, clear picture even in low light conditions for Skype.
  4. Power usage with the NVIDIA card on (Optimus off), using VirtualBox, with wifi and a bright screen is about 3 hours.

It is hard to know if this is a hardware fault (the BIOS-based self diagnostics which run for 30 mins report no problems) or a software fault. I’m inclined to think it is 10.10 and/or the Dell changes. I’m planning on trying 11.10 next in the hope that the SandyBridge chipset is better supported.

My take-home message so far is that if the manufacturer doesn’t support your OS (Dell only partially support Ubuntu), don’t buy from them. I believe HP might have been a better purchase.


Ian applies Data Science as an AI/Data Scientist for companies in ModelInsight, sign-up for Data Science tutorials in London. Historically Ian ran Mor Consulting. He also founded the image and text annotation API Annotate.io, co-authored SocialTies, programs Python, authored The Screencasting Handbook, lives in London and is a consumer of fine coffees.

No Comments | Tags: Life, Ubuntu

13 July 2011 - 19:25Dell E6420 with Ubuntu 11.04 (Natty Narwahl) 64 bit

I’ve just treated myself to a quad core (8 virtual core) Dell E6420 with 8GB RAM, 128GB SSD, NVIDIA NVS 4200M GPU (with integrated Intel GPU) and the high spec screen. It is rather nice. It comes with Win 7 Pro installed (fine for some libraries I’ll need and maybe some MS Office), I’ve just installed Ubuntu 11.04 64 bit. Obviously there were some hiccups…

Update – I’ve regressed to the 10.10 drivers from Dell (see: E6420 with Ubuntu 10.10) which seems to fix all the problems I had with 11.04.

Out of the box Ubuntu Natty Narwahl installed just fine, it took about 10 minutes from CD. It reported on first boot that I had an incompatible graphics card. This is because the Optimus NVIDIA technology (which swaps between low-power mode on the Intel card and high-power GPU mode with the 4200M) only works on Windows. I rebooted, dropped to the BIOS and disabled Optimus, after this I got no more warnings from X Windows. Details on Optimus here.

On my next boot I asked the Additional Drivers system to enable my NVIDIA card. It installed, then I had to run ‘sudo nvidia-xconfig’. This worked fine. Note that at first (before disabling Optimus) I tried this and got a ‘VALIDATION ERROR: Data incomplete in file /etc/X11/xorg.conf.’ error – just disable Optimus and that problem goes away. You might need to manually move your /etc/X11/xorg.conf.backup file to xorg.conf if you’ve lost graphics along the way.

The touchpad (as a basic mouse), sound and wifi worked fine. Flash ran in YouTube but there was no sound – under the Sound dialog I had to go to Output and choose ‘Internal Audio Analogue Stereo’ rather than ‘HDA NVIDIA Digital Stereo (HDMI)’, in the background the YouTube video that was playing suddenly played through the speakers.

The touchpad doesn’t have multitouch features yet – there’s a proprietary driver on Windows which doesn’t exist on Linux yet (though progress is being made). Details here. If you touch the touchpad and it makes a click and you don’t like the behaviour, disable it here. Personally I’m happy with the touch behaviour.

I upgraded to the latest NVIDIA drivers using:

  • sudo add-apt-repository ppa:ubuntu-x-swat/x-updates
  • sudo apt-get update
  • sudo apt-get install nvidia-current
  • sudo apt-get install nvidia-settings

After a reboot I could see the latest drivers (via Synaptic). Running a video in VLC took no extra CPU (meaning that the work is done on the GPU via VDPAU). There is an upcoming project called Bumblebee that will let us use the lower-power Intel GPU for normal graphics and will only switch to the NVIDIA card for intensive work (I want it for CUDA programming) but for now it looks like a bit of a faff. I’m just going to leave the NVIDIA 4200M running full time.

I’m also happy to see that HDMI support worked out of the box – I plugged in a cable, then had to go to the NVIDIA Settings tool to Auto-detect the monitors, then enabled TwinView and had a double-width monitor setup. The built-in Monitor tool didn’t see my extra monitor (but I guess this is all controlled by the NVIDIA stuff).

I’ve had about 3 hours for this session on the regular 6 Cell battery, that included downloading a lot of stuff and rebooting a number of times. This seems reasonable given that the NVIDIA 4200M is power hungry and runs all the time. Apparently in Windows I’d see up to 7 hours on the same hardware if Optimus is enabled. Ho hum.

I’m very impressed with the SSD – it is silent and everything feels much snappier. It was worth spending a few hundred extra pounds and losing a lot of space, the experience is far nicer. The screen is also beautiful (though the viewing angle isn’t amazing – but fine for single-use).

EDIT – installing CUDA 4 takes 10 minutes with these great instructions. It is fun to see the GPU clocking in at 90 degrees C whilst running randomFog, smokeParticles and the nbody demos.

EDIT – this CPU Frequency meter is nice as is this CPU/MEM/GPU meter.

EDIT – installing mongodb auto-starts it, it is controlled as a system service using ‘sudo [start|stop] mongodb’ as detailed here.

EDIT – installing matplotlib on Ubuntu 11.04 was a touch annoying. ‘pip install’ got a really old version (0.91!). Instead I grabbed the src for 1.0.1 and then manually had to install libfreetype6-dev2.4.4-1ubuntu2 and libpng12-dev. After that the usual setup.py process worked fine.

EDIT – to fix the hang-on-reboot issue that I’ve noticed I followed this and added “reboot=pci” as noted to /etc/default/grub (and then ran ‘sudo update-grub’). Now reboots work correctly (previously only a Shutdown would work correctly).

EDIT – I was running laptop-mode-tools and PowerTop reported 13-15W usage. I’ve disabled it for now as I have a Suspend/Hibernate bug, without laptop-mode PowerTop is reporting 15-18W usage (both after boot, doing almost nothing – the CUDA card is power hungry!).


Ian applies Data Science as an AI/Data Scientist for companies in ModelInsight, sign-up for Data Science tutorials in London. Historically Ian ran Mor Consulting. He also founded the image and text annotation API Annotate.io, co-authored SocialTies, programs Python, authored The Screencasting Handbook, lives in London and is a consumer of fine coffees.

2 Comments | Tags: Ubuntu

20 June 2009 - 14:54Running Skype on Ubuntu + QuickCam Pro 9000

I use Skype on my Win desktop and MacBook as a matter of course now, I rather like to use the video feed via the MacBook when co-working with my team on our screencasts.

Since the desktop box usually runs Ubuntu 9.04, I wanted to try my new QuickCam Pro 9000.  The short story is – it works.  I had to faff for 15 minutes figuring out the right config for Skype, the winning combo with my old Audigy 2 ZS sound-card was:

  • Sound In: QuickCam Pro 9000 (hw:Q9000,0)
  • Sound Out: Audigy 2 ZS [SB0350] (hw:Audigy2,0)
  • Ringing: Audigy 2 ZS [SB0350] (hw:Audigy2,0)

To test that the Quickcam was running first I used ‘lsusb’ to list the recognised USB devices:

ian@NewMESH:~$ lsusb
Bus 001 Device 011: ID 046d:0990 Logitech, Inc. QuickCam Pro 9000

and then I ran cheese and had to choose 640×480 to 960×720 to test the video.

My version of the QuickCam 9000 has USB BCD 0×0008 which makes it one of the buggy versions (buggy only for Linux <sigh>), instructions of figuring out if you have one of the fingered models is here.  Thankfully I’ve not seen any bugs yet in an hour’s use of audio and video Skype calls.  The quick command to get the required ‘bcd’ info is:

ian@NewMESH:~$ lsusb -d 046d:0990 -v | grep bcd
  bcdUSB               2.00
  bcdDevice            0.08

I found some notes for Ubuntu Skype along with the Linux Changelog.

Update – on a subsequent boot Skype had forgotten my audio settings and couldn’t find the webcam.  I reset the audio settings (as per above notes), then ran cheese and verified that my webcam was on /dev/video0 (I had to change the resolution before cheese picked up an image though), then I reconfigured Skype and all was well.


Ian applies Data Science as an AI/Data Scientist for companies in ModelInsight, sign-up for Data Science tutorials in London. Historically Ian ran Mor Consulting. He also founded the image and text annotation API Annotate.io, co-authored SocialTies, programs Python, authored The Screencasting Handbook, lives in London and is a consumer of fine coffees.

2 Comments | Tags: Ubuntu

16 May 2009 - 12:52Enabling Marble Mouse Scroll-wheel on Ubuntu 9.04, PulseAudio

In my recent upgrade from Ubuntu 8.04 to the latest 9.04 I lost the scroll wheel on my Marble Mouse.  The solutions is in this Logitech Marblemouse USB help page at Ubuntu via this forum entry.

I’m also annoyed by PulseAudio (again).  Once again it takes over from ALSA but doesn’t output any sound, currently I have to kill pulseaudio in the task manager and then set the volume to 100% (it gets set to 0% by pulseaudio).  So, now I’ll uninstall pulseaudio like I did before.


Ian applies Data Science as an AI/Data Scientist for companies in ModelInsight, sign-up for Data Science tutorials in London. Historically Ian ran Mor Consulting. He also founded the image and text annotation API Annotate.io, co-authored SocialTies, programs Python, authored The Screencasting Handbook, lives in London and is a consumer of fine coffees.

1 Comment | Tags: Life, Ubuntu

10 May 2009 - 12:12Ubuntu Jaunty 9.04 on Dell 9400 Laptop

We’ve just installed Ubuntu Jaunty 9.04 onto Emily’s Dell 9400 laptop.  As ever there are some wrinkles, I’ll note them here.

First – installation was fine and easy.  I used a 10Gb root partition and the rest (about 100Gb) as /home.  Sound worked straight away.

Video is, as ever, a bit more interesting.  This machine has an ATI Mobility x1400 card.  Ubuntu uses the open-source ‘ati’ driver by default.  This works but video tears which is distracting.  TV Out works (we run at 1024×768 to the LCD TV).

I’d assumed we could use the closed-source ‘fglrx’ driver (I’ve used it in the past) but ATI doesn’t support the r500 chipset (as used in the x1400) from the current release of the X display system, as used in the latest Ubuntu. So – we can’t use ‘fglrx’ and we can only use the ‘ati’ driver.

The only improvement I’ve found so far is to use the EXA (notes) switch in xorg.conf, EXA is an improvement to the XAA graphics subsystem.  By using it we can use the ‘og’ (open-gl) graphics system in mplayer along with ‘xv’.  Previously if I tried ‘og’ it wouldn’t go fullsize, only ‘xv’ would go fullsize.  Both still tear though.  Hmm, further reading suggests that Jaunty was released with EXA enabled by default as it fixed many problems.

I’ve also tried: AGPMode to 8 – no change. AccelDFS – no change.  EXAVsync to True on the radeon man page – no change.  These bits – no change.

To try – ‘man exa’, man page for radeon.  Look for other xorg.conf options.  Also  ideas.  The log in /var/log/Xorg.0.log seems to be fine although AIGLX reports that DRI2 isn’t supported.  More notes and more.

Anyone else have any suggestions? Drat – it looks like it is dual-head support that isn’t quite fixed!  The current 2D support is tear-free for this card but I think only for the primary screen, not the LCD output (which is what I’ve been viewing all along).  Looking at the laptop’s screen I don’t see any tears.  Back to searching.

For the following message see this fix: “The application ‘NetworkManager Applet’ (/usr/bin/nm-applet) wants access to the default keyring, but it is locked.”


Ian applies Data Science as an AI/Data Scientist for companies in ModelInsight, sign-up for Data Science tutorials in London. Historically Ian ran Mor Consulting. He also founded the image and text annotation API Annotate.io, co-authored SocialTies, programs Python, authored The Screencasting Handbook, lives in London and is a consumer of fine coffees.

No Comments | Tags: Ubuntu

18 April 2009 - 21:39Installing IE6 on to Ubuntu Jaunty 9.04 with ies4linux

I’m pleased to say that my earlier post on Installing IE6 on Ubuntu Gutsy (7.10) needs only a minor modification to work with Jaunty 9.04.

For Jaunty you just need to replace any references to ‘edgy’ with ‘jaunty’ on the ies4linux Ubuntu install page.

The ./ies4linux installer took 15 minutes to complete under VirtualBox (I was running Jaunty Release Candidate for a preview) – slow but…it worked.  Given that IE6 is over 10 years old, this is rather impressive.

Too see wine in action running Internet Explorer 6 on the Jaunty RC, watch this screencast of ie6update that I recorded last night in ProCasts:


Ian applies Data Science as an AI/Data Scientist for companies in ModelInsight, sign-up for Data Science tutorials in London. Historically Ian ran Mor Consulting. He also founded the image and text annotation API Annotate.io, co-authored SocialTies, programs Python, authored The Screencasting Handbook, lives in London and is a consumer of fine coffees.

6 Comments | Tags: Ubuntu