Kyran and I are starting work on a new project – strongsteam offers a web API with artificial intelligence and data mining tools. The goal is to make it easy for you to do things like:

• get the text out of images using optical character recognition
• determine whether two images look the same and if one object (e.g. a certain book or a can of coke) can be found in another
• use natural language processing to analyse, cluster and compare text
• extract text from audio (e.g. to pull out keywords from podcasts)
• use machine learning on text to derive new data

If you’d like to join the closed alpha then visit strongsteam and add your email to the announce list on the homepage.

We’ve started with Python bindings which make it easy to talk to the strongsteam web service. Initially we’ll wrap open source tools that we’ve used along with lots of our own A.I. data mining tools from years of work in my Mor Consulting A.I. consultancy.

At EuroSciPy last week I demo’d using O.C.R. to extract the words from plant labels at Wakehurst Place gardens so you can lookup the plant on Wikipedia once you’ve taken a photo like this one:

Now we’re looking at applying O.C.R. to conference name-badges, this will be a bit of a mash-up from data used in our SocialTies conference app and Lanyrd.com‘s data. Next we’ll look at image matching and some text processing tools.

---

Ian applies Artificial Intelligence as an Artificial Intelligence Researcher for companies (Mor Consulting), co-founded the StrongSteam A.I. datamining toolkit, co-authored SocialTies, programs Python, writes The Screencasting Handbook and is also a sea-side dweller and consumer of fine coffees.

My updated High Performance Python tutorial is now available as a 55 page PDF. The goal is to take you on several journeys which show you different ways of making Python code run much faster (up to 75* on the CPU, faster with a GPU).

UPDATE this talk is superseded by my High Performance Python 1 tutorial from PyCon 2012.

This is an update to the 49 page v0.1 I published three weeks ago after running the tutorial at EuroPython 2011 in Florence.

Topics covered:

• Python profiling (cProfile, RunSnake, line_profiler) – find bottlenecks
• PyPy – Python’s new Just In Time compiler, a note on the new numpy module
• Cython – annotate your code and compile to C
• numpy integration with Cython – fast numerical Python library wrapped by Cython
• ShedSkin – automatic code annotation and conversion to C
• numpy vectors – fast vector operations using numpy arrays
• NumExpr on numpy vectors – automatic numpy compilation to multiple CPUs and vector units
• multiprocessing – built-in module to use multiple CPUs
• ParallelPython – run tasks on multiple computers
• pyCUDA – run tasks on your Graphics Processing Unit
• Other algorithmic choices and options you have

The improvement over the last version (v0.1) is that I’ve filled in all the sections now including pyCUDA (there are still a few IAN_TODOs marked, I hope to finish these in a future v0.3). I’ve also added a short section on Algorithmic Choices, link to the new Cython prange operator and show the new numpy module in PyPy.

The source code is on my github page. The original slides are on slideshare too. If you’re after a challenge then at the end of the report I suggest some ported versions of the code that I’d like to see.

The report is licensed Creative Commons by Attribution (please link back here) – I’ll also happily accept a beer if you meet me in person! If you’re curious about this sort of work then note that I offer A.I. and high performance computing consulting and training via my Mor Consulting.

Update – ShedSkin 0.9 adds faster complex number support. I haven’t added it to the report yet, evidence in the ShedSkin Group suggests it gets closer to the non-complex-number version (i.e. you don’t have to do more work but you get a nice speed boost whilst still using complex numbers).

Update (Nov 2011) – Antonio and Armin posted a note which explains some of the slowness in PyPy and show how it is competitive, under the right conditions. Armin also contributed a C version which shows PyPy to run as fast as C (for their chosen configuration).

---

Ian applies Artificial Intelligence as an Artificial Intelligence Researcher for companies (Mor Consulting), co-founded the StrongSteam A.I. datamining toolkit, co-authored SocialTies, programs Python, writes The Screencasting Handbook and is also a sea-side dweller and consumer of fine coffees.

I’ve just given my yearly lecture at Sussex Uni talking about Artificial Intelligence in the real-world. It details some of my exploits over the last 10 years. The presentation is below, I’ve also linked to some of the videos below.

For my real world examples I covered cars and a few other projects. Audi have their automated racing car which climbs Pikes Peak in 27 minutes (a human racer does it in 17 minutes). I haven’t managed to find a video of the race, there is this video taken earlier in the year.

More interestingly Google have an automated car that drives the streets (has video) – this is a real automated vehicle really driving on the real roads. This is darned impressive. Sooner or later we’ll have low-accident-rate automatic cars which are far cheaper to drive than human-power-cars, this’ll have big economic implications (but could be years off yet).

IBM’s Watson played Jeopardy recently (has video), it is pretty scary watching the machine beat humans at tricky general knowledge questions.

I also mentioned Word Lens (a real time OCR-based translation system using a phone camera) and I demo’d Google’s Voice Translator with some French to English on my Android-based Galaxy S.

Finally I linked to my own project – Social Ties does data mining and natural language processing to give you a mobile ‘people radar’ – it helps you find interesting people at events and places. We’re in alpha at present, sign up on the site if you’d like to join the beta.

At the end of the talk I spoke about some local events that will help people move forwards with A.I. company ideas, these are:

• £5 App
• dotBrighton
• OpenCoffeeSussex
• Likemind Coffee

---

Ian applies Artificial Intelligence as an Artificial Intelligence Researcher for companies (Mor Consulting), co-founded the StrongSteam A.I. datamining toolkit, co-authored SocialTies, programs Python, writes The Screencasting Handbook and is also a sea-side dweller and consumer of fine coffees.

Python Text Processing with NLTK 2.0 Cookbook (Amazon US, UK) is a cookbook for Python’s Natural Language Processing Toolkit. I’d suggest that this book is seen as a companion for O’Reilly’s Natural Language Processing with Python (available for free at nltk.org). The older O’Reilly book gives a lot of explanation for how to use NLTK’s component, Packt’s new book shows you lots of little recipes which build to larger projects giving you a great hands-on toolkit.

Overall the book is easy to read, has a huge set of sample recipes and feels very useful. I’ll be referring to it for our upcoming @socialties mobile app.

You’ll need to download NLTK, you can also refer to some sample articles at Packt’s site and get Chapter 3 as a free PDF (see below). The author is Jacob Perkins, his blog links to many related articles, he also has a nice ‘how it started‘ article.

Here are my thoughts on the book. Disclosure – I was sent a free copy of the book by Packt for review, the thoughts below are entirely my own.

Chapter 1: Tokenizing Text and WordNet Basics

If you haven’t tried tokenising text before you may not realise how complicated it can be (expressing even basic rules for English is jolly hard!). This chapter has a good overview of tokenisation and the excellent WordNet library. Filtering stopwords (low value words like ‘the’, ‘of’) and synsets approaches (synonym groups in WordNet) are also covered. The word similarity measure was new to me, the book certainly throws up nice nuggets.

Chapter 2: Replacing and Correcting Words

Stemming approaches are covered, the goal is to find common root words (e.g. “running”, “runs” and “run” can each have “run” as their stem) to simplify your input text. Synonym replacement (e.g. converting “bday” to “birthday”) and negating words using antonyms are nicely treated. Babelfish is provided through NLTK for translation and the PyEnchant spellchecker is introduced.

Chapter 3: Creating Custom Corpora (sample PDF chapter)

This chapter discusses MongoDB (a NoSQL document store) as a way to store your own corpora in NLTK’s format, it also introduces part of speech tagging. File locking using lockfile is mentioned in case you’re using multiple processes (discussed later).

Chapter 4: Part-of-Speech Tagging and Chapter 5: Extracting Chunks

I was less interested in this part, I’ve had to extract Named Entities before and there’s a nice discussion in Chapter 5.

Chapter 6: Transforming Chunks and Trees

The section on filtering out insignificant words using part of speech tags was interesting (i.e. using the Determiner tag DT to filter words like “a”, “all”, “an”, “that”, “that”). Cardinals (numbers) are discussed, I liked the recipe for swapping noun cardinal phrases so e.g. “Dec 10″ becomes “10 Dec” (whilst “10 Dec” doesn’t change).

Chapter 7: Text Classification

This feels like it will be useful – bag of words classification and the Naive Bayes Classifier are discussed (along with some other classifiers). Here the author starts to build a movie rating classifier. Precision and Recall are explained nicely. A high-information classifier is built, this is useful as we can then remove low-information words (those that aren’t biased to a single class in the classifier) which can improve classification results. Combining classifiers to further improve results is also covered.

Chapter 8: Distributed Processing and Handling Large Datasets

This chapter has promise – I wasn’t aware of the share-nothing distributed execution engine execnet. Redis is also used, Jacob builds towards a distributed word scoring engine which uses Redis as a single storage system. I’ve yet to use Redis but really want to hook it into our future @socialtiesapp, distributed processing will definitely be on the agenda too.

Chapter 9: Parsing Specific Data

This is a little gem, tucked at the end of the book. Ages ago I’d come across a date parsing module (which I then forgot about), having needed it recently I was super-happy to see dateutil discussed. It makes the parsing of different date formats incredibly easy and also handles timezones.

The timex module in NLTK is introduced (I’d never heard of it before) – it takes a fuzzy reference to a date or time and marks it up. An example would be “let’s go sometime <TIMEX2>this week</TIMEX2>”, you can then extract the fuzzy reference and decide how to interpret it in your application.

lxml, Beautiful Soup and chardet (another gem) are used to write a web page scraper.

Overall I recommend this book, if you have the original O’Reilly book (and you really ought to) then this makes for a great companion. I also spotted these two other reviews.

---

Ian applies Artificial Intelligence as an Artificial Intelligence Researcher for companies (Mor Consulting), co-founded the StrongSteam A.I. datamining toolkit, co-authored SocialTies, programs Python, writes The Screencasting Handbook and is also a sea-side dweller and consumer of fine coffees.

I’m rather excited to have received a review copy of Packt’s new NLP book “Python Text Processing with NLTK 2.0 Cookbook“, it is based around Python’s Natural Language Processing toolkit.

I’ve been using the O’Reilly book for over a year, I’m curious to see what’s different between the two. I’ll post a full review once I’ve been through it.

---

Ian applies Artificial Intelligence as an Artificial Intelligence Researcher for companies (Mor Consulting), co-founded the StrongSteam A.I. datamining toolkit, co-authored SocialTies, programs Python, writes The Screencasting Handbook and is also a sea-side dweller and consumer of fine coffees.

Over the weekend at BarCampBrighton5 I demonstrated a quick visualisation that Kyran and I built over breakfast in Berlin last Friday. It looks like:

To see it yourself open Bar Camp Brighton 5 Visualisation using Chrome or WebKit (it’ll work in Firefox but might be rather slow). It is interactive so it is worth opening, about 60 people are shown here.

If you reload the page you’ll see the force directed graph bouncing around as it settles into a low energy configuration. The nodes are people attending the event, edges are friend links to other people at the event. The image sizes for the nodes reflect the number of links a person has at that event.

As you can see above Jot (the main host and co-organiser) is most connected at the event. Two people aren’t following anyone at the event, they’ve been pushed to the bottom left of the window.

You can drag nodes using the grab-handles (blue circles) or move the entire graph by dragging the image.

For a larger example (80 people) see the Flash on the Beach 2010 Visualisation:

Here you can see that seb_ly is the most connected, closely followed by niqui and bitchwhocodes. At the bottom left is a sub graph of two nodes – these two people follow each other but don’t follow anyone in the main graph.

In both cases the data is extracted from the relevant Lanyrd pages (BCB5, FOTB), friends for each attendee are read from Twitter and then a graph is built as a JSON dictionary which links nodes (screen_names) to friends (lists of screen_names). Ready to run Python source code is at github: LanyrdViewerUsingProtoVis.

Both of these links should work on a mobile device but they’ll be awfully slow (they’re useless on my iPhone 3G!)

Kyran used ProtoVis to build the force directed graph, it includes a bit of a hack to make images work on the nodes.

If you’re interested in seeing more of this stuff then Kyran will have more to demo at our upcoming £5 App show and tell.

---

Ian applies Artificial Intelligence as an Artificial Intelligence Researcher for companies (Mor Consulting), co-founded the StrongSteam A.I. datamining toolkit, co-authored SocialTies, programs Python, writes The Screencasting Handbook and is also a sea-side dweller and consumer of fine coffees.

A couple of weeks ago I sold ProCasts.co.uk, the screencasting business I built over the last two years. Some of you know that I moved away from the business back at Christmas and left it idle (a rather silly thing to do), here are some notes on how I sold it and how you could sell your business. This is my first business sale, some valuable lessons were learned.

I listed the business on flippa.com a month back, flippa specialises in matching buyers and sellers of domain names and small businesses. Since ProCasts was, after 8 months of inactivity, essentially a website that generated leads with a client list – I figured a listing on flippa would find some interested parties. I didn’t sell The Screencasting Handbook, I’m still happily developing the Handbook’s sales.

The new owners are Tintisha Technologies, a Leicester based video production company who wanted to expand their screencasting brand. Rich of Tintisha discovered the ProCasts sale through flippa by (happy!) accident, made a couple of bids at the end of the auction and came out on top. We completed the handover last week.

The reason for selling ProCasts was simple – I’d moved away from screencasting back at Christmas as I’d decided to return to my historic trade of artificial intelligence research and data science. I knew that a few of ProCasts’ competitors might be interested in the site and that a listing on flippa with money sent through escrow.com would make for a clean, safe sale.

I listed the site as an “Established lead generating screencasting site” with a two week auction. Flippa works differently to eBay – it uses an open auction (though private sales are possible) with a rolling end-time (if a bid is placed within 4 hours of the end of the auction the end time is advanced by another 4 hours).

Take a look at the listing to see the details that I included, I added:

• Full business and site description
• Details of past clients and warm leads
• Bank statements to prove income
• Verified Google Analytics traffic data
• A Transfer Agreement listing all assets/processes for the sale

I made a point of responding to all questions (lots came via the private email channel) and updating the listing with new information. Fortuitously a couple of older leads came back with requests for work during the auction so these ‘very warm leads’ got a mention in the comments too.

At the end of the day the site sold for $4,002 (£2,500), minus the sale fee (£100) and escrow.com’s fees I took away £2,400. Not bad for a site that was otherwise of no value to me but obviously not an ‘interesting exit’.

Here are some of the takehome lessons:

• If you’re selling a business, a pure consultancy (with no consultants) isn’t super interesting to buyers, only to existing market players
• Building a consultancy in a super-small niche (when I started I had 4 US competitors and 0 in the UK) means few buyers when you decide to exit (in fairness – I didn’t build the business to sell it, I know better for next time)
• Design your business with an exit in mind – recurring or passive income has real value to a buyer, make sure you can be removed from the business without damaging it
• A two week auction was fine but four weeks would have made more sense
• Soliciting private bids from competitors should have been done sooner rather than later
• Adding a product or recurring income stream to the business would have added a lot of value (I decided to keep The Screencasting Handbook as an experimental platform)
• BusinessesForSale is an alternative site, I didn’t know about it when I started, their companies tend to have higher value (flippa isn’t really for consultancy businesses, just simple web businesses)

What next?

Some of you know that I’ve been working in the field of artificial intelligence research for industry over the last 10 years (as both senior programmer, product designer and pure r&d bod) in my Mor Consulting. This role is evolving and I’m turning into a “Data Scientist” (the new shiny term for A.I. researchers!).

I’m also building some new IP by way of web services using A.I. technologies, these are designed with an exit in mind (I’m learning!). If you’re curious about using A.I. in industry see my new A.I.Cookbook.

I’m also continuing to develop The Screencasting Handbook, it is a useful experimental platform and I still very much enjoy teaching the art of screencasting.

If you have any questions, ask away.

---

Ian applies Artificial Intelligence as an Artificial Intelligence Researcher for companies (Mor Consulting), co-founded the StrongSteam A.I. datamining toolkit, co-authored SocialTies, programs Python, writes The Screencasting Handbook and is also a sea-side dweller and consumer of fine coffees.

Having built ProCasts during 2009 (from ‘just me’ to a fab team of four) and then letting it sit quietly for the first half of this year I’m now selling the site as a lead-generation opportunity for a lucky fellow screencaster.

The site continues to generate leads each week – this has lead to some odd conversations where I try to explain that I’m not taking on any work – even though the client might be rather desperate for a UK solution. After another odd conversation a week back I decided it was time to pass the site on.

The site is listed at flippa.com, the auction ends in 11 days. Via the message boards I’m having good conversations with existing screencasters and a few new entrants. It feels good to have built an asset that’ll help someone else win more business. Not only does the site generate leads but it also has solid traffic with a Page Rank of 6 and the business gets linked in forums by friendly third parties. Flippa.com has the full details, there’s also a short link on the ProCasts blog.

Personally I’ve returned to my long-term field of artificial intelligence research. The A.I. Cookbook is my new project site, I’m pushing out the ideas that I’ve sat on for years. Times continue to be interesting.

---

Ian applies Artificial Intelligence as an Artificial Intelligence Researcher for companies (Mor Consulting), co-founded the StrongSteam A.I. datamining toolkit, co-authored SocialTies, programs Python, writes The Screencasting Handbook and is also a sea-side dweller and consumer of fine coffees.

I’m hugely looking forward to EuroPython in Birmingham from Monday. I’m driving up Monday very early (I wish I’d booked the hotel room for Sunday night too…). Browsing through the abstracts I’d say all the following look darned interesting!

• C++ integration
• concurrent sequential processes
• Arduino hacking
• javascript
• OpenData
• aerodynamics
• PyPy and Unladen Swallow
• game programming
• OpenGL
• Pyjamas
• idiomatic Python
• MediaCore
• Twisted and gevent
• science and maths
• SHOGUN machine learning

I’ll bring Headroid along and I hope to organise a Birds of a Feather session on Artificial Intelligence and robotics. If you’re interested in these topics, I’d love to say hi!

---

Ian applies Artificial Intelligence as an Artificial Intelligence Researcher for companies (Mor Consulting), co-founded the StrongSteam A.I. datamining toolkit, co-authored SocialTies, programs Python, writes The Screencasting Handbook and is also a sea-side dweller and consumer of fine coffees.

I’ve just uploaded a new Mandelbrot.py demo for pyCUDA, it adds a new calculation routine that straddles the numpy (C based math) and the pure-CUDA implementations. In total there are 4 variants to choose from. The speed differences are huge!

Update – this Reddit thread has more details including real-world timings for two client problems (showing 10-3,677* speed-ups over a C task).

Update – I’ve written a High Performance Python tutorial (July 2011, 55 pages) which covers pyCUDA and other technologies, you might find it useful.

This post builds upon my earlier pyCUDA on Windows and Mac for super-fast Python math using CUDA.

You’ll need CUDA 3.1 and pyCUDA installed with a compatible NVIDIA graphics card. This version of the Mandelbrot code forces single precision math – this means it’ll work on all CUDA cards (even the older ones – full list). It runs on my MacBook (Leopard) and Windows, the Windows machines use a 9800 GT and GTX 480. Here’s what it generates:

The big-beast graphics card for my physics client is a GTX 480 – this is NVIDIA’s top of the line consumer card (costing £420GBP in the UK a few weeks back). It is huge – it covers two slots, uses one PCIe 2.0×16 slot and has a requirement for 300-400W of power (I’m using a 750W PSU to be safe on a Gigabyte GA H55M S2H motherboard):

The mandelbrot.py demo has four options (e.g. ‘python mandelbrot.py gpu’):

• ‘gpu’ is a pure CUDA solution on the GPU
• ‘gpuarray’ uses a numpy-like CUDA wrapper in Python on the GPU
• ‘numpy’ is a pure Numpy (C-based) solution on the CPU
• ‘python’ is a pure Python solution on the CPU with numpy arrays

The default problem is a 1000*1000 Mandelbrot plot with 1000 max iterations. I’m running this on a 2.9GHz dual core Windows XP SP3 with Python 2.6 (only 1 thread is used for all CPU tests). The timings:

• ‘gpu’ – 0.07 seconds
• ‘gpuarray’ – 3.45 seconds – 49* slower than GPU version
• ‘numpy’ – 43.4 seconds – 620* slower than GPU version
• ‘python’ – 1605.6 seconds – 22,937* slower than GPU version
• ‘python’ with psyco.full() – 1428.3 seconds – 20,404* slower than GPU version

By default mandelbrot.py forces single precision for all the math. Interestingly on my box if I let numpy default to numpy.complex128 (two double precision floating point numbers rather than numpy.complex64 with two single precision floats) then the Python result is faster:

• ‘numpy’ – 34.0 seconds (double precision)
• ‘python’ – 627 seconds (double precision) – 2.5* faster than the single precision version

The ’22,937*’ figure is a little unfair in light of the 627 second result (which is 8,957* slower) but I wanted to use only single precision math for consistency and compatibility across all CUDA cards (the older cards can only do single precision math).

On my older dual core 2.66GHz machine with a 9800 GT I get:

• ‘gpu’ – 1.5 seconds
• ‘gpuarray’ – 7.1 seconds – 4.7* slower than GPU version
• ‘numpy’ – 51 seconds – 34* slower than GPU version
• ‘python’ – 1994.3 seconds – 1,329* slower than GPU version

If we compare the 0.07 seconds for the GTX 480 against the 1.5 seconds for the 9800 GT (albeit on different machines but the runtime is just measuring the GPU work) then the GTX 480 is 21* faster than the 9800 GT. That’s not a bad speed-up for a couple of years difference in architectures.

If you take a look at the source code you’ll see that the ‘gpu’ option uses a lump of C-like CUDA code, behind the scenes all pyCUDA code is converted into this C-like code and then down to PTX via their compiler. This is the way to go if you understand the memory model and you want to write very fast code.

The gpuarray option uses a numpy-like interface to pyCUDA which, behind the scenes, is converted into CUDA code. Because it is compiled from Python code the resulting CUDA code isn’t as efficient – the compiler can’t make the same assumptions about memory usage as I can make when hand-crafting CUDA code (at least – that’s my best understanding at present!).

The numpy version uses C-based math running on the CPU – generally it is regarded as being ‘pretty darned fast’. The python version uses numpy arrays with straight Python arithmetic, this makes it awfully slow. Psyco 2.0.0 makes it a bit faster.

Feedback and extensions are welcomed via the wiki!

If you want to get started then make sure you have a compatible CUDA card, get pyCUDA (installation instructions), compile pyCUDA (takes 30 minutes from scratch if you’re on a well-known system), try the examples and run mandelbrot.py. The mailing list is helpful.

It’d be nice to see some comparisons with PyPy, ShedSkin and other Python implementations. You’ll find links in my older ShedSkin post. It’ll also be interesting to tie this in to some of the A.I. projects in the A.I. Cookbook, I’ll have to ponder some of the problems that might be tackled.

Books:

The following two books will be useful if you’re new to CUDA. The first is very friendly, I’m still finding it very useful.

---

Ian applies Artificial Intelligence as an Artificial Intelligence Researcher for companies (Mor Consulting), co-founded the StrongSteam A.I. datamining toolkit, co-authored SocialTies, programs Python, writes The Screencasting Handbook and is also a sea-side dweller and consumer of fine coffees.