Artificial Intelligence problems in Industry (things I’ve worked on)

A few days back Mihai commented an interest in the Artificial Intelligence work that I’ve undertaken in the past. I figure that a short run-down of the kinds of problems I’ve tackled might be interesting.

Since 2004 I have run my own A.I. research consultancy – I’m blogging about the experience of becoming a freelance programmer and researcher at the moment (part 2, part 3, several more to come).

Currently I’m working with PANalytical in the UK R&D lab (under Professor. Paul Fewster) to apply evolutionary search algorithms and statistical analysis to multi-dimensional search problems along with an old colleague from MASA (John Anderson).

We’re looking at improving their highly-regarded Epitaxy and Reflectivity X-Ray analysis tools so that they can solve more complex problems more quickly and reliably.

Techniques include evolutionary algorithms, pattern matching, statistical signal processing and a lot of lateral thinking. I won’t say any more as the details are confidential and my work is on-going.

I enjoyed some ad-hoc work at Ambiental previously on their flood-modelling software. The interesting side of things is thinking about what you can when you can accurately model floods – can you predict the best place for flood defences? Can you apply the same techniques to crowds or gas dispersal (e.g. bombs)?

During 2003 and 2004 I worked at Algorithmix on Natural Language Processing problems under Nick Jakobi (now at Google). Up until Corpora acquired Algorithmix I worked on cutting-edge approaches to sentiment analysis and for new-news ‘burst’ reporting.

During that time I also did my own work looking at the use of Bayesian Algorithms (which were becoming rather hot for personal spam filtering) for network-based spam filtering. I worked on the assumption that ISPs saw lots of the same spam so training a filter would be much more efficient at the ISP than on the end-user’s machine.

Algorithmix was spun out of the French MASA Group where I was Senior Programmer for 5 years. I worked on the logistics optimisation side of the business (competing with iLog) into what is now the Blue Kaizen division.

The general work was to use evolutionary search algorithms on heavily-constrained logistics problems to e.g.

  • Route postmen efficiently in vans to collect mail
  • Route petrol tankers to deliver fuel to many cities on complex road networks with varying traffic levels
  • Route waste-collection trucks which handle different types of waste to the appropriate management facility whilst respecting French hours-worked rules and road systems.

These problems were reasonably representative of the hardest logistics problems that high-end desktop computers could solve at the time, given the constraints of the problems.

I’ve always had an interest in electronic circuit design and in my early days at MASA I did some of my own research into floor planning, routing and device placement. Each of these are hard problems which will only get harder as e.g. our CPUs become more complex.

Another area of research at MASA was in the world of financial trading. I was involved in a long project on straight-forward stock market prediction (and no, it wasn’t successful and don’t get me started).

Later, separate from MASA, I was involved in a short piece of work looking at baskets of tradable financial instruments for statistical arbitrage which was fun.

Is AI alive and well? Yes, of course it is. It isn’t necessarily GOFAI and robots don’t clean our houses but there’s a heck of a lot that AI offers us.

One of the big reasons that I like AI is that it can be used to relieve humans of a lot of the tedium of analysing large amounts of data:

  • Spam classifiers will have to get more-AI-ish to deal with the visual and language elements that spammers keep bringing to the party
  • Logistics optimisation will get more complex as we have more constraints, more things to do and less time for planning
  • Circuit designs continue to follow Moore’s law and get more complex at a frightening rate.
  • As physics analysis machines become more complex the wealth of data becomes un-navigable unless you have the appropriate analysis tools.

What to get involved? If you don’t have a background in the area then find a subject that interests you, do some reading and choose a flexible dynamic language so that you can iterate quickly (I favour Python for all my AI work with number crunching in C++).

[I shall quickly plug our ShowMeDo’s Python tutorial videos, there is nothing directly for A.I. in the list but there are videos for programming, physics, graphics and useful utilities that are associated with the domain.]

Search Amazon for terms like ‘artificial intelligence’, ‘evolutionary algorithms’ and ‘natural language processing’. I like ‘New ideas in Optimisation‘ by Corne, Dorigo and Glover (click the link and click the author’s names to see the AI books they published themselves).

You’ll find plenty of resources on the web and feel free to leave a comment if you’d like a bit of guidance on how to get started.


  • Hi Ian, thank you for giving course to my request. The work you do is very interesting and it is nice to have a glimpse of how AI is used in the industry. I am also interested in how can AI techniques be used more in mainstream applications and one thing you mentioned caught my eye in particular: the spam filters. Indeed applications like that could make life a lot easier for the server administrators that have to deal with these problems daily. This can also spur a lot of other uses like automatic moderation for forums and blogs and anti-vandalism tools for public wikies - I am sure Wikipedia would be interested in such tings :). I am glad to see that AI is very much alive and used in real life applications and I believe that a better marketing for this kind of technology would help in a more widespread adoption in mainstream applications, which in turn might lead to further advancements in the field. I recently started learning Python and reading about AI techniques and I intend to do some work in my spare time in this regard, so you will most probably hear from me again bugging you for insights and advice :), if it's ok with you.
  • Hi Mihai, I'm more than happy to give a few pointers - feel free to comment here and I'll see what I can help you with. Cheers, Ian.
  • Well for starters, I would be interested in what Open Source Python libraries and modules can be found to make life a bit easier when implementing AI algorithms. Since you have more experience with Python and AI programming than I do, perhaps you can recommend some good APIs that you used for your applications. Thanks again, Mihai
  • Hmmm, that's a bit of a tall order. A lot of my code is custom as each problem varies from the others, but some common libraries I use are: IPython shell - MatPlotLib graphics - PIL graphics - SciPy libraries - There aren't any other obvious libraries that I can think of for AI work - as I say I tend to build everything I need myself! Ian.
  • Thank you Ian, I'll be sure to check them out.
  • Dean
    Another Python AI toolkit worth checking out is NLTK: It includes lots of lovely NLP tools and an open-content book on "Natural Language Processing in Python".
  • I've heard that NLTK is a good library though I've never used it. I've heard about it on-and-off for a while so presumably it is rather well featured by now. What I'd love to see are some screencasts on for A.I. tools - anybody up for sharing some knowledge? It'll take 10 minutes to start recording screencasts with the super (java) web-enabled: then you grab the resulting MOV and upload it to ShowMeDo... Ian.