Scrapy, libxml + libxslt, Mac, “checking for libxml libraries >= 2.6.8… configure: error:”

In the hope that this’ll save someone else the bother…if you’re installing the web scraping Python library scrapy on your Mac (I’m on Leopard 10.5.8) and you come across an error like:

checking for libxml libraries >= 2.6.8... configure: error:
Version 2.6.7 found. You need at least libxml2 2.6.8 for this
version of libxslt

then here’s the solution.

Presumably you’ll be following the Scrapy install instructions. I used the supplied links for libxml2-2.7.3 and libxslt-1.1.24. libxml built and installed to /usr/local/lib just fine. libxslt wouldn’t ./configure – it kept reporting that it could only see the older libxml from /usr/lib, not the newer one in /usr/local/lib.

The fix is here, and this is my configure line:

 $ ./configure 
    --with-python=/Library/Frameworks/Python.framework/Versions/2.5/ 
    --prefix=/usr/local 
    --with-libxml-prefix=/usr/local 
    --with-libxml-include-prefix=/usr/local/include 
    --with-libxml-libs-prefix=/usr/local/lib

At this point libxslt configured, built and installed just fine. To make python see it I had to update my .bash_profile so PYTHONPATH linked to the default output directory:

export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python2.5/site-packages

Side note – whatever you do, don’t mess with /usr/lib. I tried moving the default libxml and libxslt libraries and I had the same consequence mentioned by Kevin Watters – lots of system tools (including su!) depend on libxslt to be in /usr/lib. I had to boot to Single User Mode to copy the files back before the system would work again.


Ian applies Data Science as an AI/Data Scientist for companies in ModelInsight and in his Mor Consulting, sign-up for Data Science tutorials in London. He also founded the image and text annotation API Annotate.io, lives in London and is a consumer of fine coffees.