Kaggle’s Quora Question Pairs Competition

Kaggle‘s Quora Question Pairs competition has just closed, I’m pleased to say that with 10 days effort I ranked in the top 39th percentile (rank 1346 of 3396 in the private leaderboard). Having just run and spoken at PyDataLondon 2017, taught ML in Romania and worked on several client projects I only freed up time right at the end of this competition. Despite joining at the end I had immense fun – this was my first ‘proper’ Kaggle competition.

I figured a short retrospective here might be a useful reminder to myself in the future. Things that worked well:

• Use of github, Jupyter Notebooks, my research module template
• Python 3.6, scikit-learn, pandas
• RandomForests (some XGBoost but ultimately just RFs)
• Dask (great for using all cores when feature engineering with Pandas apply)
• Lots of text similarity measures, word2vec, some Part of Speech tagging
• Some light text clean-up (punctuation, whitespace, some mixed case normalisation)
• Spacy for PoS noun extraction, some NLTK
• Splitting feature generation and ML exploitation into different Notebooks
• Lots of visualisation of each distance measure by class (mainly matplotlib histograms on single features)
• Fully reproducible Notebooks with fixed seeds
• Debugging code to diagnose the most-wrong guesses from the model (pulling out features and the raw questions was often enough to get a feel for “what it missed” which lead to thoughts on new features that might help)

Things that I didn’t get around to trying due to lack of time:

• PoS named entities in Spacy, my own entity recogniser
• GloVe, wordrank, fasttext
• Clustering around topics
• Text clean-up (synonyms, weights & measures normalisation)
• Use of external corpus (e.g. Stackoverflow) for TF-IDF counts
• Dask on EC2

Things that didn’t work so well:

• Fully reproducible Notebooks (great!) to generate features with no caching of no-need-to-rebuild-yet-again features, so I did a lot of recalculating features (which really hurt in the last 2 days) – possible solution below with named columns
• Notebooks are still a PITA for debugging, attaching a console with –existing works ok until things start to crash and then it gets sticky
• Running out of 32GB of RAM several times on my laptop and having a semi-broken system whilst trying to persist partial models to disk – I should have started with an AWS deployment earlier so I could easily turn on more cores+RAM as needed
• I barely checked the Kaggle forums (only reading the Notebooks concerning the negative resampling requirement) so I missed a whole pile of tricks shared by others, some I folded in on the last day but there’s a huge pile that I missed – I think I might have snuck into the top 20% of rankings if I’d have used this public information
• Calibrating RandomForests (I’m pretty convinced I did this correctly but it didn’t improve things, I’m not sure why)

Dask definitely made parallelisation easier with only a few lines of overhead in a function beyond a normal call to apply. The caching, if using something like luigi, would add a lot of extra engineered overhead – not so useful in a rapidly iterating 10 day competition.

I think next time I’ll try using version-named columns in my DataFrames. Rather than having e.g. “unigram_distance_raw_sentences” I might add “_v0”, if that calculation process is never updated then I can just use a pre-built version of the column. This is a poor-mans caching strategy. If any dependencies existed then I guess luigi/airflow would be the next step. For now at least I think a version number will solve my most immediate time-sink in recent days.

I hope to enter another competition soon. I’m also hoping to attend the London Kaggle meetup at some point to learn from others.