Music Recommendation Engine

The Million Song Dataset Challenge gives you half of each user's listening history and asks you to predict the other half. No ratings, no explicit feedback, just play counts: 386,213 songs, 110,000 users and 1,450,933 visible evaluation triplets. The interesting part of the task is that a dumb baseline is genuinely hard to beat, so the point of the coursework was building four models and finding out where each one stops helping.

It is worth being precise about why that baseline is hard. There are no negatives in the data. A song a user never played is not a song they dislike; it is a song they never encountered, and roughly 386,000 of the 386,213 are in that category for any given person. So the usual machinery of rating prediction has nothing to predict, and the task collapses to ranking a very long tail against a very short head. Play counts are the only signal and they are heavily skewed toward a few thousand tracks, which means recommending the globally most-played songs to everyone already scores respectably. The organisers' own getting-started guide puts that baseline at about 0.02255 mean average precision, and says so up front, which is effectively a dare.

The four models

• Popularity. Count plays per song across the visible triplets, sort descending, and recommend the global top 500 to every user with their already-heard songs removed. Writes a submission file in the challenge's user song1 song2 ... format and scores itself with a self-contained apk/mapk implementation.
• Item-item collaborative filtering. Built on the challenge's published reference kit, a PredSI similarity predictor over the song-to-user inverted index, parameterised by two exponents A and Q that control how much a song's popularity is discounted when computing similarity. Scored with average precision truncated at tau = 500. It is still Python 2, and it takes a user-ID range and an output file on the command line so a long run can be sharded across processes.
• Latent factors. Surprise's SVD with 100 factors on an 80/20 split, ranking the candidate songs by predicted score.
• KNN. scikit-learn NearestNeighbors with n_neighbors=50 and a ball tree over per-song feature vectors, taking the union of the neighbours of every song a user has already played.

The collaborative filter is the only one with a second predictor behind it. Alongside the item-item scorer there is a user-user variant at A = 0.3 and Q = 5, and an aggregation layer that is meant to draw each recommendation from one of the two lists according to a mixing distribution. In the configuration that was actually run the distribution is a single weight, so the user-user branch never contributes. The comment left in the code next to it explains why nobody chased that: on its own the user-based predictor scored 0.0 across a ten-user sample against 0.33 for the item-based one.

The exponent that does the work

The similarity is not cosine. For a candidate song i and a song j the user already played, it is the size of the intersection of their listener sets divided by |U(i)|^A · |U(j)|^(1-A), with A = 0.15. At A = 0.5 that is exactly cosine similarity and both songs' popularity are discounted equally. Pushing A down to 0.15 divides mostly by the popularity of the song the user already has, and barely at all by the popularity of the candidate, which is what keeps globally popular tracks reachable instead of normalising them out of the ranking. Q = 3 is the other half: each pairwise similarity is raised to the third power before the scores are summed, so a single strong neighbour dominates a handful of weak ones rather than being averaged away by them. Those two numbers are the whole model, and they are where the tuning went.

The plumbing nobody mentions

The dataset ships 40-character hex identifiers for songs and users. Two numberify scripts remap both to sequential integers before anything else runs, which is what makes the matrices fit in memory. kaggle_songs.txt carries the canonical song-to-index mapping, and kaggle_users.txt fixes the user ordering the submission file must follow. Get either ordering wrong and the score is zero without any error.

The two scripts do not agree with each other, which is itself instructive. One reads the index already printed beside each song in kaggle_songs.txt; the other renumbers songs by their line position in a different feature file, and emits a CSV header the first one omits. Both are correct for their own consumer and neither is interchangeable, and nothing in the repository says so. That is the kind of thing that costs an afternoon and never appears in a write-up of the model.

Scale is the other unstated cost. The submission file for a single run is 375 MB, 110,000 rows of 500 song indices each, and every algorithm folder carries its own full copy of the ~200 MB input set because the scripts all open bare relative filenames with no path configuration.

What comparing them actually showed

The four scripts do not evaluate on the same footing, and noticing that was the most useful thing to come out of the exercise. Popularity scores mean average precision at k = 10. The collaborative filter builds a 500-long recommendation list but scores it with the same default k = 10; the MAP-at-500 path exists in the code and is commented out. KNN passes k = 10000, which with fifty neighbours per listened song is a candidate list long enough that truncation stops meaning anything. And the SVD script only ranks songs already inside the user's hidden set and stops after 40 users, which measures ordering rather than retrieval. Four numbers that look comparable and are not. Fixing the metric before comparing models is the part of this I would do first next time.

A side experiment

The same repository carries a genre classifier over GTZAN. It computes MFCCs per clip with a 20 ms window and no energy term, reduces each clip to a mean vector plus a covariance matrix, and classifies by k-nearest neighbour under a symmetrised Kullback-Leibler divergence between the two Gaussians rather than a Euclidean distance. Different problem, same lesson about picking the distance function that matches the representation.

The divergence is the closed-form expression for two multivariate normals, evaluated in both directions and summed, which makes it the Jeffreys divergence rather than a true metric. Its one real defect is that the same k is used for the neighbour count and for the dimensionality term subtracted at the end. Since that term is a constant offset it does not change the ranking, so the classifier still works; it is simply wrong in a way the output would never reveal.

What is not finished

Three of the four scripts do not produce a recorded number. The SVD file stops at a comment saying to print the final result, with the accumulator built and never divided. The KNN script defines a save helper and never calls it. Several intermediate files that the later stages read were never checked in, so only the popularity model runs end to end from what is on disk, and the submission files sitting in the other three folders are copies of its output. The genre notebook has never been executed to completion on this machine; its imports fail on a numpy ABI mismatch, and the 10,000-file Million Song subset that sits beside it is read by nothing in the repository at all. The reference kit's own core module survives only as a Python 2 .pyc whose embedded paths still point at the original author's Dropbox folder.