Recommender Systems Tutorial

In this tutorial, we’ll provide a simple walkthrough of how to use Snorkel to build a recommender system. We consider a setting similar to the Netflix challenge, but with books instead of movies. We have a set of users and books, and for each user we know the set of books they have interacted with (read or marked as to-read). We don’t have the user’s numerical ratings for the books they read, except in a small number of cases. We also have some text reviews written by users.

Our goal is to build a recommender system by training a classifier to predict whether a user will read and like any given book. We’ll train our model over a user-book pair to predict a rating (a rating of 1 means the user will read and like the book). To simplify inference, we’ll represent a user by the set of books they interacted with (rather than learning a specific representation for each user). Once we have this model trained, we can use it to recommend books to a user when they visit the site. For example, we can just predict the rating for the user paired with a book for a few thousand likely books, then pick the books with the top ten predicted ratings.

Of course, there are many other ways to approach this problem. The field of recommender systems is a very well studied area with a wide variety of settings and approaches, and we just focus on one of them.

We will use the Goodreads dataset, from “Item Recommendation on Monotonic Behavior Chains”, RecSys’18 (Mengting Wan, Julian McAuley), and “Fine-Grained Spoiler Detection from Large-Scale Review Corpora”, ACL’19 (Mengting Wan, Rishabh Misra, Ndapa Nakashole, Julian McAuley). In this dataset, we have user interactions and reviews for Young Adult novels from the Goodreads website, along with metadata (like title and authors) for the novels.

Loading Data

We start by running the download_and_process_data function. The function returns the df_train, df_test, df_dev, df_valid dataframes, which correspond to our training, test, development, and validation sets. Each of those dataframes has the following fields:

user_idx: A unique identifier for a user.
book_idx: A unique identifier for a book that is being rated by the user.
book_idxs: The set of books that the user has interacted with (read or planned to read).
review_text: Optional text review written by the user for the book.
rating: Either 0 (which means the user did not read or did not like the book) or 1 (which means the user read and liked the book). The rating field is missing for df_train. Our objective is to predict whether a given user (represented by the set of book_idxs the user has interacted with) will read and like any given book. That is, we want to train a model that takes a set of book_idxs (the user) and a single book_idx (the book to rate) and predicts the rating.

In addition, download_and_process_data also returns the df_books dataframe, which contains one row per book, along with metadata for that book (such as title and first_author).

from utils import download_and_process_data

(df_train, df_test, df_dev, df_valid), df_books = download_and_process_data()

df_books.head()

We look at a sample of the labeled development set. As an example, we want our final recommendations model to be able to predict that a user who has interacted with book_idxs (25743, 22318, 7662, 6857, 83, 14495, 30664, …) would either not read or not like the book with book_idx 22764 (first row), while a user who has interacted with book_idxs (3880, 18078, 9092, 29933, 1511, 8560, …) would read and like the book with book_idx 3181 (second row).

df_dev.sample(frac=1, random_state=12).head()

	user_idx	book_idxs	book_idx	rating	review_text
159765	6236	(16739, 19021, 18074, 10278, 31179, 9850, 1198...	5174	0	NaN
593901	22944	(29404, 19963, 19186, 13863, 29540, 7375, 2370...	13470	1	NaN
199550	7732	(1101, 26775, 13761, 5265, 3269, 21272, 7424, ...	5912	1	NaN
476454	18375	(23305, 24591, 221, 24819, 11955, 998, 16323, ...	10774	0	NaN
248831	9615	(27162, 21741, 28317, 30890, 8239, 11392, 2499...	11392	1	NaN

Writing Labeling Functions

POSITIVE = 1
NEGATIVE = 0
ABSTAIN = -1

If a user has interacted with several books written by an author, there is a good chance that the user will read and like other books by the same author. We express this as a labeling function, using the first_author field in the df_books dataframe. We picked the threshold 15 by plotting histograms and running error analysis using the dev set.

from snorkel.labeling.lf import labeling_function

book_to_first_author = dict(zip(df_books.book_idx, df_books.first_author))
first_author_to_books_df = df_books.groupby("first_author")[["book_idx"]].agg(set)
first_author_to_books = dict(
    zip(first_author_to_books_df.index, first_author_to_books_df.book_idx)
)


@labeling_function(
    resources=dict(
        book_to_first_author=book_to_first_author,
        first_author_to_books=first_author_to_books,
    )
)
def shared_first_author(x, book_to_first_author, first_author_to_books):
    author = book_to_first_author[x.book_idx]
    same_author_books = first_author_to_books[author]
    num_read = len(set(x.book_idxs).intersection(same_author_books))
    return POSITIVE if num_read > 15 else ABSTAIN

We can also leverage the long text reviews written by users to guess whether they liked or disliked a book. For example, the third df_dev entry above has a review with the text '4.5 STARS', which indicates that the user liked the book. We write a simple LF that looks for similar phrases to guess the user’s rating of a book. We interpret >= 4 stars to indicate a positive rating, while < 4 stars is negative.

low_rating_strs = [
    "one star",
    "1 star",
    "two star",
    "2 star",
    "3 star",
    "three star",
    "3.5 star",
    "2.5 star",
    "1 out of 5",
    "2 out of 5",
    "3 out of 5",
]
high_rating_strs = ["5 stars", "five stars", "four stars", "4 stars", "4.5 stars"]


@labeling_function(
    resources=dict(low_rating_strs=low_rating_strs, high_rating_strs=high_rating_strs)
)
def stars_in_review(x, low_rating_strs, high_rating_strs):
    if not isinstance(x.review_text, str):
        return ABSTAIN
    for low_rating_str in low_rating_strs:
        if low_rating_str in x.review_text.lower():
            return NEGATIVE
    for high_rating_str in high_rating_strs:
        if high_rating_str in x.review_text.lower():
            return POSITIVE
    return ABSTAIN

We can also run TextBlob, a tool that provides a pretrained sentiment analyzer, on the reviews, and use its polarity and subjectivity scores to estimate the user’s rating for the book. As usual, these thresholds were picked by analyzing the score distributions and running error analysis.

from snorkel.preprocess import preprocessor
from textblob import TextBlob


@preprocessor(memoize=True)
def textblob_polarity(x):
    if isinstance(x.review_text, str):
        x.blob = TextBlob(x.review_text)
    else:
        x.blob = None
    return x


# Label high polarity reviews as positive.
@labeling_function(pre=[textblob_polarity])
def polarity_positive(x):
    if x.blob:
        if x.blob.polarity > 0.3:
            return POSITIVE
    return ABSTAIN


# Label high subjectivity reviews as positive.
@labeling_function(pre=[textblob_polarity])
def subjectivity_positive(x):
    if x.blob:
        if x.blob.subjectivity > 0.75:
            return POSITIVE
    return ABSTAIN


# Label low polarity reviews as negative.
@labeling_function(pre=[textblob_polarity])
def polarity_negative(x):
    if x.blob:
        if x.blob.polarity < 0.0:
            return NEGATIVE
    return ABSTAIN

from snorkel.labeling import PandasLFApplier, LFAnalysis

lfs = [
    stars_in_review,
    shared_first_author,
    polarity_positive,
    subjectivity_positive,
    polarity_negative,
]

applier = PandasLFApplier(lfs)
L_dev = applier.apply(df_dev)

LFAnalysis(L_dev, lfs).lf_summary(df_dev.rating.values)

	j	Polarity	Coverage	Overlaps	Conflicts	Correct	Incorrect	Emp. Acc.
stars_in_review	0	[0, 1]	0.019500	0.004375	0.001375	129	27	0.826923
shared_first_author	1	[1]	0.061875	0.002625	0.000750	335	160	0.676768
polarity_positive	2	[1]	0.044000	0.014000	0.000625	297	55	0.843750
subjectivity_positive	3	[1]	0.018000	0.014625	0.003750	110	34	0.763889
polarity_negative	4	[0]	0.019625	0.005500	0.004375	87	70	0.554140

Applying labeling functions to the training set

We apply the labeling functions to the training set, and then filter out data points unlabeled by any LF to form our final training set.

from snorkel.labeling.model import LabelModel

L_train = applier.apply(df_train)
label_model = LabelModel(cardinality=2, verbose=True)
label_model.fit(L_train, n_epochs=5000, seed=123, log_freq=20, lr=0.01)
preds_train = label_model.predict(L_train)

from snorkel.labeling import filter_unlabeled_dataframe

df_train_filtered, preds_train_filtered = filter_unlabeled_dataframe(
    df_train, preds_train, L_train
)
df_train_filtered["rating"] = preds_train_filtered

Rating Prediction Model

We write a Keras model for predicting ratings given a user’s book list and a book (which is being rated). The model represents the list of books the user interacted with, books_idxs, by learning an embedding for each idx, and averaging the embeddings in book_idxs. It learns another embedding for the book_idx, the book to be rated. Then it concatenates the two embeddings and uses an MLP to compute the probability of the rating being 1. This type of model is common in large-scale recommender systems, for example, the YouTube recommender system.

import numpy as np
import tensorflow as tf
from utils import precision_batch, recall_batch, f1_batch

n_books = max([max(df.book_idx) for df in [df_train, df_test, df_dev, df_valid]])


# Keras model to predict rating given book_idxs and book_idx.
def get_model(embed_dim=64, hidden_layer_sizes=[32]):
    # Compute embedding for book_idxs.
    len_book_idxs = tf.keras.layers.Input([])
    book_idxs = tf.keras.layers.Input([None])
    # book_idxs % n_books is to prevent crashing if a book_idx in book_idxs is > n_books.
    book_idxs_emb = tf.keras.layers.Embedding(n_books, embed_dim)(book_idxs % n_books)
    book_idxs_emb = tf.math.divide(
        tf.keras.backend.sum(book_idxs_emb, axis=1), tf.expand_dims(len_book_idxs, 1)
    )
    # Compute embedding for book_idx.
    book_idx = tf.keras.layers.Input([])
    book_idx_emb = tf.keras.layers.Embedding(n_books, embed_dim)(book_idx)
    input_layer = tf.keras.layers.concatenate([book_idxs_emb, book_idx_emb], 1)
    # Build Multi Layer Perceptron on input layer.
    cur_layer = input_layer
    for size in hidden_layer_sizes:
        tf.keras.layers.Dense(size, activation=tf.nn.relu)(cur_layer)
    output_layer = tf.keras.layers.Dense(1, activation=tf.nn.sigmoid)(cur_layer)
    # Create and compile keras model.
    model = tf.keras.Model(
        inputs=[len_book_idxs, book_idxs, book_idx], outputs=[output_layer]
    )
    model.compile(
        "Adagrad",
        "binary_crossentropy",
        metrics=["accuracy", f1_batch, precision_batch, recall_batch],
    )
    return model

We use triples of (book_idxs, book_idx, rating) from our dataframes as training data points. In addition, we want to train the model to recognize when a user will not read a book. To create data points for that, we randomly sample a book_id not in book_idxs and use that with a rating of 0 as a random negative data point. We create one such random negative data point for every positive (rating 1) data point in our dataframe so that positive and negative data points are roughly balanced.

# Generator to turn dataframe into data points.
def get_data_points_generator(df):
    def generator():
        for book_idxs, book_idx, rating in zip(df.book_idxs, df.book_idx, df.rating):
            # Remove book_idx from book_idxs so the model can't just look it up.
            book_idxs = tuple(filter(lambda x: x != book_idx, book_idxs))
            yield {
                "len_book_idxs": len(book_idxs),
                "book_idxs": book_idxs,
                "book_idx": book_idx,
                "label": rating,
            }
            if rating == 1:
                # Generate a random negative book_id not in book_idxs.
                random_negative = np.random.randint(0, n_books)
                while random_negative in book_idxs:
                    random_negative = np.random.randint(0, n_books)
                yield {
                    "len_book_idxs": len(book_idxs),
                    "book_idxs": book_idxs,
                    "book_idx": random_negative,
                    "label": 0,
                }

    return generator


def get_data_tensors(df):
    # Use generator to get data points each epoch, along with shuffling and batching.
    padded_shapes = {
        "len_book_idxs": [],
        "book_idxs": [None],
        "book_idx": [],
        "label": [],
    }
    dataset = (
        tf.data.Dataset.from_generator(
            get_data_points_generator(df), {k: tf.int64 for k in padded_shapes}
        )
        .shuffle(123)
        .repeat(None)
        .padded_batch(batch_size=256, padded_shapes=padded_shapes)
    )
    tensor_dict = tf.compat.v1.data.make_one_shot_iterator(dataset).get_next()
    return (
        (
            tensor_dict["len_book_idxs"],
            tensor_dict["book_idxs"],
            tensor_dict["book_idx"],
        ),
        tensor_dict["label"],
    )

We now train the model on our combined training data (data labeled by LFs plus dev data).

from utils import get_n_epochs

model = get_model()

X_train, Y_train = get_data_tensors(df_train_filtered)
X_valid, Y_valid = get_data_tensors(df_valid)
model.fit(
    X_train,
    Y_train,
    steps_per_epoch=300,
    validation_data=(X_valid, Y_valid),
    validation_steps=40,
    epochs=get_n_epochs(),
    verbose=1,
)

Finally, we evaluate the model’s predicted ratings on our test data.

X_test, Y_test = get_data_tensors(df_test)
_ = model.evaluate(X_test, Y_test, steps=30)

30/30 [==============================] - 1s 33ms/step - loss: 0.6534 - acc: 0.6570 - f1_batch: 0.5226 - precision_batch: 0.5720 - recall_batch: 0.4931

Our model has generalized quite well to our test set! Note that we should additionally measure ranking metrics, like precision@10, before deploying to production.

Summary

In this tutorial, we showed one way to use Snorkel for recommendations. We used book metadata and review text to create LFs that estimate user ratings. We used Snorkel’s LabelModel to combine the outputs of those LFs. Finally, we trained a model to predict whether a user will read and like a given book (and therefore what books should be recommended to the user) based only on what books the user has interacted with in the past.

Here we demonstrated one way to use Snorkel for training a recommender system. Note, however, that this approach could easily be adapted to take advantage of additional information as it is available (e.g., user profile data, denser user ratings, and so on.)