A while ago, I was playing around with our Trustpilot review data on one of our hackdays and I noticed something interesting; a lot of our reviews mention that the customer, and their dog, love our food and our service.
This got me wondering, are we special? Or do lots of customers declare their love for their favourite companies on Trustpilot? When I dug into it, it turns out that yes, we are quite special. In fact, based on Trustpilot reviews we are the most loved pet food brand out there!
Reaching this conclusion required a lot of patient data gathering, and then NLP and Empirical Bayes methods to analyse the results. Here, I'll walk through the main points of this piece of work.
Collecting the data
To start with, I needed some data. Using Trustpilot's API, I downloaded all English language reviews for companies registered in the UK. There are tens of millions of reviews, and so this took some time. I let it do its thing for a few days, then came back to see what we had.
The analysis
How do you decide if a customer loves the product from the review they write? For this piece of work, we looked for any reviews or titles that mentions "love" (or "loved", "loving" etc) at least once in a positive way. Then, we look at the proportion of these love reviews for each company.
To start with, we need to do some data cleaning. I started by stripping out the company name from reviews so we don't get any false positive matches from the name (sorry, Love Leggings). I also corrected some common misspellings of "love" ("luv" for example) so these count in my analysis as well.
Interesting side note on the impact of a business name, any company with "love" in the name tended to have a fairly high rate of love reviews, even after removing the business name. What I think is happening here is that people's choice of vocabulary is being primed, consciously or unconsciously, by the business name itself.
After stripping out the business name, I reduced all words to the stem, to pick up any words similar to "love" that essentially convey the same meaning. For this part, I used PorterStemmer from Python's nltk library. So for example, the review:
"Love love love the service from Tails, Tiggr loves his kibble, it has enabled him to lose the excess weight easily and maintain an excellent weight for over a year now. The vet is happy and said what ever your doing keep doing it!"
would become:
"love love love the servic from, tiggr love hi kibble, it ha enabl him to lose the excess weight easili and maintain an excel weight for over a year now. the vet is happi and said what ever your do keep do it!"
The review after the stemming clearly does not read well and it loses some meaning. However, the word "loves" is stemmed to "love" which is all we're really looking for here. We could have also used Lemmatization, which would reduce a word to a meaningful base word whilst taking into account the context the word is used in. It's also much more computationally expensive and with millions of reviews to analyse, we stick to stemming.
But what about reviews that mention love in a negative context? For example:
"...It is not a matter that my Dachshund puppy does not like their food, in fact he loves it. However, their customer service falls way below what one would expect..."
Thankfully, these reviews for tails.com are few and far between. But we wouldn't want to misinterpret an unfortunately negative experience for a customer of ours or any other business as something positive. To handle this, we automatically discount any 1 or 2 star reviews from the love reviews, so that an overall negative review can't count positively towards the love rating for a company. For the other reviews, before stemming we look at the sentiment using VADER (also in Python's nltk library) and we discount any review with a negative sentiment score. We use VADER here because it is trained on social media sentiments, and so will work well for our dataset.
Summing Up
At this point, we have a list of companies on Trustpilot, the total number of reviews and the proportion of qualifying love reviews. So, can we now just see who is the most loved company on Trustpilot?
Not quite.
The issue here is that there's a huge disparity between the number of reviews that a company has. For example, is a company with 40 out of 100 love reviews more loved than a company with 380 out 1000 love reviews? Of course, 40% is a higher proportion than 38%, but we have a lot less data on the first company and so we should be a lot less confident about the rate.
One approach could be to just discount companies with too few reviews. But, where do we draw the line? 100 reviews? 500? 1000? It feels a little arbitrary to just pick a threshold. Plus, we are throwing away potentially important data in our analysis. Instead, we use Empirical Bayes modelling to get a better estimation for who is the most loved company on Trustpilot.
What this technique does is allow us to take into account data from companies with a relatively small number of reviews in our calculation to find the most loved, and removes the need to set a threshold and discard data for companies with too few reviews.
In practice how this works is that we start with a prior distribution of the overall love rate of reviews for all companies on Trustpilot. Then for each company, we start with the expected love rate of reviews from the prior and update it based on the reviews we have so far for that company to get an estimate of their love reviews rate. In other words, before we know anything about an individual company or their reviews, we start with a sensible average for the love rate, and then reconsider when we know more. For a more thorough explanation into Empirical Bayes, I'd recommend this blog post and the accompanying book. Here, I'll just illustrate how this applies to our problem.
To begin with, we need to estimate a prior for this data. As our data can be represented by a binomial distribution, we use the beta distribution for our prior. We find the best beta distribution using a negative log-likelihood fit, and on our data we get hyperparameters:
$$ \alpha = 0.62 \quad \beta = 9.19 $$
and a fit:
which is pretty good, a little inaccurate for lower rates but a decent enough prior. The actual rate of love reviews stretches far beyond 30%, but we've just truncated it here to get a better view of the majority of the data. Note also that to get a more reliable prior estimate, we exclude companies with fewer than 500 reviews for the fit.
To get an estimate for the rate of love reviews for a company, we start with the average of our beta prior, and update based on the number of reviews and love reviews that a company has. The mean of a beta distribution is
$$ \frac{\alpha}{\alpha + \beta} $$
and for each company with \( T \) total reviews and \( L \) reviews that mention love, we find an estimate for the rate of love reviews as:
$$ \frac{L + \alpha}{T + \alpha + \beta} $$
If we recall what the Empirical Bayes methodology is doing, we can see that intuitively this formula makes sense, because the expected love review rate will be close to the mean from our prior for a company with only a handful of reviews, and closer to the current average love review rate for a company with several thousand reviews.
We do this calculation for all the companies in our data and the top three are:
\[ \begin{align} \text{Clubzone: }\quad & 52.1 \% \\ \textbf{Tails.com: } \quad & \mathbf{51.1}\% \\ \text{Pyjama Drama: }\quad & 50.3 \% \end{align}\]
The other two companies in the top three are both services aimed at children, which is perhaps telling about the type of business that a customer is really inclined to love. After all, dogs are just fur babies in many households.
Looking beyond the top three in the list, tails.com stands out as having far more Trustpilot reviews than any other company with a high love review rate. Perhaps this is because people tend to feel more passionately about smaller, local brands and services. Or, it's just really hard for a company to consistently deliver a great product or service at scale that customers really, genuinely love and be able to maintain a consistently high rate of positive reviews reflecting this. Either way, we're really pleased at tails.com that we're bucking the trend and delivering a product and service that our customers love.
So there you have it, tails.com is in the UK's top three most loved companies on Trustpilot, and is the most loved pet food brand!
Aww shucks guys, we love you too.
