The science of Maths and Whisky

Given maths and whisky are two of my favourite things (yes I did say I like maths), I have been using a unique website to refresh my standard deviations as well as my knowledge of the Dr David Wishart flavour classification scheme.

It's a website which teaches you about both whisky and statistics. It also guides you to your dream whisky. Impossible I hear you say. Well maybe not. Read on as we interview the author and hear how he came up with the concept.

WhiskyAnalysis is the work of Eric, going under the name of selfbuilt. He works on WhiskyAnalysis in his spare time, that is, after the time he dedicates to flashlight reviews. Err yes, when I say flashlight reviews, I really do mean reviews of the battery and bulb things we use in the dark.

We contacted Eric to ask a bit more about his motivation, and his thoughts on the future whisky landscape. I hope you enjoy the interview.

Interview with Eric (aka Selfbuilt) of WhiskyAnalysis.com

How come you know so much about statistics?

I'm not a statistician, but I have had advanced training in stats while doing my doctorate in neuroscience. At its heart, statistics is an extension of the scientific method – a way to critically examine data, applying intellectual and mathematical rigour to its analysis. But it is also a very practical discipline, as it can yield novel insights into underlying relationships that are not immediately obvious (as I discuss a bit on the background pages on my site).

As I developed an interest in whisky, I started looking to see how statistical methods could best be applied to help make sense of this complex field. Cluster analysis and principal component analysis (for whisky flavour) have been done before, but I'm not aware of anyone integrating reviewer scores in the rigorous manner used on my site. Fortunately, I know a few statisticians whom I can draw on for advice.

Do you think you have enough data sources to make the Meta-critic scores reliable?

That question is actually a very good example of statistical thinking.

When developing the site, I did some statistical bootstrapping experiments on the data. I looked at cases where I had a large number of reviewer scores, and watched what happened to the Meta-Critic scores if I started randomly removing reviews. I was interested to know at what point removing scores results in a meaningful change in the Meta-Critic score.

Depending on how you define the magnitude of a significant change (and I tested this several ways), I found that the Meta-Critic results are fairly robust once you have at least 7-8 reviewer scores. More than that, and additional reviews don't typically change the Meta-Critic score much. But less than that, and additional scores have the potential to move the average a noticeable amount.

This is why I always include the number of reviews for each whisky in the Meta-Critic database (along with the average and standard deviation). Below 8 reviews, I recommend you treat the results as provisional. And I won't post data for any whisky that has less than 3 reviews, since the results are impossible to interpret at that level.

Do you use the Meta-critic scores yourself when buying whisky?

Absolutely. One of the main benefits of the site, in my view, is arbitrage. When faced with deciding which new whisky to buy or try, it allows you to start with the highest ranked for that price point - in a given class or flavour category. I find the scores to be a very useful starting point, although you will want to explore flavour descriptions in more detail as well.

Craft spirits have become increasingly popular - first in the USA, then in Europe - how would you explain this trend?

Generally speaking, I think there is growing concern over the environmental footprint of long-distance international trade (i.e., the various grow/eat local movements). Similarly, I think there is a feeling that innovation tends to come from small producers, willing to experiment with new materials and small batch methods. And the internet is a great equalizer – small businesses are starting up all the time, and able to interact with a growing customer base through the reach of social media and true personalized service. While the online world is not a true meritocracy, I think there is a feeling that the playing field is a lot more level now than it was previously.

In the case of whisky, we are without a doubt in a period of high volume sales and high interest (relative to most other spirits), which is bound to stimulate craft interest. I suspect the success of the craft beer industry (especially in North America) has helped spawned the idea of <a href"https://www.craftwhiskyclub.com/subscribe">craft whisky distilling. Indeed, in my own town, one of the two major craft brewers has recently expanded into whisky making.

Have you thought about how to get the new small batch and emerging craft whiskies into the database?

When initially building the database, I tried to focus on the most commonly-available expressions – in order to make the site as widely useful as possible to most people. But for any whisky to get added to the database, it needs to be reviewed by a good number of expert reviewers (i.e., at least 3, as explained above). The number of available reviews is ultimately the limiting factor.

If so, what is the timeframe to get the emerging craft whiskies represented in the database?

At this point in the evolution of the site, expansion is really being driven by the whiskies the expert reviewers are choosing to look at. I think the craft industry would be well served by trying to get themselves on those experts' radars. I am sure most would be open to receiving samples from craft distillers. One thing I can attest to as a product review in another field (I also do rigorous flashlight testing and reviewing), is that it is always interesting to review something a little bit "different" – as long as the quality proposition is still there.

Are you going to update with the latest scores from sources such as the Whisky Bible each year?

I am constantly updating the database all the time with new scores from all the reviewers, including Mr. Murray. And that includes not just new whiskies, but re-reviews of expressions previously examined. Whisky is a very batch-driven enterprise, so it is important to continually assess how a given expression is changing over time.

I have used as a cut-off in my database of five years – that is, I only look at reviewer scores for expressions produced in the last five years. If a given reviewer has scored numerous batches of the same whisky, I average their scores over that time period. If there seems to be a sharp dividing point when the scores diverge, I expand the database to explicitly compare pre- and post-reviews around that time (along with the overall average). A recent example is the Highland Park 12yo, where reviews appear to have turned less positive in late 2014. I now provide both the overall average, as well as the specific reviewer averages before and after this time.

Which reviewer do you rely on most?

For quality ranking, I don't rely on any one reviewer more than any other – I rely on the Meta-Critic score. I have done correlational analyses of my own personal scores, and I consistently correlate better to the Meta-Critic average than I do to any one reviewer. And that is the main finding of the analysis in general – EVERYONE correlates better to the Meta-Critic than they do to any one reviewer. That includes all the individual reviewers that make up the database. The statistics are quite compelling.

Which reviewer would you say is the most consistent in their reviews?

There are a number of ways you could look at that. There are certainly some reviewers who correlate better with the Meta-Critic score than others. But it's hard to know how to interpret this, since no one is truly independent (i.e., they may simply be more influenced by what other reviewers are saying). That said, all reviewers in my Meta-Critic database seem fairly independent in their scoring.

On a related point, It is all too easy to fall into the common fallacy that you can find reviewers whose "palate" best matches your own. When assembling the Meta-Critic database, there were certain reviewers who I subjectively felt were a particularly good match to my personal tastes. But once I ran the numbers, I discovered that none of them even came close to how well I correlate with the Meta-Critic score. And this is true for each of the expert reviewers as well (which gets back to the point about their relative independence).

Beyond scoring, I think consistency in terms of reviewers' flavour descriptions also matter greatly to readers. But that is hard to assess quantitatively, given the more subjective nature of taste. It may also have nothing to do with how consistently a reviewer scores (if they even provide scores).

Which reviewer would you say is the least consistent, or has odd outlier scores?

I am not touching that one. But if you look at some of the individual whisky commentaries on my site, you'll see I highlight a few cases where there are more discrepancies than normal. You can draw your own conclusions as to which reviewer name keeps coming up the most.

But this is an important point when considering adding new reviewers to the Meta-Critic. I need to carefully review all their scores, to see if they seem both internally consistent and roughly consistent with the rest of the group. The same goes for how they apply their particular scoring range, from a statistical perspective. This is a topic I explore in a bit of detail on my recent blog posting, "So You Want to Be a Whisky Reviewer …"

Do you have preference for Scottish whisky in particular, or do you value whisky from all over?

I personally value a wide range. I have a particular fondness for malt whisky as well as high-rye whiskies. But the more I try, the less I feel bound to classic distinctions based on geography (which is to say, specific production methods in long-standing use in certain geographical regions). The variety of approaches in use world-wide means that you can find outstanding whiskies produced the world over.

It is very interesting that you recently reviewed the Mackmyra Svensk Ek. I am thinking that we will feature this whisky in a future subscription box.

Yes, for its flavour cluster, Mackmyra Svensk Ek (formerly known as Första Utgåvan, the First Edition) is very highly ranked. This makes it an outstanding value, given the typical cost (there is that arbitrage again).

Mackmyra is a good example of why it is worthwhile to try whiskies produced outside of classic jurisdictions. The use of a small proportion of Swedish Oak casks appears to have introduced some novel properties (just as Mizunara Oak casks do in Japanese whisky). And the First Edition's "tongue-tingle" reminds me of some of the early Stalk & Barrel whiskies (a local Canadian craft distillery).

I think this speaks to the value of craft distilling. While Mackmyra has certainly grown considerably in size, it was not too long ago that it began as a largely craft enterprise. And I think a healthy craft distilling sector bodes well for the industry in general.

WhiskyAnalysis is a site I will be referring to many times. I'd like to personally thank Eric for all of his hard work, and his openness when it comes to sharing his methodology and data.

David Nicol, Craft Whisky Club

At Craft Whisky Club we want to help you discover this new whisky landscape as it unfolds.

Why not Join us to learn more.