How do we rate?

by Justin D'Olier on September 15, 2010 | (10) Comments |

As much as some of you may hate me for saying it, we need a rating system.

I say this fully aware of the fact that alcohol ratings are purely subjective and almost completely dependent upon the palate of the reviewer. When Robert Parker gives a wine 95 points it doesn’t mean you’ll like it (though you can be reasonably confident it’s better than Matthew Fox Shirax), it means Robert Parker likes it. If you like the same type of wine as Mr. Parker, that’s great; but what if you don’t enjoy wine with fruit that’s forward enough to make you file a restraining order or an alcohol level that would fatally maim a mid-sized rhinoceros?

Sadly, you’re out of luck. A rating system can never be all things for all people. But that doesn’t mean there’s no value in them.

Rating systems are a starting point. A vote of approval. A straight forward way to recommend noteworthy wine. Not everyone has time to read 450 words about Tamas Pinot Grigio (though, if you have the time, you should). They want answers and they want them now. A rating gives that to them, quickly and easily.

Furthermore, a rating is a great foil for discussion. It’s one thing to say you don’t like Miller Chill, completely another to give it a 5 out of 100 points (for the record, on a scale from 1 to 100, I’d give Miller Chill a negative 75. I wouldn’t use that abomination of a beverage to clean the rims on my car. Frankly, the only positive aspect of Miller Chill is it’s ability to instantly inform me that the person drinking it and I will never be friends, but I digress…). A rating provides a starting point for discussion about the relative quality of a beverage. You might agree, you might not, but at least now you have a framework for discussing the difference in opinion.

So, regardless of any potential moral misgivings we have about beverage rating systems, we still need to use one. Grudgingly and with full disclosure of its subjectivity, but using it nonetheless.

Now comes the fun part: What rating system should we use?

We can employ a current familiar system (like Parker’s aforementioned 100 point scale) or create a new rating scale altogether. I think the following methodologies merit consideration and I’m interested to hear your opinion of them or suggestions for different methods.

(Note: I was initially going to send this as an internal email to the Drink with Aloha team, but thought the discussion of rating systems was relevant (and, hopefully, entertaining) to readers of Drink with Aloha. When someone asks why we use our chosen system, we can direct them to this discussion.)

Potential Beverage Rating Systems

The Robert Parker 100-point scale

Most wine consumers are familiar with Parker’s scale, which runs from 50-100. It is fairly widespread, used by the preeminent wine publication, Wine Spectator, and assigns a nice, big, round number to the ratings. People are used to dealing with scoring systems that go to 100 and associate 100 percent with a perfect score, so that ratings are very intuitive.

On the downside, since it’s already widely used in the wine community and has some negative connotations. Not everyone likes or agrees with Robert Parker and we don’t necessarily align ourselves with the somewhat snooty wine crowd. We don’t want our ratings to seem aloof or pretentious and that might be somewhat unavoidable with a 100 point scale.

A standard A-F letter grade system

Everyone who went to school (in the United States at least) is familiar with letter grades. A is exemplary; B is good; C is average; D is poor; and F is a failing grade. Pluses and minus provide additional granularity. A letter grade assigned to a beverage provides a clear and simple indication of overall quality. It requires no further explanation.

That said, since the system only has 12 potential ratings (A, A-, B+, B, B-, etc…) it does not provide as much room for detail or subtle difference as a 100-point scale. Furthermore, because of the association to scholastic endeavors, negative grades, particularly an F, carry potentially undue negative connotations. F isn’t just bad, it’s an abject failure. (And, yes, for the record, Miller Chill would receive an F-minus).

The rating system is currently used by Christopher Null, the author of popular drinking blog, which I suppose is an argument for and against the system.

A new kitschy rating system (e.g. 1-10 Cocktail Umbrellas)

Drink with Aloha is independent and a little irreverent, so perhaps we need a rating system to match.

We could use a simple 10 point scale, but instead of points, incorporate our cocktail umbrella logo (e.g. a Chimay Blue might score 8 out of 10 cocktail umbrellas, while a Schlitz Malt Liquor might receive only 2 cocktail umbrellas). A  logo based system would reinforce our branding and differentiate our rankings from others.

On the flip side, a completely new rating system would require additional explanation and might not get taken as seriously, especially until Drink with Aloha becomes more of a household name. We have a lot of work to do before people associate Drink with Aloha with quality, reliable drinking information and establishing a new ranking system would represent another hurdle. I’m fully confident that we’re up to the task and am not, in any way, suggesting that we curtail our enthusiasm or originality, but I think it’s important to keep our current situation in perspective.

Lastly, a 10-point rating system provides even fewer options than a letter grade system, which might limit our ratings more than we want. A smaller scale is simpler and forces us to make hard decisions — if Netflix uses a 5-point scale for movies, who’s to say we can’t use it for alcohol — but it also limits our ability to differentiate between the good, the great, the amazing, and the truly transcendent.

A multi-tiered system

Most rating systems consist of a single grade, an all encompassing measure of quality. While that makes it very easy to compare beverages, it oversimplifies the rating process. Does a 90-point rating mean a wine is a good value? Easy to obtain? Does it have a nice aroma? Will it age? Does it have an impressively designed bottle or cool back story? Does it pair well with food? Is it crowd pleasing?

Beverages are inherently multi-faceted and perhaps our rating system should reflect this.

What if we had a four part-rating system that consisted of Quality, Value, Wow factor, Overall Rating?

A bottle of Opus One might score highly on quality and wow-factor (in large part due to the name recognition), but comparatively lower in value and a lower overall rating as a result.

A bottle of Gazela Vino Verde might have an average quality rating, but it’s incredible value and the novelty wow-factor of the light green wine, brightly colored bottle, and smooth fizzy finish would give it a higher overall rating than one would normally expect.

A Miller Chill could score a zero across the board, but end up with a negative overall rating to emphasize just how truly embarrassing it is to drink a can of artificially flavored moose urine.

No matter what categories we selected for the final system, a multi-tiered rating system would provide a more well-rounded and accurate representation of each beverage.

Unfortunately, the additional specificity adds an increased level of complexity and, potential, confusion to the rankings. Rather than planting our flag in the sand and providing a definitive opinion about a beverage, it creates a more subjective, somewhat ambiguous scale for comparison. This will make it more difficult to compare previously rated beverages.

If we liked the general idea of a multi-tiered rating system, but wanted to avoid the additional complexity, we could rate simply on quality, but also provide a list of “best buys” or “staff picks” that deserved mention due to good value or a certain je ne sais quoi.

In summary, I think all the options have merit, but I want to know what you think. Please leave your thoughts, questions, comments, concerns, musings, ramblings, and wild machinations below and let’s make this decision happen.

The deadline for selecting a rating system is next week Friday, September 24 (a completely arbitrary date that I selected just now), so speak now or forever hold your drinks.

10 responses to “How do we rate?”

  1. Joshua Hampton says:

    I’ve always liked a single concrete number (1-100, A-F, whatever) that is qualified by a more subjective and comprehensive tiered rating. The overall would be the single rating, allowing people to sort ratings at-a-glance, but then expound upon the single rating with more specific ratings. See examples below:

  2. Elton Nichols says:

    I have thought long and hard on this (get your mind out of the gutter Justin), and I have gone back and forth on whether or not to do a multifaceted rating system. I think we do need to address the fact that expensive wines should have to live up to their price tag while cheap wines are cheap for a reason and don’t necessarily need to hold up to the focused scrutiny of their bigger brothers and sisters. I have had many guests over the years order wines because of the point based reviews the wines have received in the Wine Speculator and I have been shocked at the inflexibility they have to look beyond the simple number. Even Mr. Parker loudly states that ratings alone mean nothing if the accompanying review isn’t taken into account.

    I had initially thought that we should rate wines separately in their respective price tiers but I think that still might be too ambiguous, and worse yet that only takes wines into account and not also spirits and beer. It’s “Drink With Aloha” not “Winos With Rainbows” (shoot, we should have thought the name through a little better!). I next thought that we should have a rating that gave an indicator as to the general style of the wine – like what tried to get off the ground – but again I think that people want their information distilled down to a fine neutral spirit.

    In the end my vote is to measure things on a “Bang for Buck” scale (call it what you will, “B4$”, “BfB”, “B/$”, etc.). I think an open ended scale to dispel any concept of absolute perfection in a beverage would be nice. Too many people are obsessed with a 100pt wine and what that means, and they end up sounding like Double Rainbow Guy ( ).

    The B4$ scale would be x/10, so a beverage like Miller Chill would get 0/10 or 1/10 whereas a beverage that is a break-even return on investment like Mollydooker “The Boxer” Shiraz would be 10/10. The open ended idea is that if a beverage is undervalued or over-performs for its price it would get a fraction that exceeds 1. For instance I would rate Segura Viudas Brut Reserve Cava a 15/10. Why is ten the baseline? If we set it at 100 people will immediately think of the WS/WA scoring system and their brains won’t follow us. If we set it at 1 then the numerical ratings will seem like splitting hairs. We could do a 20pt scale like my favorite wine website, .

    In the end if people see a rating of 10/10 or above they will know that we are stating that they will at least get their money’s worth out of what they are buying. Again I would caution the general public not to blindly buy based on our ratings without at least reading our quick recap (imagine the shock of buying a Chablis that is 15/10 without realizing that it is made of unoaked Chardonnay), but I think this would be an easy and intuitive benchmark towards estimating quality quickly.

    Regardless of what we choose as our measuring stick we should hyper-link each rating to a pop-out page that explains our scale.

    Anyone else have any thoughts on this? Should it be X/99Bananas instead? Let me know.

  3. Elton Nichols says:

    That last bit should read $1.40 / $1 (why can’t we edit our own comments?)

  4. Meltron says:

    Well I just wrote a very lengthy and in-depth expose’ regarding my opinion on a ratings system and then it got deleted. FML. In short, I like Elton’s idea. It’s descriptive and doesn’t detract from the articles. It most likely will even pique interest in the articles, like “Why did this drink receive a rating higher than its denominator?? I must read its description!”. So, yeah.

  5. Les JoceSip says:

    Dave and I were playing around with our own five score system for fun…
    WB, A, WG, WFG, WP:
    Wicked Bad
    Alright (for those of us deep in the city “Aight”)
    Wicked Good
    Wicked Fuckin’ Good
    Wicked Pissa

    otherwise i’m not sure a rating is necessary. I think more directional like “Buy it all” “Run don’t walk” “only if your choice is between this and a white zin that’s been left in your car for a week” “good for Sangria”

    I personally like to spin it all positively since we all taste differently. I firmly believe that if it’s not corked, cooked or off there is a way to celebrate it! =)

  6. Erin says:

    I’m a personal fan of the multi-tiered system…mostly because of the control freak in me who needs/wants as much information at any given point as possible… But also, I think with anything there are many layers that, if possible, deserve a bit of explanation as everyone has different agendas/priorities when it comes to choosing beverages.

    I do however think that some sort of numeric rating system would be helpful to distinguish exactly where we feel the product falls in the grand scheme of things. I agree with Elton that we should have a scoring key either on each post or in an easily accessible place where people can reference it.

    Either way we go, as long as we are consistent we’ll be good.

  7. […] to sometimes deviate from your usual course.  I’m not trying to say that you have to drink Miller Chill.  Ever.  Just maybe try something new like sitting on your hand for a few minutes before using it […]

  8. […] the chemically introduced flavored is based upon unnatural limes. After seeing the introduction of Miller Chill – another 0% unnatural lime flavored beer – into the American marketplace, […]