Ruth Vs. Maris, meet “Twist” Vs. “Smooth”: All-Time Hot 100 Pits Old Against New

Baseball is such a rich sport for data geeks, what with every move made by a player having a number attached to it. But the inconsistencies and outright mistakes in the sport’s long history, from the number of games in a season to the exclusion of the Negro Leagues, make the sport even moregeek-friendly–those missteps become something else to debate, at least when fans aren’t mulling the current statistical effects of controlled substances.

Mistakes and inconsistencies are two of many things baseball geekery and music-chart geekery have in common. (Lord knows the music business has its substances.) Billboard has changed the rules behind the Hot 100 often enough to keep guys like me talking for days on end.

From flip-flopping rules about B-sides, to shifting radio genres, to the belated inclusion of album cuts on the singles charts, chronicling chart history means making allowances for a slew of discrepancies. The biggest discrepancy of all is the 1991 shift in the charts wrought by SoundScan, a B.C.-A.D. moment that makes comparing old and new hits an exercise in near-futility.

You need to keep this top-of-mind when you take in the so-called All-Time Hot 100, released by Billboard this week as part of its celebration of the chart’s 50th anniversary. The final result, led by Chubby Checker’s “The Twist” and Santana and Rob Thomas’ “Smooth,” is less interesting than the contortions the magazine went through to put them there.

As Maura noted yesterday, the list Billboard has compiled is delightfully argument-starting. A Beatles single did make the Top 10, but it’s beaten by Bobby Darin and Olivia Newton-John. Michael Jackson doesn’t appear until the mid-30s, and it’s with a Paul McCartney duet, not “Billie Jean.” Elvis doesn’t appear until the bottom 20. Even the bad stuff that you’d expect to place higher, doesn’t: Elton John’s 1997 tribute to Princess Diana, the biggest-selling single of all time, doesn’t show up until No. 41; “My Heart Will Go On” doesn’t appear at all.

Of course, there are good explanations for all of these phenomena, and I could spend two months’ worth of this column trying to pick apart the reasons Billboard‘s data helped this or that song. (One big hint: longevity is generally more important than peak position.) I welcome column regulars to fill the comments section below with queries or ideas, and we can pull the chart apart together.

My main message to anyone who questions this chart is this: Unless you really, really loved the ’90s–and I don’t mean grunge or vintage indie, I mean vintage Mariah and Ace of Base–be thankful for this chart. It could have been a whole lot worse.

That’s because, if you look at other recent attempts to chart the last half-century, you come to the bizarre conclusion that the 1990s was the dominant decade of the Rock Era. Previous all-time charts compiled by Billboard (at the Hot 100’s 40th anniversary in 1998) and by the magazine’s chart-trivia god Fred Bronson (his book The Hottest Hot 100 Hits) have songs like “Smooth,” “Macarena,” and “Un-Break My Heart” completely dominating the field.

It is impossible to overstate how radically the advent of accurate counts of sales and airplay changed the charts in 1991. A system that previously had consisted of “surveys”–Billboard‘s polite term–of retail chain managers and radio program directors, who could basically tell the magazine whatever was in their best interest, was transformed into a data-rich compilation of computer-collected cash-register and broadcast-spin tallies. I will never forget the week I picked up Billboard in November 1991 and saw the first post-SoundScan Hot 100, my mouth agape: literally dozens of singles shifted tens of places up and down the chart, reflecting the new reality and deflating songs that had risen largely on payola and illusory sales. (Wilson Phillips, Paula Abdul and Michael Bolton were forevermore evicted from the Top Five.)

But getting an accurate read on the day’s big hits came with a big downside: you learned just how slow the nation’s music metabolism really was. Songs just didn’t turn over as quickly as the industry or the public has previously been led to believe, especially in the post-SoundScan, pre-blog era of the 1990s. Starting in 1992, songs dawdled on the Hot 100, staying at No. 1 longer (the all-time record was broken three times in four years, by Boyz II Men then Whitney Houston then Mariah Carey), in the Top 10 longer (30 weeks, set by “Smooth” in 1999-2000), on the Hot 100 longer (69 weeks, set by LeAnn Rimes’s “How Do I Live” in 1997-98).

What this means is that, if you look at just plain, unfiltered Hot 100 data–peak position, weeks on the list–the songs of 1990s and 2000s are unbeatable. Until 1992, no song had spent more than 10 weeks atop the Hot 100 (one pre-Hot 100 hit, Elvis Presley’s “Don’t Be Cruel”/”Hound Dog,” topped a predecessor chart for 11 weeks). Since 1992, 14 songs have been on top for more than 10 weeks. In the 1960s and 1970s, the average Hot 100 hit had a lifespan of maybe a dozen weeks, the big hits riding the charts for, at most, about four or five months; and in the early 1980s, when Soft Cell’s slow-rising hit “Tainted Love” spent 43 weeks on the list, it set a seemingly unbeatable record. Now, total-week chart runs in the low 40s are a regular phenomenon.

Back in January, Idolator regular Michaelangelo Matos pored through Bronson’s latest “Hottest” book in his “Project X” column. Bronson’s list, which relies on a straight reading of chart position and longevity, was capped by “Smooth.” Surely, the Santana/Rob Thomas smash has an impressive chart run: 12 weeks at No. 1, that record-setting stay in the Top 10, and more than 50 weeks on the Hot 100 overall. But what left Bronson’s readers with a bad taste was the fact that “Smooth” wasn’t alone among its 1990s peer group of outperformers. As Matos noted, the early-’90s shift to better data “has precipitously skewed the upper reaches of [the all-time list]: only 22 of Bronson’s first 100 entries predate 1992. Even if you take into account the payola of the ’50s and ’60s–and the ’70s, and the ’80s–the shift is manifest.”

Matos’s “Project X” post evolved into discussion of pitfalls in trying to compile an all-time list of the greatest pop-chart hits in the first place. As he and I continued to chat offline, we fantasized about an all-time list of hits that found a way to weight older hits to give them a fair shake, accounting for the lack of decent data pre-1991 and the shorter chart runs common back then.

That is exactly what Billboard has tried to do with its 50th-anniversary chart. The mere fact that the list is headed not by “Smooth” but by “The Twist”–a deserving chart champion, given its two separate chart runs in 1960 and 1962, both capped by a stay at No. 1–shows the lengths Billboard went to handicap pre-1991 hits. The rest of the All-Time chart’s Top 10 includes such oldies as Bobby Darin’s “Mack the Knife,” Olivia Newton-John’s “Physical,” Debby Boone’s “You Light Up My Life” and the Beatles’ “Hey Jude.” In short, the winners’ circle is split 50-50 between pre-1991 and post-1991 hits, a trend that persists through most of the chart.

In a good-natured, if somewhat defensive, Frequently Asked Questions page Billboard has posted alongside the chart, the mag attempts to head off the inevitable complaints sure to be lodged by music fans around the world. (Judging by the hundreds of comments already appearing on the site, the rules explanation doesn’t seem to be mollifying anyone.)

Here’s a few key paragraphs from the top of the FAQ:

Prior to the Hot 100’s implementation in 1991 of enhanced radio and sales information from Nielsen BDS and Nielsen SoundScan, songs had shorter reigns at No. 1 and shorter chart lives.

To ensure equitable representation of the biggest hits from all 50 years, earlier time frames were each weighted to compensate for the differences in the faster turnover rates from those earlier decades, compared to the slower churn the Hot 100 has seen since the advent of Nielsen Music data.

The FAQ goes on to explain that older songs were weighted differently, given a multiplier to account for the average length of a chart run when they charted. So in other words, a 20-week run for a 1965 single counts for more than a similar run for a 2005 single.

This, to me, is fair and just. Who’s to say that in 1983, the Police’s “Every Breath You Take” wouldn’t have spent 12 weeks at No. 1, instead of the recorded eight weeks, if more accurate data had been available? If we can give both Babe Ruth and Roger Maris credit for home runs hit in seasons of different lengths, surely we can place Checker and Darin alongside Santana and Thomas.

Like all things chart-related, you can’t take this stuff too seriously. This is a list based on data, not taste. I wouldn’t want to sit and listen to these 100 songs in one sitting. (And I’m not sure I want to hear Flo Rida’s “Low,” No. 23 on the All-Time list, ever again.) But given the limited data available for the Hot 100’s first 33 years of existence, I give Billboard props for at least trying to even out the playing field.

Not that Beatles fans and rock purists won’t complain anyway…

Top 10s Last week’s position and total weeks charted in parentheses (Digital Songs chart includes total downloads/percentage change in parentheses):

Hot 100 1. T.I., “Whatever You Like” (LW No. 1, 5 weeks) 2. Pink, “So What” (LW No. 3, 3 weeks) 3. Rihanna, “Disturbia” (LW No. 2, 12 weeks) 4. Chris Brown, “Forever” (LW No. 4, 20 weeks) 5. M.I.A., “Paper Planes” (LW No. 5, 8 weeks) 6. Jason Mraz, “I’m Yours” (LW No. 9, 21 weeks) 7. Coldplay, “Viva la Vida” (LW No. 7, 18 weeks) 8. Ne-Yo, “Closer” (LW No. 8, 21 weeks) 9. Flo Rida feat. will.i.am, “In the Ayer” (LW No. 21, 12 weeks) 10. Kardinal Offishall feat. Akon, “Dangerous” (LW No. 6, 18 weeks)

Hot Digital Songs 1. Pink, “So What” (LW No. 2, 197,000 downloads) 2. T.I., “Whatever You Like” (LW No. 1, 145,000 downloads) 3. Jason Mraz, “I’m Yours” (LW No. 5, 128,000 downloads) 4. Rihanna, “Disturbia” (LW No. 3) 5. M.I.A., “Paper Planes” (LW No. 4) 6. Flo Rida feat. will.i.am, “In the Ayer” (LW No. 13, 110,000 downloads) 7. Artists Stand Up to Cancer, “Just Stand Up!” (CHART DEBUT, 110,000 downloads) 8. Katy Perry, “Hot N Cold” (LW No. 8) 9. The Pussycat Dolls, “When I Grow Up” (LW No. 7) 10. Coldplay, “Viva la Vida” (LW No. 6)

Hot R&B/Hip-Hop Songs 1. Jazmine Sullivan, “Need U Bad” (LW No. 1, 19 weeks) 2. T.I., “Whatever You Like” (LW No. 2, 8 weeks) 3. Jennifer Hudson, “Spotlight” (LW No. 8, 17 weeks) 4. T-Pain feat. Lil Wayne, “Can’t Believe It,” (LW No. 7, 9 weeks) 5. Young Jeezy feat. Kanye West, “Put On” (LW No. 3, 18 weeks) 6. Lil Wayne feat. Bobby Valentino, “Mrs. Officer” (LW No. 12, 10 weeks) 7. Rihanna, “Take a Bow” (LW No. 4, 20 weeks) 8. Robin Thicke, “Magic” (LW No. 11, 16 weeks) 9. Ne-Yo, “Miss Independent” (LW No. 16, 7 weeks) 10. Keyshia Cole, “Heaven Sent” (LW No. 5, 24 weeks)

Hot Country Songs 1. Brad Paisley, “Waitin’ on a Woman” (LW No. 2, 13 weeks) 2. Jimmy Wayne, “Do You Believe Me Now” (LW No. 1, 24 weeks) 3. Darius Rucker, “Don’t Think I Don’t Think About It” (LW No. 5, 21 weeks) 4. Keith Urban, “You Look Good in My Shirt” (LW No. 3, 16 weeks) 5. Kenny Chesney, “Everybody Wants to Go to Heaven” (LW No. 6, 6 weeks) 6. Kid Rock, “All Summer Long” (LW No. 7, 17 weeks) 7. Keith Anderson, “I Still Miss You” (LW No. 4, 32 weeks) 8. Toby Keith, “She Never Cried in Front of Me” (LW No. 9, 11 weeks) 9. George Strait, “Troubadour” (LW No. 8, 15 weeks) 10. The Lost Trailers, “Holler Back” (LW No. 10, 29 weeks)

Hot Modern Rock Tracks 1. Staind, “Believe” (LW No. 1, 11 weeks) 2. Foo Fighters, “Let It Die” (LW No. 2, 23 weeks) 3. Carolina Liar, “I’m Not Over” (LW No. 4, 19 weeks) 4. Coldplay, “Viva la Vida” (LW No. 1, 14 weeks) 5. Weezer, “Troublemaker” (LW No. 8, 9 weeks) 6. The Offspring, “You’re Gonna Go Far, Kid” (LW No. 9, 7 weeks) 7. Metallica, “The Day That Never Comes” (LW No. 10, 3 weeks) 8. Weezer, “Pork & Beans” (LW No. 5, 21 weeks) 9. Disturbed, “Inside the Fire” (LW No. 6, 24 weeks) 10. Saving Abel, ” Addicted” (LW No. 7, 25 weeks)