Tyranny of the Scrobble

I spent too much of my holiday weekend thrashing around trying to find a hack to get KPIG playlist extracts into my Last.fm scrobble data. Because I have been avoiding listening to KPIG, because streaming radio doesn't scrobble. And not-scrobbling is bad bad bad ... . headdesk

[Scrobbling is a proprietary technology owned by Last.fm that sucks up your iTunes, WMPLayer, and Last.fm playlists in real time and matches them up with track metadata, and stores everything permanently in your listening profile on their server. Uh, it's a big list of everything you listen to. Except it's not really everything ... ]

In a spasm of clarity, I asked myself just why this was so important.

Last.fm presents itself as a music-specialized social-networking site, and it has those features, but that is not what it brings to the table. The heart of the system is a music-recommending system tied to artist and music info. It looks at your playlist, matches you up with "neighbors" (people who have generally similar listening habits), then recommends artists who are similar to artists you like or who are liked by your neighbors. It also lets you play "radio stations" with tracks from artists similar to one you name, or from your historic playlist, or from your recommendations, or from the playlist of one of your neighbors, and so on. And then it scrobbles those tracks into your profile. (And all this is free--we get back to that point later.)

The quality of music recommendations depends on a lot of things. They have to have useful and accurate information about all the artists. They have to have a good system for matching up similar artists and similar listeners. They have to have a way to distinguish an artist who is played by a few people constantly from an artist who is played by a lot of different people. To work well in general, it has to have a VERY BIG database of scrobbled play data. The more data they have, the better they can build neighborhoods, make recommendations, etc. (Also, the more users they have scrobbling, the more they can persuade artists to give permission for them to stream out their tracks.)

I have no control over any of that. I can only vote with my feet based on how good their recommendations are. But my recommendations also depend a lot on my own play history: more play data, accurately spread across my music preferences will get me better recommendations.

And that is what has been bothering me. Usually at work I listen to RealRadio for hard stuff; KPIG for country, western, and other old-timey stuff; and Batanga.com for Latin stuff. I also do a lot of clicking around to other streaming stations to see what is around. I have been so satisfied with that, I haven't been buying music in those categories, mostly. But if I stream that way, the playlists don't scrobble.

I have found the Last.fm "stations" too narrowly and mechanically programmed for me so far (although I may yet find the magic entry point that will lead to a broader range of artists in one "station"). It is missing the personality and serendipity you get with even corporate radio programming that can lead you into new threads of music. It is good for helping you focus in on one kind of music, but not so much for broadening.

I don't own nearly the range of music that I listen to, so I can't play that range in iTunes, so I can't scrobble that way. And if I stop listening to music from a variety of sources, how will I find the new stuff to scrobble in to my Last.fm profile so I can get good recommendations there so I can find good new stuff? Last.fm will only recommend music of the same general kind(s) as music it has in my profile, so if whole genres never get in there, Last.fm will never recommend any of it.

This is most important for the country, western, old-timey patch of the musical landscape, because it has been a lot harder for me to find new stuff there than in other flavors of music. KPIG has been a godsend for that, so that is how I came to spend the weekend trying to figure out how to trick Last.fm into scrobbling KPIG playlists that I had listened to. No joy, of course.

This is a place where open standards would save us. If there was a standard format for sending and capturing playlist data, then Last.fm could scrobble off of anything. Radio streaming services could deliver play data to my local client, and Last.fm could scrobble away merrily.

There are a lot of reasons why this doesn't happen, and what would get us from here to there is interesting to think about.

But what I was thinking about today was business relationships. Apple has pretty much committed to open iTunes standards, as far as I can tell. Real has pretty much committed to being poopyheads about datasharing, again as far as I can see from my chair. PayPlay.fm apparently has some kind of business relationship with Last.fm, because their recommender can read your Last.fm profile.

It's in the interest of any company like Last.fm to keep the important data flowing only inward, because because all that data is extremely valuable. Their service is not ad-free, but it looks pretty ad-lite, so how are they compensating the artists whose music they stream? What incentives can they give artists to sign on? That has to come out of the value of the data, somehow. How exactly, I don't know, although I could make some good guesses what record labels could mine out of that data. (There is also the specter of shilling--tilting the recommendations to favor artists or labels who pay for that.)

This is not just idle speculation. CBS just bought Last.fm for $280 million. Although the autoscrobbling system is pretty slick and they could well set up their CBS streaming radio stations to support scrobbling, it's not worth that much. CBS was pretty frank that they bought it for the reasons that indusry watchers figured--to get their hands on that mountain of play data.

Music isn't the only kind of data. Think of Google Maps. That's a ginormous database with an nice open interface that Google maintains for free so that they can have the intelligence to put the little ads on the side of my screen. Any one can lay their own application on top of the Google Maps database--Google makes it as easy as possible. Google Maps on my handheld will use real-time GPS data to let me track myself and anyone can make and share Google Earth overlays using GPS data. People are willing to keep putting data into the various ginormous Google databases because Google gives good value back without a lot of arm-twisting. Google can serve up the raw data and sweet clients generously, because they have smart ways behind the scenes of grabbing and using METAdata on how people use that subject data.

So will CBS try to be the Google of music data? Or will they try to make a quick obvious buck and drive everyone away? I am really interested to see how it goes.

In the meantime, I will keep pumping in data and using the Last.fm service as well as I can. If anyone knows how to import a table of play data from a delimited text file into a Last.fm profile, please let me know.