A random Shuffle
This Guardian UK story on the randomness of the shuffle feature in Apple’s iPods and iTunes was certainly interesting. Like many, I have also wondered about the feature. How does it work, is there a detectable pattern to it, and so on. I used to wonder about it a lot when I was using an iPod Shuffle. For one thing, the pool of songs on it wasn’t nearly as large as what I’ve got on my iPod Video (and I’m not even using half of my 30 GBs!), and for another the Shuffle would sometimes get stuck in a circle of a varying number of songs. I never figured out why or how, but every now and again it would shuffle only 10, or 20 – or five – songs, over and over. And then it would resume normal shuffle a few days later. It was baffling. The iPod Video has never done that, but even so, it seems to cluster just a bit – picking songs that were added to it around the same time. It’s not that noticeable, and it turns out that my assumption about that might just be all in my head.
There’s certainly a worthy perceptual isssue here.
The article quotes Jeff Robbins, a member of the iTunes development team says:
“We’ve many times proved to ourselves that it is truly random, because every now and again, at least once a year, we get the ‘Is this really random?’ question, or someone asks if you guys just have some sort of bug,” Robbin says. “No, no, no – it’s truly random.”
Robbin is talking randomness in terms that software can reasonably produce, which is not perfect randomness. True randomness, it turns out, is very difficult to produce. This subject was most famously examined by Claude Shannon, arguably the Father of Randomness. Shannon himself expressed some random behaviour: the MIT maths professor was known for his eccentric habits, which included riding a unicycle. But his papers on information theory are rock solid. Basically, he defined randomness as a question of unpredictability. If a series of numbers is truly random, you have no possible way of guessing what comes next. If something isn’t random (as in the case of what letter might follow another in a message written in English), you have a better chance of figuring out what comes next. That’s why it’s so crucial to remove the natural redundancy of language from an encoded message and make the coded text look random…
But perfect randomness is an elusive ideal. For instance, if you’re flipping a coin, a minuscule weight imbalance might, over the course of millions of tosses, make heads come up slightly more than tails. And if you’re randomising on a computer, you have to introduce a “seed”, which is a starting point for the algorithm that mixes up the selections. The seed must draw on some unpredictable input of time that begins outside the computer. Otherwise, the results would be the same over and over again.
Even then, a peculiarity in the computer hardware may prevent you from attaining absolutely pure randomness. In certain cryptosystems, the search for the most unpredictable seed relies on quantum behaviour of atomic particles.
“Apple doesn’t need iTunes to be random to the degree that you need randomness for cryptography,” says Paul Kocher, CEO of Cryptography Research. The consequences of less-than-perfect randomness on an iPod aren’t as dire as a broken national-security cipher, say. So, for purposes of mixing up songs, you don’t really need to draw on quantum disintegration, just a reasonably strong pseudorandomising function in your shuffle, which Apple insists it has.
That bit about the difficulty in creating something “truly random” is extremely fascinating. It seems to suggest that the world is very much an ordered place, even if the nature of that order isn’t obvious to most of us, most of the time. It makes you wonder, for example, how the big bang could “just happen”, how evolutionary mutations “just happen”, or what a metaphysical “seed” might be. I’m certainly no math expert and am at risk here of being rapidly out of my depth. The article continues:
We think the shuffle is flawed, but the problem is actually in our heads. Even if we know something about maths, cryptography and statistics, we still can’t deal with randomness when it comes up at the spin of a click wheel. Steven D Levitt, the self-described “rogue economist” who co-wrote the bestselling Freakonomics, also fell into the trap. Writing on his blog, he professed constant surprise at how often his iPod shuffle “plays two, three or even four songs by the same artist, even though I have songs by dozens of different artists on it”. But as a statistics maven, Levitt understood that the bottom line is that “the human mind does badly with randomness”.
Indeed, says Kocher, “Our brains aren’t wired to understand randomness – there’s even a huge industry that takes advantage of people’s inability to deal with random distributions. It’s called gambling.”
So why does Autofill produce nine Springsteen songs out of 188? Because that is what almost always happens in normal distributions of items from databases. Clusters of something are to be expected. Here’s a classic maths trick: gather 40 people in a room and have everyone write down the day he or she was born. What are the odds that two people will have the same birthday? Nearly 100%.
Putting aside the issue of clustering, which revolves around a difference in scientific and lay use of the word “random”, what we have here are mathematicians telling us that 1) it is hard to create anything that is “truly random” and 2) that the human mind is much prone to pattern finding. Both of those claims seem to be quite reasonable, and to fit with the poetry of mankind “made in the image of God.” As I understand it, one of the things that flows from that is God is the “seed” from which everything else flows and depends, and we can find that ultimate pattern because we are made in his image.
If I’ve oversimplified anything here, I sure would like to know about it. Comment away if you feel you can help.