How to Measure Ghosts: Arthur C. Nielsen and the Invention of Big Data

How to Measure Ghosts: Arthur C. Nielsen and the Invention of Big Data Family watching Television, 1958. Credit: WikimediaThis is a story about ghosts. About how the technological networks of the twentieth century made us more distant, and yet more connected, than ever before. About the struggle to measure, analyse and interpret the way we use these networks. And at the centre of this story is a man who is in many ways a ghost himself.He was responsible for a series of innovations that helped build the empires of mass media, retail, and digital technologies over the last century, and yet there has never been a biography written about him. His company, as large and influential today as it was when he founded it nearly a century ago, still bears his name, but he is rarely mentioned in literature on business and innovation.So this is his story — how Arthur C. Nielsen discovered how to measure ghosts, and in doing so, created a world built around data. At the most elemental level, stories are about the feedback loop between storytellers and audiences. Storytellers need to be able to imagine, or ideally to viscerally feel, a connection with the audience. Without a connection between a storyteller and an audience, there is no story.At the turn of the twentieth century, this connection between storytellers and audiences was abruptly cut. Before then, the distribution of stories relied on physical networks — printing presses churned out magazines, newspapers and books that were shipped to high street stores and corner news-stands, and gargantuan telegraph and telephone cables laid across ocean floors, or crowded the skies above rapidly growing cities. Stockholm Telephone Tower, 1890But in the 1880s, Heinrich Hertz, a German inventor, demonstrated how to send and detect ‘Hertzian Waves’ — electromagnetic waves that could invisibly travel through the air, crossing through physical objects as easily as open space. Reporters at the time realised the potential to use this new discovery for communication:“Here is unfolded to us and new and astonishing world […] Rays of light will not pierce through a wall, nor, as we know only too well, a London fog. But the electrical vibrations of a yard or more in wave length […] will easily pierce such mediums, which to them will be transparent. Here, then is revealed the bewildering possibility of telegraph without wires, posts, cables or any of our present costly appliances.”London Fortnightly Review, 1892In 1894, Guglielmo Marconi picked up a journal with an account of Hertz’s experiments, and became obsessed with developing the technology. First, he found ways to use Hertzian waves to ring a bell on the other side of his room. Within a few years, he had invented the antennae, and was sending signals further and further, across the fields, hills, then mountains near his family villa.Marconi took his invention to London, where the British Government— then at its imperial height, connected by undersea cables to its colonies around the world — funded his experiments for military use. By 1920 wireless was starting to be used for entertainment — in that year The Daily Mail, one of the most popular British newspapers, broadcast a concert by Dame Nelly Melba across Europe.The growing hype around wireless technology created tension between enthusiastic amateur ‘wireless’ enthusiasts, the British Military, and early entrepreneurs like the Daily Mail. In the UK, the Government introduced legislation to regulate these new invisible networks, and created the British Broadcasting Corporation, putting it under the charge of John Reith, the son of a Presbyterian minister. Reith had a clear-sighted and egalitarian vision for how wireless networks would transform the world, set out in his 1924 book ‘Broadcast over Britain’:“Wireless ignores the puny and often artificial barriers which have estranged men from their fellows. It will soon take continents in its stride, outstripping the winds; the divisions of oceans, mountain ranges and deserts will be passed unheeded. it will cast a girdle round the earth with bands that are all the stronger because invisible.” Not everyone was so sure about the emerging wireless broadcast networks, however. For advertisers in particular, there was a major issue — how can you know who is listening when there’s no feedback loop with the audience? Newspapers and magazines measured circulation by counting unsold copies returned by shops and distributors, and cinema and theatre had ticket sales as a tangible measure of the audience. But for the new broadcast networks, there was no way of knowing who was listening.Radio shows were broadcast via invisible Hertzian waves, flowing across cities and country, rivers and highways, before being captured by antennae in people’s living rooms. Because of this, early broadcast radio audiences were ghosts — invisible, unheard, and, most worringly, unmeasured. In the UK, the BBC was owned by the government and funded by a license fee, with no need of advertising. But in the USA, local networks were being established by entrepreneurs in every city, county and state, and they were struggling to convince advertisers to divert money traditionally spent on newspapers and magazines to this new medium.Radio broadcasters realised that they needed to agree on a consistent measure of attention to convince advertisers to invest in radio. In 1929, the Cooperative Analysis of Broadcasting was established as a collaboration between advertisers and broadcasters, and they hired Archibald Crossley to travel the country to check that the ads bought by advertisers were actually being broadcast by the local stations they were booked on.Crossley was a pioneer of polling — along with George Gallup and Elmo Roper he devised the first opinion polls that used scientifically selected samples to represent large groups. In the 1936 Presidential election, Crossley’s sampled poll correctly predicted a Roosevelt victory, whilst a competing poll that used brute scale rather than scientific samples, issuing and counting millions of random ballots, got it wrong.After doing an initial audit of radio ads for the CAB, Crossley suggested that he measure the audience instead of measuring the ads. He argued that this information would be more valuable for the advertisers, and would reveal which shows audiences preferred, helping broadcasters increase ad prices for the top shows. Crossley invented the term ‘ratings’ to describe this new service, which quickly became the common standard across the new radio networks.To gather data for his ratings system, Crossley had to find a way of contacting the audience across as large an area as possible. To do this, he had to use the physical network that wireless technologies had initially sought to disrupt— the telephone. His team created lists of radio owners, and would call them asking them to recall what shows they had listened to the previous day. But telephone surveys didn’t give a complete picture — they missed out homes that owned a radio but not a telephone, and relied on audience’s ability to remember and recall radio shows accurately. The race was on to find a better system. Solving the ratings problem would require both technical and statistical innovation, and one person had both these skills — Arthur C. Nielsen. He had trained as an electrical engineer, but spent his career innovating in the world of market research. Unlike Crossley, he came not from the world of opinion polling, but from performance and sales measurement. He started the A C Nielsen company in 1923 to provide benchmarks for manufacturing processes, and after the Great Depression launched the Nielsen Food and Drug Index, a sampled report on sales and distribution in America’s fast-growing chains of stores and supermarkets.This was an era of increasing distance between producers and customers. In the late 19th century, manufacturers sold direct to local shopkeepers, who in turn knew their local customers well. By the time Nielsen launched his Food and Drug Index, this had all changed — Clarence Saunders had launched the first self-service chain stores in 1914, and the Sears Catalogue was at its height, using the postal networks and railways of America to sell everything from clothes and wallpaper to guns and baby carriages.Where Crossley, Gallup and Roper were interested in people’s opinions, Nielsen was interested in numbers. He once said to his son, who would later inherit the Nielsen company, “If you can put a number on it, then you know something.” Nielsen reports were driven by data — numbers that represented actual things like sales, not woolly concepts like politics. He already knew how to turn the behaviour of distant consumers into data, and it turned out that this was just what early radio broadcasters and advertisers needed.In 1936, at a meeting of the US Market Research Council at New York’s Yale Club, Nielsen saw the technology that would revolutionise broadcast metrics. It was a demonstration of the Audimeter, a mechanical device for measuring radio listening, developed by Robert Elder and Louis Woodruff from MIT. The Audimeter worked by using a waxed paper tape which turned constantly under a stylus. The stylus was connected to the tuning dial on the radio, and when the radio was switched on, it would come in contact with the tape, etching a line that corresponded to the tuning of the radio. Using this device, Nielsen could precisely measure data about radio listening, based on the listeners’ actual behaviour, not recalled telephone conversations.Nielsen acquired the patent, and over the next six years worked on improving the technology, and the logistics of sending the completed rolls to his analysts to be turned into data. To speed this up, he introduced mailable Audimeter rolls, which dispensed a quarter when replaced with a new cartridge to incentivise radio owners to return them.Unlike Crossley’s recalled ratings, the Audimeter measured all activity, including when audiences switched over from shows they found boring, and they could also capture data from homes that did not yet have a telephone. But Nielsen had a problem convincing advertisers to adopt the Audimeter ratings, as his initial service only included 1,500 households. Nielsen knew that although the sample size was smaller, the data was far more detailed — he estimated that over a year each Audimeter provided the same amount of data points as 500,000 telephone surveys.By March 1948, Nielsen had expanded his sample size across the US, and the amount of data his company was processing led to an interest in the emerging world of computing. The Nielsen company was almost one of the first customers for the UNIVAC, the first commercially available computer, but pulled their $150,000 order when the company making the UNIVAC were struggling to meet orders.Nielsen’s ratings system would find their most influential customer in another technological innovation that was starting to emerge in the 1940s — Television. The analogue Audimeter technology didn’t work with Television sets, so for decades Nielsen reverted to capturing TV ratings using diaries instead. But then in 1971 they launched the Storage Instantaneous Audimeter using microchip technology. This device measured minute by minute data about how audiences were watching TV, just like the original Audimeter did with radio. But even better, the data was sent directly back to Nielsen’s analysis teams using phone lines. By combining new electronic technology with the phone lines that Crossley had used to collect the first ever broadcast ratings, Nielsen closed the feedback loop between broadcasters and audience. Finally, there was a real-time, electronic way to measure ghosts. The Storage Instantaneous Audimeter. Photo Credit: Nielsen It’s difficult to underestimate how important ratings were to the creative and commercial development of television. Even today, planners and schedulers pore over the data from Nielsen’s sampled ratings, using their insight to make multi-million dollar decisions. The ‘sweeps weeks’ — quarterly events in which US TV ratings are measured to set advertising prices — would prompt networks to try and game the metrics by hosting celebrity guest appearances on peak time shows, or scheduling the dramatic resolution of cliff hanger storylines. In 1997, the popular hospital drama E.R. staged a live broadcast episode, performing the same story twice for East Coast and West Coast timezones, just to create a ratings bump for sweeps week.Nielsen’s legacy is not limited to television, though. Starting with his work on market research indexes, Nielsen created a business empire built around data, the size and scale of which is reflected in the current Nielsen Company’s corporate tagline: “Nielsen: What People Watch, Listen To, and Buy”.Nielsen wasn’t the first person to convert audiences into numbers, but he was a pioneer of using databases, statistical analysis and then real-time computer networks to do this at scale, just as the early networked technologies of railways, telephones and broadcasting were making it possible for companies to serve not just local and regional audiences, but national or international ones.If the 21st century has seen the emergence of a world built on big data, then that world is very much the creation of the Nielsen Company. The networks of 20th century mass commerce and media stretched the connection between producer and consumer, turning people from local neighbourhood customers to distant ghosts at the other end of a telephone line, mail-order catalogue or radio broadcast.There is a direct line between the stylus etching a line on the wax tape of the Audimeter as a hand turns a radio dial, and the software recording every swipe and click as our thumbs move around a smartphone screen. From early radio networks to Google, Facebook, Amazon and Apple, we have been turning audiences into data for almost a century. The history of the last century is the history of measuring ghosts, and at the heart of this story is Arthur C. Nielsen.
SIGUE LEYENDO…
Ir a la fuente / Author: Matt Locke