ON THE MORNING OF 27 JANUARY 2004—the day after India’s 55th Republic Day—Prime Minister Atal Bihari Vajpayee drove to Rashtrapati Bhavan to meet President Abdul Kalam. He was carrying a letter from his cabinet, which recommended that the Lok Sabha be dissolved so that the National Democratic Alliance (NDA) government could face elections eight months before the expiry of its term. The decision was bolstered by the confidence gained from its stellar performance in the state elections held the previous month; the Bharatiya Janata Party had defeated the Congress in three crucial states: Rajasthan, Madhya Pradesh, and Chhattisgarh.
Four days later, India Today published an opinion poll predicting that the NDA would win a whopping 330 seats. No party or alliance had achieved such a victory since the Congress won 415 seats in 1984. A reassured Vajpayee called on the president again, on 5 February, and told him that all consitutional formalities—including passing a finance bill that would allow the new government to pay for essential goods and services while it drafted a new budget—were complete. Kalam dissolved the 13th Lok Sabha the next day, and on 1 March the Election Commission announced that voting would begin in April, six months earlier than it was due.
Following the announcement, at least five more polls by major news networks and pollsters projected that the NDA would win somewhere in the region of 270 Lok Sabha berths. The Congress and its pre-election allies were predicted to get anything between 150 and 170. With no outliers, everyone in the media was confident that they were more or less right. In mid April, Vir Sanghvi, then an editorial director for HT Media, which publishes the Hindustan Times, wrote in that paper, “I don’t know of a single person who thinks that the Congress will get more than 120 seats and most people say it will get even less. Plus, I suspect that Vajpayee as Prime Minister is probably unbeatable.”
By late April, it was clear the Congress had made some gains, but pollsters and journalists were still confident of an NDA victory. The cover of the 26 April edition of India Today peddled an “exclusive” survey (commissioned in partnership with Aaj Tak and Dainik Bhaskar, and conducted by the market research firm ORG-MARG), which it claimed was the “most exhaustive opinion poll ever undertaken in India.” After surveying over 50,600 voters from 185 Lok Sabha constituencies across the country, it predicted that even as the Congress “gained momentum”, the NDA would win 282 seats. Even in early May, after three rounds of the four-phased election, exit polls had the NDA winning—although it was predicted that the coalition might miss the majority mark of 272 by a whisker.
Three weeks later, when the Election Commission announced the results, the bedrock of the NDA, Vajpayee’s Bharatiya Janata Party, had lost about 1.6 percentage points from the vote share it had garnered in the last Lok Sabha elections, in 1999, sliding from 23.75 percent to 22.16 percent. The Congress lost 1.8 percentage points, dropping to 26.53 percent of votes nationwide. But despite similar declines in vote share, the two parties achieved contradictory results. While the BJP lost 44 seats, its main challenger gained 31. In the end, the NDA did not even touch 200, falling 15 short. The Congress’s alliance marshalled 217, and eventually secured enough external support to form a new government.
It wasn’t the first time that political polls had been inaccurate in India—but calling the 2004 polls inaccurate is like calling a tsunami a ripple. Not only did every major pollster and media outlet get the numbers wrong, they miserably failed to predict the overall trend. Worse, they had oversold their polls in a way that now seemed disingenuous. India Today, which had conducted 155 opinion polls since 1978, would have known that even the most rigorous survey can get the final outcome wrong. But the magazine, and others in the media, presented their various polls as definitive forecasts.
Some members of the press, humiliated by the discrepancy between their headlines and the results, laid the blame on pollsters. “I don’t see any reason why this magazine should carry the can for the incompetence and ineptitude of desi and foreign pollwallahs who use the media in order to pontificate on their brilliant ‘scientific models’,” an angry Vinod Mehta, then the editor-in-chief of Outlook, wrote. “Some readers are convinced Outlook manipulates its polls. I would like to inform them that I only pay huge sums of money to pollsters so that they can embarrass me with hopelessly inaccurate predictions.”
TODAY, despite the embarrassment of years like 2004, there is little to suggest that the media has become more cautious in selling opinion polls. Survey after survey is proclaimed to be the definitive exercise in election forecasting. Every seat, every state, every alliance and, finally, every election, is the subject of self-assured prophesying.
Perhaps as a corollary, cynicism about opinion polling seems fairly widespread. Since 1997, the Election Commission has contemplated banning opinion polls, claiming, without clear evidence, that voters will be unfairly swayed by survey results. In a letter written to the Election Commission on 30 October this year, the Congress supported the proposal to prohibit the dissemination of surveys once elections are announced. The party said polls “lack credibility” and had the potential to be “manipulated”.
But the scepticism isn’t limited to political parties, which are generally eager to protect the morale of their cadres from negative survey results. In the last two general elections, the majority of opinion polls have been off the mark when it comes to the number of seats the two major parties will win, and this seems to have fostered public distrust of pollsters and their methods. If the final number is wrong, many people tend to assume a poll was either manipulated or based on flawed science.
In reality, however, most of the opinion polls published and broadcast by national media houses are fairly robust exercises led by social scientists with decades of experience. Although they sometimes get the final tally of seats wrong, this isn’t necessarily the result of poor polling methods or bias. Rather, it’s a reflection of the inherent difficulty of distilling into a handful of numbers the vastness and diversity of the Indian electorate and the fickleness of Indian politics—a challenge of which pollsters and editors are acutely aware.
But opinion polls are nevertheless presented as a sort of political gospel, and the media often gives viewers and readers only the final outcome of a survey—seat projections. But these projections are subject to greater uncertainty than any other product of the entire polling exercise. Behind the numbers are two complex, expensive processes. The first—comparatively easy but by no means simple—involves holding structured conversations with a large number of people to understand how they are likely to vote and to determine a party or candidate’s probable vote share. The second is to create seat projections by subjecting vote share to various mathematical models and to adjustments based on historical precedents and a dizzying array of political factors.
Rarely does the complexity of these processes get conveyed by journalists, many of whom feel the public are only interested in seat projections. “If we don’t give seat number projections based on vote shares, the viewer feels cheated,” Rajdeep Sardesai, the editor-in-chief of CNN-IBN, told me. “Then the viewer says ‘tum cop out kar rahe ho’ (You are copping out). You are not telling me who will get how many seats. I am not interested in the vote share.’ So after having spent so much money, if you give only vote share then viewers and readers are upset.” Unfortunately, Sardesai added, “most viewers and readers expect it to be arithmetic, saying, ‘why didn’t you get this number right?’”
Most of the opinion polls available for public consumption are solely commissioned by the media. But there is little consensus on why they do it. Almost all the editors I met while reporting this story believe their audiences are interested in the surveys, but there’s no way to measure that. Sardesai and other television journalists said it does not make much of a difference to their viewership numbers. Newspaper and magazine editors told me that in India, publications are largely subscription-driven and readers seldom buy papers off the stands, so it’s impossible to detect short-term fluctuations in readership. But Sardesai claimed that some of the data can be used to improve reporting on topics that survey respondents care about, and that, he thinks, is extremely valuable for any editor.
Some journalists I spoke with called polls a publicity stunt, even though it’s not clear that surveys contribute to a media organisation’s bottom line. Others said that polls add value to their socio-political understanding and reporting. A good opinion poll—one that is transparent about its methodology, adequate and representative in its sample, and honest about its funding—is a tool that can help journalists understand the mood of broad segments of the population. It can elucidate voting behaviour even if the seat numbers are off the mark, providing insight not only into who the electorate wants in power, but also why they are choosing one candidate over another—whether it’s a local representative they favour or a national leader, or a party, or an alliance. As Sardesai put it, “polls should only be a starting point, frankly, for you to analyse what is happening across the country.”
Unfortunately, the public appetite for seat projections is fuelled by journalists who like to talk up their polls and boast about knowing the pulse of the nation. The gap between the best possible use of surveys and the reductive way in which they’re presented has led to a widespread distrust of polls—the outcome of a sometimes lazy, sometimes careless and sometimes dishonest approach some journalists take towards their responsibilities. Too often, the running mantra is: the pollster takes the blame if the numbers are wrong, the editor takes the credit if they’re right.
LATE ONE EVENING IN EARLY AUGUST, I bumped into Sardesai in the foyer of the Noida Film City building that houses CNN-IBN’s newsroom. About a week earlier, the channel had run poll-based forecasts on what would have happened if general elections had been held in July. The survey indicated that the NDA could get up to 180 seats, beating out the incumbent United Progressive Alliance, which was projected to get only 153. Sardesai told me that he had strong opinions on polling, and agreed to meet to discuss them further.
A week later, on a Sunday evening, I sat across from Sardesai at his bungalow in a quiet, leafy south Delhi neighbourhood. He was relaxed—wearing shorts—and spoke thoughtfully. “My sense is, looking at this from the television I have done for 18 years, first when I was at NDTV and now at CNN-IBN—so I have been privileged in a way to have done it with people who I consider the best in the business—the sense I get is that opinion polls can be done seriously and can lend themselves to serious political analysis,” he said. “But opinion polls can also become a business and a gimmick.”
In a way, though, opinion polls have always been about both business and analysis—about the relationship between the media and its consumers as much as about the relationship between the citizenry and its political choices. Although much of the polling that’s undertaken today looks more like statistical modelling than beat reporting, the practice has its roots in a very journalistic enterprise. “Opinion polling is a child of the newspaper world,” the Swedish sociologist and pollster Hans Zetterberg has written. “Only later did the academic world of social science enter as a stern stepfather.”
George Horace Gallup’s doctoral thesis in journalism at the State University of Iowa in the late 1920s described a method for gauging reactions to newspaper features by sampling the opinions of a carefully selected set of readers. Over the next few years, he developed his techniques and founded the American Institute of Public Opinion, which ran what would become the famous Gallup Polls—fortnightly surveys of American public sentiment that he syndicated to newspapers across the country. Although the polls were partly meant to gauge the mood of the nation, Gallup also expected them to improve newspaper sales, and they soon became popular with editors.
In 1936, when Gallup’s company was barely a year old, it became famous for correctly predicting not only that Franklin D Roosevelt would be re-elected to the United States presidency, but also that the country’s most popular survey at the time—run by the magazine Literary Digest, which collated postcard questionnaires from roughly 2 million people—would get its forecast wrong. Gallup’s major insight was that the number of people polled was less important than the extent to which those people represented the balance of views in the country’s various constituencies. A humiliating failure to predict the outcome of the 1948 presidential race taught Gallup—who was so confident of his result that he stopped collecting data several weeks before election day—that opinion polls also had to keep up with the dynamic nature of public sentiment.
MANY PEOPLE CONSIDER 1980 to be the dawn of Indian polling, when two young economists (Prannoy Roy and Ashok Lahiri) teamed up with two young market researchers (Dorab Sopariwala and KMS “Titoo” Ahluwalia) to correctly forecast the general elections on behalf of Aroon Purie’s five-year-old fortnightly magazine, India Today. But the “father of Indian polling,” Ahluwalia told me, “was the one who got it famously wrong”.
Unlike in America, it wasn’t journalists who originally felt the need to conduct opinion polls in India, but an Oxford-trained economist named Eric da Costa. By the early 1950s, da Costa had left a job in the civil service and became the editor of a publication called Eastern Economist. He soon met Gallup and Henry Durant, the director of Social Surveys Limited (the British counterpart of the American Institute of Public Opinion). Based on his conversations with them, da Costa founded the Indian Institute of Public Opinion, in Delhi, which was modelled on Gallup’s US operation. IIPO conducted and published its first national poll before the 1957 general elections.
The challenges facing da Costa in India were much tougher than the ones Gallup and Durant had to account for in the US and the UK. Polling and election forecasting is a complicated process in any democracy, but in India it is compounded by a multitude of ethnic, caste, religious, linguistic and regional identities; by the size of the electorate and the remoteness of many of its members; and by the proliferation of major parties—sometimes as many as seven in a race—battling it out in multi-pronged contests at both the state and national levels. Added to this is the fact that, although the public is often interested in who will become prime minister, it’s ultimately not their votes that decide, but the personalities and internal politics of the winning party or coalition. Not for nothing has India been called a “pollster’s nightmare”.
In his later years, Ahluwalia told me, da Costa was “really a very impressive old gentleman, very refined, very erudite. But quite opinionated, I thought.” For the fourth Lok Sabha elections in 1967, da Costa “did several polls that I think were front-paged in the Indian Express,” Ahluwalia said. According to an article by the historian Ramachandra Guha, published in The Hindu, a 1967 report by da Costa forecasted “the disintegration of the monolothic exercise of power by the Congress party”.
Although the Congress ultimately lost power in many states, Indira was able to maintain control at the centre for the next ten years. In many people’s eyes, da Costa got the most important part of his assessment—the overall picture—wrong. (In fact, however, da Costa’s numbers appear to have been fairly right on.) “So I think pollsters went into hiding for a while,” Ahluwalia said, laughing. “That got opinion polling on to a very bad start.”
The resurgence in Indian political polling began in 1979, when Roy and Lahiri, then young professors at the Delhi School of Economics, created an election forecasting tool called the Index of Opposition Unity. At the time, Congress (Indira) was the single largest party and, even though it wasn’t in power, Roy and Lahiri theorised that the likelihood of a successful challenge to the party depended upon the solidarity of its opponents. Roy approached Purie, whom he knew from their days at the Doon School, to see if he wanted to publish the results of their predictions.
Before starting India Today, Purie had studied with Sopariwala at the London School of Economics. Sopariwala was now working with the Indian Market Research Bureau under the leadership of Ahluwalia, who was only in his late twenties but already heading one of the largest private market research firms in the country. Ahluwalia and Sopariwala had previously done some commercial polls for India Today and some political polls for other publications. Purie brought them together with Roy and Lahiri. “And we got it right, several times,” Ahluwalia said as we chatted in his apartment overlooking the Arabian Sea in south Mumbai. “Including, most famously, the Rajiv Gandhi victory.”
In 1980, the India Today team predicted a solid majority for the Congress (Indira), which ended up getting 353 seats. In 1984, after Indira’s assassination, they officially gave Rajiv Gandhi up to 400 seats, but said that these results might be “understated” and he could actually go on to win even more. He won 415.
Five years later, they hit the bullseye with an exit poll accurately predicting the Congress would win 193 seats. Purie threw a party at his house to celebrate the poll just as the final results were being announced, Sopariwala told me over email. The results for Uttar Pradesh’s Pratapgarh constituency came in around 10 pm, pushing the final tally to exactly 193. “There was celebration all around and bottles of champagne were opened,” Sopariwala said. The file that contained the polling data was named “Spot On”. “For years, we referred to that election in our discussions and hoped that we’d get another ‘Spot On’,” Sopariwala added. They never did.
“It was just bloody good luck!” Ahluwalia remembered. “We suddenly began to be seen as whiz kids.” This seems to have set the precedent for the way opinion polling is often viewed today—as an exercise whose sole purpose is to foretell the number of seats the winning party will bag on election day, and which is only of value to the extent that it gets this number right.
ON 7 MARCH 2012, CNN-IBN aired a half-hour show called “BATTLE FOR THE STATES: Ask Yogendra Yadav”. In the opening minute, a man’s voice intoned: “Uttar Pradesh has given an extraordinary verdict for the Samajwadi Party. When everyone spoke of a hung assembly, the CNN-IBN–The Week–CSDS post-poll survey predicted exactly this verdict.” Then came a video message from Uttar Pradesh’s chief minister-elect, Akhilesh Yadav: “I really congratulate the people at CNN-IBN. They brought in a wave of celebrations to our cadre even before the sixth [of March].” Reacting to the results for Madhya Pradesh, the Congress leader Digvijaya Singh appeared on a split screen opposite a very pleased Sardesai. “You won the bet Rajdeep,” Singh said. “I owe you a dinner.” A graphic popped up on screen: “CNN-IBN = ELECTIONS”.
The camera cut to a beaming Sagarika Ghose in her Noida studio. “Hi there, good evening. Yes, CNN-IBN equals elections,” she said. “The assembly elections of 2012 saw some hits and misses. But once again the CNN-IBN–Week–CSDS post-poll survey got the assembly elections results of 2012 right.”
“Over the next 30 minutes, we will put your questions to my old friend Yogendra Yadav, senior fellow at the Centre for Studies of Developing Societies [CSDS] someone who many call India’s election oracle,” Ghose continued. “Oracle of Delhi, not the oracle of Delphi.” She looked over at Yadav, who sat with her in the studio. “Just to take it off, let me begin by asking you a question,” she said. “Are you satisfied with this particular post-poll survey that you have done?”
“In fact I was thinking about it when you said we got it right, asking myself, did we really get it right?” Yadav said, looking slightly uncomfortable. “I mean, we got the big picture right. That’s important … Did we get the exact things right? No, we did not. We overestimated the extent of Mulayam Singh’s victory in terms of seats, but even more in terms of votes.”
Ghose jumped in: “He’s ended up with a 29 percent vote share while we have given him 34 percent.”
“Which is a significant difference,” Yadav replied. “And basically, you see, our ambition in the long run, Sagarika, is to make polls something which is not discussed too often. Doctors don’t discuss thermometers.”
Ghose tried to summarise: “It is to catch the trend.”
Earlier this year, I met Yadav, one of India’s foremost psephologists, in his office at the CSDS campus in north Delhi. Yadav founded the Lokniti Project at CSDS in 1997 to collect data on voters’ opinions before and after every major state and national election. (He stepped down as the head of the project last year, when he joined the Aam Aadmi Party. Though he is still a part of the CSDS team, he does not participate in producing opinion polls or seat forecasts now.) The project’s goal is partly to monitor “the working of Indian democracy”.
“My sense is there is unnecessary mystique” and a feeling of “black magic about polls in our country,” Yadav said to me. “Largely because these are new things and people are not familiar with this stuff. Essentially, opinion poll is nothing but a very systematic way of holding conversations with a very large number of people. And the findings are presented not in the form of quotations from those conversations but in the form of numbers, because you can’t report 5,000 conversations. In that sense they are not in principle different from news reporting.”
“The trouble is that in our country opinion polls have been reduced to election-related polls, election-related polls have been reduced to election forecasting,” he added. He spoke softly, confidently, calculating each word. “Now that is a very limiting way of looking at polls.” The poll CSDS did for CNN-IBN in 2012 estimated that 34 percent of votes in Uttar Pradesh would go to the Samajwadi Party, and 24 percent to Mayawati’s Bahujan Samaj Party. This 10 percentage point lead was three times the actual difference—well beyond the survey’s stated margin of error—and it seemed to bother Yadav. But Ghose was happy that her channel had picked the winning side.
Part of the problem, pollsters feel, is that journalists don’t understand the usefulness of the data that underlie seat projections. “They are uncomfortable with numbers,” said Yashvant Deshmukh, the head of CVoter, one of the largest private socio-political polling agencies in the country. He told me journalists don’t look behind the final numbers to see what went into the result. “That is why media is coming up always with so superficial, over-simplistic analysis, virtually pedestrian analysis, for every mandate,” he told me.
Since 1993, when Deshmukh founded CVoter after graduating in journalism from the Indian Institute of Mass Communication, he has conducted polls for almost all major English- and Hindi-language national news channels; some regional ones; magazines including The Week and India Today; the Hindustan Times and Indian Express; and various other institutions, according to his company’s website.
“Over the period of all these years the media was not educated about the polls,” Deshmukh said. “The numbers—what exactly a poll means, what is the probability, what is the margin of error, what do you get out of the poll. Scientifically speaking, what you get out of a good poll is the vote share.”
He gave me an example of how vote share often gets misunderstood. “Suppose if you say today that Congress will get 30 percent of the votes, that means scientifically speaking it will get anywhere between 27 to 33 percent of the votes. If you are projecting Congress 30 percent and BJP 28 percent, that means your margin of victory or defeat are very much within the margin of error. So Congress getting 28 and BJP getting 30 is also a probability. People don’t look into this. People don’t understand this.”
Projecting how many seats a party will win is an entirely separate affair. “Conversion of those vote shares into seat share has nothing to do with the polling business. It’s not part of the polling at all. Survey stops at the projection of the vote share. Unfortunately in India nobody wishes to understand that. Everybody wishes to look at the seat. Seats batao kitni hain. Vote chhodo, seat batao kitni hain. (Tell us how many seats. Leave the votes, tell us how many seats.)”
Later, I asked Deshmukh if the media was interested in understanding these complexities of polling and projection. “No, not at all,” he said, “It is painful.” He didn’t fault the public for not realising what a good poll actually tells them. In India, “my vote doesn’t really get translated into number of seats,” he said. “It’s not a presidential system. Popularity ratings don’t convert into votes, votes don’t convert into seats. Seats don’t convert into alliances. Alliances don’t convert into government, in majority figures. When you have so many intangibles and the entire population including the media is highly illiterate about what to expect from the polls—if the entire media is asking, number batao, kisko kitni seatein? Toh agar media hi yeh pooch raha hai, public kya poochegi? Public kya poochegi? (Tell us the number, how many seats to whom? So if the media itself is asking this, what will the public ask? What will the public ask?)”
AT THE START OF TIMES NOW’S national poll projection show on 29 July, a methodology note flashed up on the bottom of the screen. It stayed for three minutes—more than ample time to read the description: “National representative sample of 13052 randomly selected respondents across all states during 18th July to 24th July 2013; Date weighted to known population profile; margin of error is +/-3% at national level and +/-5% at regional level.”
Arnab Goswami got the ball rolling for the two-hour programme. “Who is going to form the next government in India if elections were to be held now?” he asked, before introducing his panellists. “To my right is Yashvant Deshmukh, the man who takes all the blame, and we get all the credit. That’s a terrible cliché, but I know Yashvant is used to it.”
That Deshmukh takes the blame and Goswami’s channel gets the credit holds true for more than just the final forecasts. Opinion polls are often criticised for a lack of information on how they were conducted. In June, for example, the economist Vivek Dehejia criticised polls in India generally (and a specific poll in particular) for their poor reporting practices. “The GFK poll tells us only that interviews were conducted “in respondents’ homes and in street corners” but gives us no indication that subjects were picked randomly,” Dehejia wrote in Business Standard. “Also, as is typical with Indian polls, we are not told the margin of error, so have absolutely no way to assess the accuracy of the predictions.”
Dehejia seemed to suggest that the lack of transparency in Indian polling was the fault of pollsters. But methodology notes such as the one aired by Times Now—although it would have failed standards established by various international polling bodies—satisfy the limited guidelines published by the Press Council of India (PCI). The PCI recommends that a newspaper publishing a survey should indicate which institutions carried out the survey, who commissioned it, the size and nature of the selected sample, the method of selection of the sample, and the possible margin of error in the findings. Even though the PCI’s jurisdiction is restricted to the print industry, it is the only institution in the country that produces any sort of recommendations for publishing polls. Most of the media houses adhere to them as indifferently as the PCI seems to lay them down.
Yogendra Yadav, who has been conducting polls and surveys since 1996, said he wasn’t aware of these recommendations: “As people who carry it out, at least we have never received any guidelines from anyone.” He agreed that there is a need to make opinion polls more transparent. “I have personally been fighting for it,” he said. “I have written about it. I have written to all kinds of people who matter to say please make it mandatory for every opinion poll to disclose their methodology in great length. I should exactly know the methodology followed for the survey. I should exactly know the method for vote-to-seat conversion. I should know who paid for this survey, who was the customer, who commissioned it, who paid money for the survey. And what is the track record of the agency who is doing it? Is there any conflict of interest? These are absolutely standard procedures.”
Accusing just the pollster of opacity in an environment where there is no code of conduct or monitoring body is unfair. It gives the gatekeepers of information—the news channels and publications—a guilt-free pass. Some outlets, such as CNN-IBN, which works exclusively with CSDS, do an admirable job of ensuring the transparency of their surveys, putting thorough methodological notes on their websites. But others fall far short of this standard. If a poll is like reporting, only multiplied, it’s the editor’s job to make sure his correspondent isn’t relying on shoddy research, personal bias or an outright plant. Like a bad story, if a bad poll is published, the editor and the publication have to share the blame.
DORAB SOPARIWALA IS IMMACULATE AND POLITE. He is also the only person I have ever met who talks about god as the “woman up there”. His former colleague Titoo Ahluwalia called him, along with Yogendra Yadav “amongst, I would say, the finest pollsters anywhere on this planet. They really are extremely knowledgeable, with impeccable integrities.”
I met Sopariwala in Prannoy Roy’s office. Roy was not present, busy with one of the many duties of running a media company. The office, hidden on one side of the NDTV newsroom in south Delhi, is small but extraordinarily pleasant, lacking the pomp one might expect from the founder of a major national television network.
As we discussed issues of political polling and election forecasts, Sopariwala told me about the first poll he got wrong. It was “when NTR came”, he said. In a paper Sopariwala and Roy wrote in 1990, they described how the Telugu “cinestar of mythical proportions” Nandamuri Taraka Rama Rao founded the Telegu Desam Party and entered Andhra Pradesh politics on a platform of his own fame in 1982. Sopariwala was hired by India Today to do a statewide poll ahead of the January 1983 assembly election. “To meet the deadline of the sponsor, the fieldwork began just after the nominations closed,” with three weeks to go before elections. “The poll indicated a victory for the ruling Congress.” But Rao won a “massive victory” and became the chief minister. In a “post-mortem” by Sopariwala’s team, they learnt that the pollsters had “gone in too early and missed a strong ‘late’ swing”.
Sopariwala told me that after Rao’s “Chaitanya Ratham” campaign—the actor drove a converted Chevrolet across the state—which was quite close to the election date, “he swept” the votes. “We couldn’t catch the wave. I was so embarrassed I didn’t charge my fees.” He joked that he should have charged double, because he had to go back to a lot of the people they had interviewed in order to understand what went wrong. “I don’t work as hard on anything else,” he said of election forecasts. “I kill myself doing it. You think I want to get it wrong?”
The process of opinion polling starts with understanding the objectives of the poll—the topics on which it’s meant to sample opinions. A questionnaire is then designed that, in most cases, is supposed to solicit people’s views without distorting them (for instance, by asking leading questions or using certain loaded terms). Often, several questions are asked to get at the same issue—such as preferences for a given politician—from different angles. A method is then devised for selecting and interviewing a representative sample of the electorate, and a sufficient sample size is decided. Most pollsters either send teams to conduct face-to-face interviews, or they call people up on their mobile phones. The data collected are then used to calculate vote share for each party, and this is finally converted into the likely number of seats each party would grab if the elections happened when the poll was carried out. Every step has its possible pitfalls, all of which are compounded by the diversity of the country’s population and the complexity of its politics.
Insufficient sample size is perhaps the most frequent criticism put to pollsters in India. How can a sample size of 30,000 or 50,000 be representative of more than 1.2 billion people? These sorts of questions, Ahluwalia told me, are “just as old as the polls. And they just show a very low level of debate on the subject. And unfortunately they still happen to this day.” Sample size, he said, hasn’t got “anything to do with the size of the population.” Instead, it’s all about heterogeneity. “How much tea do you need to taste to know whether the sugar is alright or not? If the sugar is stirred, one sip will tell you. If it’s not stirred, you can drink it right up to the end and not know.” Pollsters try to solve this problem by identifying homogenous clusters at sub-regional levels, and then building up a picture of the the regional, state and national electorate through the amalgamation of these units.
Sanjay Kumar, who now leads the Lokniti project at CSDS, said that just having a bigger sample size doesn’t help, as it also “multiplies the errors”. In theory, the more you spread the sample, the more representative it is. “But spreading the sample also has some restrictions,” he said. “Ideal would be to go to all the constituencies. But one has to think about manageability, whether you can manage it or not.”
One of the many problems pollsters face is whether to conduct telephone or face-to-face interviews. The former is cheaper, but some critics believe mobile penetration in India is not yet deep enough to yield representative survey results. In-person interviews are beset by their own difficulties. For one, the interviewer has to win the confidence of the respondent. According to both Deshmukh and Kumar, in rural areas, members of marginalised communities often misrepresent their views out of intimidation and fear of reprisal from the often urban interviewers. This is apparently the reason Mayawati’s electoral base is always under-represented in polls.
To overcome some of these distortions, Sopariwala and Ahluwalia introduced to India a “secret ballot” technique that was pioneered by Gallup in America in the late 1940s. Respondents are asked to mark their candidate preferences on a sheet of paper and put it in a dummy ballot box. The paper is coded with a serial number to match a serial number on the respondent’s main questionnaire.
To try and ensure that their samples are representative, pollsters take a systematic approach to selecting interviewees at random. “Systematic random—it is kind of a lottery, where there is no bias applied,” Kumar told me. CSDS has a method for randomly picking voters from the electoral roll. The list is then given to interviewers in the field, who never decide whom to poll. For telephone surveys, a computer automatically connects random voters from a given region with interviewers who speak the local language. In either case, the pollster checks for under-represented populations, and then does additional polling or weights the data to account for this.
However a poll is conducted, there are inevitably many rejections, or people are unavailable, or government data used to generate addresses and telephone numbers are outdated, so the pollsters have to attempt to sample more people than they need for a robust poll. The data are then collated, and the vote share of each party is derived by tallying up the individual responses.
All this makes polling a very expensive exercise. The price differs from agency to agency and according to what is expected of the pollster. In the last general elections, CSDS, which is a non-profit organisation, polled roughly 30,000 people around the country. The total cost was around
Rs 1 crore, Kumar said. About 20 percent of that was covered by CSDS’s media partners; the rest came in the form of grants from academic and research institutions and government bodies.
Few for-profit agencies, if any, are able to make money from political opinion polls for the Indian media. CVoter makes most of its money outside of India, doing polls for foreign governments and international organisations, according to Deshmukh. Market research firms, such as ORG (formerly ORG-MARG), use the publicity generated by these surveys to market themselves to corporate clients interested in understanding consumer behaviour, which is how they make most of their income. Sopariwala said that election forecasting is “less than one percent of their business, but ninety percent of the public face”.
Many media houses share their costs and data with another partner. CNN-IBN had The Week as a print partner for the assembly election polls last year. This year, their election tracker was shared by The Hindu in July and The Weekin October. CVoter is in a tri-party agreement with Times Now and India TV, and is also doing polls for the India Today group. Deshmukh did not tell me how much it costs him to do his polls, but it is presumably cheaper than CSDS; the former relies mostly on telephone interviews, while the latter only interviews their respondents face-to-face.
TO GET FROM THE RESULTS OF THE POLL to the seat numbers is a mind-bogglingly complex process. Sopariwala and Ahluwalia think that the biggest challenge is identifying those respondents who give interviews but do not vote, thus distorting the analysis of the data. But this is only one of a nearly endless list of factors that can go into seat projections.
Various indices and formulas have been developed over the years by Indian psephologists. Prannoy Roy and Ashok Lahiri started with their Index of Opposition Unity. Then, building on Roy and Lahiri’s work, came the economist Surjit Bhalla’s Lying Index, which tried to offset the effects of electors who don’t vote. The process to fine tune these techniques is ongoing.
Pollsters are continually trying to account for the ways that various layers of political representation—the local MLA, the state’s incumbent party, the member of parliament, the national incumbent party, the governing alliance, the executive, and the prime minister—affect voter preferences. An individual may be unhappy with a politician or party’s performance at any one of these levels, and still vote for the same party at other levels.
For two-way races in first-past-the-post democracies, the basic formula that governs conversions from vote share to seats won is the cube law, which states that seats will be divided up in the same proportion as the ratio of vote shares raised to the power of three. In India, however, there is often the challenge of accounting for more than two parties, for parties splitting between elections, and for fluid alliances.
Kumar gave me an example of how unpredictable the results can be. In Uttar Pradesh in 2012, the Samajwadi Party had 29 percent of the votes and got 224 seats. In 2007, the Bahujan Samaj Party had over 30 percent of the votes, but won only 206 seats. “Same state, same voters, same
parties. Party gets 1 percent less votes and gets 20 extra seats,” Kumar said. These surpises also happen at the national level: the increase in Congress’s vote share between 2004 and 2009 was only 2.02 percent, but they picked up 60 extra seats.
“Every election I find something new, and fine tune it,” Deshmukh said. “It can never be perfect. It is still not foolproof. Arre public opinion is as dynamic as it can be. Every damn election we learn something or the other, and it is an unending process.”
IN 2009, on the eve of the second phase of India’s general elections, Prannoy Roy sat in front of an externally lit Hyderabad Public School building, facing Dorab Sopariwala and Shekhar Gupta, the editor-in-chief of Indian Express. It was two days prior to the second round of assembly elections in Andhra Pradesh, and the Election Commission had recently prohibited the broadcasting or publication of data based on exit polls before the last phase of the elections was over. All three men were dressed in kurta-pajama for an NDTV show titled Battleground 2009.
“We travelled all across Karnataka, and we also travelled all across Andhra Pradesh,” Roy said. “So now these two gentlemen, they know everything. They are experts on these two states. They’re gonna tell us, by the end of the show, who’s gonna win in both of these states. Not only that, by travelling in these areas—it’s amazing—they have an idea of the entire country. So if you want to know who’s gonna win the entire country, hang on.”
They discussed the prospects of three of the state’s most significant contenders: the Telugu Desam Party, the Congress, and the former actor Chiranjeevi’s Praja Rajyam Party. “Let’s move on to what’s happened in phase one of this election,” Roy continued. “Now we’ve been talking to a lot of people. This is not really an opinion poll, but it’s an analysis for which we sat together and tried to figure out what happened in that phase one.” They examined Chiranjeevi’s chances in exceptional quantitative detail, tossing out likely percentages of vote share, then moved on to the odds bookies were offering on the general election.
“Now many people believe that bookies know little more than Dorab,” Roy said in jest. “Sometimes. Though most of the time you know more than them.” They discussed a survey, based on “lakhs and lakhs” of text messages from NDTV’s viewers, which suggested that Manmohan Singh was the top prime ministerial candidate, followed distantly by LK Advani.
“I think that’s the general mood, at least in the SMS class,” Gupta said.
“But SMS class—there are 400 million mobile phones in this country,” Roy offered.
“I think the fact that we are here discussing projections of bookies and SMS polls is a very unfortunate thing,” Gupta replied. “I would rather go by the—by the figures thrown up by your opinion poll.”
Roy laughed and raised a finger to his lips. “Shh … You can’t even mention the word. Hahaha. No, I know the point you are getting to.”
“I know I can’t,” Gupta said. “So I am not mentioning the figures. But, but, but I want to say that Election Commission has done great things, running up in these elections. But, this is the most terrible decision. That’s my editorial viewpoint. I think it would have had been fantastic to have some kind of exit polls now. Figures coming in—it just enriches the discussion. I think if the Election Commission is watching, or—the new Election Commissioner has taken over today—Navin Chawla, if he watching, I think this is one decision that they have to revisit. Because it’s a terrible decision. It doesn’t help anybody … If bookies can talk about figures, why can’t legitimate people?”
“We know all over the world, most bans don’t work,” Yogendra Yadav told me when I met with him recently. The ban on exit polls still stands, and the Election Commission is also considering banning surveys during the period after elections are announced. (At the time of writing, the ban was under consultation with political parties.)
The proposal to ban the polls, Yadav said, was based on four flawed assumptions of the Election Commission: most polls are either mistaken or “vicious”; they can influence voting behaviour; these problems cannot be tackled by any other means; and bans will work. While admitting there are some “increasing unprofessional” practices, he said most polls are still infinitely better than any other way of tracking the popular mood, and whether surveys affect voters in some undue way is an empirical question that remains unanswered. If “we want to ban something, and if we have been discussing a ban for ten years, should we as a country not gather evidence about these things?” he asked.
Prohibition should be the last resort to tackle any problem, Yadav argued; ensuring transparency through some basic regulation would be a far better way to curb malpractice. “Bans are counterproductive,” he continued. He said that although political parties conduct their own surveys, some are now trying to create more opacity on whatever limited data are available. Prohibiting polls “would create a black market of information … So, instead of actually creating public information it would create an information elite. Second, it would create rumours—‘I know they have done it, but I can’t release it legally, but let me tell you what the survey is.’ It creates, absolutely, a culture of duplicity, rumours and so on.”
He mentioned Battleground 2009: “Prannoy Roy, Dorab Sopariwala and Shekhar Gupta sat and said, ‘According to my intuition Congress will get 23 seats in Andhra Pradesh.’ Of course they had done a poll. Everyone knew they had done a poll.”
The ramifications of these bans, Yadav suggested, could go well beyond surveys. “The simple fact is you cannot say Prannoy Roy cannot have intuition. Tomorrow I can have intuition. The question is, will we impose this ban on people who write on the edit page? Will we impose this ban on reporters—that they cannot say someone is ahead, someone is behind? If we don’t ban all that, how can we possibly
ban this?”
EDITORS ACROSS TELEVISION AND PRINT MEDIA told me that while some “bogus”, “rogue” or manipulated polls find space in regional and local news outlets, it is rare to find them on national broadcasts, or in national publications. There are “malpractices” in the Indian industry, Yadav agreed. But, “If you look at some of the more professional polls, I would say they compare with some of the best in the world. Nowhere in the world are election forecasts 100 percent correct.”
Still, there are valid criticisms to be made even of polls publicised in the national media. Foremost among these is that pollsters’ methods and the calculus that produces seat projections remain outside public scrutiny. Almost every other suspicion about the technicalities of polling—about the size and representativeness of samples, margins of error, statistical confidence, unresponsive interviewees, and adjustments based on historical precedents—could be dispelled if pollsters and their media partners were absolutely transparent about how they conduct their polls. And this would enhance the credibility of the process. It should be mandatory, Yadav said, for pollsters to “disclose their methodology at great length. Now with web there is no space problem. You can ask them to put it up on the web.”
Other sceptics question the motivations and political bias of the pollsters. Recently, on blogs, Twitter and in an article in Open magazine, CVoter has been called out for gravely overestimating the BJP’s prospects in many forecasts since 2004. Deshmukh, the founder, is a nephew of Nana Saheb Deshmukh, one of the Rashtriya Swayamsevak Sangh’s most revered ideologues. For their part, Yadav and Sopariwala have faced accusations of being sympathetic towards Congress. But none of this proves bias. Here, too, transparency with regard to methods, and clear disclosures about potential conflicts of interest, would help shed light on the validity of such claims, and prevent bias from entering polls in the future.
In the end, however, it may be that people only believe the polls that reinforce their beliefs. Part of this is human psychology, and part of this may be that whatever lack of trust exists between the public and the media is extended to surveys and forecasts. Readers or viewers may pay less attention to a poll than to the medium that brought it to them.
I asked Sardesai why he still paid for polls, despite their uncertain effects on his network’s popularity and the technical and reputational pitfalls that beset the process. “You ask me a question which I have no answer to, barring saying that I am an election junkie,” he said. “I still believe it’s fun to do.”
“See this is the mug’s game,” Sardesai continued. “If we are going to go by what television audiences want, then we are going to end up doing only naach gaana [dance numbers]. That’s—that’s not a factor. I think you have to do things that are intelligent. I think the viewer respects an intelligent opinion, or an intelligent poll. A viewer knows when the pollster is also faking. That I think is becoming increasingly clear.
“It’s one more weapon available to try and explain the large election landscape of this country. I still believe in that, which is why we still do polls. Maybe in the future we may have to stop it altogether. I hope that day doesn’t come.”
Correction: The spellings of Rashtriya Swayamsevak Sangh and Bahujan Samaj Party have been corrected. The Caravan regrets the errors.