Pearls and Irritations

The Australian Bureau of Statistics has another debacle approaching brought on by its direction to conduct the government’s proposed postal plebiscite on same sex marriage. Little more than an outmoded postal survey it will be flawed from the start, plagued by biases both known and unknown. The survey will seriously erode the public’s confidence in this once peerless official statistical agency. It needs to start work now to salvage what it can of its reputation for trust and integrity. 

The Australian Bureau of Statistics is Australia’s official statistical agency with a worldwide reputation for excellence. For years it has stood as a bastion of best practice in the collection, processing and dissemination of trusted information about our economy and society and it has been a world leader in survey methodology and applications.

Last year its reputation was on the line when a devastating breakdown of its computers threatened completion of the Australian Census. Now there is another more serious threat. Apparently it has been cornered by its political masters into conducting a “postal plebiscite” on same sex marriage. In effect, this will amount to little more than a survey of public opinion significantly flawed before it is even launched. It is hard to understand that an esteemed institution like the ABS can’t see this disaster approaching let alone resist the urgings of its masters.

This represents a massive departure from the Bureau’s mission of providing finite quantitative indicators of the health of the economy, such as unemployment levels and retail sales. The carefully crafted trust associated with these national indicators is critical to the integrity of public planning processes and policy formulation.

For years, the Bureau has been scrupulous in avoiding the collection and dissemination of data about public opinions and attitudes, leaving it to private sector survey companies and specialised academic research centres. In the past, the ABS has even commissioned private survey companies to conduct surveys of its own image with the public!

Already, the Turnbull government’s directive that the ABS will conduct the “plebiscite” has been seriously questioned by persons prominent in the opinion polling and market research business (SMH August 10, 2017). Not only do they consider it a waste of money — where a sample survey would produce more reliable results at a fraction of the cost – they also point to serious flaws in the methodology of the proposed “plebiscite” which will harbour all of the problems of one of the weakest and these days, fast-disappearing forms of survey research, the postal survey.

The Australian Statistician’s neck will surely be on the chopping block as a raft of accusations, innuendos and doubts about methodology will float to the surfaces along with the political gamesmanship that is likely to follow in the wake of the announcement of results.

First there is the contentious issue of coverage of the population. Does everyone have an equal chance of participating in the survey? No. At any point in time a significant proportion of the resident population is on the move and out of reach for a fully representative survey, travelling domestically and internationally and even living overseas; young people move house frequently; people of indigenous communities do not have personal letterboxes, and so on. Ironically, the Bureau keeps the books on these issues through its many collections. Surely, it could sound the alarm bells to its masters and warn of the biases of non-coverage that will diminish the representativeness of the planned survey.

Then there is the formidable problem of non-response to the survey. Postal surveys suffer from indifference, lack of interest and ignorance on the part of their target audiences. This cripples response and therefore, representativeness. Hence the use of incentives to stimulate response, but it’s hard to imagine an appropriate incentive for this one.

Another way of boosting response is to send reminders to those who have not responded to the initial mail-out. Reminders, if prosecuted in the best manner can bring about as much as a 50 per cent response from those who failed to respond to the first and in turn, subsequent waves of mailings. Can we seriously envision the book keeping necessary to keep track of who has and who has not responded to a mailing out to millions of Australians? No. This will be a one-shot mailing with a woeful response rate.

Critically, there will be a sizeable slab of the population whose opinions we will never know. Worse, unless information on at least the gender, age, and location of respondents is collected we will never know how faithfully the resulting sample reflects the known demographic structure of the Australian population measured by the Census.

Among the many other operational problems is how to measure passionate attitudes. The issue about same sex marriage is highly passionate and it will become even more passionate as the yes and no campaigns progress in coming weeks. The hidden reality is that people who say yes or no to a proposition hold their beliefs with varying intensities. Two people may hold the same pro or con view but to one of them, the same view may be of greater personal importance than it is to the other. Weak agreement is not the same as strong agreement, and so on.

While not excluding anyone else, those in the market and social research profession should advocate for the ABS to undertake and publish what it knows about coverage problems, non-response and question design for this so-called plebiscite. It could show that our revered official statistical agency has addressed doing its homework instead of remaining silent. The ABS should start work on this right now.

In a promise of integrity to the Australian people, the market and social research industry long ago adopted ISO 20252, an international survey quality assurance standard with which most of its members comply in the collection, processing and disclosure of information gathered from the public. It means the clients of the survey companies, as well as the public who participate in their surveys can have confidence and trust in the reliability of the findings. Surely, the Australian public can expect the same from the organisation which it pays for, the ABS. It might just help to salvage the dignity of a once-revered public institution.

Dr Terence Beed is a former opinion pollster and academic survey researcher. He founded Australian Nationwide Opinion Polls (ANOP), was foundation Director of the University of Sydney Sample Survey Centre and headed the survey quality assurance unit at KPMG. He is an Honorary Associate Professor in the University of Sydney Business School and Senior Research Fellow at the Law and Justice Foundation of New South Wales.

Share and Enjoy !

SHARE TWEET SHARE COPY EMAIL PRINT