Why collaboration is key in designing better surveys

The usefulness of surveys in which respondents make predictions about the economy and other future events can be much-improved by better survey design, research has shown.

Studies have looked at ways to improve consumer expectation surveys of inflation - used by central banks to inform key monetary policy decisions – and surveys in which teenagers make predictions about significant life events, which can feed into decisions of social policies.

When Professor Wandi Bruine de Bruin of Leeds University Business School and Professor Baruch Fischhoff of Carnegie Mellon University, both psychologists, undertook studies on these topics and others, they were concerned that questions used in earlier surveys were open to misinterpretation, casting doubt on the use of the results.

In their work, they were joined by leading economists, including those working in the research branch of the US Federal Reserve.

The economists and psychologists collaborated on all stages of survey design and analysis, particularly the phrasing of the questions, with psychologists working to make the economists’ concepts comprehensible to survey respondents.

Professor Bruine de Bruin said: “Consumer expectation surveys of inflation provide an insight into what is happening ‘on the ground’ economically and are used by central banks and economists to inform financial decisions affecting interest rates, durable goods purchases and wage negotiations.

“The main US survey, used for many years, asked respondents whether they think ‘prices in general’ will go up or down in the next 12 months and by what percentage they think this might change. However, responses tend to show large variations and people’s expectations of inflation often prove quite different from the actual rate of inflation. This has called into doubt the surveys’ usefulness.

“Similarly, previous surveys in which teenagers were asked about the chances that they would go to university, get married and have children asked questions in ways that teenagers sometimes misunderstood or felt uncomfortable answering.”

Professor Baruch Fischhoff of Carnegie Mellon University said that the problem lies partly in conventional academic training.  He said: “Economists involved in consumer expectation surveys aren’t trained in survey design. As a result, the questions they ask may not be understood as intended. Psychologists have training in survey design, but do not know the questions central to economic research. As a result, they may ask questions the responses to which are not that useful.”

Professor Bruine de Bruin summarized the findings: “We identified four elements central to sound survey design: having a shared research goal; having common ground in shared methodology; genuine collaboration, with common language and ownership; and gaining mutual benefit from the research and its products.”

The research is published in the Proceedings of the National Academy of Sciences (PNAS).

Read more about: