A personal comment by Simon Fitall
A common thread through the past 40 years of the market research industry is that every few years a development arises that appears to threaten the future of the industry. These “threats” are usually associated with a significant shift in one of the big three elements of market research
- How do we access the source of our data?
- How do we capture the data we need?
- How do we gain valuable insights from the data and distribute those to our customers?
I can only speak to the last 40 years because prior to that I wasn’t in the industry, although much of what I have observed appears to have been relevant to that earlier time as well.
In exploring current threats, I think it is worthwhile assessing the impact of previous threats. Then we can address those things that may be perceived as current threats and attempt to foresee how things might work themselves out. I put it that way because we cannot accurately predict threats over any period any more than we can accurately forecast, and I say this having spent a good part of my career running a forecasting company.
An element of the question we are addressing is ambiguous. Are we considering threats to market research or to the market research industry? I can see the argument that would suggest that these are one and the same, but I would argue that a threat to individual elements of the industry are less likely to be threats to market research as a concept.
For example, in the early 80’s AC Nielsen were capturing nearly all their retail sales data by hand. A business model that had built the company was still going strong as the major retail chains started to introduce bar coding that led to scanning data. ACN did not react to the shift as quickly as some competitors and their retail sales business collapsed. What was a threat to one of the mainstay companies in the industry was treated as an opportunity by others and the market shifted.
In this example market research benefited hugely because the scanning data were of significantly higher quality than the old retail audit data, they arrived faster, and the clients benefited. The losers were companies (like ACN) who didn’t respond to the opportunity.
Similarly, in the world of primary MR the “threats” of moving from face-to-face interviews to telephone interviews, to internet interviews, were each seen as major threats but they were each taken up as means of improving how we access respondents and how we capture data. The potential threat only manifested itself in companies that were slow to react. The same can be said for change from individual MR agencies having their own field recruitment department to the use of panel companies.
Here I think it is worth discussing a more recent trend which is, I think a candidate for classification as a current strategic threat – the question of fake respondents.
The moves towards the digitization of data capture (paper to phone to the internet) combined with the near universal adoption of panels for sample access, combined with the most recent developments in computing technology have created a new industry in bad actors creating fake respondents that provide fake data in response to surveys.
At the same time, many PMR agencies are now promoting the idea of creating digital personas, digital twins, synthetic respondents (they are all similar) to support the generation of data in areas where it is difficult to access “real” respondents. For example, using PMR to create personas that can then be quizzed regarding how they are likely to react to different stimuli, be it a new advertisement or a new product. This is being marketed by MR agencies as a major enhancement to the PMR process as it eases access to the data source and makes the data capture process more efficient (the first two of our core areas of activity).
This is all very well when managed and controlled by good actor MR agencies with suitable transparency for the client and the necessary governance. But what is the difference between good actors who create digital respondents for the purpose of augmenting existing PMR and bad actors creating millions of digital personas with the purpose of defrauding the MR industry? Perhaps more importantly, how do we ensure that it is possible to differentiate between the two and manage the threat from bad actors?
Are you ready for what Professor Malcolm MacDonald used to describe as “a blinding glimpse of the bloody obvious”? I think that the increased attention being paid to potential threats to market research is being driven by the hype surrounding AI (there, and you thought I might be old and out of touch!). Of course it is and the previous point about fake respondents is entirely driven by AI. But this isn’t, in my opinion, the biggest potential threat from AI; that is the analysis and reporting of findings, the third core element of the MR industry.
I go back in time again to explore a (very) brief history of the analysis and reporting of findings. We used to produce tables of data that were then analysed either by statistical and mathematical analysts, interpreted by researchers, who would then write a report of the research. That report would then be summarised into a presentation which would involve flip charts, overhead acetates or even slides (as in 35mm slides that were projected). Old school.
Then came Microsoft Office and PowerPoint and over the years the written report became less common, “decks” became the preferred delivery mechanism and even tables were relegated to near obscurity as speed of delivery became more important. This then morphed into dashboards where tables and charts were combined into an all-in-one data repository / analysis / reporting tool.
In the world of secondary data, we used to have printed volumes of tables which would be pored over by analysts which evolved into databases that could be filtered and cross-tabulated automatically prior to being downloaded. Then came the relational database and associated mathematical and statistical tools that in turn became “big data” with neural networks and Monte Carlo analyses and all that went with them.
All of these methods of analysis and reporting involved direct human interaction with the data, frequently many different analysts and researchers would be examining any given data set to ensure that the most valuable insights were extracted. Reports were generated, delivered and stored and at any time could be referenced and, if necessary, analyses could be rerun using modified inputs or audited for validation. In fact, a continuous process of validation was considered essential to the process as this was the way in which agencies were able to demonstrate to the client that their findings were reliable and actionable.
What if we remove the assurances of reliability and entrust our analytics and insight generation to AI tools? Some in the industry will tell you that this is the future of the industry. That all MR – qual, quant, secondary, open-source, all of it – can and in fact should be analysed using AI to find the hidden nuggets, the insights that other methods couldn’t find, the real value.
I say that we have to remove the assurance of reliability because AI cannot show its working. You can’t look inside to see what it did to get from A to Q. Even worse, unless you have an independent reference you cannot be entirely sure that what it found is even there and that the AI didn’t create a ghost finding.
In the ‘90’s there was a huge trend for a few years towards the use of neural networks. This was going to be the way of the future with large datasets (as yet not called big data). The thing that diluted this trend towards neural networks was “black box” syndrome. Clients were loathe to give credence to a solution that came from a black box that couldn’t be explained in a step-by-step way. Well, there are many agencies currently suggesting that the client doesn’t need to know what is going on with AI because it is so smart that you will never need to question the findings, the AI will have looked at everything and come back with the optimum, whatever that might be.
Here I think is a real threat to the industry because the neural network trend was killed by experienced clients who knew not to trust the snake oil. Their knowledge of the industry, market research expertise, understanding of all the pitfalls of well-run MR projects (let alone the poorly run projects) told them that they had to be able to see the workings. Now the rush to AI is frequently being driven from the top down. CFOs are looking at what they are being told about AI and believing the hype. They are saying that we don’t need expensive PMR as much because AI can do it all for us; we don’t need so many expensive experts in MR because the AI will do it for us. And the agencies are leaping on the bandwagon and saying “yes, AI can do it and we have the solution for you” – even when they have no idea how the AI works or whether it is really working properly at all (and that assumes that they know what “properly” would look like in the first place).
Properly applied, AI can transform the way we analyse, interpret and report research data and findings. With appropriate levels of human oversight and intervention – the human/AI loop – this could be the making of the entire industry. But, misplaced, poorly applied and incorrectly managed AI could lead to a total loss of trust in the industry. One or two big companies having major failures because of AI-led market research could set the industry back years.
And so, I come to the nightmare scenario. Imagine a client that buys into the AI hype and insists that all their agencies must have strong AI capabilities. Agencies respond by finding all sorts of ways in which they can incorporate AI into their MR offerings, even though they have to buy the tools from AI vendors rather than create their own. Data are captured, but the agency doesn’t know that half their respondents are digital fakes because those fakes look just the same as the digital personas the agency created for the project. The AI tools used to analyse and interpret the data were not trained for the specific purpose but can fake it because that’s what they do, and the agency and client have no idea that the “findings” are all nonsense and don’t suspect anything because the AI generated the most persuasive set of slides, reports, executive summaries and action points.
All these things have already happened.
