The biggest threat to ROI: the problem with fast and agile testing
Last week we presented at the Research and Results conference about how to improve the ROI of research. The biggest threat to ROI is probably the lack of testing and the reliance on gut feeling and experience. So, let me ask two simple questions:
The answer is simple: most introductions and/or campaigns are not tested or are tested in the wrong way.
In our Brand Growth study, we also looked into this and found three main reasons. The first two are obvious: companies say they don't have the time or the money to test and need to trust their experience and belief. The third one is more worrying:
60% of the marketers that we interviewed said they are not satisfied with the insights they generate.
The solution of the time and money issue is related to research automation and creating a new and agile way of testing. Almost every week a research company announces that they built an agile testing environment that makes it possible to test fast and with low costs.
I think this is a great step forward for market research and a sign that we are listening to our clients and adjust our testing to their way of working. However, if you look more in-depth into these agile solutions we see that there are two main aspects missing, namely:
Most agile solutions are driven by the Do-It-Yourself (DIY) trend and it allows research buyers the possibility to create a questionnaire and to access their target audience often in an intuitive user interface environment, so speed is guaranteed. As a market research professional, I’ve filled in so many questionnaires. So, I know that making a good questionnaire is a profession. It is not something that can be done by every intern or junior researcher. A question needs to be tested and validated before you can use it in practice. So, although cheap and fast, this development will never lead to good research.
The second possibility is that the questionnaire is pre-designed. The problem here is more or less the same. Because these methods are built for speed and easy analysis and delivery, they are inefficient.
One of the best examples I have ever seen presented is by James Dodkins.
James plays a role on stage where he is the interviewer and someone from the audience the respondent. His conclusion is that there is no better way to let a customer know that you don’t care about their opinion by forcing them onto a 5-point scale.
I really hope that the developers of agile tools spend some time on creating a questionnaire that is tested, validated and can be used for its purpose. Resulting in more efficient research.
Saving time and money is in most cases re-shifting time and money. Most agile tools or research shops are so complex that they can hardly be used by a bigger audience. You need skilled researchers to start the process. Let me give you an example.
In our road to understanding all the available tools, we tried several research shops for pre-testing. First, you have to go through an enormous set of questions to clarify what you want. Probably complete, but way too complex for a marketer. Then you have to select the methodology, the way an ad is exposed to the respondent and the questions that you want to ask. Also impossible for the marketer and a skilled and trained research professional should do it. So, no time or cost saving, only a shift towards the research expert within a company who in most cases already has too much work on his or her plate.
If you managed to set up the process of testing, then the report comes in. A beautifully designed report, but nothing more than a questionnaire with percentages. What is relevant and how you should interpret this is difficult because it needs to be complete. Marketers need a summary and reading hierarchy where you start with the first most important KPI and compare it with a benchmark, then you have the second most important KPI, the third one and so on.
A good automated report needs hierarchy and benchmarking, but in most cases that is missing.
So, we understand the lack of satisfaction of all marketers we interviewed in our Brand Growth Study. Agile testing should never be quick and dirty. Because with worthless research you get worthless results leading to worthless decisions. And worthless decisions will not do anything good for your ROI.
But let’s look at it from the bright side. Suppose that agile testing becomes good enough and you can support your decision with it. It will mean that companies get the possibility to test more and more and create more control. It will prevent companies from top line damage and failure. With some of our clients, we learned that a disciplined approach towards pre-testing can lead to savings of 150-200 GRP’s per campaign.
Blog Lucas Hulsebos – CEO DVJ Insights
It’s very important to keep panel members involved and motivated. After all, we need our panel members to provide our clients with valuable insights.
If you look at the past 10 years, we can conclude that mobile phones have changed our lives. It not only influenced our personal lives, but it also changed business.
Catching a movie
A seemingly obvious mistake, when you take that second glance, but not so obvious when writing your own questionnaire.
Using social stories in questionnaires
Wouldn’t it be more useful for a respondent to see these commercials in environments where they are normally showed?
Shopping for tea
I recently came across a technique that caught my attention since it really improved both my engagement and the reliability of my answers.
The solution to screening-out respondents
Screening-out respondents can disappoint respondents, because they can’t give their opinion.
Take your time
Recently, I received an invitation to fill out a survey, which surprised me quite a bit.
Graphics, colours and interaction
Online questionnaires and surveys are still at the forefront of market research for many companies, as it opens up a vast number of possibilities.
The limitations of a respondent’s memory are sometimes forgotten when programming a questionnaire.
To make questionnaires more engaging, it’s important to create a certain interaction for respondents.