The biggest threat to ROI: the problem with fast and agile testing

Last week we presented at the Research and Results conference about how to improve the ROI of research. The biggest threat to ROI is probably the lack of testing and the reliance on gut feeling and experience. So, let me ask two simple questions:

1. Why do so many introductions fail in the first year after launch?
2. Why do so many campaigns create not enough impact?

The answer is simple: most introductions and/or campaigns are not tested or are tested in the wrong way.

The reasons

In our Brand Growth study, we also looked into this and found three main reasons. The first two are obvious: companies say they don't have the time or the money to test and need to trust their experience and belief. The third one is more worrying:

60% of the marketers that we interviewed said they are not satisfied with the insights they generate.

The solution of the time and money issue is related to research automation and creating a new and agile way of testing. Almost every week a research company announces that they built an agile testing environment that makes it possible to test fast and with low costs.

I think this is a great step forward for market research and a sign that we are listening to our clients and adjust our testing to their way of working. However, if you look more in-depth into these agile solutions we see that there are two main aspects missing, namely:

• Quality
• Easiness

About quality testing

Most agile solutions are driven by the Do-It-Yourself (DIY) trend and it allows research buyers the possibility to create a questionnaire and to access their target audience often in an intuitive user interface environment, so speed is guaranteed. As a market research professional, I’ve filled in so many questionnaires. So, I know that making a good questionnaire is a profession. It is not something that can be done by every intern or junior researcher. A question needs to be tested and validated before you can use it in practice. So, although cheap and fast, this development will never lead to good research.

The second possibility is that the questionnaire is pre-designed. The problem here is more or less the same. Because these methods are built for speed and easy analysis and delivery, they are inefficient.

One of the best examples I have ever seen presented is by James Dodkins.

James plays a role on stage where he is the interviewer and someone from the audience the respondent. His conclusion is that there is no better way to let a customer know that you don’t care about their opinion by forcing them onto a 5-point scale.

I really hope that the developers of agile tools spend some time on creating a questionnaire that is tested, validated and can be used for its purpose. Resulting in more efficient research.

The issue with easy

Saving time and money is in most cases re-shifting time and money. Most agile tools or research shops are so complex that they can hardly be used by a bigger audience. You need skilled researchers to start the process. Let me give you an example.

In our road to understanding all the available tools, we tried several research shops for pre-testing. First, you have to go through an enormous set of questions to clarify what you want. Probably complete, but way too complex for a marketer. Then you have to select the methodology, the way an ad is exposed to the respondent and the questions that you want to ask. Also impossible for the marketer and a skilled and trained research professional should do it. So, no time or cost saving, only a shift towards the research expert within a company who in most cases already has too much work on his or her plate.

If you managed to set up the process of testing, then the report comes in. A beautifully designed report, but nothing more than a questionnaire with percentages. What is relevant and how you should interpret this is difficult because it needs to be complete. Marketers need a summary and reading hierarchy where you start with the first most important KPI and compare it with a benchmark, then you have the second most important KPI, the third one and so on.

A good automated report needs hierarchy and benchmarking, but in most cases that is missing.

Worthless results leading to worthless decisions

So, we understand the lack of satisfaction of all marketers we interviewed in our Brand Growth Study. Agile testing should never be quick and dirty. Because with worthless research you get worthless results leading to worthless decisions. And worthless decisions will not do anything good for your ROI.

But let’s look at it from the bright side. Suppose that agile testing becomes good enough and you can support your decision with it. It will mean that companies get the possibility to test more and more and create more control. It will prevent companies from top line damage and failure. With some of our clients, we learned that a disciplined approach towards pre-testing can lead to savings of 150-200 GRP’s per campaign.

Blog Lucas Hulsebos – CEO DVJ Insights