Online Market Research Surveys -- Maximizing Your ROI

The current economic climate has caused some companies to defer or cancel market research projects. In many cases, a company may have correctly determined that the cost of completing a survey within the parameters required for actionable data is no longer possible given its current budget. However, defining exactly what those parameters should be is not always clear, and that is what mostly determines the project cost. This article focuses on the concept of incidence rates, which can be one of the most important cost considerations in designing market research surveys.

The term "incidence rate" refers to the percentage of respondents who qualify to complete a research study. Most vendors incorporate the incidence rate and cost of awards into a quoted cost per complete, referred to as the CPI ("cost per interview") or CPC ("cost per complete"). The lower the incidence rate, the higher the cost; with more pronounced increases as the incidence drops below 20%.

For example, a B2B market research study with a 70% incidence might have a CPI of $7.00; while a study targeting high-level executives with a 3% incidence might have a CPI of $50.00. The CPI is then multiplied by the number of completes to determine the cost of sample for the project. As the number of completes increase, there may be a volume discount in the quoted CPI, but it is generally a small reduction (such as $7.00 for 350 completes and $6.50 for 600 completes).

From a cost perspective, online market research surveys are different from RDD (random digit dialing) phone surveys where the general population is randomly called and the incidence rate is simply the percentage of respondents who qualify. Most survey panels have hundreds of pre-identified information cells of panelists ("selects") and can target in advance participants who qualify for participation. This can result in significant cost savings, particularly in lower incidence studies.

When vendors quote an incidence rate for an online study and provide alternative CPIs for different incidence rates (such as a $12.00 CPI for a 25% incidence and a $16.00 CPI for a 15% incidence), the client might not fully understand how the vendor is defining "incidence". For example, if the targeted select for an online consumer survey is "Owners of 2008 New Ford Mustang Convertibles", and the vendor has "Owners of Ford Mustangs" pre-identified in its database, in theory the quoted incidence rate should be based on the percentage of those pre-identified Owners of Ford Mustangs who own a New 2008 Convertible. Yet, some vendors quote incidence based on the percentage of Owners of 2008 New Ford Mustang Convertibles within the general population, which is confusing since the general population is not targeted to participate in the survey.

SUGGESTION: For lower incidence rate studies, ask the vendor if the targeted select is pre-identified. Keep in mind that some vendors will list a select as being identified not because it is pre-identified, but because the vendor knows it can reach that select through a general population blast to its survey panel. However, the CPI on a pre-identified select is almost always less expensive in a low incidence study because there is less risk to the vendor and no need to blast out emails to a large portion of its database. The variation from vendor to vendor in the CPI could be $30.00 or higher depending on whether the select is pre-identified. Wide variations in quoted CPIs are particularly evident in B2B market research surveys.

Selection of the specific criteria for respondent qualification, and the wording on the questionnaire of the associated screening questions, is a critical element of any project, particularly lower incidence studies. Differences in qualification requirements such as "final decision maker" versus "influencer", "C-level executive" versus "Manager, Director, VP or higher", or "Owner of 2008 New Ford Mustang Convertible" versus "Owner/Lessee of New or Used 2006-2008 Ford Mustang Convertible", can often have a significant cost impact. Even a screener such as "How often do you travel out-of-town on business..." can have very different pricing depending on whether the screening requirement is twice a year or four times a year.

While the goal of a survey should always be to reach the targeted audience that can provide the best feedback; there is often some flexibility in defining the screening criteria. Is there a good reason why a 2008 model year is the only qualifier for an auto study? Does a decision to purchase a product or service really need to be limited to a final decision maker and not include input from others in the decision-making process? Those determinations directly impact the incidence rate and can quickly move a project from too expensive to affordability, or vice versa.

SUGGESTION: Review the specific goals of the study with the vendor's statistician or other questionnaire design expert to determine what options are available for defining respondent criteria that can reduce costs but still produce actionable data. Keep in mind, for example, that the cost difference between surveying "final decision makers" and "final decision makers or influencers" can be as much as $20.00 a complete or higher.

Some online market research companies provide survey hosting services in addition to survey sample. In such cases, they may offer incidence rate testing at no charge or for a small fee. An "incidence rate test" is a small study using the screening questions only which is conducted in advance of the actual project. With this information, a prediction can be made as to the actual incidence rate and its cost in the field. The client can then determine how many completes will fit within budget, what changes might be needed in respondent qualifications to increase the incidence rate, or if the project still makes sense from a cost perspective.

SUGGESTION: Take advantage of this option if the study is low incidence and the incidence rate is uncertain. In many cases, the incidence rate established in the test might be higher than expected, resulting in a lower project cost. Or, if this is not the case, an informed decision can be made on what adjustments are needed in other parts of the project to lower the overall cost.

Working with a full-service market research supplier that is involved in the different phases of the project including questionnaire design, sample, hosting, and analytics brings the parties together with the common goal of providing high-quality, actionable data in a cost-effective manner. Take advantage of the vendor's expertise in each of these phases and discuss the scope of the project with the vendor's statistician or other questionnaire design expert. Respondent criteria, respondent targeting, number of completes, and other cost sensitive items are legitimate topics of discussion on which the vendor should be able to provide expert advice. There is rarely only one approach to a survey project.

Marc I. Tillman is a member of the professional services staff at Amplitude Research, Inc., a full-service survey company headquartered in Boca Raton, Florida. Amplitude is an industry leader in conducting b2b market research and consumer market research surveys, providing questionnaire design, survey programming, survey hosting, and survey reporting services using its software technologies and proprietary survey panels.