Customer research and its abuses

There are many different ways to abuse information about customers. Simply claiming that customers are important doesn't signify much. It takes no work to say "We care about customers" or "Customer satisfaction is important" because rarely does anyone ask how those beliefs map to organizational behavior. Even though in the last decade much progress has been made in refining methods for researching and understanding customers, most of it has not penetrated through to management- or engineering-centric organizations. It's still uncommon for project teams to have an expert in customer research, interface design, or usability available to decision makers.

By far, the most prevalent mistake I've seen in customer research is over-reliance on a single research method as the source for decision making. The fundamental problem with all research, scientific or otherwise, is that a given study assesses only one point of view on an issue (we'll discuss this again in Chapter 8). Each method for examining something is good at measuring certain attributes and horrible at measuring others (see Table 3-2). Just as you would never use a speedometer to measure your weight, or your bank account to measure your blood pressure (though they may be related), there are some things that surveys and focus groups are good for and others that they are not.

Table 3-2. Common customer research methods


What is it?



Focus group

A group of potential customers are brought together to view prototypes and give opinions in a facilitated discussion.

Can get many opinions at once. Allows for extended suggestions and open dialog.

Discussions are difficult to analyze and easy to misinterpret. Poorly trained facilitators create deceptive data.a


A series of questions are given to potential customers.

Low-cost way to get information from large numbers of people. Good for very broad trends.

Information reliability is low.b Authoring surveys without biasing answers is difficult. Easy to misinterpret data.

Site visits

Experts or team members go to the customers' work sites and observe them doing their work.

Observe the true customer experience. Often this is the most memorable and powerful experience for the team.

The data is most valuable to those who did the visit: it's hard to transfer to others or to use quantitatively.

Usability study

Selected customers use a design in a controlled environment. Measurements are taken for how many scenarios they can complete, in how much time, and with how many errors.

Quantifies how easy it is to use anything. Provides evidence for specific problems. Most valuable when done early, before project begins.

Little direct value for business or technological questions. Can be wasted effort if done late or if engineering team doesn't watch often.

Market research

The market of the product is examined to see how many customers there are, what the competing products cost, and what the revenue projections are.

Only way to capture the business view of a market or industry.

Doesn't explain why products are successful, and it focuses on trends and spending, rather than people and their behaviors.

a Focus groups tend to bias people toward being helpful. They don't want to insult their hosts, and they will often be more positive and generous in considering ideas than they would otherwise.

b Consider how diligent you were in answering questions in the last survey you took. If you never take surveys, ask yourself about the kinds of people likely to spend lots of time taking surveys.

Experts at customer research do two things: they choose the method based on the questions the project team needs to answer, and they make use of multiple methods to counteract the limitations and biases of individual approaches. Table 3-2 outlines some of the major research methods and their high-level tradeoffs.

As a program manager at Microsoft, on the best project teams I worked on, I had access to many of these sources of information. I'd often have to request answers to specific questions that went beyond what I was provided with, but there were dedicated experts in the organization who would generally do this for me. On other teams with less support, I'd have to go and make do on my own (typically with less success because I had many other things to do as well, and I wasn't as proficient at getting results as a full-time expert would be).

Even with no resources or budget, a few hours of work toward answering those planning questions can sometimes provide useful results. Focused energy spent on smart web searches and library inquiries (real librarians are often more powerful tools than web sites) can reveal sources that are infinitely more useful than nothing. Over time, the skills and experience in doing this kind of research will grow, and it can take less time in the future. More importantly, having done some of this kind of work on your own will put you in a more informed position to hire someone to do it for you, should the budget or headcount finally be offered to you.

With any source of data, skepticism and healthy scrutiny help refine and improve its value. Assumptions should be questioned, and known biases of different kinds of research should be called out at the same time the research is presented in a discussion. This doesn't mean that that data should be thrown out simply because there isn't enough of it or because there are valid questions about it. Instead, the team should try to look past the flaws to find the valuable parts that can be used to influence discussions and give a better perspective on what the reality of the customer's experience is like. No form of data is perfect: there are always biases, caveats, margins of error, and hidden details. The project manager has to be able to see past the biases and make intelligent use of what's available to make better decisions.

Was this article helpful?

0 0
Search Engine Optimization Overview

Search Engine Optimization Overview

This is the 2nd volume of a 9 volume series called the Webmasters Toolbox package. Search engines are the number one way that internet users find websites. In most cases, a listing in a search engine is free. So, it's no surprise that Search Engine Optimization SEO is often the first priority when marketing a website.

Get My Free Ebook

Post a comment