What is MaxDiff, or as some other sources may refer to it, Maximum Difference Scaling, or Best-Worst Scaling? Simply put, it is a survey research technique used to determine preferences between a predefined set of items. By analyzing MaxDiff survey data, we can draw conclusions such as which one is the most/least preferred or rank them.

Introduction to MaxDiff

On a higher level, a MaxDiff questionnaire has multiple pages and on each of them, a subset of the items are displayed. The respondent is asked to pick two of them per page: the most and the least preferred one.

There are, of course, other more traditional approaches that can be used for creating a similar output. An example of this is using rating questions: let’s ask respondents to rate the items to set up a ranking based on these ratings. This method may seem to be the most straightforward way to find out preferences, but they also have many limitations. With such an approach, we can run into multiple problems. One of these is that a respondent might turn out to be a "straightliner" and enter the same value for each item, completing the questionnaire with minimal effort. Another limiting aspect of a ranking question is that it doesn’t inspire respondents to think of relations/comparisons between the various items included in the question, so creating the ranking, in the end, may be a tough or even a futile task.

These are just some of the many associated problems with the attempts to find a “work-around” for a MaxDiff technique. There are many advantages in using MaxDiff : first of all, it’s straightforward for a respondent to complete. It requires only two clicks per question. For this reason, a large number of items can be compared with each other without overwhelming a respondent. Moreover, this method is considerably more diverse than any other. Not only the ranking can be determined, but other aspects as well like how much the difference between several items is.

To make this fair, it’s necessary to mention the disadvantages of this technique as well. The most important one is its complexity, which is a tradeoff that regularly comes with diversity. Analyzing the data is not an easy task at all as it requires quite some knowledge of statistics. The good news is that because MaxDiff is gaining more and more popularity, you can easily find help and support.

Designing the study

Now that we know the basics of MaxDiff let’s dive a little deeper. How can we design a study?

First of all, we need to determine which items (typically brands or services to compare, in the case of Nebu's MaxDiff demo survey coffee brands) we want to include in the questionnaire.

After this, we can create the design, which involves subsetting the items and combining these subsets as questions. Then their combination can be interpreted as a complete questionnaire, where on each of the pages, the respondents will need to select their most and least preferred entries.

It is important to mention now that even though we will now dive deep into the logic of a MaxDiff study, you won’t have to worry about creating this design from scratch because we can offer you a ready-to-use solution (see below, or if you are a registered Nebu Data Suite user you can go to this support article to learn in more technical detail on how to implement a MaxDiff). 

This design is usually a Balanced Incomplete Block Design, or BIBD for short. Essentially setting this up is the process of subsetting items and creating combinations of subsets so that both the individual items and pairs of items appear together an equal number of times. The main advantage of this is that we can cut down on the enormous number of comparisons, that would have been required of a respondent to answer to in a classical ranking questionnaire, while maintaining the reliability of the study.

One of the design attributes is called version, which is essentially a single combination of subsets. It is important if the design has multiple versions, which is usually the case. When dealing with a small number of items, we may design a single-version study, but usually, it is not a good idea. Having multiple versions ensures the balance while keeping a small number of comparisons.

Each version contains an equal number of sets, which can be considered a single page/a question of the questionnaire. As previously mentioned, it is a subset of the original item list.

The next table represents a very simple MaxDiff BIBD. It consists of two versions, each version contains 3 sets and in each set 4 items are compared. The option column contains the ID of the items that will be part of the given set.

Once the design is set up, the next step is to build the actual questionnaire. Of course, the implementation depends on the data collection tool. Still the best practice would be to create a general template that can dynamically build the questionnaire based on the design created in the previous step.

The questionnaire's dataset should contain the following information: items displayed on the page, the item chosen as most preferred, and the one that was chosen as least preferred. 

As previously mentioned, Nebu Data Suite has a ready-to-use implementation for MaxDiff questionnaires that you can add to your projects with relatively low effort. It consists of two main parts: an R script that is used for creating the BIBD. It can be parameterized and run in Nebu Data Hub. The second part is a Nebu InterViewer questionnaire which can pick up the design and build its pages accordingly. You can check out a Nebu MaxDiff demo questionnaire here. What is great about this solution is that it is ready after setting a few parameters but can also be easily customized if you need something more advanced.

Now let’s talk about what we can do with the data of a MaxDiff survey. There are three groups of frequently used analyzing methods, from which this article discusses two in more detail.

Some additional information and examples can be found on this Nebu’s MaxDiff demo dashboard Nebu’s MaxDiff demo dashboard, which is also the output created from the previously referenced, coffee-themed survey.

Analyzing the data

The first and most basic technique is Count Analysis. It involves counting the number of times a certain item was chosen as most and least preferred. By subtracting the latter value from the first, we can assign a score to each of the items and rank them accordingly.

While it’s a simple method, it has several flaws. For instance, it doesn’t take into account which items were compared with which other ones, even though it is a significant factor. The other disadvantage is that Count Analysis offers only the overall ranking of items, which may not be enough for some more detail-sensitive studies.

Logit Modeling can be considered a more advanced approach. A Logit Model is a statistical model with which the probability of certain outcomes of an event can be predicted. Using this method, we can calculate choice probabilities for the items. In other words, these are the probabilities that a certain item will be chosen as most preferred. These values also allow us to set up a ranking, but we can also create different models for different purposes, such as observing data per gender or age group.

The fact that this method includes which items were compared with each other makes this a more refined and insightful way to handle the survey data.

The most advanced method that we can use for analyzing MaxDiff data is Hierarchical Bayes. It’s a very robust method and requires quite a complex statistical background and for these reasons, we will discuss it in a future article.

If you want to learn more about Nebu Data Suite and how it can help you automate processes and increase work efficiency submit a form to the right. Our expert will reach out soon to schedule a call!