Course · 4. MaxDiff

The MaxDiff model

Chosŏn fits the standard model for MaxDiff, the hierarchical Bayes (HB) best-worst logit. Nothing here is our own method.

One score per person and item

Each respondent has a score for every item. In each set:

• Best is a logit choice among the items shown: an item with a score 1 higher is about 2.7 times as likely to be picked.
• Worst is a logit choice on the reversed scores among the rest (the item picked best cannot also be worst).

A dozen sets are not enough to pin down 16 scores for one person on their own. HB fills the gap: people's scores are drawn from a common distribution, so each person's estimate leans toward the sample's average where their own answers say little, and away from it where they say a lot. The fit runs 20,000 rounds of sampling and keeps the second half.

Scores are relative: only differences between items mean anything. Chosŏn centres each person's scores so the average item is 0.

The comparison model

Chosŏn also fits an aggregate logit: one set of scores for everyone. It is used only in the holdout check, to show how much person-level scores add. When people mostly agree, the two predict about as well.

Ranges

Every output has a 95% range. It combines two kinds of uncertainty: the fit's (200 samples of every person's scores) and the sample's (who happened to answer, by reweighting respondents at random). On 10 simulated surveys the ranges for share of preference contained the truth 91% of the time, and for the top 3, 89%: a little less often than 95%. The truth there had three groups of people with different tastes, which the model does not assume.

Try it in the tutorial: model setup

Next: Reading the results

Tutorial links open the example project in the app; it is free and needs a free account.