Course · 4. MaxDiff
MaxDiff: what to expect
Chosŏn's MaxDiff is the standard method. Its results should match those of other tools using the HB best-worst logit, up to sampling noise.
How close it gets
The tutorial's data was simulated from a known truth: 16 fitness-app features, 300 respondents, three groups of people with different tastes (40% training-focused, 35% wellbeing, 25% social). On 10 surveys simulated this way:
| output | average error | truth inside the 95% range |
|---|---|---|
| Share of preference | 0.5 points | 91% |
| In the top 3 | 3.2 points | 89% |
Shares are recovered closely; the top 3 less so, because it depends on each person's ordering of their favourites, which a dozen sets only roughly reveal. Items liked strongly by one group and little by others are the hardest: in the tutorial, Habit reminders is in the top 3 of 9% of people in the truth, and the fit says 2% (0% to 5%).
On the held-out sets, person-level scores picked the best item 42% of the time on average, against 39% for one set of scores for everyone and 25% by chance. On a single survey of 300 that gain is within the noise.
Not in Chosŏn yet
- Anchored MaxDiff: an extra question per set ("would you use any of these?") or a threshold item, which would say whether people want an item at all, and give reach a real meaning.
- Segments from the person-level scores.
End of chapter 4. Next: TURF