Course · 1. Background and motivation
Uncertainty: ranges and holdouts
A survey of a few hundred people gives estimates, not facts. Chosŏn reports every number with a 95% range: if the study were repeated many times, ranges built this way would contain the true value about 95% of the time. We checked this on simulated data where the truth is known: across five repeated tutorial-sized studies, 93% of the ranges contained the truth (95% range for that check: 87% to 97%).
How to read a range
- "$0.32 (95% range $0.21 to $0.43)": the best estimate is $0.32; values from about $0.21 to $0.43 are consistent with the data.
- A range that includes zero means the data cannot tell whether the effect is positive or negative.
- Overlapping ranges for two products do not by themselves mean they are equal; the simulator's "chance that A outsells B" answers that question directly.
Holdouts
By default Chosŏn leaves out each respondent's last choice when fitting, then checks how well each model predicts it. Two scores:
- Hit rate: how often the model's most likely choice is the one the person made. Easy to read, but coarse.
- Log-likelihood: rewards giving high probability to what actually happened. Higher is better; Chosŏn reports the difference between models with a 95% range, so you can see whether a gap is real.
Errors in absolute terms
When we compare a forecast to the truth we lead with the absolute error ("off by 2 share points") rather than only a percentage, because a 50% error on a 2% share and on a 40% share are very different things.
Next chapter: Choice-based conjoint