Menu
▾
▴
#176 Logit/probit results inconsistency
I found possible inconsistency in logit/probit output.
Les's estimate simple logit and compute 3 'out-of-sample' forecasts:
<hansl>
set verbose off
open wagedisc.gdt
list X = 1 2 3
smpl 1 $nobs-3
logit Sex 0 X --quiet
fcast --out-of-sample
</hansl>
What we get as a results is a matrix with prediction column containing probabilities - which is fine:
Sex prediction
98 0.00 0.01
99 0.00 0.00
100 0.00 0.01
But if we estimate ordered logit:
<hansl>
set verbose off
open wtp.gdt
list X = 1 2 9 10 11
smpl 1 $nobs-3
logit depvar 0 X --quiet
fcast --out-of-sample
</hansl>
what we get as a results are integers, not probabilities:
depvar prediction
310 3.00 1.00
311 2.00 1.00
312 3.00 1.00
What's is more, if we look at $yhat accessor, is case of ordinary logit we get probabilities again, but in ordered logit case we get values of random variable instead.
Diff:
IMO this makes some sense: in the simple binary case, once you have the estimated probability, you're free to use whatever criterion you choose to make a bet on the outcome. On the contrary, for ordered/multinomial models you have a vector of probabilites, so in order to coerce the prediction into a series you need to choose some criterion beforehand.
The solution I would like best would be changing the output to the fcast command to something similar to what we already have for the "var" command (therefore turning the $fcast accessor to a multi-column matrix), but in our online meeting other solutions have emerged.
I like this solution and - if I may suggest - such additional column should include random variable values (to be inline with $yhat). Then user - if she/he wants - may convert it using Allin's 'oprobit_predict' package.
After this evening's online meeting, there was a broad consensus on this idea: for ordered and multinomial models we could modify $yhat to report a matrix, with the probabilities for every possible outcome, whereas $fcast could report just the point forecast, maybe by choosing the outcome with the highest probability, or some other method to be duly documented.
New discussion: "everything stays as is", except that a new option to fcast is introduced (only for non-binary discrete models, like multinom, ordered...). This option triggers returning probabilities in $fcast. (And making $mnlprobs obsolete.) Plus, in order to get the analogous thing after estimation (complementing $yhat), a new accessor, say $allprobs, is introduced for that. (And the new option to fcast could also be named --allprobs.)
$allprobs now exists, $mnlprobs is deprecated, and the new option to the fcast command is --all-probs (with a dash/hyphen in the middle). The only thing that remains to be done is to document the latter.
From what I understand, also the doc should now be completed, so I'm closing this.
Reopening this because we offer a menu entry for $allprobs under the "Analysis" menu for multinomial models but not for ordinal ones. Should be easy to add.
I've found that my package oprobit_predict already offers to attach to the Analysis menu in a window showing ordered probit or logit estimates, and its output is better than that of $allprobs. Plus it's very well documented, if I say so myself! So I'm not sure what we should do. Obviously we could upgrade $allprobs to match what my package produces.