This is indeed correct. Somehow I missed this. Setting label_mode = 'all' gets the behavior I wanted.
Though, the top and right side are technically on the outside too. Perhaps that should be an available or the default setting? Perhaps the top row should default to labeling on the top, the right column default to labeling on the right, etc?
Isn't it a feature? Axes_grid lets you choose the label mode which is
'L' by default. This means that only the outer labels are shown. Or
am I missing something in the description of the problem?