Firstly, I will say that the option works fine in Heretic. It's just Doom that has this issue.
With the 3d movement option for bad guys, enabled, flying bad guys instantly jump to the top of a mob they try to climb over, rather than ascend it.
They will also instantly jump to the top of a mob, even if there is no space above the mob, for it to fit through. This causes the flying bad guy to potentially jump into the ceiling above the mob it's trying to climb or jump into the void above the ceiling.