By accident, I read theĀ Introduction to Cross Compilation again and took Sakoman's kernel configuration from there. The kernel from the omap-3.2 branch compiled :)