I found the problem while transitioning to a 64bit machine

The problem seems to exist for physical memory exceeding 504MB because that is the precise limit which tips VMALLOC_START with MSB being 1

That would be a problem for 64bit machines since in symbol relocation function, namely apply_relocate_add in arch/x86/kernel/module.c:249,
evaluating R_X86_64_32S sign extends 64bit values

The root of the problem though, is that  VMALLOC_START  start is too big to begin with since START is too big in Makefile-x86_64 

Changing to a smaller value, like say 0x8048000 used for i386 machines seems to do the trick

Any reason it was 0x60000000 ??