On OpenBSD, dlopen() and friends are provided within libc and there is no libdl. So instead of linking libdl unconditionally, check for presence of dlopen in libdl and only if present link against it. The patch has been tested on OpenBSD-5.2-current and Debian/wheezy.
I believe NetBSD doesn't provide libdl as well according to their documentation, so the patch could help there as well. FreeBSD might provide libdl for compatibility, so this patch shouldn't hurt.
The patch is against trunk revision 2189.