I think I found a bug in the fopen functions of devkitPPC when it is used with a custom devoptab device.
The main problem is that the Wii's memory starts at 0x80000000 which will always become negative when casted to an integer and make one check in fopen fail.
devkitPPC's _open_r function allocates the handle in the following way (this code is taken from buildscripts/dkppc/patches/newlib-1.15.0.patch line 5273 in the CVS):
handle = (handle *)malloc( sizeof(handle) + devoptab_list[dev]->structSize );
The memory address of the handle will therefore be somewhere above 0x80000000. This handle is then casted to an int and passed to the custom devoptab _open_r function and later returned to fopen.
newlib's fopen functions does the call to _open_r in the following way (taken from src/newlib/stdio/fopen.c)
if ((f = _open_r (ptr, file, oflags, 0666)) < 0)
fp->_flags = 0; / release /
__sfp_lock_release (); return NULL; }
Because the return value of _open_r will always be negative as it's a memory address casted to int fopen will always fail no matter what the function specified in the devoptab struct does.
A possible fix is to just make fopen check if f is -1, which is the only error value returned by _open_r.
I attached a text file with both full functions and two comments to demonstrate where the error occurs.