While I agree it would be nice to have the plugins as a separate package (in fact this is already planned) I disagree with your rationale. If a plugin does not compile with a certain compiler option, that is a bug that needs to be filed and fixed.

Compiling the panel and the plugins with different CFLAGS is error prone and makes things unpredictable, thus it is harder for us to support.