I've been playing with OpenBTS for a few weeks now and would like to share some experiences with the list.
I first tried using the USRP 64 MHz clock. It was far off and none of my phones found the signal. Purchased a 13 Mhz TCXO on ebay and soldered it to a 4x clock multiplier from TI (free sample). Works perfectly.
Tx/Rx isolation
I'm using a single RFX900. After I removed C202 the isolation was app. 50 dB and it was possible to send to and receive from a phone anywhere in the same room. I then grounded pin 8 on U209 and that raised the isolation to 65 dB. That made it possible to take the phone to the next room. I wanted more isolation and tried shielding the TX amplifer. Took me a whole day to make and solder a box around the tx amplifers, but the improvement was only 2 - 3 dB. At this point I actually considered cutting the RFX900 in two to separate the transmitter and receiver, but I had one more idea first. I completely removed the trace from U209 pin 3 to C205 and soldered a new SMA connector to the board, placed directly above C205 with the center pin soldered to C205. This seems to have (I'm not able to measure accurately) increased the isolation to 90 dB.
Rx gain
Even with the improved isolation, it was still not possible to setup a call from outside the house. To avoid receiver saturation because of tx leaking into rx, I had to reduce the rx gain from 80 dB to 30 dB. To compensate for the loss of sensitivity, a 40 dB amplifer (Lucent KS21583 L9, $20 on ebay) was connected outside the USRP. I now have a good balance between up-/down-link range, and are able to setup a call when the Nokia Field Test Display indicates a signal strength of -100 dBm. With a small whip antenna on the BTS that is 1/2 mile away. With a decent antenna it would be a lot more.
I'm using a "G-way Microwave Diplexer P/N CD940/6SK-E" (I know a diplexer is supposed to be different from a duplexer, but it works fine, so I don't care). It has 6 cavities on each side and provides more isolation than I can measure. This was also $20 on ebay.