tujikawa - 2012-10-16

The file here means file descriptors and they includes the number of open regular files and sockets connected to the peers. Normally, it is limited to 1024 per process. You can limit the number of open regular files for each torrent using --bt-max-open-files. You can also limit the number of connected peers using --bt-max-peers. You wrote you tried --bt-max-open-files.
If these options don't work, you can increase the number of file descriptors per process using ulimit command. For example to increase it to 2048:

$ ulimit -n 2048