When using s3cmd to upload to a bucket where the name has an underscore in it, it falls back to signature_v2, and when the bucket has no underscore it uses signature_v4. I dont think buckets with underscores and capitals need to be changed or not used, its just that new buckets can not be created with those in the names.
I have tested this on the latest version (2.0.2).
Issue solved in release 2.1.0.