You could try using the same settings I use for uploading to Youtube:

Container: Matroska (.mkv): matroskamux
Video codec: H.264: x264enc
Audio codec: AAC