If you want to do template docking, I recommend generating a large ensemble of conformers for each of your ligands, aligning to the common substructure, and then minimizing with smina rather than docking.
--custom_atoms is what you want. To see the format of the file it is expecting run smina with --print_atom_types. If you are using the default Vina scoring function the number you want to change is the xs_radius.
The ligand is charged which means OpenBabel wants to add a hydrogen to the charged nitrogen, but it doesn't really know how and uses a random vector resulting in non-deterministic output. If the charge is removed or if the ligand is first converted to a different format (e.g. pdbqt) you get deterministic output. You also get deterministic output with gnina, since gnina explicitly initializes the random number generator that OpenBabel uses with the user provided seed. I've backported this behavior...
Set RNG with seed for deterministic openbabel
Add features requested by Amedeo Caflisch
smina only samples torsions. If there are issues with the rest of the geometry they will not be fixed. Pretty much any conformer generator will produce reasonable conformations that do not need additional energy minimization.
No, it is a commandline tool. But rdkit GetBestRMS will also do what you want.
You should be able to use obrms (from openbabel) and not have to worry about file format conversion (and the RMSD will be symmetry corrected if need be).