Jai and whomever may be interested:
Many larger Open-Source Softwares will usually have premium support of some sort. Some will accomplish this by dual-licensing their product (a free, unsupported version under GPL or similar, and a closed license for pay support). Others will provide contract support for their product that is licensed under just an open-source license.
Especially with software such as an ILS (where operational uptime could be critical), companies can be contracted for support. Examples would be Koha (support provided in the most part by LibLime) and Evergreen (support provided in the most part by Equinox Software Inc.). In these cases, the majority of (recent) development was contributed by the companies, and that's how they establish themselves as a credible support service (above and beyond standard industry practices).
From a business POV, a commercial ILS that decides to open up their product will surely retain the majority of their customer base. The community contributions will ultimately end up in a better product (imagine having hundreds of potential developers and code critics but are only paying a handful), and the potential competition will keep the ILS support from doing evil (Under most OSS licenses such as the GPL, anybody can provide support... As such, a vendor will need to continuously "prove their supremacy").
For products as large as an ILS, it can be fairly common to have different portions under different licenses. For example, the majority of the ILS can be under a FOSS license (such as the GPL), while tools like data migration kits can remain under a proprietary license (of course, there's nothing preventing a 3rd-party from developing their own kits and potentially release them under a FOSS-license).
A business model which I've previously admired was that used by Mandarin Library Automation. Their ILS product, Mandarin M3, has the core (database, circ, standalone PAC, etc.) released under a no-cost license (it's still closed source, though). Additional functionality (such as a Web OPAC, Z39.50 server, Authority Control, and the like) can be purchased additionally.
I hope this answers some of your questions! If you have any more, please do not hesitate to contact the list again!