It would be a nice feature to create a different PulseAudio source per user. This would allows you to filter, record and modify streams of individual users. For example, you can mute and make people louder, or and an effect to some user.
I think that this should be configurable because not everyone will want this feature and there might be a performance overhead when it is active.
Another thing I would think is that streams would need to stay open even when users are silent, so that you don't lose track of users every time they stop talking.