Nope, Mojang TOS says Minecraft Mods can 'do whatever they want' and accordingly this includes specifying an own ToS for the mod itself
I would be wary about using that sort of sloppy legal language to support your argument. "Mods can do whatever they want" is fine in conversation, but in actual terms of service, it's just about meaningless.
Even if Mojang's TOS document says that minecraft mods can 'do whatever they want', that doesn't grant legal backing to everything a minecraft mod does. For a ridiculous example, let's say I made a mod with terms of service that transfer ownership of the user's computer to me, as well as sovereignty over the continent of Antarctica. Would I have the legal ability to do that? No, of course not, but it falls within the extremely vague guidelines of Mojang's terms of service.
You weed out the trolls and troublemaking players by using whitelists and inviting your friends to play. Since they're already your friends, why make them waste their time downloading, configuring, and possibly tweaking a ton of mods to play nice together?
I find forcing users to download all of the mods individually to just be a crutch for the lazy and dumb. Not understanding the critical importance of ease-of-use is a sign of technical incompetence.