If you're talking about X-ray, nodus, etc. then your server log should output when it detects such mods.
Not really, most of those mods are base-file modifications and are installed over forge (if it has compatibility), so the server does NOT detect it. Only way (i know) to detect them are to use plugins (bukkit) or watching the suspect player.
@OP : Não há a necessidade de restringir clientes com outros mods, ja que estes não serão utilizaveis caso o servidor nao os contenha tambem. Ja se tratando de "cheat" mods, as unicas formas que conheço é plugins bukkit e monitoramento de players (invisibility).