We have fantasy stories about magic being manipulated through a specific magical language. Say the word for "fire" in this language and BOOM, fire.
What if magic could manipulated by any language but only certain people had the capability to do it? If you're not a magician, nothing will happen when you say "fire," but if you are then you need to be careful. Magicians that don't restrict themselves to sign language on a regular basis are centers for cataclysmic change, continually reshaping the world around them with every word spoken- if they were sane when they decided to be such things, they don't remain so for long but quickly become more like a bodily force of nature with increasingly inhuman drives.
Moreover, when magicians speak, they don't get specific results. The magic is locked down for as long as they are expressing a thought (and with greater practice comes the ability to lock it down for longer periods of time, with young magicians unable to keep it for longer than it takes to say a single word) but when they're done it takes shape and is bound only by the words used. "Fire" can do anything from shoot fire from your fingertips to turn your enemy to fire to turn yourself into fire and only the very slightest of mental nudging is possible (if you're doing it with the intent of harming someone then the blast of fire will be directed at your enemy) but it's based on raw emotion (it's really not so much your enemy as it is "whoever you most want to hurt," which means it's not so easy as just shooting the guy with the knife in front of you but still too far to be an immediate threat if you haven't gotten over the fact that your boyfriend has been cheating on you and he's in the same room).
Magicians need to be kept from speaking until they're old enough to be careful with what they say (maybe stitching the mouth shut, but if this has been going on for long enough some other techniques have certainly been developed) and mostly use sign language. Magicians need to know what the word means (it needs to have some meaning in their minds to be shaped by) but that doesn't help with communication because a language that can't be shaped by magic is then also a language that they don't understand (or else it could be shaped).