By coherent I assume you mean consistent. Yes an inconsistent theory isn't considered useful in mathematics. In philosophical logic there's an idea of "paraconsistency" that means something like "inconsistent but only slightly" but I think it's not used much in math.

Russell's fix to Frege's inconsistent system was quite complicated, much more than just adding an axiom disallowing certain types of sets. ZFC handles it differently too, by saying you can only create new sets by following certain rules designed to keep things consistent. Frege's system let you do whatever you wanted and it went sideways quickly, as Russel found.