There is nothing in the constitution directly disallowing extreme racial gerrymanders. Those are unlawful not because they're unconditional, but because they're prohibited by the voting rights act.

Congress could very well have passed simple laws banning racial and gender disenfranchisement in federal elections. The amendments were necessary to impose a rule on sub-federal elections and to keep a mere majority from taking the franchise away.

The US Constituon is neither very long nor hard to read, and it has oodles of text that Congress could invoke to ban the gerrymandering of congressional districts:

US Constitution article 1 section 4:

The Times, Places and Manner of holding Elections for Senators and Representatives, shall be prescribed in each State by the Legislature thereof; but the Congress may at any time by Law make or alter such Regulations, except as to the Places of chusing Senators.

Article 4, section 4:

The United States shall guarantee to every State in this Union a Republican Form of Government,...

14th amendment section 2:

Representatives shall be apportioned among the several States according to their respective numbers, counting the whole number of persons in each State, excluding Indians not taxed. But when the right to vote at any election for the choice of electors for President and Vice-President of the United States, Representatives in Congress, the Executive and Judicial officers of a State, or the members of the Legislature thereof, is denied to any of the male inhabitants of such State, being twenty-one years of age, and citizens of the United States, or in any way abridged, except for participation in rebellion, or other crime, the basis of representation therein shall be reduced in the proportion which the number of such male citizens shall bear to the whole number of male citizens twenty-one years of age in such State.