Mine would be not slacking in learning Chinese because I never though I would be a weeb.

I thought "weeb" slur was refering to one way into Japanese pop culture, not Chinese.

Now I need to re-learn tons of Kanji

Kanji would be one of the three Japanese writing systems. The written language symbols in Chinese would be Hanzi.

That said, its never too late to learn any language, and any work you did before still helps today as you have a basis for understanding to build on. Depending on how young you learned what you did, you could have some very helpful hard wiring in your brain for other languages than your native one.