I used to wear jeans only and I pretty much hated it. I now wear those mildly stretchy pants that normally go under suits. On top, I wear polo's when the setting is semi formal. I try to find the ones that are loosely fitting because I'm kinda self conscious about wearing stuff that's too short or tight.
Idk where you are located but for me, www.wefashion.nl has what I need. You could have a look, even if you're not in the Netherlands. Maybe you'll find a similar store in your area that also deals in large sizes like that. I'm close to 2m tall and I kind of have a beer belly but narrow shoulders so only XL really works for me.