Every time I open qbittorrent it slowly more and more memory util it runs out.
Does anything happen after that, or is this just an observation? Your OS should know when to swap to disk to avoid things like crashes, etc.
You may want to update your post with more information on your OS, setup, if you're actively uploading/download torrents, how many torrents you're seeding, things like that.
When I was on Windows I do remember seeing qBittorrent/Libtorrent 2.x use as much available RAM as it could but nothing bad ever happened, I just ignored it. In fact at one point the qBittorrent devs even added an option to limit RAM use and I used to just set that to the maximum RAM & just let qBittorrent/Libtorrent do its thing. Not sure if the Windows version of qBittorrent still has that option but it used to be a setting under Tools/Preferences/Advanced at the top under the qBittorrent section. I don't see it in the current version under Linux.
Barring any special settings the usual advice would be to limit your maximum connections and/or upload/download bandwidth, probably tweaking the Global Maximum Connections will be enough. The more connections qBittorrent/Libtorrent establishes the more RAM gets used. Libtorrent 2.x historically used to use more RAM but I don't know nowadays how much of a problem that is.
In case it helps I haven't seen anything abnormal on qBittorrent 5.3.0/Libtorrent 2.x + Debian Linux.