Same can be said for your phone having a browser and brokerages and banks being accessible via browser.
Make sure the brokerage/bank logs you out after a few minutes of activity, and has password or biometric verification for things like transfers or adding new external accounts (surly all do), and you’ll be fine.
Further, is your phone not locked at all times when you’re not using it? If I put mine down, it’s locked for for whoever picks it up unless it’s me and the biometric (faceID) authenticates that it’s me and unlocks automatically.