when I started using vim mode in zsh.

I'm an emacs user myself, but if you're not aware, readline

which handles a considerable portion of the "prompt for text" stuff in many terminal programs, like input for bash and such

can be put into vi mode.

https://tiswww.case.edu/php/chet/readline/rluserman.html#Readline-vi-Mode

In order to switch interactively between emacs and vi editing modes, use the command M-C-j (bound to emacs-editing-mode when in vi mode and to vi-editing-mode in emacs mode). The Readline default is emacs mode.

When you enter a line in vi mode, you are already placed in ‘insertion’ mode, as if you had typed an ‘i’. Pressing ESC switches you into ‘command’ mode, where you can edit the text of the line with the standard vi movement keys, move to previous history lines with ‘k’ and subsequent lines with ‘j’, and so forth.

Or, in ~/.inputrc:

set editing-mode vi

To set the default.