This is surprisingly non-shitty by Google. I must admit that I didn't know that before. Can you limit such a passcode to just IMAP/SMTP, or can it be used to log in to other parts of Google?
This passcode is inherently limited to the service it bound to (IMAP or POP3), that's the whole point: don't expose your account password to something which only needs a finer-grained access.
You're directly contradicting your sibling comment. I guess I'll experiment with this in the coming days, although I'm a little worried tinkering too much will just completely lock me out of my account.
If you can't switch to OAuth, you can simply create an app password and continue using that as your IMAP password as usual.