Anaconda (and miniconda) is a very good solution if you need a container-like environment. I can easily go back to old projects with different python versions and do maintenance in them with no hassle.
Use docker/podman if you are worried about ABI changes and isolating them from the entire operating system.
Most people's problems getting their Python toolchains to work optimally are caused by using operating systems that don't come with build utils. That's a cultural problem solved by using an operating system with a culture of distributing those tools.
Use poetry/Debian. Standard Python tooling is acceptable in 2024; using some busted 3-plus-year-old "supported" environment with a package manager with a really busted constraint solver, which doesn't even come bundled with compilers, is unnecessary.
I've installed plenty of compilers using conda - how long ago was it that you last tried it? even ROS is available in conda now, instead of requiring specific obsolete versions of Ubuntu.