Indeed. I recommend you look into one of the projects that is trying to mitigate this problem by allowing you to precisely specify, inspect, and change the dependency graph of software on your computer. That is, Guix or Nix.

Guix (and I presume Nix) can then take advantage of that transparency and control to build containers, virtual machines, and bare-metal systems to your specifications.