> It's weird how it's not considered a basic human right to be able travel to where you want, and even live where you want as long as you can support yourself and comply with the local laws and customs
The "as long as you can" is exactly the reason. There is no way to ensure the travelers or migrants can (and want) to do that, if you don't have a border.
Really? If I go to another country and commit a crime they can't enforce their laws on me unless they stop me at the border? I don't see how stopping me at the border stops me from committing a crime when I get past it or stops the country from enforcing their laws when I am caught. What did the border magically do to change enforcement?
The "as long as you can" is exactly the reason. There is no way to ensure the travelers or migrants can (and want) to do that, if you don't have a border.