They are invasive (in the US) and grow like weeds completely taking over and crowding out native species. Having many of them screws up the nitrogen level of the soil further crowding out/killing native plants.

This alters the ecosystem.