The fact that it only kind of sort of incidentally works some times shows clearly that people who consider PyTorch a vital library for their work aren't the target audience for poetry. Another use case they don't really care about are people for whom compiling C++ etc. is important part of their build process.

Poetry is great if you're developing the sort of software that the poetry devs priorities, but since poetry is only targeting a subset of the python community it will never be a good default tool for all python development, and thus the fragmentation continues.