Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I wanted to download a few hundreds of files, but the server was enabling only 4 simultaneous connections.

I did a makefile like

    file1: 
        wget http://example.com/file1

    file2: 
        wget http://example.com/file2

    file3: 
        wget http://example.com/file3

And used make -j4 to download all of them, but only 4 parallel tasks at once. It starts another download when one finishes


Check out aria2 next time.

https://aria2.github.io/


Xargs parallel processing or Gnu parallel would probably help a lot here.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: