Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Interesting!

If any puppeteer developers would also find it useful, let us know - it's always good to know something is wanted before you make it.



Check out Headless Recorder. It records browser interactions and generates Puppeteer and Playwright scripts.

https://github.com/checkly/headless-recorder


So what I'm suggesting is slightly different.

This outputs a series of clicks and keyboard events. It doesn't handle the logic necessary for most JS-heavy web-apps.

Scraping an infinite scroll is a good example. A naive algorithm is - scroll, wait, scrape, scroll, wait, scrape...

We have a slightly more sophisticated algorithm to deal with this. Also, what happens when you encounter a page with multiple iframes?

I'm essentially talking about a library which works with higher-level abstractions than just click and type, perhaps 'ScrapeInfiniteScroll' - and similar operations.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: