I always liked the Yahoo Pipes concept... but it didn' seem take off... and I personally found it too limited for everything I tried to do with it. Perhaps it's just another case of the old "visual programming language" is harder than it looks.
I hope Huginn does better. I like their copywriting "You always know who has your data. You do."
Agreed. I did a multipart Yahoo Pipes project to find my current apartment. It grabbed info from two sites, tossed out the uninteresting ones, filtered it a bit, then texted me if a new apartment in my price/location range appeared.
Very useful, if a little awkward. The Huginn project sounds like a great alternative!
It's hard to tell -- I tweaked it quite a bit, and rewrote it from scratch 2-3x. I'd say 4-10 hours.
Was it worth it? As a programmer, no. I'm very familiar with scraping (raw) web/RSS feeds for data, then processing it. I was hoping Pipes would have enough intelligence, so that I could subscribe to (cooked) data sources, then split and refine the results.
In practice, Pipes worked, but the data always required further post-processing, which was awkward to do in Pipes. You have to be a dev to understand what your system is doing, but you don't have easy access to all the standard dev things.
I look forward to seeing Pipes take off, or another technology (Huginn? Ifttt?) replace it. It was a lot of fun to wire things up graphically then for example get a text when someone's RSS feed changed.
You have to be a dev to understand what your system is doing, but you don't have
easy access to all the standard dev things.
Interesting, this mismatch may be a good description of the problem of visual languages.
Curious: what do you think is the minimal subset of unix tools to do this? i.e. instead of pretending the problem is simpler than it is, accept the complexity, but minimize it.
I'm thinking of a tool like "jq" (sed for json) for json data sources... but I don't think its raw-text manipulation is up to the task (and of course you need tools to monitor the feeds etc).
Python :-) there are libraries specifically for parsing malformed html. I'm happy using Unix tools for scraping and parsing, but you run into a brick wall rather quickly. Python is more reliable, flexible, and easier to integrate.
They're all quite simple. The most complex one uses the "parse location into lat/long" Pipes feature to automatically find me jobs in the Los Angeles area.
I hope Huginn does better. I like their copywriting "You always know who has your data. You do."