Alternative, real life scenario: navigate through 6 months of daily MySQL dumps, assorted YAML files and Rails production.log, looking for some cross product between tables, requests and serialised entities, for analysis and/or data recovery (pinpoint or retrieval).
zcat/cut/sed/grep/awk/perl crawled through it in a couple of minutes and required less than half an hour to craft a reliable enough implementation (including looking up relations from foreign keys straight from the SQL dumps).
My colleagues, who still don't get the point of a command line, would still be restoring each dump individually and running SQL requests manually to this day (or more probably declare it "too complex" and mark it as a loss). Side note: I'm torn between leaving this place where nobody seems to understand the point of anything remotely like engineering or keeping this job where I'm obviously being extremely useful to our customers.
> Side note: I'm torn between leaving this place where nobody seems to understand the point of anything remotely like engineering or keeping this job where I'm obviously being extremely useful to our customers.
You should always aim at working with people who are smarter or better than you. Unless they have stack ranking.
I use the multiprocessing module in python all the time for quick parallelism <https://medium.com/@thechriskiehl/parallelism-in-one-line-40... Let's you easily map to multiple cores. I use it a lot for image processing tasks. Quickly crawl through directories to find all the files, then spin up all the cores on my machine to crunch through them. Wish there was an easy way to enlist multiple machines.
Yeah. This is a great article. While I was reading it, I was thinking about what I would have done, and my answer was Python. Bash is just too easy to do wrong (see the recent Steam rm -rf bug), and I don't code in it often enough to know the pitfalls by heart.
I'd be interested to see another article about doing this job in Python and how its performance compares to this simple one-liner.
Python is a great tool for this, and is even better when used with the pythonpy tool, which allows for convenient integration of python commands inside unix pipelines - https://github.com/Russell91/pythonpy
zcat/cut/sed/grep/awk/perl crawled through it in a couple of minutes and required less than half an hour to craft a reliable enough implementation (including looking up relations from foreign keys straight from the SQL dumps).
My colleagues, who still don't get the point of a command line, would still be restoring each dump individually and running SQL requests manually to this day (or more probably declare it "too complex" and mark it as a loss). Side note: I'm torn between leaving this place where nobody seems to understand the point of anything remotely like engineering or keeping this job where I'm obviously being extremely useful to our customers.