Our experience was as a Python shop who was backed into a corner to use Apache P... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		pixelmonkey on Feb 17, 2015 \| parent \| context \| favorite \| on: Introducing DataFrames in Spark for Large Scale Da... Our experience was as a Python shop who was backed into a corner to use Apache Pig for our Hadoop batch jobs. We decided to rewrite some of those jobs from Pig to PySpark, and though there was a little bit of a learning curve and some sharp edges, the development experience is so much better than Pig that my team is generally happy with the switch.

rjurney on Feb 17, 2015 [–]

PySpark is really compelling for Pig/Python shops. If it weren't for Pig on Spark, I'd fear for Pig's future.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact