Things have gotten much more mature in the past few years.
Check out edwardlib.org which is adapting Tensorflow to support probabilistic modeling (with a heavy focus on variational inference, less so on MCMC methods). If you’ve got trillions of observations you can use stochastic VI. And tensorflow now can do distributed computation graphs, or you could just go data parallel and then average your parameters at the end.
In general David Blei’s group at Columbia does a lot of work in scalable probabilistic inference.
The other big option is of course Stan, which is really well optimized but I don’t think is particularly intended for “big data” of this kind. If you have “medium data” that fits on one machine though, it’s blazing fast.
I've been really excited about Edward but when I tried it for a project last year I could never get it to come together in the right way. I got the sense that it wasn't quite ready for prime time yet, although very promising. My memory of it was that a lot of claimed flexibility in how to specify models wasn't really implemented fully. The experience also turned me off of TensorFlow a bit. But that was a year ago, so maybe it's improved?
I ended up doing it in Stan in part because I was more familiar with that, and it worked out fairly well.
Check out edwardlib.org which is adapting Tensorflow to support probabilistic modeling (with a heavy focus on variational inference, less so on MCMC methods). If you’ve got trillions of observations you can use stochastic VI. And tensorflow now can do distributed computation graphs, or you could just go data parallel and then average your parameters at the end.
In general David Blei’s group at Columbia does a lot of work in scalable probabilistic inference.
The other big option is of course Stan, which is really well optimized but I don’t think is particularly intended for “big data” of this kind. If you have “medium data” that fits on one machine though, it’s blazing fast.