Depending on the algorithm, there is this: https://homes.cs.washington.edu/~marcotcr/blog/lime/

It's a neat approach that I've used with some random forest classifiers.