Ways to visualize the outcome of machine learning interpretability techniques (for image classification)

“Machine Learning Interpretability” or “Explainable Artificial Intelligence” has become quite popular in the machine learning community and in recent research. The goal is to make complex (deep learning) models explainable such that one can understand why the model made a particular decision. I had a look at various algorithms which do this (prominent ones like LIME, SHAP, Grad-Cam, but I've also skimmed over many papers that present very “special” approaches).

Since I am working with image data, I am particularly interested in explaining image classification decisions. However, what I realized is that, even though there is a manifold of algorithms, the (final) representation of an explanation of an image classification decision for the end user is always the same: It is the original image with some marks showing what part of the image had the most influence (whatever this means based on the underlying mathematical explanation algorithm) on the decision. Concretely, the explanations are always the original image and some sort of heatmap or outlines on top, marking the "most influential" pixels.

What other representation of an explanation of an image classification could be used? Are there any explanation methods that allow a representation as a textual or numeric explanation (instead of just heatmaps or outlines on the original image)?