Fingers-On Tutorial for Making use of Grad-CAMs for Explaining Picture Classifiers Utilizing Keras and TensorFlow | by Aditya Bhattacharya | Oct, 2022

It’s also possible to undergo my ebook Utilized Machine Studying Explainability Strategies and check out the code repository for getting hands-on publicity to different XAI strategies. On this article, I’ll confer with the hands-on utility of one of many in style LRP strategies known as Grad-CAM for explaining picture classifiers. I will even current the step-by-step code tutorial to use Grad-CAMs utilizing Keras and TensorFlow.

Now, let’s get began!

To clarify DL fashions, LRP is without doubt one of the most distinguished approaches. Intuitively talking, this technique makes use of the weights within the community and the ahead go neural activations to propagate the output again to the enter layer via the assorted layers within the community. So, with the assistance of the community weights, we will visualize the info components (pixels within the case of photographs and phrases within the case of textual content information) that contributed most towards the ultimate mannequin output. The contribution of those information components is a qualitative measure of relevance that will get propagated all through the community layers. Furthermore, for deep neural networks with a number of layers, studying occurs when the move of knowledge via the gradient move course of between the layers is maintained persistently. So, to elucidate any deep studying mannequin, the LRP technique permits us to visualise the activated or most influential information components all through the completely different layers of the community and qualitatively examine the functioning of the algorithm.

Class Activation Maps (CAMs) are visualization strategies used for explaining deep studying fashions. On this technique, the mannequin predicted class scores are traced again to the final convolution layer to spotlight discriminative areas of curiosity within the picture which are class-specific and never even generic to different laptop imaginative and prescient or picture processing algorithms. Gradient CAM or popularly known as as Grad-CAMs combines the impact of guided backpropagation and CAM to spotlight class discriminative areas of curiosity with out highlighting the granular pixel significance. However Grad-CAM could be utilized to any CNN architectures, in contrast to CAM, which could be utilized to architectures that carry out international common pooling over output characteristic maps coming from the convolution layer, simply previous to the prediction layer. To get a extra detailed understanding on the Grad-CAM course of, you possibly can take a look at this analysis paper Grad-CAM: Visible Explanations from Deep Networks by way of Gradient-based Localization, Ramprasaath et. al — https://arxiv.org/abs/1610.02391.

Now comes the enjoyable a part of this text: studying how one can apply Grad-CAMs! We are going to use Keras and TensorFlow to use Grad-CAMs for explaining pre-trained photographs classifiers. You will want the next Python frameworks to use Grad-CAMs which could be put in utilizing Python pip installer:

!pip set up --upgrade numpy matplotlib tensorflow

Let’s begin by loading the required modules in Python. I’d advocate utilizing native Jupyter notebooks or Google colab to run this code tutorial.

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cm as c_map
from IPython.show import Picture, show
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.purposes.xception import Xception, preprocess_input, decode_predictions
from tensorflow.keras.preprocessing import picture
import os

We are going to use TensorFlow and Keras framework to get a pretrained community on the ImageNet dataset and take a look at the strategy on a pattern open picture obtained from the supply: https://photographs.unsplash.com/photo-1615963244664-5b845b2025ee?ixlib=rb-4.0.3&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&match=crop&w=464&q=80. For extra examples utilizing Keras and TensorFlow please go to: https://keras.io/examples/.

model_builder = Xception
preprocess_input = preprocess_input
decode_predictions = decode_predictions
IMG_SIZE = (299, 299)
last_conv_layer = "block14_sepconv2_act"# The native path to our goal picture
image_path = keras.utils.get_file(
"tiger.jpg", "https://photographs.unsplash.com/photo-1615963244664-5b845b2025ee?ixlib=rb-4.0.3&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&match=crop&w=464&q=80"
)
show(Picture(image_path))

As soon as the picture is loaded, you would wish to use the pre-processing layer. Since we will probably be utilizing the pre-trained Xception mannequin from Keras and TensorFlow, we might want to apply the identical pre-processing.

def vectorize_image(img_path, measurement):
'''
Vectorize the given picture to get a numpy array
'''
img = picture.load_img(img_path, target_size=measurement)
array = picture.img_to_array(img)
array = np.expand_dims(array, axis=0) # Including dimension to transform array right into a batch of measurement (1,299,299,3)
return array

Now, let’s apply the pre-trained mannequin on our pre-processed picture and see the prediction.

vectorized_image = preprocess_input(vectorize_image(image_path, measurement=IMG_SIZE))
mannequin = model_builder(weights="imagenet")
mannequin.layers[-1].activation = None # Eradicating the final layer as it's the softmax layer used for classification
model_prediction = mannequin.predict(vectorized_image)
print(f"The anticipated class is : {decode_predictions(model_prediction, high=1)[0][0][1]}")

That is the output that we get:

The anticipated class is : tiger

So, our mannequin accurately predicted our inference picture as tiger. Now, let’s perceive the rationale behind the prediction utilizing Grad-CAMs.

We are going to construct a Grad-CAM Warmth-map visualizer to spotlight the influential super-pixels of the mannequin.

def get_heatmap(vectorized_image, mannequin, last_conv_layer, pred_index=None):
'''
Operate to visualise grad-cam heatmaps
'''
gradient_model = tf.keras.fashions.Mannequin(
[model.inputs], [model.get_layer(last_conv_layer).output, model.output]
)
# Gradient Computations
with tf.GradientTape() as tape:
last_conv_layer_output, preds = gradient_model(vectorized_image)
if pred_index is None:
pred_index = tf.argmax(preds[0])
class_channel = preds[:, pred_index]
grads = tape.gradient(class_channel, last_conv_layer_output)
pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))
last_conv_layer_output = last_conv_layer_output[0]
heatmap = last_conv_layer_output @ pooled_grads[..., tf.newaxis]
heatmap = tf.squeeze(heatmap)
heatmap = tf.most(heatmap, 0) / tf.math.reduce_max(heatmap) # Normalize the heatmap
return heatmap.numpy()
plt.matshow(get_heatmap(vectorized_image, mannequin, last_conv_layer))
plt.present()

As we apply the heatmap on the 4th convolution layer of our pre-trained mannequin, that is the output heatmap picture that we get:

However this doesn’t inform us something until we superimpose this picture on our inference picture. So, let’s see how to do that utilizing the next code snippet:

def superimpose_gradcam(img_path, heatmap, output_path="grad_cam_image.jpg", alpha=0.4):
'''
Superimpose Grad-CAM Heatmap on picture
'''
img = picture.load_img(img_path)
img = picture.img_to_array(img)
heatmap = np.uint8(255 * heatmap) # Again scaling to 0-255 from 0 - 1
jet = c_map.get_cmap("jet") # Colorizing heatmap
jet_colors = jet(np.arange(256))[:, :3] # Utilizing RGB values
jet_heatmap = jet_colors[heatmap]
jet_heatmap = picture.array_to_img(jet_heatmap)
jet_heatmap = jet_heatmap.resize((img.form[1], img.form[0]))
jet_heatmap = picture.img_to_array(jet_heatmap)
superimposed_img = jet_heatmap * alpha + img # Superimposing the heatmap on authentic picture
superimposed_img = picture.array_to_img(superimposed_img)
superimposed_img.save(output_path) # Saving the superimposed picture
show(Picture(output_path)) # Displaying Grad-CAM Superimposed Picture
superimpose_gradcam(image_path, get_heatmap(vectorized_image, mannequin, last_conv_layer))

And, voila! We get the next superimposed heatmap picture of the reference picture:

Was that too tough to use Grad-CAMs? Completely not! Keras and TensorFlow makes it even simpler to use such an explainability approach for picture classifiers! This can be a very highly effective approach that’s used to elucidate the working of complicated Deep Studying algorithms on unstructured information like photographs. Though this technique is obscure for newbie learners. Nevertheless, when you get a hold of it, it’s a very highly effective technique and really useful for mannequin explainability.

Hope you could have loved this text! The total tutorial pocket book is on the market at: https://github.com/PacktPublishing/Utilized-Machine-Studying-Explainability-Strategies/blob/major/Chapter02/Layerwisepercent20Propagation.ipynb.

Observe me on Medium and LinkedIn to be taught extra about Explainable AI and Machine Studying.

1. Explainable Machine Studying for Fashions Skilled on Textual content Information: Combining SHAP with Transformer Fashions
2. EUCA — An efficient XAI framework to deliver synthetic intelligence nearer to end-users
3. Perceive the Workings of SHAP and Shapley Values Utilized in Explainable AI
4. Easy methods to Clarify Picture Classifiers Utilizing LIME

1. Keras Tensorflow Tutorial Examples — https://keras.io/examples/
2. Grad-CAM: Visible Explanations from Deep Networks by way of Gradient-based Localization, Ramprasaath et. al — https://arxiv.org/abs/1610.02391.
3. Utilized Machine Studying Explainability Strategies
4. GitHub repo from the ebook Utilized Machine Studying Explainability Strategies — https://github.com/PacktPublishing/Utilized-Machine-Studying-Explainability-Strategies/