Calculate/Visualize Tensorflow Keras Dense model l

Here is my tensorflow keras model,(you can ignore dropout layer if it makes things tough)

import tensorflow as tf
optimizers = tf.keras.optimizers
Sequential = tf.keras.models.Sequential
Dense = tf.keras.layers.Dense
Dropout = tf.keras.layers.Dropout
to_categorical = tf.keras.utils.to_categorical

model = Sequential()
model.add(Dense(256, input_shape=(20,), activation="relu"))
model.add(Dropout(0.1))
model.add(Dense(256, activation="relu"))
model.add(Dropout(0.1))
model.add(Dense(256, activation="relu"))
model.add(Dropout(0.1))
model.add(Dense(3, activation="softmax"))

adam = optimizers.Adam(lr=1e-3) # I don't mind rmsprop either
model.compile(optimizer=adam, loss='categorical_crossentropy', metrics=['accuracy'])  
model.summary()

and I have saved the model structure and weights as

model.save("sim_score.h5", overwrite=True)
model.save_weights('sim_score_weights.h5', overwrite=True)

on doing model.predict(X_test) I get [0.23, 0.63, 0.14] which is the prediction probability of the 3 output classes.

How would I visualize how much weight/importance each of my initial 20 features have in this model w.r.t the 3 output softmax?

For example, my 2nd column could have negligible impact on the end result, while the 5th column could have 3x more impact on the output prediction than the 20th column. It does not matter what the absolute effect of the 5th column is, just figuring out the relative importance is good enough, for example 5th column = 0.3, 20th column = 0.1 and so on for a 20 x 3 matrix.

See this animation for intuition or Tensorflow playground. The visualization doesn't have to show how the weights change during training, but can just show a snapshot image of how it looks at the end of training.

In fact, the solution does not even have to be a visualization, it can even be an array of 20 elements x 3 outputs having the relative importance of each feature w.r.t the 3 output softmax and importance relative to the other features.

Getting the importance of intermediate layers is just an added bonus.

The reason I want to visualize the 20 features is for transparency purposes(currently the model feels like a black box). I am comfortable in matplotlib, pyplot, seaborn. I am aware of Tensorboard as well, but couldn't find any examples for simple Dense Relu network with Softmax.

I feel a time-consuming way to get the 20 x 3 weights would be to do a domain search of 20 features from 0 - 1 with delta of 0.5 by sending various inputs and trying to deduce the importance of features based off of that(which would have 3 to the power of 20 ~= 3.4 billion possible sample space and gets exponentially worse the more features I add) and then applying conditional probability to reverse engineer the relative weights, but I am not sure if there is a simpler/optimized way via TensorBoard or some custom logic.

Can someone help visualize the model/calculate the 20 x 3 = 60 relative weights of the 20 features w.r.t the 3 outputs with code snippets or provide a reference on how to achieve this?

I had come across a similar problem, but i was concerned more about the visualization of the model parameters(weights and biases) rather than the model features [since i wanted to explore and view the black box as well].

For example, the following is a snippet of a shallow neural network with 2 hidden layers.

model = Sequential()
model.add(Dense(128, input_dim=13, kernel_initializer='uniform', activation='relu'))
model.add(Dropout(0.1))
model.add(Dense(64, kernel_initializer='uniform', activation='relu'))
model.add(Dropout(0.1))
model.add(Dense(64, kernel_initializer='uniform', activation='relu'))
model.add(Dropout(0.1))
model.add(Dense(8, kernel_initializer='uniform', activation='softmax'))

# Compile model
model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# Using TensorBoard to visualise the Model
ks=TensorBoard(log_dir="/your_full_path/logs/{}".format(time()), histogram_freq=1, write_graph=True, write_grads=True, batch_size=10)

# Fit the model   
model.fit(X, Y, epochs = 64, shuffle = True, batch_size=10, verbose = 2, validation_split=0.2, callbacks=[ks])

For one to be able to visualize the parameters, there are few important things to be kept in mind:

Always ensure to have a validation_split in the model.fit() function[else histograms cannot be visualised].
Make sure the value of histogram_freq > 0 always!![histograms won't be computed otherwise].
The callbacks to TensorBoard have to be specified in the model.fit() as a list.

Once, this is done; goto cmd and type the folllowing command:

tensorboard --logdir=logs/

This gives you a local address with which you can access TensorBoard on your web browser. All histograms, distributions, loss and accuracy functions will be available as plots and can be selected from the menu bar at the top.

Hope this answer gives a hint about the procedure to visualize model parameters(I myself had a bit of a struggle as the above points weren't available together).

Do let me know if it helped.

Below is the keras documentation link for your reference:

https://keras.io/callbacks/#tensorboard