Keras Functional model giving high validation accu

I am trying to do transfer learning for VGG16 architecture with 'ImageNet' pretrained weights on PASCAL VOC 2012 dataset. PASCAL VOC is a multi label image dataset with 20 classes, and so I have modified the inbuilt VGG16 model like this:

def VGG16_modified():
    base_model = vgg16.VGG16(include_top=True,weights='imagenet',input_shape=(224,224,3))
    print(base_model.summary())
    x = base_model.get_layer('block5_pool').output
    x = (GlobalAveragePooling2D())(x)
    predictions = Dense(20,activation='sigmoid')(x)

    final_model = Model(input = base_model.input, output = predictions)
    print(final_model.summary())
    return final_model

and my input image preprocessing is like this:

img_val = []
for i in tqdm(range(dfval.shape[0])):
        img = image.load_img(train_images+y_val[0][i],target_size=(224,224))
        img = image.img_to_array(img)
        img_val.append(img)
x_val = np.array(img_val

I have converted the categorical labels like this with pd.get_dummies for 20 classes [[0 0 0 0 1 0 0 0 0 1 0 .... ]] and The corresponding labels are of the shape (number of image samples, 20). The input images are of shape (number of image samples, 224,224, 3)

When I trained the model for several epochs, I see very good validation accuracy (around 90%) but when I used the same validation data set to predict the images, it is giving the same class output for every image.

I trained the model like this:

model = VGG16_modified()
model.summary()
model.compile(optimizer=Adam(),loss='binary_crossentropy',metrics = ['accuracy'])
model.fit(x_train, y_train, epochs=100, validation_data=(x_val, yval), batch_size=4)
model.save('CAMVGG16trainall.h5')
model.save_weights('CAMVGG16weightstrainall.h5')

Later I loaded the model and tried to predict the labels for the same validation data set.

model = load_model(model)
preds = model.predict(image)

But I am getting same output for every image. The output is of shape [[0 0 0 ......1 0 0 0...]] I tried with more number of epochs,less number of epochs, by setting a few layers non trainable, by setting all layers trainable,changing the learning rate, using different optimizer (SGD), not using Imagenet weights and training from scratch but none of them are giving me the correct results. Can anyone tell me where have I gone wrong.

Mentioning the Resolution here for the benefit of the community, as there are many comments to know the solution.

Issue here was that the Model was Freezed, i.e., the Layers were not Trained on the PASCAL VOC Dataset.

Weights of the Pre-Trained Model should be Freezed and the Weights of the Layers of the Model Trained on our Dataset shouldn't be.

Issue is resolved by setting, layer.trainable = True. This can be better understood by the screenshot below.