Tensorflow/Keras: Model accuracy during training i

Question

I'm beginner with deep learning and keras/tensorflow. I have followed the first tutorial on tensorflow.org: a basic classification with fashion MNIST.

In this case the input data are 60000, 28x28 images and the model is this:

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation=tf.nn.relu),
    keras.layers.Dense(10, activation=tf.nn.softmax)
])

Compiled with:

model.compile(optimizer=tf.train.AdamOptimizer(), 
                loss='sparse_categorical_crossentropy',
                metrics=['accuracy'])

At the end of training the model has this accuracy:

10000/10000 [==============================] - 0s 21us/step
Test accuracy: 0.8769

It's ok. Now I'm trying to duplicate this model with another set of datas. New input is a dataset downloaded from kaggle.

The dataset has images with different sized of dogs and cats, so I have create a simple script that get the images, resize in 28x28 pixel and convert in a numpy array.

This is the code to do this:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
from tensorflow.keras.models import load_model
from PIL import Image

import os

# Helper libraries
import numpy as np

# base path dataset
base_path = './dataset/'
training_path = base_path + "training_set/"
test_path = base_path + "test_set/"

# size rate of images
size = 28, 28

# 
train_images = []
train_labels = []
test_images = []
test_labels = []

classes = ['dogs', 'cats']

# Scorre sulle cartelle contenute nel path e trasforma le immagini in nparray
def from_files_to_nparray(path):
    images = []
    labels = []
    for subfolder in os.listdir(path):
        if subfolder == '.DS_Store':
            continue

        for image_name in os.listdir(path + subfolder):
            if not image_name.endswith('.jpg'):
                continue

            img = Image.open(path + subfolder + "/" + image_name).convert("L").resize(size) # convert to grayscale and resize
            npimage = np.asarray(img)

            images.append(npimage)
            labels.append(classes.index(subfolder))

            img.close()

    # convertt to np arrays
    images = np.asarray(images)
    labels = np.asarray(labels)

    # Normalize to [0, 1]
    images = images / 255.0 
    return (images, labels)

(train_images, train_labels) = from_files_to_nparray(training_path)
(test_images, test_labels) = from_files_to_nparray(test_path)

At the end I have these shapes:

Train images shape   :  (8000, 128, 128)
Labels images shape  :  (8000,)
Test images shape    :  (2000, 128, 128)
Test images shape    :  (2000,)

After training the same model (but with the last dense layer format by 2 neurons) I have this result, that should be ok:

Train images shape   :  (8000, 28, 28)
Labels images shape  :  (8000,)
Test images shape    :  (2000, 28, 28)
Test images shape    :  (2000,)


_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
flatten (Flatten)            (None, 784)               0         
_________________________________________________________________
dense (Dense)                (None, 128)               100480    
_________________________________________________________________
dense_1 (Dense)              (None, 2)                 258       
=================================================================
Total params: 100,738
Trainable params: 100,738
Non-trainable params: 0
_________________________________________________________________
None

Epoch 1/5
2018-07-27 15:25:51.283117: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
8000/8000 [==============================] - 1s 66us/step - loss: 0.6924 - acc: 0.5466
Epoch 2/5
8000/8000 [==============================] - 0s 39us/step - loss: 0.6679 - acc: 0.5822
Epoch 3/5
8000/8000 [==============================] - 0s 41us/step - loss: 0.6593 - acc: 0.6048
Epoch 4/5
8000/8000 [==============================] - 0s 39us/step - loss: 0.6545 - acc: 0.6134
Epoch 5/5
8000/8000 [==============================] - 0s 39us/step - loss: 0.6559 - acc: 0.6039
2000/2000 [==============================] - 0s 33us/step

Test accuracy:  0.592

Now, the question is, if I try to change the input size from 28x28 to, for example 128x128 the result is this:

Train images shape   :  (8000, 128, 128)
Labels images shape  :  (8000,)
Test images shape    :  (2000, 128, 128)
Test images shape    :  (2000,)


_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
flatten (Flatten)            (None, 16384)             0         
_________________________________________________________________
dense (Dense)                (None, 128)               2097280   
_________________________________________________________________
dense_1 (Dense)              (None, 2)                 258       
=================================================================
Total params: 2,097,538
Trainable params: 2,097,538
Non-trainable params: 0
_________________________________________________________________
None

Epoch 1/5
2018-07-27 15:27:41.966860: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
8000/8000 [==============================] - 4s 483us/step - loss: 8.0341 - acc: 0.4993
Epoch 2/5
8000/8000 [==============================] - 3s 362us/step - loss: 8.0590 - acc: 0.5000
Epoch 3/5
8000/8000 [==============================] - 3s 351us/step - loss: 8.0590 - acc: 0.5000
Epoch 4/5
8000/8000 [==============================] - 3s 342us/step - loss: 8.0590 - acc: 0.5000
Epoch 5/5
8000/8000 [==============================] - 3s 342us/step - loss: 8.0590 - acc: 0.5000
2000/2000 [==============================] - 0s 217us/step

Test accuracy:  0.5

Why? Though adding a new dense layer or increasing the neuron numbers the result is the same.

What is the connection between the input size and the model layers? Thanks!

Answer 1

The problem is that you have more parameters to train in the second example. In the first example you just have 100k Parameters. You train them with 8k images.

In the second example you have 2000k Parameters and you try to train them with the same amount of images. This does not work because there is a relation between the free parameters and the number of samples. There is no exact formula to calculate this relation, but there is a rule of thumb that you should have more samples than trainable parameters.

What you can try it to train more epochs and to look how it works but in general you need more data for more complex models.