What is Tensorflow?

Tensorflow is one of the most popular tools for developing neural networks. Tensorflow is not a library for making neural networks per se. Rather, Tensorflow is for efficiently handling the mathematic operations that neural networks rely on. ANNs utilize a lot of linear algebra, matrix operations and node-edge graph manipulations. Tensorflow provides a set of functionality to accomplish these operations efficiently with C++ code and allows the developer to use these functions with a python interface.

What is Keras?

As discussed, Tensorflow itself does not directly provide neural network functionality. Instead, the programmer who wants to develop an ANN will need another library. Keras provides programmers a way to declare what kind of ANN they want and then Keras accomplishes the interfacing with Tensorflow to generate the network. Tensorflow focusses on accomplishing the mathematics efficiently, while Keras provides programmers an easy way to develop their models. Although Keras and Tensorflow are very often used together, they are actually independent. Keras can be used to interface with an alternative to Tensorflow, like Theano. Similarly, developers can use an alternatives to Keras, like TFLearn and still use Tensorflow in the backend.

The predictive features (X)

Our X value is a matrix of floats where each row is a flower and each column is a measurement.

pd.DataFrame(X,columns = feature_names).head(5)

The dependant (y)

For each row in X, there is a corresponding item in the vector y, which represents the species of flower. There are three unique species in our dataset as shown in the table below. However, our dependant y must come in the form of a One Hot Encoded (OHE) matrix, not a vector. In this format, there are three columns with a zero or one value to indicate which species of flower is this subject.

pd.DataFrame(y[:5],columns = dataset.target_names)

Let us inspect this same data when converted to an array, this will be used to ‘flatten’ the matrix form of y to a vector. Each row is a ‘setosa’ (0).

y[:5].argmax(axis=1)

array([0, 0, 0, 0, 0], dtype=int64)

Terminology

• Batch size: This parameter instructs the ANN to trial that number of cases with a set of random weights before trialing another batch with a new set of weights. A large batch size enhances computational efficiency, at the potential cost of accuracy.

• Epochs: The number of times the ANN shall repeatedly use the training data to find the optimum set of weights. A large number of epochs may increase accuracy, but takes longer to compute.

• Optimizer: In the fitting process, random weights are trialed for every neuron and the best combination is selected. However, trying every possible random combination is very inefficient. Instead, optimizers apply algorithms that aim to discover which combinations of parameters for each node is best. Instead of randomly selecting weights, the optimizer applies a guided process to discovering the best settings. These algorithms for determining the optimum weights are called ‘gradient descent’ algorithms.

• Learning rate: The incremental value by which the optimizer varies weights. A larger learning rate may be quicker to compute, but less accurate.

• Loss Function: During the fitting process, the loss function is how we calculate if a given trial was better or worse that others. Here we use ‘Binary Cross Entropy’ which is a common selection for classification problems. Mean-squared error would be an appropriate choice for regression problems.

• Metrics: Machine learning practitioners know there are many ways to evaluate a model’s performance. We can record multiple metrics to evaluate performance. For example, we can examine the model’s accuracy (the portion of correct predictions) and the recall (the portion of correctly classified cases of a given class among all cases of that class).

def get_simple_ann(
        input_shape: int|tuple,
        output_shape:int|tuple
        ) -> keras.models.Sequential:
    """
    This function returns an instance of a 
    Keras Sequential model which is ready to 
    fit with training data.
    
    Args:
    input_shape: int or tuple. Describes the 
    dimensions of a single case of input.
    output_shape: int or tuple. The desired 
    shape of output. 
    
    Returns:
    A compiled instance of a keras Sequential model
    """

    ### Step 1: Instantiate an empty Sequential model object
    ann = keras.models.Sequential()
    
    ### Step 2: Adding the input layer with same number 
    # of nodes as we have input features
    ann.add(
        keras.layers.Dense(
            units = 16,
            activation='relu', 
            input_dim=input_shape
            )
        )
    
    # Step 3: Adding 10% dropout to prevent overfitting
    ann.add(keras.layers.Dropout(rate=0.1))
    
    ### Step 4: Adding another Dense-Dropout combo to 
    # make a second layer in the network
    ann.add(
            keras.layers.Dense(
            units = 16, 
            activation='relu'
            )
        )
    ann.add(keras.layers.Dropout(rate=0.1))
    
    ### Step 5: Adding the output layer. This must match the 
    # format of our y variable
    ann.add(
            keras.layers.Dense(
            units = output_shape, 
            activation='softmax'
            )
        )
    
    ### Step 6: Once we have defined the architecture of 
    # our ANN, we call compile() to convert the model to tensors.
    ann.compile(
        optimizer='adam', 
        loss='binary_crossentropy', 
        metrics=['accuracy']
        )
    return ann

### Calling our function
simple_ann = get_simple_ann(
    input_shape = input_shape,
    output_shape = n_classes
    )

### Fitting to training data
simple_ann.fit(
    X_train,
    y_train,
    batch_size=100, 
    epochs=150,
    verbose=False
    )

### Making predictions
# argmax() here is used to 'flatten' the matrix form of the 
# dependant variable (as seen above) into a vector of integers 
# which correspond to classes. 
predictions = simple_ann.predict(X_test).argmax(axis=1) 

### Accuracy score
accuracy = accuracy_score(y_test.argmax(axis=1),predictions)
print("Accuracy of this model is: "+"{:1.2f}".format(accuracy*100)+"%")

2/2 [==============================] - 0s 0s/step
Accuracy of this model is: 70.00%

Visualizing the ANN

From the code above, we are constructing an ANN like the one in this diagram. We have four features in our data, and so we have four input nodes to receive each value. If you know the theory of ANNs, you know that each node will apply a weight and an activation function to the input to determine the number it will output to every node in the next layer. However, we use a dropout layer, which will select some nodes to nullify. We have two such layers. Finally, we have three output nodes, representing the probability that the given case is of a certain species. Therefore, for each case we predict, we output three numbers. We conclude that whichever of those three numbers is largest corresponds to the most likely species for the given case.

Evaluating the model

As we can see from the output above, this model was 70% accurate, meaning it classified 70% of cases as the correct species.

Why we need hypertuning

Is 70% accuracy the best we can do? What if we had more than 12 nodes in each layer? Or fewer? What if we added more layers? Or increased the dropout rate? Hypertuning is how we experimentally verify which parameters are best.

To accomplish this, we could try repeatedly fitting the model with every possible combination of parameters (this is called ‘Grid Search’). However, for many parameters, this would require a lot of fittings. Instead, the ‘keras tuner’ library was created to efficiently trial different hyperparameter selections.

Terminology

• Objective: One of the metrics. The ‘best’ model is the one with the highest value.
• Hypergrid: Is a dictionary-like structure that specifies what values for each parameter to trial.
• Callbacks: In this context, the callback is a special type of function called at the end of each epoch. In this example, we use ‘EarlyStopping,’ which is helpful for reducing computation time. With early stopping, once each additional epoch only improves the objective slightly, we conclude we have reached a point of diminishing return and finish the fitting process without completing every epoch.
• Width: The number of units is a particular layer.
• Depth: The number of layers in an ANN.
• Validation split: By default, the tuner selects the model with the best value for the objective as calculated on training data. However, we can optionally specify a validation split size, whereby a random sample of training data will be withheld, and we select the best model as validated against that holdout data. This can be helpful in avoiding overfitting to training data.
• Activation function: a function which each node in a layer uses to convert the inputted values into the outputted values. A sigmoid function would be appropriate for the activation function of an output node doing classification, while no function would be appropriate for a regression. For a discussion on the different types of common functions, see this article.

class ANNHyperModel(kt.HyperModel):
    """
    Given a set of hyperparameters to explore, 
    uses keras-tuner to discover the best settings 
    and returns the optimum model.
    """
    
    _default_hypergrid = {
            'width':{
                'min':12,
                'max':12,
                'step':1
                },
            'depth':{
                'min':1,
                'max':1,
                'step':1
                },
            'droprate':{
                'min':.8,
                'max':.8,
                'step':.1
                },
            'activation':{
                'values':['relu']
            },
            'learning_rate':{
                'values':[1e-2]
            }
        }
    """If the user does not input values for the 
    following hyperparameters, we use these values by default."""
    
    def __init__(
                self,
                input_shape:tuple|int,
                n_outputs:int,
                hypergrid:dict = None,
                output_activation:str = 'sigmoid',
                epochs:int = 1,
                batch_size:int = 100,
                optimizer_class = keras.optimizers.Adam,
                loss: Loss = SparseCategoricalCrossentropy(
                                        from_logits=True
                                        ),
                metrics: list[str] = ['accuracy'],
                objective:str = "accuracy",
                directory:str = None,
                project_name:str = "kt_hyperband",
                factor:int = 3,
                verbose:int = 0,
                callbacks:list = []
    ):
        """
        Args:
        - input_shape (tuple | int): Dimensions of a single input case. 
        Provide an int of n if one case is a 1-d array of length n.

        - n_outputs (int): Number of output nodes. One for regression or 
        binary-classification. Multiple for 
        multi-regression/multi-classification.

        - hypergrid (dict, optional): Dictionary describing hyperparams 
        to trial, see default '_default_hypergrid' for example. 
        Defaults to None.

        - output_activation (str, optional): Output layer activation 
        function. Defaults to 'sigmoid'.

        - epochs (int, optional): Training epochs. Defaults to 1.

        - batch_size (int, optional): training batch size. 
        Defaults to 100.

        - optimizer_class (_type_, optional): Optimizer class. 
        Input the class type, not an instantiation. Defaults to 
        keras.optimizers.Adam.

        - loss (tf.keras.losses.Loss, optional): training loss function. 
        Defaults to 
        keras.losses.SparseCategoricalCrossentropy(from_logits=True).
        
        - metrics (list[str], optional): Training metrics. Defaults 
        to ['accuracy'].
        
        - objective (str, optional): Must be included in 'metrics.' 
        Tuning process will select the model with the best performance
        by this metric. Defaults to "accuracy".
        
        - directory (str, optional): When used, tuning process creates 
        a directory to hold data on disk. Deleted afterwards. 
        Be careful not to use existing dir name. Defaults to "None".
        
        - project_name (str, optional): File naming stem in directory. 
        Defaults to "kt_hyperband".

        -factor (int, optional): Reduction factor for epochs at each 
        tuning trial. Defaults to 3.

        -verbose (int, optional): Set to true to print details of 
        fitting process. Defaults to 0.
        
        - callbacks (list, optional): A keras.callback function called 
        after each epoch. Defaults to [].
        """
        self.input_shape = input_shape
        self.n_outputs = n_outputs
        self.hp = kt.HyperParameters()
        self.hypergrid = hypergrid
        self.output_activation = output_activation
        self.epochs = epochs
        self.optimizer_class = optimizer_class
        self.loss = loss
        self.metrics = metrics
        self.objective = objective
        self.callbacks = callbacks
        self.directory = directory
        self.project_name = project_name
        self.factor = factor
        self.verbose = verbose
        self.batch_size = batch_size
        self.history = None # null before training

    def get_hypergrid(self):
        """Returns the user-specified hypergrid, plus the default items 
        (from _default_hypergrid) where the user did not specify 
        values"""
        output = copy.deepcopy(ANNHyperModel._default_hypergrid)
        if self.hypergrid is None:
             self.hypergrid = {}
        output.update(self.hypergrid)
        return output

    ### If you want to try more complex types of ANN, you should extend 
    # this class and overwrite this function with your own logic. 
    # For example, instead of adding Dense layers, you could add LSTM 
    # layers instead to create an RNN. 
    def build(self,hp:kt.HyperParameters):
        """Using an instance of kt.HyperParameters, we build and 
        compile a Keras Sequential instance with 
        some parameters filled by HyperParameter objects instead of 
        literal values. The Keras Hyperband object can then execute 
        a search function using this hypermodel to discover the 
        best model setting."""

        ### here we take the hypergrid and use it to construct a set of 
        # kt.HyperParameter objects
        hypergrid = self.get_hypergrid()
        width_hyperparam = hp.Int(
              'units', 
              min_value=hypergrid['width']['min'], 
              max_value=hypergrid['width']['max'], 
              step=hypergrid['width']['step']
              )
        depth_hyperparam = hp.Int(
              'layers', 
              min_value=hypergrid['depth']['min'], 
              max_value=hypergrid['depth']['max'], 
              step=hypergrid['depth']['step']
              )
        droprate_hyperparam = hp.Float(
              'droprate', 
              min_value=hypergrid['droprate']['min'], 
              max_value=hypergrid['droprate']['max'], 
              step=hypergrid['droprate']['step']
              )
        activation_hyperparam = hp.Choice(
              'activation', 
              values=hypergrid['activation']['values']
              )
        learning_rate_hyperparam = hp.Choice(
              'learning_rate', 
              values=hypergrid['learning_rate']['values']
              )

        model = keras.Sequential()
        model.add(
              keras.layers.Input(
              shape=self.input_shape
              )
            )
        
        ### Add a Dense-Dropout combo for each layer in the depth param
        for layer_number in range(0,depth_hyperparam):
              model.add(
                    keras.layers.Dense(
                        units=width_hyperparam,
                        activation=activation_hyperparam,
                        name = 'Dense_'+str(layer_number)
                        )
                    )
              model.add(
                   keras.layers.Dropout(droprate_hyperparam)
              )
        
        model.add(
                keras.layers.Dense(
                self.n_outputs,activation=self.output_activation
                )
            ) 

        model.compile(
            optimizer=self.optimizer_class(
                learning_rate=learning_rate_hyperparam
                ),
            loss=self.loss,
            metrics=self.metrics
            )
        ### This function must return an instance of a keras 
        # model that includes kt.Hyperparamater objects
        return model 
    
    def _hypertune(self,X_train, y_train, validation_size = 0):
        """This function executes the hypertuning of the
          built hypermodel"""
        self.build(self.hp)
        
        self.tuner = kt.Hyperband(
            ### We can input an instance of the custom class here (self) 
            # and the 'build' function will be called by keras 
            # Hyperband. 
                self, 
                objective= self.objective,
                max_epochs= self.epochs,
                factor=self.factor,
                hyperband_iterations=10,
                directory=self.directory,
                project_name=self.project_name
                )
        ### use validation split if validation_size > 0
        validation_sets = None
        if validation_size > 0:
            X_train, X_val, y_train, y_val = train_test_split(
                    X_train, 
                    y_train, 
                    test_size=validation_size
                    )
            validation_sets = (X_val,y_val)

        self.tuner.search(
               X_train,
               y_train, 
               epochs=self.epochs, 
               callbacks=self.callbacks, 
               verbose=self.verbose,
               batch_size=100,
               use_multiprocessing=True,
               validation_data=validation_sets
               )
        self.best_hyperparams=self.tuner.get_best_hyperparameters()[0]

        best_model = self.tuner.hypermodel.build(self.best_hyperparams)

        return best_model
    
    def execute(self,X,y,validation_size = 0):
        """
        Fit the model on training data.

        Args:
            - X (_type_): predictors, ndarray
            - y (_type_): dependant, ndarray or 1-d array
            - validation_size (int, optional): If > 0, tuner will 
            record validation performance of the objective. Set 
            objective to 'val_<metric name>' to select best model 
            on validation data. Defaults to 0.

        Raises:
            - e: If errors occur during fitting, a 'final' clause is 
            invoked to ensure the tuning directory is still deleted. 
        """
        ### Use finally clause to ensure the tuning dir is deleted even 
        # if there is an error.This failsafe is used because 'leftover' 
        # tuning directories from previous fittings can cause errors.
        try:
            self.model = self._hypertune(X,y,validation_size)
            self.history = self.model.fit(
                X,
                y, 
                batch_size=self.batch_size, 
                epochs=self.epochs,
                callbacks=self.callbacks, 
                verbose = self.verbose)
        except Exception as e:
            raise e
        
        finally:
            try:
                  shutil.rmtree(self.directory)
            except FileNotFoundError as e:
                 pass

Making predictions

Now that we have our fitted model, let us evaluate its predictive power. In this context, we are going to use accuracy as our evaluation metric. Accuracy is the portion of classifications which were correct. Typically, this metric is suitable when there are roughly equal numbers of classes in the data and we consider any kind of misclassification as equally undesirable.

y_pred = model.model.predict(X_test).argmax(axis=1)

accuracy = accuracy_score(y_test.argmax(axis=1),y_pred)
print("Accuracy of this model is: "+"{:1.2f}".format(accuracy*100)+"%")

2/2 [==============================] - 0s 0s/step
Accuracy of this model is: 98.00%

Success!! The best model discovered by our hypertuner was 98% accurate, a big improvement over 70% in our basic model.

Fitting history

The history parameter helps us monitor how effectively the fitting process converged on the optimum accuracy. We can construct a timeline which shows us how the accuracy metric improved at each epoch as the optimizer progressively improved the weightings. Note that once a point of diminishing return is reached, the EarlyStopping callback function will conclude the fitting process early to save computation time, which is why we cannot see the requested 150 epochs.

Practitioners should expect to see a steady increase in performance over epochs. A history showing no improvement over epochs would indicate a problem with the model.

pd.Series(
    model.history.history['accuracy']
    ).plot(
        xlabel='Epochs',
        ylabel="Accuracy"
        )

<Axes: xlabel='Epochs', ylabel='Accuracy'>

What were the best hyperparameters?

The best_hyperparams property will tell us which of the parameters we trialed was best. Note that some of the ‘tuner’ parameters are internal considerations for the tuner and not important to us for now.

model.best_hyperparams.values

{'units': 52,
 'layers': 2,
 'droprate': 0.1,
 'activation': 'relu',
 'learning_rate': 0.01,
 'tuner/epochs': 2,
 'tuner/initial_epoch': 0,
 'tuner/bracket': 4,
 'tuner/round': 0}

What does our model look like?

We can use the summary() function to print out a basic summary of our model’s architecture.

model.model.summary()

Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 Dense_0 (Dense)             (None, 52)                260       
                                                                 
 dropout_1 (Dropout)         (None, 52)                0         
                                                                 
 Dense_1 (Dense)             (None, 52)                2756      
                                                                 
 dropout_2 (Dropout)         (None, 52)                0         
                                                                 
 dense_1 (Dense)             (None, 3)                 159       
                                                                 
=================================================================
Total params: 3,175
Trainable params: 3,175
Non-trainable params: 0
_________________________________________________________________

Evaluating Performance

We know the model was 98% accurate, but let us examine more deeply how well it classified each species. We shall use a confusion matrix.

matplotlib.rc('figure', figsize=(5,5))
ConfusionMatrixDisplay(
    confusion_matrix(
        y_test.argmax(axis=1), 
        y_pred)
        ).plot(
            colorbar=False
            )
plt.title('Confusion Matrix')

Text(0.5, 1.0, 'Confusion Matrix')

Interpreting the model

Most consumers of machine learning models want to know why a model gives the output it does. This can be a difficult task for a complex neural network, with many interactions nested in the algorithm. In a linear model, each feature has a positive or negative impact on the outcome. In an interactive model, the direction and size of effect is conditional and not so simple to describe. A common method for explaining why models output as they do is ‘permutational feature importance’ in which we calculate the loss of accuracy in a model when it does not have a given feature available. However, this is a lot of repeated computation when you have many features and ANNs can be slow for this.

Another approach is SHapley Additive exPlanations (SHAP), which uses an algorithm based on game-theory as well as many computational optimizations, to efficiently calculate ‘SHAP values’ for an ANN. Here we shall demonstrate SHAP. A larger SHAP value indicates the feature is more influential.

deep_explain = shap.DeepExplainer(model.model, X_test)
deep_shap_values= deep_explain.shap_values(X_test[1:5])

matplotlib.rc('figure', figsize=(10,10))
pd.Series(
    deep_shap_values[0][0],
    index=feature_names
    ).sort_values(
    ).plot.barh(
        title='SHAP Values'
        )

<Axes: title={'center': 'SHAP Values'}>

It appears that petal length and width are the most important factors when trying to classify a flower. Sepal width is also important, but sepal length is relatively inconsequential.