Vanilla Neural Network Framework

Python Version Build Status

Features:

  1. Forward Propagation
    Forward Pass

     z = input.dot(weight) + bias.T 
         if activation == "sigmoid":
             return self.sigmoid(z)
    
  2. Loss Function
    Backward Pass
     # Cross Entropy
     y_hat_clip = np.clip(y_hat, epsilon, 1 - epsilon)
     result = ((-1.0 / (m)) * np.sum(np.sum(y_train *
                 np.log(y_hat_clip), axis=1), axis=0))
    
     # MSE
     loss = np.square(np.subtract(y, y_pred)).mean()
    
     # Logistic Loss
     loss = - (y_train * np.log(y_hat) + (1 - y_train) * np.log(1-y_hat))
     result = (1.0 / m) * np.sum(loss)
    
  3. Back Propagation
    Backward Pass

     if output_layer:
         dz = out - dz_out 
     else:
         if activation == "sigmoid":
             dz = dz_out.dot(w_out.T) * self.d_sigmoid(out)
         elif activation == "tanh":
             dz = dz_out.dot(w_out.T) * self.d_tanh(out)
         elif activation == "relu":
             dz = dz_out.dot(w_out.T) * self.d_relu(out)
    
  4. Activation Functions
     def sigmoid(self, x):
         """
         [Private Function] Returns sigmoid function of x
         """
         return (1.0/(1.0+np.exp(-x)))
    
     def softmax(self, x):
         """
         [Private Function] Returns softmax function of x
         """
         exps = np.exp(x - np.max(x, axis=1, keepdims=True))
         result = exps / (np.sum(exps, axis=1, keepdims=True))
    
         if (np.any(np.isnan(result))):
             print("Error in Softmax")
             exit()
         return result
    
     def relu(self, x):
         """
         [Private Function] Returns relu function of x
         """
         return np.maximum(0.0, x)
    

    Note : Currently Vanilla only supports Gradient Descent Optimizer

  5. L1 & L2 Regularization
     def l1_reg(self, x, lam):
         """
         [Private Function] Adds L1 regularization to avoid overfitting
         """
         return (lam * np.abs(x))
    
     def l2_reg(self, x, lam):
         """
         [Private Function] Adds L2 regularization to avoid overfitting
         """
         return (lam * np.power(x, 2)) / 2.0
    
  6. Misc
    • Confusion Matrix
        cm = np.zeros(shape=(2, 2))
        for a, p in zip(y, y_pred):
            cm[int(a), int(p)] += 1
        return cm.ravel()
      
    • Classification Scores
        # Sensitivity, hit rate, recall, or true positive rate
        score['sensitivity'] = tp/(tp+fn)
        # Specificity or true negative rate
        score['specificity'] = tn/(tn+fp)
        # Precision or positive predictive value
        score['precision'] = tp/(tp+fp)
        # Negative predictive value
        score['npv'] = tn/(tn+fn)
        # Fall out or false positive rate
        score['fpr'] = fp/(fp+tn)
        # False negative rate
        score['fnr'] = fn/(tp+fn)
        # False discovery rate
        score['fdr'] = fp/(tp+fp)
        # Overall accuracy
        score['accuracy'] = (tp+tn)/(tp+fp+fn+tn)
      
    • Model Plotter & Training Progress Training Animation

Tests

1. Classification:

Dataset Model
Iris input(30) ==> L1(100) ==> L2(150) ==> L3(50) ==> out(2)
Breast Cancer input(4) ==> L1(100) ==> L2(60) ==> L3(60) ==> out(3)
Wine input(13) ==> L1(200) ==> L2(120) ==> L3(60) ==> out(3)
Parameter Iris Breast Cancer Wine
Learning Rate 0.01 0.01 0.01
Loss Function Cross Entropy Cross Entropy Cross Entropy
# of Epochs 50 100 50
Activation Sigmoid Sigmoid ReLU
Set Iris Breast Cancer Wine
Training Accuracy 100 90.93 85.25
Validation Accuracy 94.44 90.11 88.50
Test Accuracy 95.55 93.86 87.04

Iris Loss Plot
Iris Loss Plot
Cancer Loss Plot
Cancer Loss Plot
Wine Loss Plot
Wine Loss Plot

2. Regression:

Boston House Prices: input(13) ==> L1(100) ==> L2(60) ==> L3(60) ==> out(1)

Parameter Boston
Learning Rate 0.01
Loss Function MSE
# of Epochs 50
Activation Sigmoid + ReLU
Set Iris
Training MSE 65.66
Validation MSE 49.55
Test MSE 45.73

Iris Loss Plot

Actual Labels Predicted Labels
25.11 23.6
28.88 32.4
17.23 13.6
25.76 22.8
17.63 16.1
22.20 20.
24.46 17.8
20.83 14.
18.01 19.6
22.89 16.8