ETC3250/5250 Introduction to Machine Learning

Remember the logistic function:

\[\begin{align} f(x) &= \frac{e^{\beta_0+\sum_{j=1}^p\beta_jx_j}}{1+e^{\beta_0+\sum_{j=1}^p\beta_jx_j}}\\ &= \frac{1}{1+e^{-(\beta_0+\sum_{j=1}^p\beta_jx_j)}} \end{align}\]

Also,

\[\log_e\frac{f(x)}{1 - f(x)} = \beta_0+\sum_{j=1}^p\beta_jx_j\]

\[\widehat{y} =b_0+\sum_{j=1}^pb_jx_j\]

Drawing as a network model:

\(p\) inputs (predictors), multiplied by weights (coefficients), summed, add a constant, predicts output (response).

\[\begin{align} \widehat{y} =a_{0}+\sum_{k=1}^s(a_{k}(b_{0k}+\sum_{j=1}^pb_{jk}x_j)) \end{align}\]

The architecture allows for combining multiple linear models to generate non-linear classifications.

The best fit uses \(s=4\), four nodes in the hidden layer. Can you sketch four lines that would split this data well?

The models at each of the nodes of the hidden layer.

1. Define architecture

   • flatten: if you are classifying images, you need to flatten the image into a single row of data, eg 24x24 pixel image would be converted to a row of 576 values. Each pixel is a variable.
   • How many hidden layers do you need?
   • How many nodes in the hidden layer?
   • Dropout rate: proportion of nodes removed randomly at each update, for regularisation, to reduce number of parameters to be estimated

4. Training process:

   • epochs: number of times the algorithm sees the entire data set
   • batch_size: subset used at each fit
   • validation_split: proportion for hold-out set for computing error rate
   • batch_normalization: each batch is standardised during the fitting, can be helpful even if full data is standardised

• 4D data
• Simple cluster structure to classes
• How many nodes in the hidden layer?

library(keras)
tensorflow::set_random_seed(211)

# Define model
p_nn_model <- keras_model_sequential()
p_nn_model %>% 
  layer_dense(units = 2, activation = 'relu', 
              input_shape = 4) %>% 
  layer_dense(units = 3, activation = 'softmax')
p_nn_model %>% summary

loss_fn <- loss_sparse_categorical_crossentropy(
  from_logits = TRUE)

p_nn_model %>% compile(
  optimizer = "adam",
  loss      = loss_fn,
  metrics   = c('accuracy')
)

Note that the tidymodels code style does not allow easy extraction of model coefficients.

Split the data into training and test, and check it.

Fit the model

# Data needs to be matrix, and response needs to be numeric
p_train_x <- p_train %>%
  select(bl:bm) %>%
  as.matrix()
p_train_y <- p_train %>% pull(species) %>% as.numeric() 
p_train_y <- p_train_y-1 # Needs to be 0, 1, 2
p_test_x <- p_test %>%
  select(bl:bm) %>%
  as.matrix()
p_test_y <- p_test %>% pull(species) %>% as.numeric() 
p_test_y <- p_test_y-1 # Needs to be 0, 1, 2

# Fit model
p_nn_fit <- p_nn_model %>% 
  keras::fit(
    x = p_train_x, 
    y = p_train_y,
    epochs = 200,
    verbose = 0
  )

How many parameters need to be estimated?

Model: "sequential"
____________________________________________________________
 Layer (type)              Output Shape            Param #  
============================================================
 dense_1 (Dense)           (None, 2)               10       
 dense (Dense)             (None, 3)               9        
============================================================
Total params: 19 (76.00 Byte)
Trainable params: 19 (76.00 Byte)
Non-trainable params: 0 (0.00 Byte)
____________________________________________________________

Evaluate the fit

p_nn_model %>% 
  evaluate(p_test_x, p_test_y, verbose = 0)

    loss accuracy 
    0.28     0.94 

Confusion matrices for training and test

           p_train_pred_cat
            Adelie Chinstrap Gentoo
  Adelie        95         5      0
  Chinstrap      0        45      0
  Gentoo         1         0     81

           p_test_pred_cat
            Adelie Chinstrap Gentoo
  Adelie        46         3      2
  Chinstrap      0        23      0
  Gentoo         2         0     39

Note: Specifically have chosen settings so fit is not perfect

Check the fit at the hidden layer nodes