ETC3250/5250: Introduction to Machine Learning

class: middle center hide-slide-number monash-bg-gray80

.info-box.w-50.bg-white[
These slides are viewed best by Chrome or Firefox and occasionally need to be refreshed if elements did not load properly. See <a href=lecture-09b.pdf>here for the PDF <i class="fas fa-file-pdf"></i></a>. 
]

<br>

.white[Press the **right arrow** to progress to the next slide!]

---

class: title-slide
count: false
background-image: url("images/bg-02.png")

# .monash-blue[ETC3250/5250: Introduction to Machine Learning]

<br>

<h2 style="font-weight:900!important;">k-means clustering</h2>

.bottom_abs.width100[

Lecturer: *Professor Di Cook*

Department of Econometrics and Business Statistics

<i class="fas fa-envelope"></i>  ETC3250.Clayton-x@monash.edu

<i class="fas fa-calendar-alt"></i> Week 9b

<br>

]

---
## Cluster analysis

<br>

- The aim of cluster analysis is to group cases (objects) according to their similarity on the variables. It is also often called unsupervised classification, meaning that classification is the ultimate goal, but the classes (groups) are not known ahead of time. 
- Hence the first task in cluster analysis is to construct the class information. To determine closeness we start with measuring the interpoint distances.

---

## Cluster this!

---

## k-means clustering - algorithm

This is an iterative procedure. To use it the number of clusters, `$k$`, must be decided first.  The stages of the iteration are:

- Initialize by either (a) partitioning the data into k groups, and compute the `$k$` group means or (b) an initial set of `$k$` points as the first estimate of the cluster means (seed points).
- Loop over all observations reassigning them to the group with the closest mean.
- Recompute group means.
- Iterate steps 2 and 3 until convergence.

[Thean C. Lim's blog post](https://theanlim.rbind.io/post/clustering-k-means-k-means-and-gganimate/)

---
class: split-50
Some data 🤸

.column[.pad50px[

---
class: split-50

.column[.pad10px[.content[

Select `$k=2$`, and set initial seed means <br>
`$\bar{x}_1=$` (10, 11) ,
`$\bar{x}_2=$` (11, 9) <br>
]]]
.column[.pad50px[.content[

]]]

---
class: split-50

.column[.pad10px[.content[

Compute distances `$(d_1, d_2)$` between each observation and each mean.

]]]
.column[.pad50px[.content[
<table class="table table-striped" style="font-size: 24px; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;color: white !important;background-color: #3F9F7A !important;"> lbl </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> x1 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> x2 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> d1 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> d2 </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> a </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 16 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 4 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 9.2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 7.1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> b </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 8 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 9.5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> c </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 14 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 4 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.1 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 5.8 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> d </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 9 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 9.2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.0 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> e </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 10 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 21 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 10.0 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 12.0 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> f </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 10.8 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> g </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 1 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 20 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 12.7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 14.9 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> h </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 15 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.9 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 10.8 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> i </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 3 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.5 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> j </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 3 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.1 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> k </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 9.8 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.6 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> l </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 7.2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 6.4 </td>
  </tr>
</tbody>
</table>
]]]

---
class: split-50

.column[.pad10px[.content[

<br><br><br><br>
Assign the cluster membership

]]]
.column[.pad50px[.content[

<table class="table table-striped" style="font-size: 24px; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;color: white !important;background-color: #3F9F7A !important;"> lbl </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> x1 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> x2 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> d1 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> d2 </th>
   <th style="text-align:left;color: white !important;background-color: #3F9F7A !important;"> cl </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> a </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 16 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 4 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 9.2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 7.1 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> b </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 8 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 9.5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.1 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> c </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 14 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 4 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.1 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 5.8 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> d </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 9 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 9.2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.0 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> e </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 10 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 21 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 10.0 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 12.0 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> f </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 10.8 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> g </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 1 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 20 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 12.7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 14.9 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> h </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 15 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.9 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 10.8 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> i </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 3 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.5 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> j </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 3 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.1 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.2 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> k </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 9.8 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.6 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> l </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 7.2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 6.4 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
</tbody>
</table>
]]]

---
class: split-50 
layout: false

.column[.pad10px[.content[

Assign the cluster membership

<img src="images/lecture-09b/unnamed-chunk-10-1.png" width="100%" style="display: block; margin: auto;" />
]]]

---
class: split-50 
layout: false

.column[.pad10px[.content[

Recompute means, and re-assign the cluster membership

`$\bar{x}_1=$` (5, 16) ,
`$\bar{x}_2=$` (12, 5) <br>

<table class="table table-striped" style="font-size: 24px; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;color: white !important;background-color: #3F9F7A !important;"> lbl </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> x1 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> x2 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> d1 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> d2 </th>
   <th style="text-align:left;color: white !important;background-color: #3F9F7A !important;"> cl </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> a </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 16 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 4 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 16.8 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 4.4 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> b </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 8 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 16.7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 7.6 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> c </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 14 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 4 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 15.6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2.6 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> d </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 9 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 16.2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.0 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> e </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 10 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 21 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 7.1 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 15.7 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> f </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 3.5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 14.4 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> g </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 1 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 20 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 5.1 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 18.2 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> h </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 15 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 3.0 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 13.7 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> i </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 3 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 10.5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.9 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> j </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 3 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 9.5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 9.0 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> k </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 14.5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 6.8 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> l </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 11.5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 5.9 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
</tbody>
</table>
]]]
.column[.pad50px[.content[

<img src="images/lecture-09b/unnamed-chunk-13-1.png" width="100%" style="display: block; margin: auto;" />
]]]

---
class: split-50 
layout: false

.column[.pad10px[.content[

Recompute means, and re-assign the cluster membership

`$\bar{x}_1=$` (5, 19) ,
`$\bar{x}_2=$` (11, 6) <br>

<table class="table table-striped" style="font-size: 24px; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;color: white !important;background-color: #3F9F7A !important;"> lbl </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> x1 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> x2 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> d1 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> d2 </th>
   <th style="text-align:left;color: white !important;background-color: #3F9F7A !important;"> cl </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> a </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 16 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 4 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 18.4 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 5.5 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> b </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 8 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 17.7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.6 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> c </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 14 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 4 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 17.3 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 3.6 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> d </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 9 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 17.1 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.9 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> e </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 10 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 21 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 5.5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 15.4 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> f </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2.0 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 13.9 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> g </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 1 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 20 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 4.2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 17.4 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> h </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 15 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 4.8 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 12.8 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> i </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 3 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 12.9 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 7.8 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> j </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 3 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 11.9 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 7.9 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> k </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 16.8 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 6.0 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> l </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 13.8 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 4.8 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
</tbody>
</table>
]]]
.column[.pad50px[.content[

<img src="images/lecture-09b/unnamed-chunk-16-1.png" width="100%" style="display: block; margin: auto;" />
]]]

---
class: split-50 
layout: false

.column[.pad10px[.content[

Recompute means, and re-assign the cluster membership

`$\bar{x}_1=$` (5, 19) ,
`$\bar{x}_2=$` (11, 6) <br>

<table class="table table-striped" style="font-size: 24px; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;color: white !important;background-color: #3F9F7A !important;"> lbl </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> x1 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> x2 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> d1 </th>
   <th style="text-align:right;color: white !important;background-color: #3F9F7A !important;"> d2 </th>
   <th style="text-align:left;color: white !important;background-color: #3F9F7A !important;"> cl </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> a </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 16 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 4 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 18.4 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 5.5 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> b </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 8 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 17.7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.6 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> c </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 14 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 4 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 17.3 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 3.6 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> d </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 9 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 17.1 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 8.9 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> e </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 10 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 21 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 5.5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 15.4 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> f </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 19 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2.0 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 13.9 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> g </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 1 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 20 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 4.2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 17.4 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> h </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 15 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 4.8 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 12.8 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> i </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 3 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 12.9 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 7.8 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> j </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 3 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 7 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 11.9 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 7.9 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> k </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 2 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 16.8 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 6.0 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;width: 1cm; border-right:1px solid;"> l </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 6 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;"> 5 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 13.8 </td>
   <td style="text-align:right;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 4.8 </td>
   <td style="text-align:left;width: 1cm; border-right:1px solid;color: #7570B3 !important;"> 2 </td>
  </tr>
</tbody>
</table>
]]]
.column[.pad50px[.content[

<img src="images/lecture-09b/unnamed-chunk-19-1.png" width="100%" style="display: block; margin: auto;" />
]]]

---
# Watch it animate

---
count: false
# Watch it animate

---
count: false
# Watch it animate

---

## Example

---
class: split-50 
layout: false

.column[.pad50px[.content[

`$k=2$`

<img src="images/lecture-09b/unnamed-chunk-24-1.png" width="100%" style="display: block; margin: auto;" />
]]]
.column[.pad50px[.content[

`$k=3$`

<img src="images/lecture-09b/unnamed-chunk-25-1.png" width="100%" style="display: block; margin: auto;" />
]]]

---
class: split-50 
layout: false

.column[.pad50px[.content[

`$k=4$`

<img src="images/lecture-09b/unnamed-chunk-26-1.png" width="100%" style="display: block; margin: auto;" />
]]]
.column[.pad50px[.content[

`$k=5$`

<img src="images/lecture-09b/unnamed-chunk-27-1.png" width="100%" style="display: block; margin: auto;" />
]]]

---
## Choosing k

.monash-blue2[Cluster statistics]

- .monash-orange2[WBRatio]: average within/average between want it to be low, but always drops for each additional cluster so look for large drops
- .monash-orange2[Hubert Gamma]: (s+ - s-)/(s+ + s-) where `$s+=$`sum of number of within `$<$` between, `$s-=$` sum of number within `$>$` between, want this to be high
- .monash-orange2[Dunn]: smallest distance between points from different clusters/maximum distance of points within any cluster, want this to be high
- .monash-orange2[Calinski-Harabasz Index]: `$\frac{\sum_{i=1}^p B_{ii}/(k-1)}{\sum_{i=1}^p W_{ii}/(n-k)}$` want this to be high

---

## Choosing k

---
## k-means caveats

.monash-blue2[Effect of seed]

- The k-means algorithm can yield quite different results depending on the initial seed.
- Example runs used 5 random starts, and used the `within.cluster.ss` metric to decide on the best solution.

---
## Interpoint distance measures

.monash-blue2[Euclidean]

- Cluster analysis depends on the interpoint distances, points close together should be grouped together
- Euclidean distance was used for the example. Let `$A=(x_{a1}, x_{a2}, ..., x_{ap}), B=(x_{b1}, x_{b2}, ..., x_{bp})$`

`\begin{align*}
d_{EUC}(A, B) &= \sqrt{\sum_{j=1}^p (x_{aj}-x_{bj})^2} &\\
&= \sqrt{((X_A-X_B)^T (X_A-X_B))}&
\end{align*}`

---
class: split-50
layout: false

.column[.pad10px[

### Other distance metrics

- Mahalanobis (or statistical) distance

`$$\sqrt{((X_A-X_B)^TS^{-1} (X_A-X_B))}$$`

- Manhattan:

`$$\sum_{j=1}^p|(X_{aj}-X_{bj})|$$`

- Minkowski:

`$$(\sum_{j=1}^p|(X_{aj}-X_{bj})|^m)^{1/m}$$`
]]
.column[.pad10px[

### Distances for count data

- Canberra:

`$$\frac{1}{n_z}\sum_{j=1}^p\frac{X_{aj}-X_{bj}}{X_{aj}+X_{bj}}$$`

- Bray-Curtis:

`$$\frac{\sum_{j=1}^p|X_{aj}-X_{bj}|}{\sum_{j=1}^p(X_{aj}+X_{bj})}$$`

]]
---
## Interpoint distance measures - Euclidean

.monash-blue2[Rules for any metric to be a distance]

1. `$d(A, B) \geq 0$`
2. `$d(A, A) = 0$`
3. `$d(A, B) = d(B, A)$`
4. Metric dissimilarity satisfies 
`$d(A, B) \leq d(A, C) + d(C, B)$`, and an ultrametric dissimilarity satisfies
`$d(A, B) \leq max\{d(A, C), d(C, B)\}$`

---

background-size: cover
class: title-slide
background-image: url("images/bg-02.png")

<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.

.bottom_abs.width100[

Lecturer: *Professor Di Cook*

Department of Econometrics and Business Statistics

<i class="fas fa-envelope"></i>  ETC3250.Clayton-x@monash.edu

<i class="fas fa-calendar-alt"></i> Week 9b

<br>

]