# Set up training logics

## We have all the components we need

1. Raw Data: Loaded the SMILE dataset
2. Split Data: We used stratified sampling to ensure generalized distribution
3. Data For Machine Learning: we tokenized the data into `X_train, X_val`&#x20;
4. Model we are going to use: `Model = BertForSequenceClassification.from_pretrained`
5. Tools to feed data: `dataloader_train = DataLoader`
6. Tools to train: `optimizer = AdamW ; scheduler = get_linear_schedule_with_warmup`
7. Tools to measure performance: `f1_score_func ; accuracy_per_class`

## Check environment, let model know whether to use GPU or even TPU

```python
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model.to(device)
print(device)
```

## Customize evaluate function for logging:

The main training happens in `model.eval()`

We customize the `evaluate` function to evaluate and log the validation performance

```python
def evaluate(dataloader_val):
    """
    This function is customize some output of the evaluation result
    """


    # call the default evaluation function
    model.eval()
    
    loss_val_total = 0
    predictions, true_vals = [], []
    
    
    # loop through validation set, evaluate and show performance
    for batch in tqdm(dataloader_val):
        
        batch = tuple(b.to(device) for b in batch)
        
        inputs = {'input_ids':      batch[0],
                  'attention_mask': batch[1],
                  'labels':         batch[2],
                 }

        with torch.no_grad():  # disabled gradient calculation, because we just want to evaluate here      
            outputs = model(**inputs)
        
        # recored each loss and aggregate    
        loss = outputs[0]
        loss_val_total += loss.item()

        # record all prediction vs real labels        
        logits = outputs[1].detach().cpu().numpy()
        label_ids = inputs['labels'].cpu().numpy()
        predictions.append(logits)
        true_vals.append(label_ids)
    
    loss_val_avg = loss_val_total/len(dataloader_val) 
    
    predictions = np.concatenate(predictions, axis=0)
    true_vals = np.concatenate(true_vals, axis=0)
            
    return loss_val_avg, predictions, true_vals

```

## Training Loop:

`tqdm` : a progress bar library. tqdm derives from the Arabic word *taqaddum* (تقدّم) which can mean "progress," and is an abbreviation for "I love you so much" in Spanish (*te quiero demasiado*).

Logics of the loop:

1. For the training, iterate through all epochs
2. For the epoch, train on each batch
3. Log the training performance on that epoch

```python
for epoch in tqdm(range(1, epochs+1)):
    """
    for each epoch, we do a training all all instance, batch by batch
    and we also log the performance of training for that epoch
    """


    # ask the model to train
    model.train()
    
    # specify information for the progress bar
    loss_train_total = 0
    
    progress_bar = tqdm(dataloader_train, 
                        desc=f"Epoch {epochs}",
                        leave=False,
                        disable=False
                       )
    
    # similar to the evaluation function
    for batch in progress_bar:
    
        model.zero_grad()
        batch = tuple(b.to(device) for b in batch)
        
        inputs = {
            'input_ids': batch[0],
            'attention_mask': batch[1],
            'labels': batch[2]
        }
        
        outputs = model(**inputs)
        
        loss = outputs[0]
        loss_train_total ++ loss.item()
        loss.backward()
        
        
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        # proceed to next step
        optimizer.step()
        scheduler.step()
        
        progress_bar.set_postfix({'training_loss': '{:3f}'.format(loss.item()/len(batch))})
    
    # save the model
    torch.save(model.state_dict(), f"Models/BERT_ft_epoch{epoch}.model")
    
    # write training information
    tqdm.write('\nEpoch {epoch}')
    
    loss_train_avg = loss_train_total/len(dataloader)
    
    tqdm.write(f'Training loss: {loss_train_avg}')
    
    val_loss, predictions, tru_vals = evaluate(dataloader_val)
    val_f1 = f1_score_func(predictions, true_vals)
    tqdm.write(f'validataion loss: {val_loss}')
    tqdm.write(f"F1 score {val_f1},")
        
        
```
