> For the complete documentation index, see [llms.txt](https://michael-mao.gitbook.io/sentiment-analysis-bert/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://michael-mao.gitbook.io/sentiment-analysis-bert/pytorch-bert/encode-with-tokenizer-and-prepare-the-dataset-for-machine-learning.md).

# Encode with Tokenizer and Prepare the dataset for Machine Learning

Usually, what we have done in the “Train-Validation-Test Split” section is good for ML. But NLP is different, we need Tokeniser to convert text into numbers representation for the machine to learn

## Package

1. **BertTokenizer** - provided in the [transformers](https://huggingface.co/transformers/index.html) package
2. **TensorDataset** - provided in the [torch](https://pytorch.org/docs/stable/torch.html) package

```python
from transformers import BertTokenizer
from torch.utils.data import TensorDataset
```

## BertTokenizer

It is a tokenzier provided by BERT model, we can choose want kind of tokenizer we want to use by specifying the parameters. In this case: we use the following tokenizer

```python
tokenizer = BertTokenizer.from_pretrained(
    'bert-base-uncase',
    do_lower_case=True
)
```

And by using this tokenizer, we are able to transform our data into tokens

```python
encoded_data_train = tokenizer.batch_encode_plus(
    df[df.data_type=='train'].text.values, # the data we provide
    add_special_token=True, # let BERT know the begin & end of a sentence
    return_attention_mask=True, # attention mask is a fixed size that specify the amount of data we are interested in
    pad_to_max_length=True, # if size is less than attention mask, then fill pad data in
    max_length=256, # the size of the attention mask
    return_tensor='pt'
)
```

So now our input texture data is tokenized into number-based data. The returned `encoded_data_train` is a dictionary

```python
input_ids_train = encoded_data_train['input_ids']
attention_masks_train = encoded_data_train['attention_mask']
labels_train = torch.tensor(df[df.data_type=='train'].label.value)
```

## **TorchDataset**

```python
dataset_train = TensorDataset(
    input_ids_train, # data
    attention_masks_train, # mask
    labels_train # label
)
```

Then we have the data prepared to feed into the machine learning model