> For the complete documentation index, see [llms.txt](https://michael-mao.gitbook.io/sentiment-analysis-bert/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://michael-mao.gitbook.io/sentiment-analysis-bert/pytorch-bert/what-is-bert.md).

# What is BERT

A introduction to the BERT model

**BERT** is a large scale transformer-based language model That can be fine-tuned. It is developed by Google and the detail information can be found at <https://arxiv.org/abs/1810.04805>

**Transformer**: an advancement of recurrent neural net. Can parallel processing and training of instance. It make the input and output size fit, which is more convenient

**From Pre-Training to Fine-Tuning:**

![BERT model overview](https://4004916086-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MBn18XBhmp6C9tjZfuN%2F-MBqcDg0oihiM559aO7c%2F-MBqd2VJOcAZbhCDmxyk%2FBERT_diagrams.png?alt=media\&token=24a9185e-d432-432d-bd49-a8d18fce9d8e)

1. Token Embedding: specify the token for that word: e.g. `my -> 1`
2. Segment Embedding:  specify which segment the word belongs to: e.g. sentence A or sentence B
3. Position Embedding: specify the position of that word in the whole input