Edit RoBERTa is an extension of BERT with changes to the pretraining procedure. The modifications include: training the model longer, with bigger batches, over more data Nevertheless, in the vocabulary size growth in RoBERTa allows to encode almost any word or subword without using the unknown to