Overview of BERT's Masking Approach
BERT (Bidirectional Encoder Representations from Transformers) introduced a novel approach to masked language modeling that has since become a cornerstone in NLP. BERT-style masking involves randomly masking or replacing tokens in the input text and then training the model to predict these masked tokens.
The key idea behind BERT's masking strategy is to create a deep bidirectional representation by jointly conditioning on both left and right context for all layers. This approach allows the model to develop a more nuanced understanding of language context.
Key Characteristics of BERT Masking
Random Masking: BERT randomly masks 15% of the tokens in each sequence. This percentage balances the amount of context available and the number of predictions the model needs to make.
Mask Token Replacement: Of the 15% masked tokens:
80% are replaced with a special [MASK] token
10% are replaced with a random word
10% are left unchanged
Bidirectional Context: BERT considers both left and right contexts simultaneously for each token, unlike traditional unidirectional language models.
Static Masking: The masking pattern is determined once during data preprocessing and remains the same for all training epochs.
Subword Tokenization: BERT uses WordPiece tokenization, which can split words into subword units, allowing the model to handle out-of-vocabulary words effectively.
BERT Masking Process
Tokenization: The input text is tokenized using WordPiece tokenization.
Special Token Addition: [CLS] token is added at the beginning, and [SEP] tokens are used to separate sentences or mark the end of the input.
Masking:
15% of tokens are randomly selected.
These selected tokens are then masked according to the 80-10-10 rule mentioned above.
Input Preparation: The masked sequence, along with position and segment embeddings, is prepared as input for the model.
Training Overview
During training, BERT learns to predict the original tokens of the masked positions based on the surrounding context. This process encourages the model to develop a deep understanding of language patterns and relationships between words.
The training objective includes:
Masked Language Modeling (MLM): Predicting the original vocabulary id of the masked words.
Next Sentence Prediction (NSP): Determining whether two given sentences follow each other in the original text.
By training on these objectives across a large corpus of text, BERT develops powerful, context-aware language representations that can be fine-tuned for various downstream NLP tasks.
Pros & Cons of BERT-Style Masking
Pros
Bidirectional Context: BERT-style masking allows the model to capture context from both directions, leading to more nuanced language understanding.
Effective Pre-training: This technique enables efficient pre-training on large amounts of unlabeled text data.
Versatility: The representations learned through this masking approach are useful for a wide range of downstream NLP tasks.
Handling Ambiguity: By considering full context, BERT-style masking helps in better handling of ambiguous words and phrases.
Improved Performance: Models using this technique have achieved state-of-the-art results on various NLP benchmarks.
Cons
Computational Intensity: The bidirectional nature of BERT-style masking makes it computationally expensive to train.
Artificial Noise: The [MASK] token used during pre-training doesn't appear in real-world text, potentially creating a mismatch between pre-training and fine-tuning/inference.
Static Masking: In the original BERT, the same masks are used across training epochs, which may limit the model's exposure to different contexts.
Limited Sequence Length: BERT-style models typically have a maximum sequence length (e.g., 512 tokens), which can be limiting for tasks involving longer texts.
Suboptimal for Generation Tasks: The bidirectional nature of BERT-style masking makes it less suitable for autoregressive text generation tasks.
Comments