A model cannot read words
You can look at two sentences and recognize that they discuss the same idea. A machine begins with a harder problem: text is not yet a numerical object it can compare, classify, or retrieve. The first design decision in an NLP system is therefore not which model to train, but what information the representation should preserve.
A bag-of-words representation takes a blunt but useful approach. It creates a fixed vocabulary and represents each document with word counts. This discards grammar and order, yet it often works because word presence is the dominant signal in tasks such as spam detection or product-review sentiment. If a spam message contains the vocabulary you expect from spam, sentence structure may add little.
This simplicity also makes bag of words a strong baseline. Its feature weights map directly to words, inference is fast, and a failure is easier to inspect than a failure inside a large neural model. Starting here tells you whether a more complex representation is solving a real problem.
Tradeoff: bag of words is sparse and its vocabulary can reach tens of thousands of dimensions. It also treats synonyms such as “automobile” and “car” as unrelated, and it makes “dog bites man” identical to “man bites dog.”
Make counts more selective
Raw counts give frequent words too much influence. TF-IDF corrects that by combining a term's frequency inside one document with its rarity across the collection: TF-IDF(t,d) = TF(t,d) × log(N / df(t)). A word such as “the” appears almost everywhere and receives little weight, while a term such as “aneurysm” can become highly informative in a medical record.
TF-IDF remains useful for document classification, keyword extraction, and retrieval. BM25 builds on the same idea with better saturation curves and is generally preferred for retrieval, while TF-IDF remains a natural feature representation for classic machine-learning classifiers.
You can recover a little word order by adding n-grams. A bigram treats “New York” or “not good” as a feature rather than two isolated words. Character n-grams can also help with language identification, spelling variation, and words that were not in the original vocabulary.
Tradeoff: n-grams buy local context by expanding the vocabulary. A typical bigram vocabulary can be about 100 times larger than its unigram vocabulary, which increases sparsity and memory use. TF-IDF also remains non-semantic and is weak on very short texts where frequency carries little information.
Similarity must be learned
One-hot encoding gives every word a vector with one 1 at its vocabulary index and zeros everywhere else. It is a clean lookup mechanism, but it contains no similarity structure: “king” and “queen” have a dot product of zero, just like any unrelated pair. One-hot vectors are therefore useful as inputs to an embedding table, not as final representations.
The key shift comes from the distributional hypothesis: words with similar meanings tend to appear in similar contexts. If two words repeatedly occupy similar positions near similar neighbors, training can pull their vectors toward one another without anyone programming a definition of either word.
Word2Vec turns this idea into a shallow prediction task. CBOW predicts a missing center word from surrounding context, as when “the cat ___ on mat” points toward “sat.” Skip-gram reverses the task and predicts surrounding words from the center word. CBOW is faster and works well for frequent words; skip-gram is slower but often represents rare words better because each context prediction receives its own update.
Word2Vec avoids a full-vocabulary softmax with negative sampling: it contrasts the correct word with a small set of sampled wrong words. The source notes suggest 5–20 negatives for small datasets and 2–5 for large ones.
Word2Vec, GloVe, and FastText solve different gaps
Word2Vec learns from local windows. Its vector space can develop consistent offsets, which is why the famous “king” minus “man” plus “woman” example lands near “queen.” But its vocabulary is fixed. A word absent during training must be discarded or mapped to an unknown token, a serious limitation when word forms multiply.
GloVe instead builds global word-to-word co-occurrence statistics over the corpus and factorizes them so vector dot products approximate log co-occurrence probabilities. Its practical quality is similar to Word2Vec, but its corpus-level formulation is more reproducible and makes it a useful static feature source when you need broad domain coverage.
FastText addresses the unknown-word problem by representing a word as the sum of character n-gram embeddings plus a whole-word embedding. The vector for “playing,” for example, includes pieces such as “pla,” “lay,” and “ing.” An unseen word such as “unplayable” can still receive an approximate vector because many of its pieces appeared in known words. That makes FastText especially useful for specialized vocabularies and morphologically rich languages such as Finnish, Turkish, and Arabic.
Choice: use GloVe when static vectors and corpus coverage are enough. Prefer FastText when out-of-vocabulary words, specialized terminology, or rich morphology are central. Both avoid the cost of running a contextual transformer.
A useful vector can still miss the sentence
Static embeddings give each word one vector in every setting. That is efficient, but the representation for “bank” is identical in “deposit money at the bank” and “sit on the river bank.” The model has learned that words relate, yet it still cannot change a word's meaning in response to the sentence around it.
Static methods remain the right tool when you need inexpensive semantic features, CPU-scale similarity, or averaged word vectors for an sklearn or XGBoost pipeline. The right question is not whether they are old, but whether your task needs context strongly enough to justify more computation.
To make a word representation depend on what came before and after it, a model must process an ordered sequence and carry information across positions. In the next lesson, we will follow that requirement into recurrent networks, their memory problem, and eventually attention.