What is the core idea behind count-based word embeddings?

What is the core idea behind count-based word embeddings?

• A. Predicting co-occurrence frequencies with neural nets
• B. Assigning words random vectors
• C. Counting co-occurrences in the matrix
• D. Measuring sentence length with vectors

Answer: (C)

Count-based word embeddings rely on the idea that a word’s meaning emerges from the company it keeps in text. You gather a large corpus and build a word–context co-occurrence matrix, counting how often each word appears with nearby context words within a fixed window. These counts are then transformed into a lower-dimensional representation—often by weighting the counts (for example with PMI) and reducing dimensionality with a method like SVD. The resulting dense vectors place words that occur in similar contexts close to each other, capturing semantic relationships such as synonyms and related concepts. This contrasts with approaches that train a neural network to predict a word from its context or vice versa. So the key move is counting and organizing co-occurrence information in a matrix and then compressing it into embeddings.

Count-based word embeddings rely on the idea that a word’s meaning emerges from the company it keeps in text. You gather a large corpus and build a word–context co-occurrence matrix, counting how often each word appears with nearby context words within a fixed window. These counts are then transformed into a lower-dimensional representation—often by weighting the counts (for example with PMI) and reducing dimensionality with a method like SVD. The resulting dense vectors place words that occur in similar contexts close to each other, capturing semantic relationships such as synonyms and related concepts. This contrasts with approaches that train a neural network to predict a word from its context or vice versa. So the key move is counting and organizing co-occurrence information in a matrix and then compressing it into embeddings.