What is the main idea behind count-based embeddings?

What is the main idea behind count-based embeddings?

• A. Counting occurrences in a co-occurrence matrix
• B. Predicting the next word using a neural network
• C. Using one-hot vectors per word
• D. Learning through reinforcement feedback

Answer: (A)

Count-based embeddings build word representations from how often words appear near each other in text. The process starts by tallying co-occurrences within a window and forming a co-occurrence matrix, where each entry reflects how often a pair of words shows up together. By applying dimensionality reduction or factorization (for example, using PMI or SVD), these counts are transformed into dense, low-dimensional vectors. Words with similar surrounding contexts end up near each other in this embedding space, mirroring the distributional idea that meaning comes from usage. This approach is different from predicting future words with a neural network, which learns embeddings by trying to forecast context rather than counting occurrences. It also isn’t based on one-hot vectors, which are sparse and don’t encode similarity, and is not about reinforcement feedback, which is a different learning paradigm.

Count-based embeddings build word representations from how often words appear near each other in text. The process starts by tallying co-occurrences within a window and forming a co-occurrence matrix, where each entry reflects how often a pair of words shows up together. By applying dimensionality reduction or factorization (for example, using PMI or SVD), these counts are transformed into dense, low-dimensional vectors. Words with similar surrounding contexts end up near each other in this embedding space, mirroring the distributional idea that meaning comes from usage. This approach is different from predicting future words with a neural network, which learns embeddings by trying to forecast context rather than counting occurrences. It also isn’t based on one-hot vectors, which are sparse and don’t encode similarity, and is not about reinforcement feedback, which is a different learning paradigm.