What is text embedding ?
Large language model is replacing RDBMS and SQL, use vector to represent text
Did you know Cambridge dictionary works like a language model ? and Each time you lookup a word, it's as if you are obtaining a vector that represents the essence of that word! Let us dive into the fascinating world of semantic search.
Computers and computer languages are deterministic, which is why we store values in databases to facilitate this inherent aspect of computer languages. The IF statement relies on known values to determine the subsequent logic to execute. However, in semantic search or natural language processing, this deterministic nature is absent.
Words do not exist in isolation. The meaning of "I buy an Apple" depends on the context established by previous statements. To comprehend the meaning of words, it is crucial to understand their relationships with one another. To capture these relationships, we use word embeddings, which are multi-dimensional vector representations of words and their associations with other words.
Text embeddings (or text vectors) are generated by trained language models, which are neural networks developed from extensive text corpora. To obtain a text vector, a sentence or text is input into a neural network model, which then outputs the text vector. This vector captures the relationships of words in the corpus. Different models produce distinct text vectors because the training corpora and mechanisms vary.
When using text embeddings, we aim to understand the meaning of words, which is expressed through the use of other English words. This process is similar to looking up a word in a dictionary; when you search for "cat" in the Cambridge Dictionary, it uses other English words to define "cat."
Cat noun
a small animal with fur, four legs, a tail, and claws,
usually kept as a pet or for catching mice
(from Cambridge online) If you consult the Collins Dictionary, the definition is similar, but it uses a different sequence of English words. In this example, the models are Cambridge and Collins, and the vocabulary definitions are the text vectors, demonstrating the relationship of "cat" with other English words.
Kids learn the about world by using the language (i.e. word relationships) their parents use and computer scientists train AI models to understand our world by letting the NN to discover the relationships with billion words. That maybe some how explains the reasoning power of AI models. My personal view is the secret of AI will likely happen in combining linguistics with mathematics. When we can use math symbols to describe all linguistic theories, then the AI models will be able to comprehend human-centric reality.
OpenAI offers several models and charges a fee for generating text embeddings. There are also open-source models available, with some providing more accurate representations of word meanings and relationships than OpenAI's Ada model.
The study of text embedding outside of computer science is important but rare.
"Running head: Theoretical foundations and limits of word embeddings" by Alina Arseniev-Koehler
https://arxiv.org/pdf/2107.10413.pdf gives a detailed analysis of linguistic structuralism.
an extract from her paper intro:
In this paper, I focus on three of its key premises. First, that language is a system comprised of various signs (e.g., words, suffixes, or idioms) where these signs are purely defined by their relationship to other signs in the system, rather than by any external reality. For example, a word is defined by its co- occurrence relationship to all other words – not from the intrinsic properties of the letters or sounds that comprise the word, from dictionary definitions, or by its reference to some external object (Saussure, 1983, p. 113). This suggests, for example, that if a misspelled word is used in a similar way as a correctly spelled word, both spelling variants will be understood in the same way. If, however, spelling variants are used in some systematically different way (e.g., British versus American spellings), the variants will evoke slightly different interpretations– even when all variants refer to the same physical object.