I wrote the first draft of this in roughly 2017 but left lots of details out. I happened to run across this again about 2 weeks ago and it now seems quite apropos for the time we're living it, so I spruced it up a bit, added some of the missing details, and here I'm publishing it. Enjoy!
I know it has to be true – The magic of human-level cognition isn't the result of a brain in which every single piece is perfectly tuned for its very special and isolated task. Instead, there has to be some simple "principle of computation" that is repeated throughout the brain; some component that, when replicated throughout the brain, gives rise to this emergent property we call intelligence. Sure, we have a motor cortex, specialized to control movement. We have a visual cortex, specialized to process information from our eyes. And we have a pre-frontal cortex that assimilates all this information together and plans what to do next – "That's a snake! Don't step on it!" But there is evidence that the base circuitry that makes up all these modules is actually quite general. At every point on the neocortex, you see basically the same pattern - an interwoven computational "component" composed of 6 layers of stacked neurons. At every portion of our cortex this pattern is, with very little modification, repeated. Besides the similar morphology, there is other evidence that the computational components are general. In several really weird brain rewiring studies, they have redirected visual input to the auditory pathway and shown that animals can compensate quite well - effectively "seeing" with the part of their brain that was rightfully intended to hear stuff! (Mice (Lyckman et al., 2001), ferrets (Sur et al., 1988; Roe et al., 1990; Roe et al., 1992; Roe et al., 1993; Sharma et al., 2000), and hamsters (Schneider, 1973; Kalil & Schneider, 1975; Frost, 1982; Frost & Metin, 1985).)
Vector search has taken the world by storm. The idea is this - cast documents into a vector embedding space via some transformation (e.g. BERT) and then index them into a vector database that is capable of fast approximate nearest neighbors search. Then when the user makes a text query, cast their query into this same embedding space and find the nearest vectors to the query vector and return the corresponding documents.
It occurred to me today while staring at the sky blankly, that this type of search won't work when searching for grouped content. For instance, what if I go onto Reddit, and I'm not really looking for a single reddit page, but I'm looking for an entire subreddit that matches my interests. Here I'm not looking a document, but a group of documents that match my intent.
Embedding spaces are quite trendy right now machine learning. With word2vec for example, you can create an embedding for words that is capable of capturing analogy. Given the input "man is to king as woman is to what?", a word2vec embedding can be used to correctly answer "queen". (Remarkable isn't it?) Embeddings like this can be used for a wide variety of different domains. For example, facial photos can be projected into an embedding space and for tasks of facial recognition. However I wonder if embeddings fall short in a domain that I am very near to - search. Consider the facial recognition task: Each face photo is converted into an N-dimensional vector where N is often rather high (hundreds of values). Given a sample photograph of a face, if you want to find all of the photos of that person then you have to search for all the photo vectors near to the sample photo's vector. But, due to the curse of dimensionality, very high dimensional embedding spaces are not amenable to data structure commonly used for spatial search, such as k-d trees.