Vector Databases and the Power of RAG

In this episode of the AI + a16z podcast, Pinecone Founder and CEO Edo Liberty joins a16z’s Satish Talluri and Derrick Harris to discuss the promises, challenges, and opportunities for vector databases and retrieval augmented generation (RAG). He also shares insights and highlights from a decades-long career in machine learning, which includes stints running research teams at both Yahoo and Amazon Web Services.

Here are some highlights:

[9:42] “I’ll admit to being a snob and being part of the camp that was, to begin with, quite skeptical of deep learning. My roots are in theoretical machine learning and in theoretical CS, in general. So I was really trying to prove convergence and improve, asymptotic running times and really understand worst-case behaviors and so on.

“And then deep learning comes along and people just run stuff, and it does something and they find it to be interesting. . . . And so for a few good years, pretty much every conversation between a theoretician and practitioner was some flavor of the theoretician says, ‘I can prove this is impossible.’ And the practitioner said, ‘But I did it this morning.’

[18:23] “RAG accelerated semantic search significantly . . . What used to happen is . . . you still used the vector database to retrieve the semantically similar documents or passages or facts and so on that you put into context. But when you consume the results of search, as a human, you can consume 10 blue links or 10 paragraphs or like one page of content. There’s, there’s a limit to how much cognitive bandwidth you can exert when you receive the search results.

“And with RAG, we have the opportunity to take a hundred full documents as the result of a search, and then further process them with a language model or a foundational model to glean some piece of information or some fact or some output. And so that created a whole new set of opportunities to build with RAG, where just semantic search, frankly, fell short.”

[25;51] “I think RAG today is where transformers were in 2017. It’s clunky and weird and hard to get right. And it has a lot of sharp edges, but it already does something amazing. Sometimes, most of the time, the very early adopters and the very advanced users are already picking it up and running with it — and lovingly deal with all the sharp edges. . . .

“Making progress on RAG, making progress on information retrieval, and making progress on making AI more knowledgeable and less hallucinatory and more dependable, is a complete greenfield today. There’s an infinite amount of innovation that will have to go into it.”

You can listen to all the AI + a16z podcast episodes here, or on your preferred podcast platform.