Build AI-driven search applications with Elasticsearch

Elasticsearch is the world's leading search engine. Traditional keyword search cannot meet the needs of today's intelligent era. Contemporary enterprises propose semantic search, which is searching based on the semantics of text rather than simple keyword matching. Additionally, we need to search for other data types, such as images, voice, and video. Since version 8.0, Elasticsearch has provided vector search (dense vectors, sparse vectors). It can perfectly solve text semantic search and multimedia data search. However, vector search is not perfect, especially for text search. We can use hybrid search (keyword search, vector search) for multi-path recall and rank the final results. This method can improve search accuracy and recall rate. In today's AI development, combined with large models, we can integrate the search results with large models and use GenAI to obtain inference results. Since enterprise data or private data is generated every moment, and the knowledge of large models is limited to the time of model generation, and the data of large models is obtained through training on web data. Using large models to infer enterprise or private data without context often leads to hallucinations because this knowledge does not exist in the large model. By combining Elasticsearch's vector search technology to search enterprise data or private data and providing the search results as context to the large model, hallucinations can be eliminated. This technology is also known as RAG (Retrieval-Augmented Generation). This topic will provide a detailed introduction to Elasticsearch's vector search technology and how to use it for RAG application development and the latest agentic RAG.