Enabling SQL Server as a Vector Database for the Embedding Storage in the RAG Pattern

Abstract

This thesis explores the feasibility of adapting Microsoft SQL Server to function as a vector database for high-dimensional embeddings within Retrieval-Augmented Generation (RAG) systems. Traditionally reliant on specialized vector databases, RAG pipelines benefit from semantic search over embeddings. The research proposes a novel approach by using SQL Server with JSON support and stored procedures to store and query embeddings generated via OpenAI's API. A full-stack prototype was implemented, combining SQL Server, FastAPI, Semantic Kernel, and Azure OpenAI services. The system retrieves relevant document chunks based on cosine similarity and feeds them into a language model to generate grounded responses. Evaluation shows SQL Server can achieve effective semantic retrieval with sub-second latency, offering a viable alternative for organizations leveraging existing relational infrastructure.