Milvus, established in 2019, is an open-source distributed vector database that focuses on storing and managing large-scale embedding vectors primarily generated from deep neural networks and other machine learning models. Milvus excels in handling large-scale embedding vectors with its outstanding vector indexing capabilities, effortlessly addressing index problems involving trillions of vectors.
The database's underlying logic began design considerations by addressing embedding vectors derived from unstructured data, which differs from traditional relational databases that handle predefined structured data. With the growth of the internet, the prevalence of unstructured data has become increasingly common, including emails, academic papers, IoT sensor data, photos from social media, and protein structures, among others. To enable computers to process this unstructured data, we need to use embedding techniques to convert the data into vectors, and Milvus offers an excellent solution for storing and indexing these vectors.
Milvus's strength lies not only in storage and indexing but also in its ability to calculate the similarity distance between two vectors to analyze their correlations. This means that if two embedding vectors are highly similar, it is likely that their original data exhibits similarities as well. This capability is immensely helpful in understanding and processing patterns and trends within unstructured data.