Today I learned that building a successful open-source project like Milvus, which became a leading vector database, involved starting from scratch due to a lack of suitable existing solutions, making the difficult decision to completely rewrite the core system (Milvus 2.0) for better scalability and cloud-nativity, and continuously engaging with the community to evolve the product. It also highlighted that the company behind Milvus, Zilliz, focuses on the “unglamorous” but critical aspects like performance, security, and global support, and that they are now working on a “Vector Data Lake” to handle massive, less frequently accessed vector data more cost-effectively. Small Notes for the Blog

• Why Milvus Started: Back in 2017, there was no good way to store and search the special data (vector embeddings) that helps AI understand things. Old databases didn’t work, and other tools were too slow or not built for real use. So, they decided to build their own.
• Going Open Source: In 2019, Milvus became open source, meaning its code was made public for everyone. This helped many developers use it and suggest improvements, even finding bugs. They also joined a foundation (LF AI & Data Foundation) to learn how to manage a long-lasting open-source project.
• The Big Rewrite (Milvus 2.0): In 2021, even though Milvus 1.0 was working, big companies wanted it to work better with cloud systems and scale more easily. The team made a tough choice to completely rebuild Milvus from scratch. This new version, Milvus 2.0 (released in 2022), was much more powerful and ready for large company needs.
• AI Boom and Growth: In 2023, with the rise of new AI tools like RAG (retrieval-augmented generation), Milvus became essential. Its popularity grew very fast, leading to more users and questions. They expanded their support and created Zilliz Cloud, a managed version of Milvus, to help support the open-source project.
• The Team (Zilliz): Many people know Milvus, but Zilliz is the company that builds and supports it. They focus on making sure Milvus performs well, is secure, and has good documentation and support around the world.
• Real-World Use: Milvus is now used by many big companies like Bosch, Read AI, and various fintech leaders to power their AI applications, showing its real impact in everyday systems.
• Zilliz Cloud Explained: While Milvus is free and open source, running it for big companies can be complex. Zilliz Cloud is a paid, fully managed version of Milvus that handles all the technical complexities, offering better performance, security, and cost efficiency for large businesses.
• What’s Next (Vector Data Lake): The team is now working on something new called “Vector Data Lake.” This is for very large amounts of vector data that don’t need to be searched instantly. It aims to store this data more cheaply and connect well with other data tools, complementing the existing real-time vector database.
• Thank You to the Community: The blog post ends by thanking the community of developers, users, and contributors who have helped Milvus reach over 35,000 stars on GitHub, emphasizing that this number represents trust and a collaborative journey.

You can find more details by visiting the original blog post: Our Journey to 35K+ GitHub Stars: Building Milvus from Scratch