Table of Content
[RAG landing tool] Vector database Weaviate deployment and usage tutorial
Updated on:July-17th-2025
Recommendation
Master Weaviate and efficiently implement vector search and AI application.
Core content:
1. Introduction and functional features of Weaviate vector database
2. Weaviate installation steps and Docker deployment guide
3. Weaviate usage and configuration optimization suggestions
Yang Fangxian
Founder of 53AI/Most Valuable Expert of Tencent Cloud (TVP)
Weaviate deployment
1. Introduction
Weaviate is an open source vector search engine database that allows storing JSON documents in a class attribute manner and attaching machine learning vectors to these documents to represent them in vector space. Weaviate supports semantic search, question-answer extraction, classification, and other functions, and the data can be easily accessed through the GraphQL-API.
Official website: https://weaviate.io/
2. Install Weaviate
Download the latest image of Weaviate from Docker Hub:
docker pull semitechnologies/weaviate:latest
If the image pulling speed is slow, you can try to replace the image source.
2.3 Running Weaviate Container
Run the Weaviate container using the following command:
docker run -d --name weaviate \
--restart=always \
-p 8080:8080 \
-p 50051:50051 \
-e "AUTHENTICATION_APIKEY_ENABLED=true" \
-e "AUTHENTICATION_APIKEY_ALLOWED_KEYS=test-secret-key,test2-secret-key" \
-e "AUTHENTICATION_APIKEY_USERS=test@2024.com,test2@2024.com" \
-e "AUTHORIZATION_ADMINLIST_ENABLED=true" \
-e "AUTHORIZATION_ADMINLIST_USERS=test@2024.com" \
-e "AUTHORIZATION_ADMINLIST_READONLY_USERS=test2@2024.com" \
-e WEAVIATE_HOSTNAME=0.0.0.0 \
semitechnologies/weaviate:latest
Parameter Description
-d: Let the container run in the background.--name weaviate: Name the containerweaviate.--restart=always: Configure the container to automatically start after the host machine restarts.-p 8080:8080: Map port 8080 in the container to port 8080 on the host.-p 50051:50051: Map port 50051 in the container to port 50051 on the host machine.-e "AUTHENTICATION_APIKEY_ENABLED=true": Enable API key authentication.-e "AUTHENTICATION_APIKEY_ALLOWED_KEYS=test-secret-key,test2-secret-key": Specifies a list of API keys that are allowed to be used.-e "AUTHENTICATION_APIKEY_USERS=test@2024.com,test2@2024.com": Associate a key with a user's mailbox.-e "AUTHORIZATION_ADMINLIST_ENABLED=true": Enable admin list authorization.-e "AUTHORIZATION_ADMINLIST_USERS=test@2024.com": Specifies a user in the administrator list.-e "AUTHORIZATION_ADMINLIST_READONLY_USERS=test2@2024.com": Specifies a list of users with read-only permissions.-e WEAVIATE_HOSTNAME=0.0.0.0: Set Weaviate's hostname, listening on all available network interfaces.semitechnologies/weaviate:latest: Specifies the latest version of the Weaviate image to download from Docker Hub and run.
3. Test the connection
3.1 Install Weaviate Client
Install the Weaviate client using pip:
pip install -U weaviate-client
3.2 Writing test scripts
Create onetest.pyThe file has the following contents:
import weaviate
from weaviate.auth import AuthApiKey
# Connect to the locally deployed Weaviate
client = weaviate.connect_to_local(
auth_credentials=AuthApiKey( "test-secret-key" )
)
# Or custom connection
client = weaviate.connect_to_custom(
skip_init_checks= False ,
http_host= "127.0.0.1" ,
http_port = 8080 ,
http_secure = False ,
grpc_host= "127.0.0.1" ,
grpc_port = 50051 ,
grpc_secure = False ,
# Corresponding to the key in AUTHENTICATION_APIKEY_ALLOWED_KEYS
# Note: Only the key is needed here, no user name is required
auth_credentials=AuthApiKey( "test-secret-key" )
)
# Check if the connection is successful
print(client.is_ready())
# Close the connection
print(client.close())
3.3 Running the test script
Run the test script in the terminal:
python test.py
If the outputTrue, it means the connection is successful.
You can access the address through the browser:
http://localhost:8080/v1/docs
Using Python to operate Weaviate vector database
The following is a complete example of using Python to operate the Weaviate vector database, covering operations such as connecting to the database, checking whether a collection exists, creating a collection, inserting data, querying data, and deleting a collection.1. Install the Weaviate Python client
First, make sure you have the Weaviate Python client installed:
pip install weaviate-client
2. Connect to Weaviate database
import weaviate
from weaviate.auth import AuthApiKey
# Connect to the local Weaviate instance
client = weaviate.connect_to_local(
auth_credentials=AuthApiKey( "test-secret-key" )
)
# Or custom connection
client = weaviate.connect_to_custom(
http_host= "127.0.0.1" ,
http_port = 8080 ,
http_secure = False ,
grpc_host= "127.0.0.1" ,
grpc_port = 50051 ,
grpc_secure = False ,
auth_credentials=AuthApiKey( "test-secret-key" )
)
# Check if the connection is successful
print(client.is_ready())
3. Check if the collection exists
def check_collection_exists (client: weaviate.WeaviateClient, collection_name: str) -> bool:
"""
Check if a collection exists
:param client: Weaviate client
:param collection_name: collection name
:return: True or False
"""
try :
collections = client.collections.list_all()
return collection_name in collections
except Exception as e:
print( f"Check collection exception: {e} " )
return False
4. Create a collection
def create_collection (client: weaviate.WeaviateClient, collection_name: str) :
"""
Create a collection
:param client: Weaviate client
:param collection_name: collection name
"""
collection_obj = {
"class" : collection_name,
"description" : "A collection for product information" ,
"vectorizer" : "none" , # assuming you will upload your own vectors
"vectorIndexType" : "hnsw" ,
"vectorIndexConfig" : {
"distance" : "cosine" ,
"efConstruction" : 200 ,
"maxConnections" : 64
},
"properties" : [
{
"name" : "text" ,
"description" : "The text content" ,
"dataType" : [ "text" ],
"tokenization" : "word" ,
"indexFilterable" : True ,
"indexSearchable" : True
}
]
}
try :
client.collections.create_from_dict(collection_obj)
print( f"Creating collection ' {collection_name} ' successfully." )
except weaviate.exceptions.UnexpectedStatusCodeException as e:
print( f"Exception in creating collection: {e} " )
5. Insert data
def save_documents (client: weaviate.WeaviateClient, collection_name: str, documents: list) :
"""
Insert data into the collection
:param client: Weaviate client
:param collection_name: collection name
:param documents: document list
"""
collection = client.collections.get(collection_name)
for doc in documents:
content = doc # Assume the document is a simple string
vector = [ 0.1 , 0.2 , 0.3 ] # Assume this is your vector
properties = {
"text" : content
}
try :
uuid = collection.data.insert(properties=properties, vector=vector)
print( f"Document added content: {content[: 30 ]} ..., uuid: {uuid} " )
except Exception as e:
print( f"Add document exception: {e} " )
6. Query data
def query_vector_collection (client: weaviate.WeaviateClient, collection_name: str, query: str, k: int) -> list:
"""
Querying data from a collection
:param client: Weaviate client
:param collection_name: collection name
:param query: query string
:param k: the number of results returned
:return: Query result list
"""
vector = [ 0.1 , 0.2 , 0.3 ] # Assume this is your query vector
collection = client.collections.get(collection_name)
response = collection.query.near_vector(
near_vector=vector,
limit=k
)
documents = [res.properties[ 'text' ] for res in response.objects]
return documents
7. Delete a collection
def delete_collection (client: weaviate.WeaviateClient, collection_name: str) :
"""
Deleting a Collection
:param client: Weaviate client
:param collection_name: collection name
"""
try :
client.collections.delete(collection_name)
print( f"Deleted collection ' {collection_name} ' successfully." )
except Exception as e:
print( f"Delete collection exception: {e} " )
8. Example Usage
if __name__ == "__main__" :
# Connect Weaviate
client = weaviate.connect_to_local(auth_credentials=AuthApiKey( "test-secret-key" ))
collection_name = "MyCollection"
# Check if the collection exists
if not check_collection_exists(client, collection_name):
# Create a collection
create_collection(client, collection_name)
# Insert data
documents = [ "This is a test document." , "Another document for testing." ]
save_documents(client, collection_name, documents)
# Query data
query_results = query_vector_collection(client, collection_name, "test" , 2 )
print( "Query results:" , query_results)
# Delete a collection
delete_collection(client, collection_name)
# Close the connection
client.close()
How to implement semantic retrieval
In the TrusRAG project, the above tutorial is encapsulated, the specific link is as follows:https://github.com/gomate-community/TrustRAG/blob/pipeline/trustrag/modules/engine/weaviate_cli.py
WeaviateEngineThe implementation is as follows:
from typing import List, Dict, Any, Optional, Union
import numpy as np
import weaviate
from weaviate import WeaviateClient
from weaviate.collections import Collection
import weaviate.classes.config as wc
from weaviate.classes.config import Property, DataType
from trustrag.modules.retrieval.embedding import EmbeddingGenerator
from weaviate.classes.query import MetadataQuery
class WeaviateEngine:
def __init__(
self,
collection_name: str,
embedding_generator: EmbeddingGenerator,
client_params: Dict[str, Any] = {
"http_host" : "localhost" ,
"http_port" : 8080,
"http_secure" : False,
"grpc_host" : "localhost" ,
"grpc_port" : 50051,
"grpc_secure" : False,
},
):
"" "
Initialize the Weaviate vector store.
:param collection_name: Name of the Weaviate collection
:param embedding_generator: An instance of EmbeddingGenerator to generate embeddings
:param weaviate_client_params: Dictionary of parameters to pass to Weaviate client
" ""
self.collection_name = collection_name
self.embedding_generator = embedding_generator
# Initialize Weaviate client with provided parameters
self.client = weaviate.connect_to_custom(
skip_init_checks=False,
**client_params
)
# Create collection if it doesn't exist
if not self._collection_exists():
self._create_collection()
def _collection_exists(self) -> bool:
"" "Check if collection exists in Weaviate." ""
try:
collections = self.client.collections.list_all()
collection_names = [c.lower() for c in collections]
return self.collection_name in collection_names
except Exception as e:
print (f "Error checking collection existence: {e}" )
return False
def _create_collection(self):
"" "Create a new collection in Weaviate." ""
try:
self.client.collections.create(
name=self.collection_name,
# Define properties of metadata
properties=[
wc.Property(
name = "text" ,
data_type=wc.DataType.TEXT
),
wc.Property(
name = "title" ,
data_type=wc.DataType.TEXT,
skip_vectorization=True
),
]
)
except Exception as e:
print (f "Error creating collection: {e}" )
raise
def upload_vectors(
self,
vectors: Union[np.ndarray, List[List[ float ]]],
payload: List[Dict[str, Any]],
batch_size: int = 100
):
"" "
Upload vectors and payload to the Weaviate collection.
:param vectors: A numpy array or list of vectors to upload
:param payload: A list of dictionaries containing the payload for each vector
:param batch_size: Number of vectors to upload in a single batch
" ""
if not isinstance(vectors, np.ndarray):
vectors = np.array(vectors)
if len(vectors) != len(payload):
raise ValueError( "Vectors and payload must have the same length." )
collection = self.client.collections.get(self.collection_name)
# Process in batches
for i in range(0, len(vectors), batch_size):
batch_vectors = vectors[i:i + batch_size]
batch_payload = payload[i:i + batch_size]
try:
with collection.batch.dynamic() as batch:
for idx, (properties, vector) in enumerate(zip(batch_payload, batch_vectors)):
# Separate text content and other metadata
text_content = properties.get( 'description' ,
'' ) # Assuming 'description' is the main text field
metadata = {k: v for k, v in properties.items() if k != 'description' }
# Prepare the properties dictionary
properties_dict = {
"text" : text_content,
"title" : metadata.get( 'title' , f 'Object {idx}' ) # Using title from metadata or default
}
# Add the object with properties and vector
batch.add_object(
properties=properties_dict,
vector=vector
)
except Exception as e:
print (f "Error uploading batch: {e}" )
raise
def search(
self,
text: str,
query_filter: Optional[Dict[str, Any]] = None,
limit : int = 5
) -> List[Dict[str, Any]]:
"" "
Search for the closest vectors in the collection based on the input text.
:param text: The text query to search for
:param query_filter: Optional filter to apply to the search
:param limit: Number of closest results to return
:return: List of payloads from the closest vectors
" ""
# Generate embedding for the query text
vector = self.embedding_generator.generate_embedding(text)
print (vector.shape)
collection = self.client.collections.get(self.collection_name)
# Prepare query arguments
query_args = {
"near_vector" : vector,
"limit" : limit ,
"return_metadata" : MetadataQuery(distance=True)
}
# Add filter if provided
if query_filter:
query_args[ "filter" ] = query_filter
results = collection.query.near_vector(**query_args)
# Convert results to the same format as QdrantEngine
payloads = []
for obj in results.objects:
payload = obj.properties.get( 'metadata' , {})
payload[ 'text' ] = obj.properties.get( 'text' , '' )
payload[ '_distance' ] = obj.metadata.distance
payloads.append(payload)
return payloads
def build_filter(self, conditions: List[Dict[str, Any]]) -> Dict[str, Any]:
"" "
Build a Weaviate filter from a list of conditions.
:param conditions: A list of conditions, where each condition is a dictionary with:
- key: The field name to filter on
- match: The value to match
:return: A Weaviate filter object
" ""
filter_dict = {
"operator" : "And" ,
"operands" : []
}
for condition in conditions:
key = condition.get( "key" )
match_value = condition.get( "match" )
if key and match_value is not None:
filter_dict[ "operands" ].append({
"path" : [f "metadata.{key}" ],
"operator" : "Equal" ,
"valueString" : str(match_value)
})
return filter_dict if filter_dict[ "operands" ] else None
The test code is as follows:
from trustrag.modules.retrieval.embedding import SentenceTransformerEmbedding
from trustrag.modules.engine.weaviate_cli import WeaviateEngine
if __name__ == '__main__' :
# Initialize MilvusEngine
local_embedding_generator = SentenceTransformerEmbedding(model_name_or_path=r "H:\pretrained_models\mteb\all-MiniLM-L6-v2" , device= "cuda" )
weaviate_engine = WeaviateEngine(
collection_name = "startups" ,
embedding_generator=local_embedding_generator,
client_params={
"http_host" : "localhost" ,
"http_port" : 8080,
"http_secure" : False,
"grpc_host" : "localhost" ,
"grpc_port" : 50051,
"grpc_secure" : False,
}
)
documents = [
{ "name" : "SaferCodes" , "images" : "https://safer.codes/img/brand/logo-icon.png" ,
"alt" : "SaferCodes Logo QR codes generator system forms for COVID-19" ,
"description" : "QR codes systems for COVID-19.\nSimple tools for bars, restaurants, offices, and other small proximity businesses." ,
"link" : "https://safer.codes" , "city" : "Chicago" },
{ "name" : "Human Practice" ,
"images" : "https://d1qb2nb5cznatu.cloudfront.net/startups/i/373036-94d1e190f12f2c919c3566ecaecbda68-thumb_jpg.jpg?buster=1396498835" ,
"alt" : "Human Practice - health care information technology" ,
"description" : "Point-of-care word of mouth\nPreferral is a mobile platform that channels physicians\u2019 interest in networking with their peers to build referrals within a hospital system.\nHospitals are in a race to employ physicians, even though they lose billions each year ( $40B in 2014) on employment. Why ..." ,
"link" : "http://humanpractice.com" , "city" : "Chicago" },
{ "name" : "StyleSeek" ,
"images" : "https://d1qb2nb5cznatu.cloudfront.net/startups/i/3747-bb0338d641617b54f5234a1d3bfc6fd0-thumb_jpg.jpg?buster=1329158692" ,
"alt" : "StyleSeek - e-commerce fashion mass customization online shopping" ,
"description" : "Personalized e-commerce for lifestyle products\nStyleSeek is a personalized e-commerce site for lifestyle products.\nIt works across the style spectrum by enabling users (both men and women) to create and refine their unique StyleDNA.\nStyleSeek also promotes new products via its email newsletter, 100% personalized ..." ,
"link" : "http://styleseek.com" , "city" : "Chicago" },
{ "name" : "Scout" ,
"images" : "https://d1qb2nb5cznatu.cloudfront.net/startups/i/190790-dbe27fe8cda0614d644431f853b64e8f-thumb_jpg.jpg?buster=1389652078" ,
"alt" : "Scout - security consumer electronics internet of things" ,
"description" : "Hassle-free Home Security\nScout is a self-installed, wireless home security system. We've created a more open, affordable and modern system than what is available on the market today. With month-to-month contracts and portable devices, Scout is a renter-friendly solution for the other ..." ,
"link" : "http://www.scoutalarm.com" , "city" : "Chicago" },
{ "name" : "Invitation codes" , "images" : "https://invitation.codes/img/inv-brand-fb3.png" ,
"alt" : "Invitation App - Share referral codes community " ,
"description" : "The referral community\nInvitation App is a social network where people post their referral codes and collect rewards on autopilot." ,
"link" : "https://invitation.codes" , "city" : "Chicago" },
{ "name" : "Hyde Park Angels" ,
"images" : "https://d1qb2nb5cznatu.cloudfront.net/startups/i/61114-35cd9d9689b70b4dc1d0b3c5f11c26e7-thumb_jpg.jpg?buster=1427395222" ,
"alt" : "Hyde Park Angels - " ,
"description" : "Hyde Park Angels is the largest and most active angel group in the Midwest. With a membership of over 100 successful entrepreneurs, executives, and venture capitalists, the organization prides itself on providing critical strategic expertise to entrepreneurs and ..." ,
"link" : "http://hydeparkangels.com" , "city" : "Chicago" },
{ "name" : "GiveForward" ,
"images" : "https://d1qb2nb5cznatu.cloudfront.net/startups/i/1374-e472ccec267bef9432a459784455c133-thumb_jpg.jpg?buster=1397666635" ,
"alt" : "GiveForward - health care startups crowdfunding" ,
"description" : "Crowdfunding for medical and life events\nGiveForward lets anyone to create a free fundraising page for a friend or loved one's uncovered medical bills, memorial fund, adoptions or any other life events in five minutes or less. Millions of families have used GiveForward to raise more than $165M to let ..." ,
"link" : "http://giveforward.com" , "city" : "Chicago" },
{ "name" : "MentorMob" ,
"images" : "https://d1qb2nb5cznatu.cloudfront.net/startups/i/19374-3b63fcf38efde624dd79c5cbd96161db-thumb_jpg.jpg?buster=1315734490" ,
"alt" : "MentorMob - digital media education ventures for good crowdsourcing" ,
"description" : "Google of Learning, indexed by experts\nProblem: Google doesn't index for learning. Nearly 1 billion Google searches are done for \"how to\" learn various topics every month, from photography to entrepreneurship, forcing learners to waste their time sifting through the millions of results.\nMentorMob is ..." ,
"link" : "http://www.mentormob.com" , "city" : "Chicago" },
{ "name" : "The Boeing Company" ,
"images" : "https://d1qb2nb5cznatu.cloudfront.net/startups/i/49394-df6be7a1eca80e8e73cc6699fee4f772-thumb_jpg.jpg?buster=1406172049" ,
"alt" : "The Boeing Company - manufacturing transportation" , "description" : "" ,
"link" : "http://www.boeing.com" , "city" : "Berlin" },
{ "name" : "NowBoarding \u2708\ufe0f" ,
"images" : "https://static.above.flights/img/lowcost/envelope_blue.png" ,
"alt" : "Lowcost Email cheap flights alerts" ,
"description" : "Invite-only mailing list.\n\nWe search the best weekend and long-haul flight deals\nso you can book before everyone else." ,
"link" : "https://nowboarding.club/" , "city" : "Berlin" },
{ "name" : "Rocketmiles" ,
"images" : "https://d1qb2nb5cznatu.cloudfront.net/startups/i/158571-e53ddffe9fb3ed5e57080db7134117d0-thumb_jpg.jpg?buster=1361371304" ,
"alt" : "Rocketmiles - e-commerce online travel loyalty programs hotels" ,
"description" : "Fueling more vacations\nWe enable our customers to travel more, travel better and travel further. 20M+ consumers stock away miles & points to satisfy their wanderlust.\nFlying around or using credit cards are the only good ways to fill the stockpile today. We've built the third way. Customers ..." ,
"link" : "http://www.Rocketmiles.com" , "city" : "Berlin" }
]
vectors = weaviate_engine.embedding_generator.generate_embeddings([doc[ "description" ] for doc in documents])
print (vectors.shape)
payload = [doc for doc in documents]
# Upload vectors and payload
weaviate_engine.upload_vectors(vectors=vectors, payload=payload)
# Build the filter and search
conditions = [
{ "key" : "city" , "match" : "Berlin" },
]
custom_filter = weaviate_engine.build_filter(conditions)
# Search for vacation-related startups in Berlin
results = weaviate_engine.search(
text = "vacations" ,
# query_filter=custom_filter,
limit =5
)
print (results)
The output is as follows:
{ 'text' : "Fueling more vacations\nWe enable our customers to travel more, travel better and travel further. 20M+ consumers stock away miles & points to satisfy their wanderlust.\nFlying around or using credit cards are the only good ways to fill the stockpile today. We've built the third way. Customers ..." , '_distance' : 0.5216099619865417
Related PostsView More
OpenAI redefines the way Agent workflows are built with Agent SDK
OpenAI has launched a new Agent SDK, which brings changes to developers building agent applications. In the past, there were many challenges. Now the new tools optimize the core logic, orchestration and interaction processes, and lower the development threshold. It includes a new Responses API, built-in tools, observability tools, etc., and will continue to be updated in the future. Come and...
Mac Studio first test: Can a top-of-the-line Mac worth 100,000 yuan run the full version of DeepSeek?
Mac Studio's first test is coming! The new generation of top-end computers equipped with M3 Ultra chips, whether they can run DeepSeek perfectly attracts attention. In the test, the noise control and heat dissipation are good, and the processing capabilities of different models vary. However, there are compatibility issues, and the factory system is not the latest. The Thunderbolt 5 port has...
OpenAI releases new tool to make building AI Agents easier
OpenAI released a new tool for building intelligent agents last night, which may change the way developers build AI applications. From the launch of multiple related products to the challenges and solutions faced, this article elaborates on its unique value in simplifying the development process. Come and click to read for details!
