CrewAI

The sie-crewai package provides sparse embedding support for CrewAI. Use SIESparseEmbedder alongside CrewAI’s OpenAI-compatible dense embeddings for hybrid search.

Installation

pip install sie-crewai

This installs sie-sdk and crewai as dependencies.

Start the Server

# Docker (recommended)
docker run -p 8080:8080 ghcr.io/superlinked/sie:latest

# Or with GPU
docker run --gpus all -p 8080:8080 ghcr.io/superlinked/sie:latest

Embedders

SIE integrates with CrewAI through two embedding approaches:

1. Dense embeddings - Use SIE’s OpenAI-compatible API with CrewAI’s built-in embedder config
2. Sparse embeddings - Use SIESparseEmbedder for hybrid search workflows

Dense Embeddings

Configure CrewAI to use SIE’s OpenAI-compatible endpoint:

from crewai import Crew

crew = Crew(
    agents=[...],
    tasks=[...],
    embedder={
        "provider": "openai",
        "config": {
            "api_base": "http://localhost:8080/v1",
            "model": "BAAI/bge-m3"
        }
    }
)

Sparse Embeddings

Use SIESparseEmbedder for sparse vectors in hybrid search:

from sie_crewai import SIESparseEmbedder

sparse_embedder = SIESparseEmbedder(
    base_url="http://localhost:8080",
    model="BAAI/bge-m3"
)

# Embed documents
sparse_vectors = sparse_embedder.embed_documents([
    "Machine learning uses algorithms to learn from data.",
    "The weather is sunny today."
])
print(sparse_vectors[0].keys())  # dict_keys(['indices', 'values'])

# Embed a query (uses is_query=True for asymmetric models)
query_vector = sparse_embedder.embed_query("What is machine learning?")

Full Example

Complete example using SIE embeddings with a CrewAI agent for hybrid search:

from crewai import Agent, Crew, Task
from sie_crewai import SIESparseEmbedder

# 1. Configure dense embeddings via OpenAI-compatible API
embedder_config = {
    "provider": "openai",
    "config": {
        "api_base": "http://localhost:8080/v1",
        "model": "BAAI/bge-m3"
    }
}

# 2. Set up sparse embedder for hybrid search
sparse_embedder = SIESparseEmbedder(
    base_url="http://localhost:8080",
    model="BAAI/bge-m3"
)

# 3. Prepare your corpus with both dense and sparse embeddings
corpus = [
    "Machine learning is a branch of artificial intelligence.",
    "Neural networks are inspired by biological neurons.",
    "Deep learning uses multiple layers of neural networks.",
]

# Get sparse embeddings for your vector database
sparse_vectors = sparse_embedder.embed_documents(corpus)
# Store sparse_vectors in your vector DB (Qdrant, Weaviate, etc.)

# 4. Create a research agent
researcher = Agent(
    role="Research Analyst",
    goal="Find and analyze information from the knowledge base",
    backstory="Expert at finding relevant information using semantic search.",
    verbose=True
)

# 5. Define the research task
research_task = Task(
    description="Search the knowledge base for information about deep learning.",
    expected_output="A summary of findings about deep learning.",
    agent=researcher
)

# 6. Create and run the crew
crew = Crew(
    agents=[researcher],
    tasks=[research_task],
    embedder=embedder_config,
    verbose=True
)

result = crew.kickoff()
print(result)

Configuration Options

SIESparseEmbedder

Parameter	Type	Default	Description
base_url	str	http://localhost:8080	SIE server URL
model	str	BAAI/bge-m3	Model to use for sparse embeddings
gpu	str	None	Target GPU type for routing
options	dict	None	Model-specific options
timeout_s	float	180.0	Request timeout in seconds

What’s Next

• Encode Text - dense and sparse embedding details
• Model Catalog - all supported embedding models