# Decompressed Documentation
> The control layer for your vector data lifecycle. Version, rollback, and share embeddings across every RAG pipeline.

## What is Decompressed?

Decompressed is NOT a vector search engine. It is the versioning, governance, and deployment layer that sits between your embedding pipeline and your vector databases (Pinecone, Qdrant, Weaviate, Milvus).

Think of it as "Git for vectors" — immutable versions, instant rollback, incremental sync, and drift detection.

### Core Pillars
- Immutable Versioning: every change creates a new version; nothing is overwritten
- Instant Rollback: restore any previous version in seconds
- Incremental Sync: deploy only changed vectors to connected databases
- Drift Detection: spot-check destinations for external modifications
- Audit Trail: full provenance for every vector change

## Quick Start

### Python SDK

```bash
pip install decompressed-sdk
```

```python
from decompressed_sdk import DecompressedClient

client = DecompressedClient(api_key="dck_...")

# Upload a new dataset (creates version 1)
result = client.datasets.upload("vectors.parquet", name="my-embeddings")
ds_id = result.dataset_id

# Append more vectors (creates a new version)
client.datasets.append(ds_id, "more-vectors.parquet")

# Get version history
history = client.datasets.history(ds_id)

# Rollback: create a draft from a past version, then commit
dataset = client.datasets.get(ds_id)
draft = dataset.version(3).rollback(description="Rollback to v3")
committed = draft.commit()
```

### TypeScript SDK

```bash
npm install @decompressed/sdk
```

```typescript
import { DecompressedClient } from '@decompressed/sdk';

const client = new DecompressedClient({ apiKey: 'dck_...' });

const result = await client.datasets.upload('vectors.parquet', { name: 'my-embeddings' });
const dataset = await client.datasets.get(result.datasetId);
```

### CLI

```bash
pip install decompressed-cli
dcp config set-key dck_...
dcp datasets list
dcp data push my-dataset vectors.parquet
dcp sync push my-dataset my-pinecone-connector
```

## Core Concepts

### Datasets
A dataset is a versioned collection of vectors + metadata. Each dataset has a fixed dimension. Vectors are stored in immutable blocks.

### Versions
Every mutation (upload, append, delete, update) creates a new version. Versions are immutable snapshots. You can rollback to any previous version instantly.

### Syncs & Materializations
- Sync: push a dataset version to an external vector database (Pinecone, Qdrant, etc.)
- Materialization: export a dataset version as a downloadable file (Parquet, CSV)

## Connectors

### Supported Databases
- Pinecone (Serverless & Pod)
- Qdrant (Cloud & Self-hosted)
- Weaviate (Cloud & Self-hosted)
- Milvus / Zilliz (Cloud & Self-hosted)

### Connector Setup
1. Add a connector in the dashboard or via API with your vendor credentials
2. Decompressed validates the connection and index configuration
3. Sync pushes vectors to the destination; incremental sync only pushes changes

### Sync Model
- First sync: full upload of all vectors
- Subsequent syncs: incremental (only adds, updates, deletes)
- Drift detection: before incremental sync, Decompressed spot-checks the destination for external modifications
- If drift is detected, you are warned and must confirm before overwriting
- Use `--force` flag in CLI or "Force full re-upload" in UI to skip drift check

## Data Upload

### Supported Formats
- **Parquet**: vectors in a column (list of floats) + metadata columns
- **CSV**: vectors as comma-separated values in a column + metadata columns
- **NumPy (.npy)**: raw vector arrays (no metadata)

### Upload Methods
- File upload (bulk): upload a complete file via dashboard or SDK
- Append: add vectors to an existing dataset (creates new version)
- Streaming via Draft Versions: open a draft, stream batches, then commit
- CLI push: `dcp data push <dataset> <file>`

## Lifecycle Management

### Safe Experimentation
Create new versions to test different embeddings. Compare versions side-by-side. Promote the best version.

### Instant Rollbacks
If a new embedding model degrades retrieval quality, rollback to a known-good version:
```python
dataset = client.datasets.get(dataset_id)
draft = dataset.version(5).rollback(description="Rollback to v5")
committed = draft.commit()
```

### Audit & Lineage
Every version records: who created it, when, what changed, and the source of the data. Full provenance chain for compliance and debugging.

## SDK Reference

### Installation & Import
```bash
pip install decompressed-sdk
```
```python
from decompressed_sdk import DecompressedClient
# Note: the PyPI package is "decompressed-sdk" but the Python import is "decompressed_sdk"
```

### Dataset Object Fields
Fields on every Dataset returned by list() and get(). Common source of errors — use exact names:

| Field | Type | Notes |
|---|---|---|
| id | str | Unique UUID |
| name | str | Display name |
| current_version | int | Latest version number |
| dimensions | int|None | **dimensions** (plural). NOT dimension, NOT vector_dimension. |
| num_vectors | int|None | Total vectors stored |
| created_at | str|None | ISO 8601 timestamp |
| updated_at | str|None | ISO 8601 timestamp |

### Datasets — Method Parameters

**datasets.upload(file_path, name, *, compression=None, block_size=100_000, project=None, poll_interval=2.0, max_wait=600.0)**
- file_path (str|Path, required) — path to .parquet / .csv / .npy file
- name (str, required) — name for the new dataset
- compression (str) — fp16 · int8 · lossless
- block_size (int) — vectors per block, default 100,000
- Returns: UploadResult with fields: dataset_id (str), job_id (str), upload_session_id (str)

**datasets.append(dataset, file_path, *, description=None, compression=None, block_size=100_000, project=None, max_wait=600.0)**
- dataset (str, required) — dataset name or ID
- file_path (str|Path, required) — path to file
- description (str) — commit message shown in version history
- Returns: AppendResult with fields: dataset_id, job_id, append_session_id, previous_version (int|None), new_version (int|None)

**datasets.get(dataset, *, project=None)**
- dataset (str, required) — dataset name or ID
- Returns: Dataset object (see fields table above)

**datasets.list()**
- Returns: list[Dataset]

**datasets.history(dataset, *, limit=50, cursor=None)**
- dataset (str, required) — dataset name or ID
- limit (int) — max events, default 50
- Returns: list[DatasetEvent]

**datasets.delete(dataset, *, delete_file=False, project=None)**
- dataset (str, required)
- delete_file (bool) — also remove underlying storage file, default False

**datasets.update_vectors(dataset, vector_ids, new_metadata, *, project=None)**
- vector_ids (list[int], required) — integer IDs, max 1,000
- new_metadata (dict, required) — fields to set
- Returns: dict with affected_vectors (int), modified_blocks (int), operation_id (str)

**datasets.delete_vectors(dataset, vector_ids, *, confirm, project=None)**
- vector_ids (list[int], required) — integer IDs, max 5,000
- confirm (bool, required) — MUST be True or raises ValueError. Irreversible.
- Returns: dict with deleted_vectors (int), storage_freed_mb (float), operation_id (str)

**datasets.iter_blocks(dataset, *, version=None, shuffle=False, include_metadata=True, prefetch=True)**
- Yields BlockData with: block_id, vectors (np.ndarray), metadata (list[dict]), num_vectors, block_index, total_blocks

### Versions (via Dataset object)
Versions are accessed through the Dataset object returned by client.datasets.get():

**dataset.new_version(description=None, parent=None)** → DraftVersion
**dataset.version(n)** → CommittedVersion at version number n
**dataset.ref(name="main")** → CommittedVersion at a named ref
**dataset.at(timestamp)** → CommittedVersion at ISO 8601 timestamp e.g. "2024-01-28T00:00:00Z"
**dataset.latest()** → CommittedVersion for current_version

### Draft Version Methods

**draft.add_vectors(vectors, metadata=None, ids=None)**
- vectors (list[list[float]], required) — all same dimension
- metadata (list[dict]) — one dict per vector, same length as vectors
- ids (list[str]) — stable IDs; auto-assigned if omitted

**draft.remove(filter=None, ids=None)** — at least one of filter/ids required
**draft.commit(description=None)** → CommittedVersion (immutable forever)
**draft.discard()** — deletes draft without committing

### Committed Version Methods

**committed.search(query_vectors, *, top_k=10, filters=None, metric="cosine", materialization_id=None)**
- query_vectors (list[list[float]], required)
- metric — cosine · euclidean · dot
- Returns: SearchResponse with matches (list[SearchMatch]), each having: id, score, metadata

**committed.query(filters=None, *, load_vectors=False)** → DatasetQueryResponse
**committed.compare_to(other_version)** → VersionComparison with: vectors_added, vectors_removed, vectors_modified, blocks_added, blocks_removed
**committed.promote(ref="main")** — updates named ref to point at this version
**committed.rollback(description=None)** → DraftVersion reverting to this version's state

### Syncs

**syncs.from_version(version, connector_id, *, batch_size=100)**
- version (CommittedVersion, required) — object from dataset.version() or dataset.ref()
- connector_id (str, required) — connector UUID (not display name)

**syncs.set_auto_sync(dataset_id, connector_id, enabled)** — enabled is bool, required

### Connectors
- connector_id is always the UUID from Dashboard → Settings → Connectors, not the display name

### Materializations

**materializations(dataset_id)** — returns MaterializationsResource

**mats.create(name, type, config)**
- type — index · export · view
- config (dict) — e.g. {"compression": "int8"}

**mats.download(materialization_id, file_type="all")**
- file_type — blocks · index · all

### RAG Lab SDK

Strategy selector — all lab methods accept exactly one of (priority order):
1. preset_id (str) — Built-in preset ID: ghost · balanced · scholar · hybrid
2. strategy_id (str) — UUID of saved strategy (from list_strategies)
3. strategy (str) — Preset display name (e.g. "Balanced") OR saved strategy name. Backend checks preset names first, then saved strategies.

Built-in presets (from backend source of truth):
- ghost → Economy: text-embedding-3-small, 256 dims, recursive chunking, vector search
- balanced → Balanced: text-embedding-3-large, semantic chunking, vector search
- scholar → High Accuracy: text-embedding-3-large, late chunking, hybrid search + rerank (cohere-english-v3)
- hybrid → Hybrid Search: gte-large, 1024 dims, page chunking, hybrid search + rerank (jina-rerank-v2)

**client.lab.embed(texts, *, preset_id=None, strategy_id=None, strategy=None)**
- texts (list[str], required) — max 100 per call
- Returns: embeddings (list[list[float]]), model (str), dimensions (int), strategy_name (str), usage dict
- usage keys: texts_embedded, model, provider, token_count, base_cost_usd, billable_cost_usd, remaining_tokens, token_limit

**client.lab.chunk_and_embed(text, *, source=None, preset_id=None, strategy_id=None, strategy=None)**
- text (str, required) — raw document text to chunk and embed
- source (str, optional) — label attached to every chunk's metadata (e.g. filename, URL)
- Chunks using the strategy's chunking config (method, size, overlap), embeds all chunks in one batch
- Returns: chunks (list[Chunk]), model, dimensions, strategy_name, chunking_method, usage dict
- Chunk fields: chunk_index, text, embedding (list[float]), metadata dict
- metadata keys: source, start_char, end_char, strategy, model, chunking_method, chunk_size, chunk_overlap, recommended_retrieval (top_k, search_type, rerank, [hybrid_alpha])
- usage keys: chunk_count, token_count, base_cost_usd, billable_cost_usd, remaining_tokens, token_limit
- Output helpers (all verified working):
  - to_pinecone(id_prefix="chunk") → list[dict] — each: {id, values (embedding), metadata (all fields + text)} — for index.upsert(vectors=...)
  - to_qdrant(id_prefix="chunk") → list[dict] — each: {id, vector (embedding), payload (all fields + text)} — use PointStruct(**p) for each
  - to_milvus() → dict of column lists — keys: chunk_index, text, embedding, source, strategy — for collection.insert(list(result.to_milvus().values()))
  - to_records() → list[dict] — all fields flat inline: chunk_index, text, embedding, source, start_char, end_char, model, chunking_method, recommended_retrieval — for any store or json.dump

**client.lab.chunk_and_embed_many(documents, *, preset_id=None, strategy_id=None, strategy=None)**
- documents (list[dict], required) — each dict: text (str, required), source (str, optional)
- Client-side method: calls /chunk-and-embed once per document, aggregates results
- chunk_index is globally unique across corpus; metadata also includes doc_chunk_index (per-document position)
- Returns same fields as chunk_and_embed() plus document_count (int); usage is aggregated totals across all docs
- Same four output helpers as chunk_and_embed() — to_pinecone, to_qdrant, to_milvus, to_records — work identically across the full corpus

**client.lab.list_strategies()**
- Returns dict with two keys:
  - presets (list[dict]) — built-in presets, each with: id, name, model, dimensions, chunking_method, search_type
  - saved_strategies (list[dict]) — your saved strategies, each with: id, name, model, dimensions, chunking_method, search_type, usage_count

## CLI Reference

### Installation
```bash
pip install decompressed-cli
dcp config set-key <your-api-key>
```

### Dataset Commands
- `dcp datasets list` — list datasets
- `dcp datasets info <name>` — dataset details
- `dcp datasets delete <name>` — delete dataset

### Data Commands
- `dcp push <file> <dataset>` — upload/append data (file path first, then dataset name)
- `dcp pull <dataset> -o output.parquet` — download data

### Version Control
- `dcp versions log <dataset>` — version history
- `dcp versions diff <dataset> <v1> <v2>` — compare versions
- `dcp versions checkout <dataset> <version>` — rollback
- `dcp versions tag <dataset> <version> <tag>` — tag a version
- `dcp versions commit <dataset>` — commit draft version

### Sync Commands
- `dcp sync push <dataset> <connector>` — push to vector DB
- `dcp sync push <dataset> <connector> --version 3` — push specific version
- `dcp sync push <dataset> <connector> --mode full --force` — force full re-upload
- `dcp sync status <dataset>` — check sync state for all connectors

## API Base URL

Production: `https://decompressed-api-14213868466.us-central1.run.app`

All endpoints require authentication via `Authorization: Bearer dck_...` header.

## RAG Lab

RAG Lab is an experimentation environment for testing and optimizing RAG pipelines before production.

### What It Does
- Upload documents and test different chunking, embedding, and retrieval strategies
- Auto-generates gold set test questions to evaluate retrieval accuracy
- Compare strategies side-by-side on accuracy, MRR, latency, and cost
- Save winning strategies and use them via SDK in production

### Strategy Configuration
A strategy consists of:
- **Chunking**: recursive, semantic, late_chunking, or page-based
- **Embedding**: text-embedding-3-small, text-embedding-3-large, voyage-3.5-lite
- **Retrieval**: top_k, search_type (vector/keyword/hybrid), reranking

### SDK Integration

```python
from decompressed_sdk import DecompressedClient

client = DecompressedClient(api_key="dck_...")

# Embed pre-chunked texts using a built-in preset
response = client.lab.embed(texts=["Document 1", "Document 2"], preset_id="balanced")

# Or use a saved strategy name (also accepts preset display names like "Balanced")
response = client.lab.embed(texts=["Document 1", "Document 2"], strategy="My Strategy")

# Chunk and embed a single document
result = client.lab.chunk_and_embed(text=open("doc.txt").read(), source="doc.txt", preset_id="balanced")
index.upsert(vectors=result.to_pinecone(id_prefix="doc"))

# Chunk and embed an entire corpus
docs = [{"text": open(f).read(), "source": f} for f in os.listdir("./corpus")]
result = client.lab.chunk_and_embed_many(documents=docs, preset_id="balanced")
index.upsert(vectors=result.to_pinecone(id_prefix="corpus"))
```

### RAG Lab API Endpoints
- `POST /api/v1/lab/sdk/embed` — Embed pre-chunked texts
- `POST /api/v1/lab/sdk/chunk-and-embed` — Chunk raw text and embed (one document)
- `GET /api/v1/lab/sdk/strategies` — List available presets and saved strategies
- Note: chunk_and_embed_many() is client-side only — the SDK calls /chunk-and-embed per document

### Pricing
- Free: 5 experiments/day, 3 saved strategies
- Starter ($19/mo): 50 experiments/day, 10 strategies, 10K SDK embeddings/mo
- Pro ($49/mo): 200 experiments/day, 50 strategies, 100K SDK embeddings/mo
- SDK embeddings: Base model cost + 20% platform fee

## Optional Links

- [Documentation](https://decompressed.io/docs)
- [Dashboard](https://decompressed.io/dashboard)
- [RAG Lab](https://decompressed.io/protected/lab)
- [GitHub](https://github.com/decompressed)
- [PyPI: decompressed-sdk](https://pypi.org/project/decompressed-sdk/)
- [PyPI: decompressed-cli](https://pypi.org/project/decompressed-cli/)
- [npm: @decompressed/sdk](https://www.npmjs.com/package/@decompressed/sdk)