Scrubkit RAG Recipes

RAG & AI Ingestion Recipes

Complete developer recipes for zero-trust document processing, offline PII redaction, whitespace-snapped chunking, vector stores, and AI client middleware.

shield Core Principles for RAG Ingestion

1. Microsoft.Extensions.AI Middleware (Redaction)

Wrap any IChatClient or IEmbeddingGenerator with automatic, offline PII/secret redaction middleware:

using Microsoft.Extensions.AI;
using Scrubkit;

// Wrap base client with Scrubkit redaction middleware
IChatClient redactingClient = baseClient.AsRedacting(
    level: RedactionLevel.Standard
);

// Prompts are sanitized BEFORE leaving your boundary
ChatResponse response = await redactingClient.GetResponseAsync(
    "Please analyze account for user email: john.doe@acme.com"
);

2. Folder Scanning → Redaction → Vector Chunks

Extract, scrub, and chunk mixed document folders (PDF, Word, Excel, Email, Plain Text) into indexable vector windows:

using Scrubkit;

var options = new ReadOptions
{
    Redaction = RedactionLevel.Standard,
    ComputeContentHash = true,
    MaxDegreeOfParallelism = Environment.ProcessorCount
};

var scrubber = new FolderScrubber(options);
IReadOnlyList<FileRecord> records = await scrubber.ReadAsync(@"C:\data\documents");

var chunker = new Chunker(new ChunkOptions
{
    MaxChars = 500,
    OverlapChars = 50,
    RespectWordBoundaries = true
});

foreach (var record in records)
{
    foreach (var chunk in chunker.Chunk(record))
    {
        // Upsert sanitized chunk.Text & chunk.Metadata into Vector DB
        Console.WriteLine($"[Chunk #{chunk.Index}] {chunk.Name}: {chunk.Text}");
    }
}

3. Semantic Kernel Memory Ingestion

Direct ingestion into Semantic Kernel vector memories using Scrubkit.Extensions.SemanticKernel:

using Microsoft.SemanticKernel.Memory;
using Scrubkit;

var records = await new FolderScrubber(new ReadOptions { Redaction = RedactionLevel.Standard })
    .ReadAsync(@"C:\data\kb");

var chunker = new Chunker(new ChunkOptions { MaxChars = 1000, OverlapChars = 100 });

foreach (var record in records)
{
    foreach (var chunk in chunker.Chunk(record))
    {
        await memory.SaveInformationAsync(
            collection: "knowledge-base",
            text: chunk.Text,
            id: $"{chunk.Name}_{chunk.Index}",
            description: $"Ingested from {chunk.Path}"
        );
    }
}

4. Exporting Sanitized Records to Parquet

Serialize sanitized records to columnar Parquet files using Scrubkit.Parquet for large-scale offline dataset preparation:

using Scrubkit;
using Scrubkit.Parquet;

var records = await new FolderScrubber(new ReadOptions { Redaction = RedactionLevel.Standard })
    .ReadAsync(@"C:\data\ingest");

await ParquetTableWriter.WriteAsync(records, @"C:\data\output\sanitized_ingestion.parquet");

5. Qdrant Vector Database Ingestion Vector DB

Stream sanitized document chunks into Qdrant vector database with rich document metadata payload using the official Qdrant.Client:

using Qdrant.Client;
using Qdrant.Client.Grpc;
using Microsoft.Extensions.AI;
using Scrubkit;

var qdrant = new QdrantClient("localhost", port: 6334);
var scrubber = new FolderScrubber(new ReadOptions { Redaction = RedactionLevel.Standard });
var chunker = new Chunker(new ChunkOptions { MaxChars = 800, OverlapChars = 80 });

var points = new List<PointStruct>();

await foreach (var doc in scrubber.ReadStreamAsync(@"C:\KnowledgeBase"))
{
    foreach (var chunk in chunker.Chunk(doc))
    {
        // Generate embedding vector using your preferred model (e.g. OpenAI, Ollama, ONNX)
        float[] vector = await embeddingGenerator.GenerateEmbeddingVectorAsync(chunk.Text);

        var point = new PointStruct
        {
            Id = (ulong)Guid.NewGuid().GetHashCode(),
            Vectors = vector,
            Payload =
            {
                ["file_name"] = chunk.Name,
                ["file_path"] = chunk.Path,
                ["chunk_index"] = chunk.Index,
                ["type"] = chunk.TypeBucket.ToString(),
                ["text"] = chunk.Text
            }
        };
        points.Add(point);
    }
}

// Batch upsert into Qdrant collection
await qdrant.UpsertAsync("scrubkit_documents", points);

6. PostgreSQL with pgvector (Npgsql) PostgreSQL

Store redacted document chunks and vector embeddings directly in PostgreSQL using Npgsql and the pgvector extension:

using Npgsql;
using Pgvector;
using Scrubkit;

/* SQL DDL Setup:
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE doc_chunks (
    id UUID PRIMARY KEY,
    file_name TEXT NOT NULL,
    file_path TEXT NOT NULL,
    chunk_index INT NOT NULL,
    content TEXT NOT NULL,
    embedding vector(1536)
);
*/

await using var dataSource = NpgsqlDataSource.Create(connString);
var scrubber = new FolderScrubber(new ReadOptions { Redaction = RedactionLevel.Standard });
var chunker = new Chunker(new ChunkOptions { MaxChars = 1000, OverlapChars = 100 });

await using var cmd = dataSource.CreateCommand(
    "INSERT INTO doc_chunks (id, file_name, file_path, chunk_index, content, embedding) " +
    "VALUES ($1, $2, $3, $4, $5, $6)");

await foreach (var doc in scrubber.ReadStreamAsync(@"C:\Data\LegalDocs"))
{
    foreach (var chunk in chunker.Chunk(doc))
    {
        var vector = new Vector(await embeddingClient.GetEmbeddingAsync(chunk.Text));

        cmd.Parameters.Clear();
        cmd.Parameters.AddWithValue(Guid.NewGuid());
        cmd.Parameters.AddWithValue(chunk.Name);
        cmd.Parameters.AddWithValue(chunk.Path);
        cmd.Parameters.AddWithValue(chunk.Index);
        cmd.Parameters.AddWithValue(chunk.Text);
        cmd.Parameters.AddWithValue(vector);

        await cmd.ExecuteNonQueryAsync();
    }
}

7. Pinecone Vector Database Batch Ingestion Pinecone

Batch upsert sanitized RAG windows into Pinecone with automatic chunk positioning and metadata tracking:

using Pinecone;
using Scrubkit;

var pinecone = new PineconeClient(apiKey);
var index = await pinecone.GetIndexAsync("scrubkit-rag");

var records = await new FolderScrubber(new ReadOptions { Redaction = RedactionLevel.Standard })
    .ReadAsync(@"C:\Documents");

var chunker = new Chunker(new ChunkOptions { MaxChars = 600, OverlapChars = 60 });
var vectors = new List<Vector>();

foreach (var doc in records)
{
    foreach (var chunk in chunker.Chunk(doc))
    {
        float[] embedding = await embedder.EmbedAsync(chunk.Text);

        vectors.Add(new Vector
        {
            Id = $"{doc.Name}_{chunk.Index}",
            Values = embedding,
            Metadata = new Metadata
            {
                ["file_name"] = chunk.Name,
                ["chunk_index"] = chunk.Index,
                ["text"] = chunk.Text
            }
        });
    }
}

// Upsert all vectors in batches
await index.UpsertAsync(vectors);