RAG & AI Ingestion Recipes

Complete developer recipes for zero-trust document processing, offline PII redaction, whitespace-snapped chunking, vector stores, and AI client middleware.

shield Core Principles for RAG Ingestion

1. Zero Data Leaks: Redact sensitive PII, API keys, credentials, and connection strings before generating vector embeddings.
2. Local-First & Offline: Process files on-premises or on-device without cloud or network API calls.
3. Structured Chunking: Snap chunk boundaries to whitespace to avoid clipping words mid-token.

1. Microsoft.Extensions.AI Middleware (Redaction)

Wrap any IChatClient or IEmbeddingGenerator with automatic, offline PII/secret redaction middleware:

using Microsoft.Extensions.AI;
using Scrubkit;

// Wrap base client with Scrubkit redaction middleware
IChatClient redactingClient = baseClient.AsRedacting(
    level: RedactionLevel.Standard
);

// Prompts are sanitized BEFORE leaving your boundary
ChatResponse response = await redactingClient.GetResponseAsync(
    "Please analyze account for user email: john.doe@acme.com"
);

2. Folder Scanning → Redaction → Vector Chunks

Extract, scrub, and chunk mixed document folders (PDF, Word, Excel, Email, Plain Text) into indexable vector windows:

using Scrubkit;

var options = new ReadOptions
{
    Redaction = RedactionLevel.Standard,
    ComputeContentHash = true,
    MaxDegreeOfParallelism = Environment.ProcessorCount
};

var scrubber = new FolderScrubber(options);
IReadOnlyList<FileRecord> records = await scrubber.ReadAsync(@"C:\data\documents");

var chunker = new Chunker(new ChunkOptions
{
    MaxChars = 500,
    OverlapChars = 50,
    RespectWordBoundaries = true
});

foreach (var record in records)
{
    foreach (var chunk in chunker.Chunk(record))
    {
        // Upsert sanitized chunk.Text & chunk.Metadata into Vector DB
        Console.WriteLine($"[Chunk #{chunk.Index}] {chunk.Name}: {chunk.Text}");
    }
}

3. Semantic Kernel Memory Ingestion

Direct ingestion into Semantic Kernel vector memories using Scrubkit.Extensions.SemanticKernel:

using Microsoft.SemanticKernel.Memory;
using Scrubkit;

var records = await new FolderScrubber(new ReadOptions { Redaction = RedactionLevel.Standard })
    .ReadAsync(@"C:\data\kb");

var chunker = new Chunker(new ChunkOptions { MaxChars = 1000, OverlapChars = 100 });

foreach (var record in records)
{
    foreach (var chunk in chunker.Chunk(record))
    {
        await memory.SaveInformationAsync(
            collection: "knowledge-base",
            text: chunk.Text,
            id: $"{chunk.Name}_{chunk.Index}",
            description: $"Ingested from {chunk.Path}"
        );
    }
}

4. Exporting Sanitized Records to Parquet

Serialize sanitized records to columnar Parquet files using Scrubkit.Parquet for large-scale offline dataset preparation:

using Scrubkit;
using Scrubkit.Parquet;

var records = await new FolderScrubber(new ReadOptions { Redaction = RedactionLevel.Standard })
    .ReadAsync(@"C:\data\ingest");

await ParquetTableWriter.WriteAsync(records, @"C:\data\output\sanitized_ingestion.parquet");