RAG & AI Ingestion Recipes
Complete developer recipes for zero-trust document processing, offline PII redaction, whitespace-snapped chunking, vector stores, and AI client middleware.
shield Core Principles for RAG Ingestion
- 1. Zero Data Leaks: Redact sensitive PII, API keys, credentials, and connection strings before generating vector embeddings.
- 2. Local-First & Offline: Process files on-premises or on-device without cloud or network API calls.
- 3. Structured Chunking: Snap chunk boundaries to whitespace to avoid clipping words mid-token.
1. Microsoft.Extensions.AI Middleware (Redaction)
Wrap any IChatClient or IEmbeddingGenerator with automatic, offline PII/secret redaction middleware:
using Microsoft.Extensions.AI; using Scrubkit; // Wrap base client with Scrubkit redaction middleware IChatClient redactingClient = baseClient.AsRedacting( level: RedactionLevel.Standard ); // Prompts are sanitized BEFORE leaving your boundary ChatResponse response = await redactingClient.GetResponseAsync( "Please analyze account for user email: john.doe@acme.com" );
2. Folder Scanning → Redaction → Vector Chunks
Extract, scrub, and chunk mixed document folders (PDF, Word, Excel, Email, Plain Text) into indexable vector windows:
using Scrubkit; var options = new ReadOptions { Redaction = RedactionLevel.Standard, ComputeContentHash = true, MaxDegreeOfParallelism = Environment.ProcessorCount }; var scrubber = new FolderScrubber(options); IReadOnlyList<FileRecord> records = await scrubber.ReadAsync(@"C:\data\documents"); var chunker = new Chunker(new ChunkOptions { MaxChars = 500, OverlapChars = 50, RespectWordBoundaries = true }); foreach (var record in records) { foreach (var chunk in chunker.Chunk(record)) { // Upsert sanitized chunk.Text & chunk.Metadata into Vector DB Console.WriteLine($"[Chunk #{chunk.Index}] {chunk.Name}: {chunk.Text}"); } }
3. Semantic Kernel Memory Ingestion
Direct ingestion into Semantic Kernel vector memories using Scrubkit.Extensions.SemanticKernel:
using Microsoft.SemanticKernel.Memory; using Scrubkit; var records = await new FolderScrubber(new ReadOptions { Redaction = RedactionLevel.Standard }) .ReadAsync(@"C:\data\kb"); var chunker = new Chunker(new ChunkOptions { MaxChars = 1000, OverlapChars = 100 }); foreach (var record in records) { foreach (var chunk in chunker.Chunk(record)) { await memory.SaveInformationAsync( collection: "knowledge-base", text: chunk.Text, id: $"{chunk.Name}_{chunk.Index}", description: $"Ingested from {chunk.Path}" ); } }
4. Exporting Sanitized Records to Parquet
Serialize sanitized records to columnar Parquet files using Scrubkit.Parquet for large-scale offline dataset preparation:
using Scrubkit; using Scrubkit.Parquet; var records = await new FolderScrubber(new ReadOptions { Redaction = RedactionLevel.Standard }) .ReadAsync(@"C:\data\ingest"); await ParquetTableWriter.WriteAsync(records, @"C:\data\output\sanitized_ingestion.parquet");
5. Qdrant Vector Database Ingestion Vector DB
Stream sanitized document chunks into Qdrant vector database with rich document metadata payload using the official Qdrant.Client:
using Qdrant.Client; using Qdrant.Client.Grpc; using Microsoft.Extensions.AI; using Scrubkit; var qdrant = new QdrantClient("localhost", port: 6334); var scrubber = new FolderScrubber(new ReadOptions { Redaction = RedactionLevel.Standard }); var chunker = new Chunker(new ChunkOptions { MaxChars = 800, OverlapChars = 80 }); var points = new List<PointStruct>(); await foreach (var doc in scrubber.ReadStreamAsync(@"C:\KnowledgeBase")) { foreach (var chunk in chunker.Chunk(doc)) { // Generate embedding vector using your preferred model (e.g. OpenAI, Ollama, ONNX) float[] vector = await embeddingGenerator.GenerateEmbeddingVectorAsync(chunk.Text); var point = new PointStruct { Id = (ulong)Guid.NewGuid().GetHashCode(), Vectors = vector, Payload = { ["file_name"] = chunk.Name, ["file_path"] = chunk.Path, ["chunk_index"] = chunk.Index, ["type"] = chunk.TypeBucket.ToString(), ["text"] = chunk.Text } }; points.Add(point); } } // Batch upsert into Qdrant collection await qdrant.UpsertAsync("scrubkit_documents", points);
6. PostgreSQL with pgvector (Npgsql) PostgreSQL
Store redacted document chunks and vector embeddings directly in PostgreSQL using Npgsql and the pgvector extension:
using Npgsql; using Pgvector; using Scrubkit; /* SQL DDL Setup: CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE doc_chunks ( id UUID PRIMARY KEY, file_name TEXT NOT NULL, file_path TEXT NOT NULL, chunk_index INT NOT NULL, content TEXT NOT NULL, embedding vector(1536) ); */ await using var dataSource = NpgsqlDataSource.Create(connString); var scrubber = new FolderScrubber(new ReadOptions { Redaction = RedactionLevel.Standard }); var chunker = new Chunker(new ChunkOptions { MaxChars = 1000, OverlapChars = 100 }); await using var cmd = dataSource.CreateCommand( "INSERT INTO doc_chunks (id, file_name, file_path, chunk_index, content, embedding) " + "VALUES ($1, $2, $3, $4, $5, $6)"); await foreach (var doc in scrubber.ReadStreamAsync(@"C:\Data\LegalDocs")) { foreach (var chunk in chunker.Chunk(doc)) { var vector = new Vector(await embeddingClient.GetEmbeddingAsync(chunk.Text)); cmd.Parameters.Clear(); cmd.Parameters.AddWithValue(Guid.NewGuid()); cmd.Parameters.AddWithValue(chunk.Name); cmd.Parameters.AddWithValue(chunk.Path); cmd.Parameters.AddWithValue(chunk.Index); cmd.Parameters.AddWithValue(chunk.Text); cmd.Parameters.AddWithValue(vector); await cmd.ExecuteNonQueryAsync(); } }
7. Pinecone Vector Database Batch Ingestion Pinecone
Batch upsert sanitized RAG windows into Pinecone with automatic chunk positioning and metadata tracking:
using Pinecone; using Scrubkit; var pinecone = new PineconeClient(apiKey); var index = await pinecone.GetIndexAsync("scrubkit-rag"); var records = await new FolderScrubber(new ReadOptions { Redaction = RedactionLevel.Standard }) .ReadAsync(@"C:\Documents"); var chunker = new Chunker(new ChunkOptions { MaxChars = 600, OverlapChars = 60 }); var vectors = new List<Vector>(); foreach (var doc in records) { foreach (var chunk in chunker.Chunk(doc)) { float[] embedding = await embedder.EmbedAsync(chunk.Text); vectors.Add(new Vector { Id = $"{doc.Name}_{chunk.Index}", Values = embedding, Metadata = new Metadata { ["file_name"] = chunk.Name, ["chunk_index"] = chunk.Index, ["text"] = chunk.Text } }); } } // Upsert all vectors in batches await index.UpsertAsync(vectors);