Stop Using Fixed Chunk Sizes in Enterprise RAG.

Most RAG tutorials recommend a fixed chunk size, such as 512 tokens with 128-token overlap. In production, that approach often breaks business context and reduces retrieval quality. I prefer semantic chunking—splitting documents by logical sections such as policy rules, API endpoints, contract clauses, or code functions. Each chunk should represent one complete business concept.

Ablikim Nur

July 01, 2026 • 2 min read

One of the first things you'll see in most RAG tutorials is:

Chunk Size = 512
Overlap = 128

The problem?

Those numbers are usually copied from examples—not derived from your data.

After working with enterprise documents, I've found that semantic boundaries matter far more than arbitrary token counts.

Imagine chunking an HR policy like this:

Chunk 1
-------------------------
...Employees are eligible for
annual leave after...
Chunk 2
-------------------------
...12 months of continuous employment.
Exceptions apply when...

The most important sentence has just been split in half.

Now the embedding no longer represents a complete business rule.

Instead, I prefer chunking around document structure, not token limits.

For example:

Employee Leave Policy

├── Purpose
├── Eligibility
├── Annual Leave
├── Exceptions
├── Approval Process
└── References

Each section becomes an independent semantic unit.

Even better, different document types deserve different chunking strategies.

API Documentation
→ One endpoint per chunk

Contracts
→ One clause per chunk

Policies
→ One section per chunk

Knowledge Base
→ One topic per chunk

Source Code
→ One function or class per chunk

There is no universal chunk size.

The optimal chunk is the smallest unit that still preserves a complete business meaning.

Only after defining semantic boundaries should you consider:

• Maximum token size

• Overlap

• Embedding model

• Retrieval strategy

In production RAG systems, retrieval quality is often limited by how you chunk your documents, not by which LLM you're using.

A better model can't recover information that was poorly segmented in the first place.

My rule of thumb:

Chunk for meaning first. Chunk for tokens second.

How are you chunking documents in your RAG pipelines?

By fixed token count, document structure, or something else?

#AzureAI #AzureAISearch #RAG #LLM #OpenAI #DotNet #AIEngineering #SoftwareArchitecture #EnterpriseAI #VectorSearch