Prompt engineering only goes so far. The real efficiency gains are in what you feed the model.
You have a good prompt. You have a capable model. You paste in a 35-page strategy deck and ask for a competitive summary. The answer comes back and it’s… fine. It captures the obvious sections, misses one of the most important tables, and attributes a figure to the wrong company. You spend the next fifteen minutes asking follow-up questions to recover what should have been a first-pass answer.
The prompt wasn’t the problem. The deck wasn’t the problem either.
The problem was what happened between the file and the model.
Video introduction of our File to Markdown & JSON tool, Converter.
Where tokens actually go
Most conversations about saving tokens start and end at the prompt layer. Write shorter instructions. Avoid redundancy. Use system prompts. All of that is sound advice, and if you want to get serious about prompt structure, there are well-tested frameworks for that too. But none of it addresses the part that often costs the most: the source material itself.
When you send a PDF to a language model, you are not sending the document as you see it. You are sending whatever the extraction process was able to recover: a stream of text with page numbers embedded in surprising places, headers and footers repeated across dozens of pages, tables that have been flattened into ambiguous whitespace, footnotes that float mid-paragraph, and reading order that was never designed to be linear.
The model receives all of that. It has to spend attention distinguishing layout artifact from actual content before it can start doing anything useful. You pay for that attention in tokens and in reasoning quality.
This is the input problem. It is distinct from the prompt problem, and for many real workflows, it is the bigger one.
Semantic density: the metric that matters more than token count
Token count is a proxy metric. What actually determines the quality of your context window is semantic density, in other words how much meaningful signal each token carries relative to the noise surrounding it.
A 4,000-token chunk extracted from a scanned PDF might carry the same informational content as a 1,200-token Markdown version of the same document. The raw extraction is not “more complete.” It is louder. The model cannot reason more deeply because it has more tokens; it reasons less precisely because useful tokens are scattered across a field of formatting debris.
Think of it this way: a context window is not a hard drive. It is closer to working memory. Filling it with layout noise is the equivalent of asking someone to solve a problem while reading from a document full of typographic errors and misaligned columns — technically all the information is there, but the cognitive overhead of recovering it reduces what is left for actual thinking.
The question worth asking before you write your next prompt is: how dense is the input I’m about to send?
If the answer is “I haven’t thought about that,” you have found the optimization you were looking for.
The PDF problem (which is not really about PDF)
PDF gets blamed for this a lot, and it deserves most of the blame. PDF is an excellent delivery format, designed to preserve visual appearance across every device and printer, which means it prioritizes layout fidelity over semantic structure. There is no inherent concept of “this is a heading” or “these four numbers belong to the same table row.” There is only position on a page.
When a language model encounters raw PDF text, it is doing archaeology. It is inferring structure from spatial hints encoded for human visual processing, not machine reading. Sometimes it gets it right. Often enough, it misses the table that mattered, inverts two related values, or treats a recurring page header as a new section marker.
But the problem is not specific to PDF. Slide decks have the same mismatch at a different scale. A presentation is built for a room. Each slide has a few words visible on screen, visual groupings that only make sense visually, and decorative text that means nothing when extracted linearly. When you send a PPTX to a model without any conversion step, you are often sending all of that together: a stream of labels, fragments, and layout ghosts that the model has to interpret as a coherent argument.
Scanned documents add another variable: OCR quality. When the extraction layer degrades, currency values become ambiguous, dates stay locale-specific, and table line items lose their relational structure. The model can often patch those gaps by inference, but every inference costs context. The repair work displaces the reasoning work.
The format of your source material is a first-class workflow variable. Treating it as neutral (as just “the document”) leaves significant efficiency on the table.
The difference is not theoretical. Here is what the same invoice actually looks like at each end of the pipeline:
What Markdown actually does for a model
Markdown is not a compression format. It is a structure-preservation format, and the distinction matters.
When you convert a document to clean Markdown, you are not just making it shorter (though it often does). You are making the hierarchy explicit. Headings become headings. Tables remain tables with clear column relationships. Code blocks are separated from prose. Lists are lists, not paragraph-wrapped bullet points that the model has to parse as such.
Language models read Markdown well, not because they were instructed to prefer it, but because their training data contained enormous amounts of it. The structural cues are familiar. A ## heading signals a section boundary as clearly as any system instruction would, and it costs two characters.
Markdown is the right default when a human still enters the loop at some point: reviewing, editing, summarizing, building a Q&A layer over a document set, preparing a briefing. The structure helps the model, and it stays legible enough that whoever is checking the output can actually do that comfortably.
It is also a strong default for feeding context into agentic workflows, which matters as AI pipelines become the standard way to process large document volumes rather than an experimental approach.
What JSON does differently
JSON solves a different problem. It is not about readability, it is about eliminating ambiguity in extraction.
When the downstream step is a program or another model expecting structured input, “readable” stops being the goal. The goal is predictability. A well-formed JSON object has named fields. The date is always in the date field. The total is always in the total field. The line items are always an array. Downstream code does not have to guess where things are.
This changes where the savings happen. Markdown saves tokens at the input layer: cleaner context, less reconstruction overhead for the model. JSON saves tokens further downstream: fewer follow-up prompts to re-extract specific values, fewer retries when a field was misread, fewer repair steps in a pipeline that expected a schema and received prose.
The useful heuristic is to ask what happens dopo the conversion:
- If the output goes to a person reviewing or summarizing: markdown.
- If the output goes to a model reasoning over a document: markdown.
- If the output goes to a program, database, or another agent expecting data: JSON.
- If it feeds into an automated multi-step pipeline: JSON, or Markdown as an intermediate.
Neither is universally better. The choice depends entirely on the next step in the workflow.
Where this stops being a UI choice and becomes infrastructure
So far this sounds like something you do once, manually, before a big project. Convert a document, get cleaner output, move on.
That is one valid use. But the real leverage appears when you design the conversion step as a persistent layer in an automated workflow. Not a one-off cleanup, but a preprocessing stage that every document passes through before it reaches any model. This is the shift from convenience to infrastructure, and it is where teams building AI-driven operations should pay close attention.
If you are still mapping out where AI fits in your organization before designing pipelines (understanding which processes have the most automation potential and where the real bottlenecks are), that diagnostic work deserves its own structured approach. Tools like the Digital Maturity Scorecard and the Flowaudit are designed exactly for that: identifying where effort is being wasted and where a well-placed automation layer creates the most leverage. The document preprocessing question usually surfaces quickly once you start mapping real workflows at that level of detail.
The compounding cost in agentic pipelines
Multi-step AI workflows (pipelines that chain model calls, tools, and external services) have a compounding sensitivity to input quality. In a single-model interaction, a noisy input costs you one mediocre answer. In an agentic pipeline, a noisy input at step one propagates through every downstream step. A misread table in a financial report can produce incorrect extraction at step two, a wrong recommendation at step three, and a hallucinated follow-up at step four.
A reliable preprocessing step at the entry point of the pipeline is not a nice-to-have. It is the kind of architectural decision that prevents an entire class of errors from compounding downstream. If you want a broader view of how to design these multi-step AI systems (integrating data ingestion, decision logic, and feedback loops) the AI workflow automation framework covers a bit of that part.
Claude Code and the preprocessing layer
If you are building agent workflows with Claude Code, the conversion layer fits naturally at a few points in the architecture: as a skill that any session can invoke before working on a document, as a hook that pre-processes attachments before they enter context, or as a step in a longer slash-command sequence.
A concrete example: imagine a /brief-from-deck skill. A user passes a PPTX path, the skill calls a conversion endpoint to get clean Markdown, and passes only the structured output into the main context. The model never sees the raw slide deck; it sees headings, bullet points, and tables, properly formatted and ready to reason over.
# In a Claude Code skill: /brief-from-deck
# 1. Convert the deck to clean Markdown via REST API
curl -s -X POST https://converter.dishine.it/v1/convert \
-H "Authorization: Bearer $DSH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"filename": "Q2_strategy.pptx",
"contentBase64": "'$(base64 -i Q2_strategy.pptx)'",
"format": "markdown",
"language": "en"
}' | jq -r '.output' > /tmp/deck_clean.md
# 2. Feed the clean output into context — not the raw file
The model works with /tmp/deck_clean.md. Context window saved. Reasoning quality up. The deck’s layout noise stayed outside the context entirely.
This is one layer of the system, and it pairs naturally with the prompt layer above it. A clean input and a well-structured prompt are not competing optimizations; they work at different levels of the same pipeline. If you want the prompt side to be as rigorous as the input side, Prompt Workshop is a perfect tool to create, lint, improve and test you prompt on different model before to send in production.
SHINE Code: wiring it together
If you are using SHINE Code to configure your Claude Code environment for agency or team use, preprocessing integration sits naturally at the skill or hook layer, the same architectural layer where SHINE handles client memory injection, draft-before-send guardrails, and context monitoring. A conversion step becomes one more piece of infrastructure that the model does not have to think about because it is handled before the model ever sees the input.
L'architettura a full SHINE Code article and the documentation in the github covers how to build these kinds of skills and hooks in detail. If you are running Claude Code at any meaningful scale, it is worth reading alongside this one.
The REST API: making conversion programmable
The conversion step only becomes real infrastructure when it is programmable. A web UI is useful for one-off work. A REST API is what you integrate into a pipeline.
The API accepts a file as base64-encoded content, a target format (markdown or json), and an optional language hint for improved OCR accuracy. The response is synchronous and simple enough to pipe directly into any subsequent step:
// POST /v1/convert
{
"filename": "invoice_scan.pdf",
"contentBase64": "",
"format": "json",
"language": "en"
}
// Response
{
"output": "{ \"vendor\": \"Acme Corp\", \"total\": 4280.00, \"date\": \"2025-04-01\" }",
"format": "json",
"tier": "pro",
"model": "gemini-2.5-flash"
}
The URL endpoint (/v1/fetch-url) follows the same pattern, useful when your pipeline sources from web content rather than uploaded files. The URL is fetched, rendered if needed (including JavaScript-heavy pages on Pro tier), and returned as clean Markdown.
For teams processing documents repeatedly (weekly report batches, recurring invoice extractions, ongoing research feeds) this is where the tool becomes part of the stack rather than a periodic convenience.
What diShine Converter actually is
dishine Convert was designed as a preprocessing tool to cover this gap between raw source material and model-ready input.
It accepts PDFs, DOCX, PPTX, XLSX, plain text, Markdown, CSV, HTML, JSON, XML, YAML, and image formats including JPG, PNG, WEBP, and GIF. The output is either clean GitHub-Flavored Markdown or schema-validated JSON, depending on what your workflow needs next.
For scanned documents and images, it applies OCR with language hints for English, French, Italian, Spanish, German, Portuguese, Dutch, Japanese, and Chinese, not as a multilingual marketing feature, but because OCR quality is highly language-dependent, and better extraction means fewer downstream repair steps and fewer ambiguous tokens in the final context.
Free tier: individual file conversion, no account required. Designed for trying the approach on real documents before committing to workflow integration.
Pro tier: 100 MB file limit (vs 20 MB on free), batch upload with ZIP output, cloud-synced conversion history, full API access, and enhanced URL rendering for JS-heavy pages via Firecrawl. The API access is the main reason to be on Pro if you are building automation.
This converter is not the only tool in this space. marker, pymupdf4llm, Apache Tika, and Pandoc all handle variants of the same preprocessing work. The reason to use a hosted API endpoint rather than running local tooling is operational simplicity: no dependencies to manage, no environment to maintain, and server-side rendering for dynamic URLs that local tools cannot handle. If you are already running your own document pipeline infrastructure and want full control over the stack, the local tooling route makes more sense. If you want the preprocessing layer to be a reliable service rather than a maintenance obligation, a hosted endpoint is the faster path.
A note on compliance when documents enter AI workflows
Files processed through the converter are handled inside a Supabase edge function and are not stored. Filename, file size, and format are logged for usage tracking; document content is not retained. Account history and API keys are isolated with row-level security per user.
For teams operating under EU AI Act or GDPR constraints (which increasingly applies to any organization running AI on documents that contain personal data) the compliance picture around AI tools is broader than any single tool’s privacy policy. If you are deploying AI document workflows in a regulated context, compliance.dishine.it is built for exactly that audit: mapping your AI usage against EU AI Act requirements, GDPR obligations, and IP considerations. Similarly, if your workflows touch web content and your organization has cookie compliance obligations, the cookie-audit tool covers the browser-layer side of that picture.
For sensitive documents specifically (legal, financial, medical), normal due diligence applies regardless of what the tool states. Preprocessing infrastructure should be evaluated with the same care as any other step that touches private data. Self-hosting the open-source version is an option if the workflow requires it.
The practical takeaway
The mental model shift that makes this practical is straightforward: think about your input layer before your prompt layer.
Before you spend another hour tuning a system prompt, ask what the model is actually receiving. Is it seeing the document, or is it seeing extraction artifacts wrapped around the document? Are your tables still tables? Are your headings meaningful section markers, or random capitalized text in a stream?
- If you are writing one-off prompts over documents: convert your most commonly used source formats and compare output quality. The difference is usually visible immediately.
- If you are building automated workflows or agent systems: design the preprocessing step as infrastructure from the beginning. A conversion endpoint at the entry point of the pipeline is cheaper to add now than to retrofit after compounding input quality errors have already caused problems downstream.
- If you are not sure where document processing fits in your broader AI adoption picture, digital-map.dishine.it is a good place to start: a structured scorecard that maps your organization’s actual digital capability across data, AI readiness, governance, and performance dimensions.
The goal is always the same: give the model the minimum input required to do maximum reasoning. That means less noise, more structure, and a document format designed for machines to read, not just for humans to look at.
dishine Convert is available with a free tier requiring no account. The REST API is documented at converter.dishine.it/api. If you are building agent workflows with Claude Code, SHINE Code covers how to wire preprocessing steps into skills and hooks at the session level, and Prompt Workshop covers the prompt layer that sits above clean inputs.
