Why Cutting Prompts to Save AI Cost Can Make Things Worse

Conclusion first: Randomly shortening prompts reduces accuracy and drives up retry and manual-fix costs that exceed original spend. Combine caching, model routing, forced schema outputs, and eval-set validation to achieve real savings.

1. Compression versus Accuracy Trade-off

Shortening context by 40% typically cuts tokens 30-35% but raises error rate from 6% to 22% on factual extraction tasks. Measure with a 200-example eval set before deployment.

2. Enforcing Schema Outputs

Use structured outputs to eliminate post-processing tokens.

function callWithSchema(prompt, schema) {
  return openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [{role:'user', content:prompt}],
    response_format: {type:'json_object', schema}
  });
}

3. Model Routing Decision Table

Condition	Model	Expected saving
tokens < 600	gpt-4o-mini	70%
tokens > 1200 or high complexity	gpt-4o	baseline

4. Caching Architecture

5. Retry Policy and Backoff

Implement exponential backoff with jitter; reference retry-backoff note. Limit total retries to 3 to avoid runaway spend.

6. Failure Modes

7. When NOT to Use This Approach

Skip routing and caching when latency SLA is under 800 ms or when every request contains unique user data that invalidates cache hits.

8. Implementation Path

Step 1: Build 200-example eval set. Step 2: Add schema to all prompts. Step 3: Insert routing logic. Step 4: Add cache layer with TTL. Step 5: Wire retry policy.

See also: ai-cost-and-keys, retry-backoff, lockservice-concurrency, six-minute-limit, service-invoked-too-many-times.

Final review criteria

The useful question is not how many features the automation has. It is whether the workflow can be understood, recovered, and safely rerun after something goes wrong.

• Raw input is separated from the human-facing working view.
• Each run records success, failure, processed count, and error message.
• Replaying the same input does not create duplicate results.
• Permission changes, quota errors, and external API failures are visible later.

For low-risk internal tasks, that may be enough. For customer replies, booking confirmation, inventory updates, payments, or legal records, the threshold is higher: compare Apps Script against a dedicated SaaS or a small server-backed system before relying on it.