Batch Api Orchestrator

# Create a "Batch Api Orchestrator" AI Skill

I want you to help me set up a reusable AI skill that I can use in future conversations. Read the complete skill definition below, then help me install it.

## Complete Skill Definition

# Batch API Orchestrator

> **Category:** Engineering
> **Domain:** AI Engineering

## Overview

Decide when to run LLM work through an asynchronous batch API versus realtime/streaming, then design the job so it is cheap, idempotent, and resilient to partial failure. Batch APIs typically cost roughly half of realtime in exchange for higher latency (results arrive over minutes to hours, not milliseconds), which makes them ideal for evals, backfills, embeddings, and bulk classification/extraction — and wrong for anything a human is waiting on. This skill is model- and vendor-agnostic: it reasons about the batch *pattern*, not any one provider's API.

## Clarify First

Before recommending or designing a batch job, confirm these inputs. If any is unknown or vague, ASK — do not assume:

- [ ] **Latency tolerance** — is a human waiting (interactive), or can results land in minutes/hours? (sets `--latency-tolerance` and the batch-vs-realtime verdict)
- [ ] **Volume & token shape** — how many requests, and the average input/output tokens each? (sets `--requests`, `--avg-input-tokens`, `--avg-output-tokens` for the cost estimate)
- [ ] **Pricing & discount** — your realtime per-token prices and the batch discount your vendor offers (sets `--realtime-input-price`, `--realtime-output-price`, `--batch-discount`; defaults are neutral placeholders, not real prices)

Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions.

## Quick Start

```bash
cd engineering/batch-api-orchestrator

# 1. Should this be batch or realtime, and what does it cost?
python scripts/batch_cost_estimator.py \
  --requests 50000 --avg-input-tokens 800 --avg-output-tokens 200 \
  --realtime-input-price 3.0 --realtime-output-price 15.0 \
  --batch-discount 0.5 --latency-tolerance hours

# 2. Plan the chunking / idempotency / retry strategy for the job
python scripts/batch_job_planner.py \
  --total-items 50000 --max-batch-size 10000 --retry-policy exponential --json
```

## Tools Overview

| Tool | Purpose | Key Flags |
|------|---------|-----------|
| `scripts/batch_cost_estimator.py` | Compare realtime vs batch cost, show savings, and recommend batch or realtime given latency tolerance | `--requests`, `--avg-input-tokens`, `--avg-output-tokens`, `--realtime-input-price`, `--realtime-output-price`, `--batch-discount`, `--latency-tolerance`, `--json` |
| `scripts/batch_job_planner.py` | Produce a chunking + idempotency + partial-failure plan for a bulk job | `--total-items`, `--max-batch-size`, `--retry-policy`, `--max-retries`, `--json` |

Both scripts: Python 3 standard library only, argparse CLI, `--json` and human-readable output. Run `--help` for full usage.

## Workflows

### 1. Decide batch vs realtime, then size the cost

1. Gather volume and token shape (`--requests`, `--avg-input-tokens`, `--avg-output-tokens`).
2. Plug in *your* vendor prices and batch discount — never assume them.
3. Run `batch_cost_estimator.py` with the real `--latency-tolerance` (`realtime`, `minutes`, or `hours`).
4. Read the verdict: if work is interactive, the tool recommends realtime regardless of savings; otherwise it quantifies the batch savings.
5. Sanity-check against the decision tree in `references/batch-patterns-and-decision-tree.md`.

### 2. Design a resilient bulk job

1. Run `batch_job_planner.py` with `--total-items`, `--max-batch-size`, and a `--retry-policy`.
2. Adopt the generated idempotency-key scheme so re-submitting a chunk never double-charges or double-writes.
3. Wire result reconciliation: match every output back to its request id, and collect the unmatched into a dead-letter set.
4. Apply the partial-failure handling (retry only failed items, never the whole batch) from the reference.
5. Choose polling vs callback for completion, per the reference guidance.

## Reference Documentation

- **[references/batch-patterns-and-decision-tree.md](references/batch-patterns-and-decision-tree.md)** — when-to-batch decision tree; job design (idempotency keys, partial failures, reconciliation, polling vs callback); fitting use cases (evals, backfills, embeddings, bulk classification/extraction); and anti-patterns such as batching interactive requests.
- **[references/cost-and-throughput-economics.md](references/cost-and-throughput-economics.md)** — the cost/throughput tradeoff in depth: the ~half-cost rule of thumb, throughput vs latency, queueing, chunk sizing, and how to model the break-even between a faster realtime path and a cheaper batch path.

## Common Patterns

- **Batch the patient, stream the impatient** — if no human is blocked on the result, default to batch for the cost win; reserve realtime/streaming for interactive UX.
- **Idempotency key per item** — derive a stable key (e.g. hash of input + job version) so retries and re-submissions are safe and never double-billed.
- **Retry the item, not the batch** — on partial failure, re-enqueue only the failed request ids; resubmitting the whole chunk wastes money and re-runs successes.
- **Reconcile by request id** — never rely on output ordering; join results back to inputs by id and route the unmatched to a dead-letter queue for inspection.
- **Right-size chunks** — split by the vendor's max-batch limit and by your own blast-radius tolerance, not into one giant job whose failure is all-or-nothing.
- **Embeddings and evals are the sweet spot** — large, latency-insensitive, embarrassingly parallel workloads capture the full batch discount with the least risk.

---

## What I Need You to Do

First, detect which platform I'm using (Claude.ai, ChatGPT, etc.) and follow the matching instructions below.

### If I'm on Claude.ai:

Walk me through these exact steps:

1. **Create the Project:** Tell me to go to **claude.ai > Projects > Create project** and name it **"Batch Api Orchestrator"**

2. **Add Project Knowledge:** Give me the COMPLETE skill definition above as a single copyable text block inside a code fence. Tell me to click **"Add content" > "Add text content"** inside the project, then paste that entire block. Do NOT say "paste from above" -- give me the actual text to copy right there.

3. **Set Custom Instructions:** Tell me to open project settings and paste this exact instruction:
   "You are an expert Batch Api Orchestrator in the Engineering domain. Use the project knowledge as your expertise. Follow the workflows, frameworks, and templates defined there. Always provide specific, actionable output."

4. **Test It:** Give me a specific sample prompt I can use inside the new project to verify it works. Pick a real task from the skill's workflows.

### If I'm on ChatGPT:

Walk me through these exact steps:

1. **Create a Custom GPT:** Tell me to go to **chatgpt.com > Explore GPTs > Create**
2. **Configure it:**
   - Name: **"Batch Api Orchestrator"**
   - Description: "This skill should be used when the user asks to "batch LLM requests", "should I use the batch API", "estimate batch vs realtime cost", "design a bulk LLM job", or "process thousands of prompts cheaply"."
   - Instructions: Give me the COMPLETE skill definition above as a single copyable text block inside a code fence to paste into the Instructions field. Do NOT say "paste from above."
3. **Test It:** Give me a sample prompt to verify it works.

### If I'm on another platform:
Ask which tool I'm using and adapt the instructions accordingly.

## Important
- Always provide the full skill text in a ready-to-copy code block -- never tell me to "scroll up" or "copy from above"
- Keep the setup steps simple and numbered
- After setup, test it with me using a real workflow from the skill

Source: https://github.com/borghei/Claude-Skills/tree/main/engineering/batch-api-orchestrator/SKILL.md