Prompt Engineer Toolkit

# Create a "Prompt Engineer Toolkit" AI Skill

I want you to help me set up a reusable AI skill that I can use in future conversations. Read the complete skill definition below, then help me install it.

## Complete Skill Definition

# Prompt Engineer Toolkit — Production Prompt Engineering

The complete lifecycle for production prompts: design patterns that work, testing frameworks that catch regressions, versioning systems that track changes, and evaluation rubrics that replace subjective "looks good" with measurable quality. This treats prompts as production code with the same rigor — not clever tricks.

**Tags:** prompt engineering, chain-of-thought, few-shot, evaluation, testing, prompt versioning

## Core Capabilities

- **Prompt patterns** — 6-layer system-prompt architecture, chain-of-thought (standard, scratchpad, self-consistency), few-shot design + dynamic selection, JSON/section output structuring, decomposition pipelines, calibration (temperature + confidence levels).
- **Testing framework** — test-case structure, suite composition (40/30/15/15), a 5-dimension automated scoring rubric with a weighted formula, and a regression-testing protocol.
- **Versioning** — version-control layout, changelog format with rationale/baselines/rollback, and a prompt-diff risk checklist.
- **Failure-mode catalog** — instruction override, format drift, sycophancy, verbosity, hallucination, anchoring, lost-in-the-middle, each with fixes.
- **Lifecycle workflows** — design a prompt, debug a degraded prompt, migrate a prompt to a new model.

## When to Use

- Designing production prompts or building a prompt library.
- Running A/B tests or regression tests on prompt variants.
- Versioning prompts and gating changes on test scores.
- Debugging a degraded prompt or migrating prompts across models.

## Clarify First

Before designing or testing the prompt, confirm these inputs. If any is unknown or vague, ASK — do not assume:

- [ ] **Task & definition of "good"** — what the prompt must produce and how success is judged (drives the 5-dimension evaluation rubric)
- [ ] **Target model** — calibration (temperature, few-shot count) and migration paths differ by model
- [ ] **Lifecycle stage** — design new / debug a degraded prompt / migrate to a new model (selects the workflow)

Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.

## Tools

| Tool | Purpose | Command |
|------|---------|---------|
| `eval_scorer.py` | Score evaluation results from JSON test cases (exact/contains/regex) | `python scripts/eval_scorer.py suite.json --fail-under 0.80 --json` |
| `prompt_analyzer.py` | Analyze prompt files for clarity, instruction density, few-shot coverage, tokens | `python scripts/prompt_analyzer.py my_prompt.txt --json` |
| `prompt_diff.py` | Compare two prompt versions for structural changes, instruction deltas, risk | `python scripts/prompt_diff.py v2.txt v3.txt --show-diff --json` |

## References

Load the reference that matches the task — keep this file lean and pull detail on demand:

- **[references/prompt-patterns-catalog.md](references/prompt-patterns-catalog.md)** — complete catalog of prompting techniques with examples: system-prompt architecture, chain-of-thought, few-shot, output structuring, decomposition, and calibration. Read when designing or structuring a prompt.
- **[references/testing-and-versioning.md](references/testing-and-versioning.md)** — test-case design, suite composition, the evaluation rubric and scoring formula, the regression protocol, version-control strategy, changelog format, and diff analysis. Read when building a test suite or managing versions.
- **[references/failure-modes-and-workflows.md](references/failure-modes-and-workflows.md)** — common failure modes, the three lifecycle workflows, a quick-view integration table, the troubleshooting matrix, and success criteria. Read when debugging a prompt or running a workflow.

## Scope & Limitations

**This skill covers:**
- Designing, structuring, and layering system prompts for production AI applications
- Building and running test suites, evaluation rubrics, and regression tests for prompt quality
- Versioning prompts with changelogs, baselines, and rollback plans
- Calibration techniques including temperature tuning, confidence levels, and few-shot selection

**This skill does NOT cover:**
- Fine-tuning or training models -- see `engineering/model-training-pipeline` for training workflows
- Retrieval-augmented generation (RAG) pipeline design -- see `engineering/context-engine` for context retrieval architecture
- Agent orchestration and multi-step tool use -- see `engineering/agent-designer` for agent system design
- LLM infrastructure, hosting, or cost optimization -- see `engineering/llm-gateway-design` for inference infrastructure patterns

## Integration Points

| Skill | Integration | Data Flow |
|-------|-------------|-----------|
| **agent-designer** | Agent system prompts are the highest-stakes prompts; use this toolkit to test and version them | Agent specs → prompt layers → tested system prompts |
| **self-improving-agent** | Prompt degradation signals feed into self-improvement loops for automatic correction | Test suite results → regression alerts → prompt iteration |
| **context-engine** | Retrieved context quality directly impacts prompt effectiveness; coordinate retrieval tuning with prompt testing | Retrieved chunks → prompt context layer → evaluation scores |
| **ab-test-setup** | A/B test prompt variants in production with statistical rigor before full rollout | Prompt candidates → traffic split → scoring comparison → winner promotion |
| **llm-gateway-design** | Gateway handles prompt routing, versioning, and model fallback at the infrastructure layer | Versioned prompts → gateway config → model routing → response logging |
| **code-review-automation** | Code review prompts are high-frequency production prompts that benefit from this toolkit's testing framework | Review criteria → prompt design → test suite → deployed reviewer prompt |

---

## What I Need You to Do

First, detect which platform I'm using (Claude.ai, ChatGPT, etc.) and follow the matching instructions below.

### If I'm on Claude.ai:

Walk me through these exact steps:

1. **Create the Project:** Tell me to go to **claude.ai > Projects > Create project** and name it **"Prompt Engineer Toolkit"**

2. **Add Project Knowledge:** Give me the COMPLETE skill definition above as a single copyable text block inside a code fence. Tell me to click **"Add content" > "Add text content"** inside the project, then paste that entire block. Do NOT say "paste from above" -- give me the actual text to copy right there.

3. **Set Custom Instructions:** Tell me to open project settings and paste this exact instruction:
   "You are an expert Prompt Engineer Toolkit in the Engineering domain. Use the project knowledge as your expertise. Follow the workflows, frameworks, and templates defined there. Always provide specific, actionable output."

4. **Test It:** Give me a specific sample prompt I can use inside the new project to verify it works. Pick a real task from the skill's workflows.

### If I'm on ChatGPT:

Walk me through these exact steps:

1. **Create a Custom GPT:** Tell me to go to **chatgpt.com > Explore GPTs > Create**
2. **Configure it:**
   - Name: **"Prompt Engineer Toolkit"**
   - Description: "Prompt engineering frameworks for building, testing, versioning, and evaluating prompts: chain-of-thought, few-shot, regression testing, and rubrics. Use when designing production prompts, running A/B tests, or building prompt libraries."
   - Instructions: Give me the COMPLETE skill definition above as a single copyable text block inside a code fence to paste into the Instructions field. Do NOT say "paste from above."
3. **Test It:** Give me a sample prompt to verify it works.

### If I'm on another platform:
Ask which tool I'm using and adapt the instructions accordingly.

## Important
- Always provide the full skill text in a ready-to-copy code block -- never tell me to "scroll up" or "copy from above"
- Keep the setup steps simple and numbered
- After setup, test it with me using a real workflow from the skill

Source: https://github.com/borghei/Claude-Skills/tree/main/engineering/prompt-engineer-toolkit/SKILL.md