Ab Test Setup

# Create a "Ab Test Setup" AI Skill

I want you to help me set up a reusable AI skill that I can use in future conversations. Read the complete skill definition below, then help me install it.

## Complete Skill Definition

# A/B Test Setup Skill

## Overview

Production-ready A/B testing toolkit for calculating sample sizes, designing rigorous test plans, and analyzing results with statistical significance testing. Designed for growth teams, product managers, and marketers who need to make data-driven decisions from controlled experiments.

## Quick Start

```bash
# Calculate required sample sizes for a test
python scripts/sample_size_calculator.py --baseline 0.05 --mde 0.10 --power 0.80

# Design a complete A/B test plan
python scripts/test_designer.py test_config.json

# Analyze A/B test results
python scripts/results_analyzer.py results.json
```

## Tools Overview

| Tool | Purpose | Input | Output |
|------|---------|-------|--------|
| `sample_size_calculator.py` | Sample size calculation | Baseline rate, MDE, power | Required samples + duration |
| `test_designer.py` | Test plan design | JSON test config | Complete test plan document |
| `results_analyzer.py` | Results analysis | JSON with test results | Statistical analysis + recommendation |

## Workflows

### Workflow 1: New A/B Test Setup

1. Define hypothesis and success metric
2. Run `sample_size_calculator.py` with baseline conversion and minimum detectable effect
3. Create test configuration JSON (see Common Patterns)
4. Run `test_designer.py` to generate complete test plan
5. Share plan with stakeholders for alignment before launch

### Workflow 2: Test Results Analysis

1. Collect test results into JSON format
2. Run `results_analyzer.py` to get statistical significance
3. Review confidence interval, p-value, and effect size
4. Check for segment-level effects if overall result is inconclusive
5. Make ship/no-ship decision based on analysis

### Workflow 3: Experimentation Program Review

1. Compile results from multiple past tests
2. Run `results_analyzer.py --batch` on all results
3. Review win rate, average effect size, and velocity
4. Identify patterns in winning vs losing tests
5. Optimize test pipeline based on learnings

## Reference Documentation

See `references/ab-testing-guide.md` for comprehensive methodology covering:
- Statistical foundations (z-tests, confidence intervals)
- Sample size theory and trade-offs
- Common experimentation pitfalls
- Multi-variant and sequential testing
- Bayesian vs frequentist approaches

## Common Patterns

### Pattern: Test Configuration JSON
```json
{
  "test_name": "Homepage CTA Button Color",
  "hypothesis": "Changing the CTA button from blue to green will increase click-through rate",
  "metric_primary": "cta_click_rate",
  "metric_secondary": ["signup_rate", "bounce_rate"],
  "baseline_rate": 0.045,
  "minimum_detectable_effect": 0.10,
  "significance_level": 0.05,
  "power": 0.80,
  "variants": [
    {"name": "control", "description": "Current blue CTA button"},
    {"name": "treatment", "description": "Green CTA button"}
  ],
  "daily_traffic": 5000,
  "allocation": {"control": 0.50, "treatment": 0.50}
}
```

### Pattern: Test Results JSON
```json
{
  "test_name": "Homepage CTA Button Color",
  "variants": {
    "control": {"visitors": 12500, "conversions": 563},
    "treatment": {"visitors": 12500, "conversions": 625}
  },
  "metric": "cta_click_rate",
  "significance_level": 0.05
}
```

### Quick Reference: Common Effect Sizes

| Context | Small Effect | Medium Effect | Large Effect |
|---------|-------------|---------------|--------------|
| Conversion Rate | 2-5% relative | 5-15% relative | > 15% relative |
| Revenue per User | 1-3% | 3-8% | > 8% |
| Engagement Rate | 3-5% | 5-10% | > 10% |

---

## What I Need You to Do

First, detect which platform I'm using (Claude.ai, ChatGPT, etc.) and follow the matching instructions below.

### If I'm on Claude.ai:

Walk me through these exact steps:

1. **Create the Project:** Tell me to go to **claude.ai > Projects > Create project** and name it **"Ab Test Setup"**

2. **Add Project Knowledge:** Give me the COMPLETE skill definition above as a single copyable text block inside a code fence. Tell me to click **"Add content" > "Add text content"** inside the project, then paste that entire block. Do NOT say "paste from above" -- give me the actual text to copy right there.

3. **Set Custom Instructions:** Tell me to open project settings and paste this exact instruction:
   "You are an expert Ab Test Setup in the Marketing domain. Use the project knowledge as your expertise. Follow the workflows, frameworks, and templates defined there. Always provide specific, actionable output."

4. **Test It:** Give me a specific sample prompt I can use inside the new project to verify it works. Pick a real task from the skill's workflows.

### If I'm on ChatGPT:

Walk me through these exact steps:

1. **Create a Custom GPT:** Tell me to go to **chatgpt.com > Explore GPTs > Create**
2. **Configure it:**
   - Name: **"Ab Test Setup"**
   - Description: "Production-ready A/B testing toolkit for calculating sample sizes, designing rigorous test plans, and analyzing results with statistical significance testing. For growth teams, product managers, and marketers."
   - Instructions: Give me the COMPLETE skill definition above as a single copyable text block inside a code fence to paste into the Instructions field. Do NOT say "paste from above."
3. **Test It:** Give me a sample prompt to verify it works.

### If I'm on another platform:
Ask which tool I'm using and adapt the instructions accordingly.

## Important
- Always provide the full skill text in a ready-to-copy code block -- never tell me to "scroll up" or "copy from above"
- Keep the setup steps simple and numbered
- After setup, test it with me using a real workflow from the skill

Source: https://github.com/borghei/Claude-Skills/tree/main/marketing/ab-test-setup/SKILL.md