# AI visibility reference - Raegan

Companion to [`seo-content-plan.md`](../seo-content-plan.md). Preserves architecture decisions, external references, and porting notes so nothing is lost in chat.

Last updated: June 2026.

---

## What we are measuring

Two different questions:

| Question | Tooling | Example |
|----------|---------|---------|
| **Search ranking** | GSC, Bing | "Do we show in blue links for `ai chief of staff`?" |
| **AI citation** | Citation audits | "When someone asks Perplexity, does it name or link Raegan?" |

Citation audits do **not** replace SEO. They tell you whether answer engines retrieve your pages and directories when users ask category questions.

---

## Three-layer audit stack (use all three)

```
┌─────────────────────────────────────────────────────────────────┐
│ Layer A - OpenRouter baseline (weekly, cheap)                   │
│ scripts/ai-visibility-audit.py  OR  Cursor + openrouter-mcp    │
│ "Would models mention Raegan from parametric knowledge?"        │
│ Output: ai-visibility-log.csv (script) or benchmark export (MCP)│
└─────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────┐
│ Layer B - Provider API live retrieval (biweekly, planned)       │
│ scripts/ai-visibility-live-audit.py (port from citation tool)   │
│ Perplexity API with return_citations; optional OpenAI/Gemini    │
│ Output: ai-visibility-live-log.csv                              │
└─────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────┐
│ Layer C - Manual consumer surfaces (quarterly, required)        │
│ chatgpt.com, perplexity.ai UI, Google AI Overview               │
│ Output: ai-visibility-manual-log.csv                            │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
              Post-audit decision tree (seo-content-plan.md)
```

### Layer comparison

| Layer | Cadence | API keys | Trust level for citations |
|-------|---------|----------|---------------------------|
| **A - OpenRouter** | Weekly | `OPENROUTER_API_KEY` | Low for URLs; OK for brand mentions in text |
| **A′ - openrouter-mcp** (optional) | Ad hoc | Same key via `openrouter-mcp init` | Same as A; interactive in Cursor, not a citation layer |
| **B - Provider APIs** | Biweekly | `PERPLEXITY_API_KEY`, optional `OPENAI_API_KEY`, `GEMINI_API_KEY` | **High for Perplexity**; medium for OpenAI/Gemini (see gaps below) |
| **C - Manual UI** | Quarterly | None (browser) | **Ground truth** for ChatGPT.com and Google AI Overview |

**Rule:** If Layer A says Raegan appears but Layer B/C do not, trust B/C. Fix discovery (directories, indexing), not model prompting.

---

## Canonical query set

- **File:** [`ai-visibility-queries.json`](../ai-visibility-queries.json)
- **Count:** 15 queries (category + branded)
- **Also tracks:** `brands_to_track`, `directory_domains_to_track`, OpenRouter model defaults

All three layers must use the **same 15 queries** so results are comparable across runs and dates.

---

## Layer A tooling: Python script vs openrouter-mcp

Two ways to run the same Layer A audits. Use **both** if you want: script for scheduled CSV, MCP for interactive work.

| | `scripts/ai-visibility-audit.py` | [openrouter-mcp](https://github.com/physics91/openrouter-mcp) in Cursor |
|--|-----------------------------------|------------------------------------------------------------------------|
| **Trigger** | Terminal, cron, GitHub Actions | Ask in Cursor chat |
| **Output** | `ai-visibility-log.csv` (fixed schema) | Chat text, or benchmark JSON/CSV via MCP tools |
| **Repeatability** | Same 15 queries every run | Depends on prompt; anchor to `ai-visibility-queries.json` |
| **Multi-model** | Model list in JSON | `benchmark_models`, `compare_model_performance` |
| **Consensus** | No | `collective_chat_completion`, `ensemble_reasoning`, `cross_model_validation` |
| **Free models** | Manual model IDs | `free_chat`, `list_free_models` |
| **Key storage** | `.env` | `npx @physics91/openrouter-mcp init` (keychain / encrypted file) |
| **Best for** | Weekly regression, CI | Copy drafts, ad-hoc audits, model pick, comparison QA |

**Neither Layer A path** provides live web citations or replaces Perplexity API (Layer B) or manual UI (Layer C).

### What openrouter-mcp is

MCP server ([physics91/openrouter-mcp](https://github.com/physics91/openrouter-mcp)) that exposes OpenRouter as Cursor/Claude Desktop tools: chat, vision, benchmarking, and multi-model "collective intelligence" workflows. npm: `@physics91/openrouter-mcp`.

**Prerequisites:** Node.js 16+, Python 3.10+, OpenRouter API key.

### Install and connect (Cursor)

```bash
npx @physics91/openrouter-mcp init
npx @physics91/openrouter-mcp status
```

Add to Cursor MCP settings (user or project `.cursor/mcp.json`):

```json
{
  "mcpServers": {
    "openrouter": {
      "command": "npx",
      "args": ["-y", "@physics91/openrouter-mcp", "start"]
    }
  }
}
```

Run `init` once so the key is stored securely; do not put `OPENROUTER_API_KEY` in the JSON config unless required.

Official guides: [MCP_CLIENT_GUIDE.md](https://github.com/physics91/openrouter-mcp/blob/main/docs/MCP_CLIENT_GUIDE.md), [BENCHMARK_GUIDE.md](https://github.com/physics91/openrouter-mcp/blob/main/docs/BENCHMARK_GUIDE.md), [API.md](https://github.com/physics91/openrouter-mcp/blob/main/docs/API.md).

### MCP tools useful for Raegan

| Tool | Raegan use |
|------|------------|
| `chat_with_model` | Run one visibility query on one model |
| `benchmark_models` | Same prompt × N models; export CSV/JSON |
| `list_available_models` / `list_free_models` | Pick cheap models for weekly audits |
| `compare_model_performance` | Cost vs quality for audit model selection |
| `collective_chat_completion` | "Do any models recommend Raegan for q02?" |
| `cross_model_validation` | Sanity-check comparison page claims before ship |
| `export_benchmark_report` | Save benchmark run as markdown/csv/json |
| `chat_with_vision` | QA homepage or SEO page screenshots |

### Example Cursor prompts (Layer A)

Anchor every run to the canonical query file:

1. *"Read `ai-visibility-queries.json`. For q02, q05, and q09, call benchmark_models with openai/gpt-4o-mini and anthropic/claude-sonnet-4. Report whether Raegan or any `brands_to_track` appear in each response."*

2. *"Use collective_chat_completion on q14 (who makes an AI chief of staff product). List every product name mentioned."*

3. *"Draft a Futurepedia long description from `directory-submissions.md` tone rules; use cross_model_validation with two models for factual claims about Raegan only (no Hermes deep-dive)."*

4. *"compare_model_performance on gpt-4o-mini vs claude-sonnet-4 with weights cost 50%, quality 50% - pick default for weekly `ai-visibility-audit.py` runs."*

### Script vs MCP: when to use which

| Task | Use |
|------|-----|
| Weekly scheduled 15-query CSV | `python3 scripts/ai-visibility-audit.py` |
| Exploring a new competitor on 2-3 queries | openrouter-mcp in Cursor |
| Directory copy variants | openrouter-mcp (faster iteration) |
| Quarterly citation ground truth | Manual Layer C only |
| Live URL citations | Layer B Perplexity API (planned script) |
| GitHub Action cron | Python script, not MCP |

---

## Reference project: ai-citation-tool

**Repo:** [github.com/ps0394/ai-citation-tool](https://github.com/ps0394/ai-citation-tool)

Microsoft-internal benchmark that tests how **learn.microsoft.com** is cited by ChatGPT, Gemini, and Perplexity. Use as an **architecture example**, not a dependency.

### What to borrow

| Pattern | Their implementation | Raegan adaptation |
|---------|---------------------|-----------------|
| Query file | `prompts.csv` (`prompt_id`, `category`, `prompt`) | Already: `ai-visibility-queries.json` |
| Multi-provider loop | `run_benchmark.py` loops providers x prompts | Port into `ai-visibility-live-audit.py` |
| CSV schema | `date`, `provider`, `model`, `prompt_id`, `prompt`, `response`, `citation_url`, `citation_domain` | Add `raegan_named`, `raegan_url_cited`, `brands_mentioned` |
| Perplexity citations | `return_citations: true` on Perplexity chat API | **Primary automated citation source** |
| Scheduling | GitHub Actions daily + `results_YYYY-MM-DD.csv` | Optional private repo workflow with secrets |
| Dashboard | `docs/index.html` + Chart.js | Optional later; CSV is enough pre-launch |

### What NOT to use as-is

| Gap | Why it matters for Raegan |
|-----|---------------------------|
| **Microsoft-only domain** | `normalize_domain()` targets `learn.microsoft.com` | Replace with `raegan.ai` + directory domains |
| **ChatGPT ≠ chatgpt.com** | Uses `chat.completions`; extracts URLs via regex from prose | Does not equal consumer ChatGPT with browsing |
| **`use_web_search=True` unused** | Flag exists but OpenAI path is plain completions | Need OpenAI **Responses API + web search** for a real ChatGPT proxy |
| **Gemini citations** | Regex on generated text | Not guaranteed grounding metadata |
| **No Google AI Overview** | Not in repo | Stay manual quarterly |
| **Stub mode** | Fake Microsoft URLs when keys missing | Always fail loudly if keys absent |
| **Small / niche repo** | 1 star, Azure-focused | Fork patterns into `raegan-ai`; do not submodule |

### Their Perplexity pattern (worth porting)

From `run_benchmark.py`:

- Endpoint: `https://api.perplexity.ai/chat/completions`
- Model: online sonar family (e.g. `llama-3.1-sonar-small-128k-online` - verify current model IDs in Perplexity docs)
- Payload includes `"return_citations": true`
- Citations may appear in `result["citations"]` or fall back to URL regex on response text

This is the **most reliable automated citation layer** because Perplexity actually retrieves the web.

### Their OpenAI pattern (weak proxy)

- `client.chat.completions.create()` with a system prompt asking for URLs
- URLs extracted with regex from answer text
- **Not** the same as ChatGPT with search enabled in the product UI

To improve later: OpenAI Responses API with web search tool (document in porting checklist when implemented).

---

## Log files (keep separate)

| File | Layer | Gitignored |
|------|-------|------------|
| `ai-visibility-log.csv` | A - OpenRouter | Yes |
| `ai-visibility-live-log.csv` | B - Provider APIs | Yes |
| `ai-visibility-manual-log.csv` | C - Manual UI | Yes |

Never merge layers into one CSV without an `audit_type` column - easy to misread parametric mentions as live citations.

### Manual log columns (Layer C)

| Column | Description |
|--------|-------------|
| `date` | ISO date |
| `surface` | `chatgpt` / `perplexity` / `google_aio` |
| `query_id` | q01-q15 |
| `query` | Full query text |
| `raegan_named` | Y/N |
| `raegan_url` | Y/N - `raegan.ai` in citations |
| `brands_named` | Semicolon-separated |
| `urls_cited` | Semicolon-separated domains or URLs |
| `notes` | Pattern ("listicle dominated", "no AIO", etc.) |

### Planned live API log columns (Layer B)

Same as ai-citation-tool plus Raegan fields:

| Column | Description |
|--------|-------------|
| `audit_type` | `live_api` |
| `provider` | `perplexity` / `openai` / `gemini` |
| `model` | Model ID string |
| `prompt_id` | q01-q15 |
| `prompt` | Query text |
| `response` | Full answer text (truncate in storage if huge) |
| `citation_url` | One row per citation URL |
| `citation_domain` | Parsed domain |
| `raegan_citation` | Y/N - citation URL is on `raegan.ai` |
| `target_citation` | Y/N - matches tracked domain (raegan or directory) |
| `brands_in_response` | Parsed brand mentions |

---

## API keys and env

Store in local `.env` only (gitignored). Never commit. Never paste in chat.

| Variable | Layer | Required |
|----------|-------|----------|
| `OPENROUTER_API_KEY` | A | For OpenRouter baseline |
| `PERPLEXITY_API_KEY` | B | **Recommended first** for live citations |
| `OPENAI_API_KEY` | B | Optional; weak citation proxy unless Responses+search |
| `GEMINI_API_KEY` | B | Optional |

Example `.env` (local only):

```env
OPENROUTER_API_KEY=sk-or-...
PERPLEXITY_API_KEY=pplx-...
# OPENAI_API_KEY=sk-proj-...
# GEMINI_API_KEY=...
```

---

## Porting checklist: ai-citation-tool → raegan-ai

When implementing `scripts/ai-visibility-live-audit.py`:

- [ ] Read queries from `ai-visibility-queries.json` (not `prompts.csv`)
- [ ] Implement Perplexity provider first with `return_citations: true`
- [ ] `normalize_domain(url)` → track `raegan.ai` and `directory_domains_to_track`
- [ ] `categorize_citations()` → `raegan` vs `directory` vs `competitor` vs `other`
- [ ] Detect `brands_to_track` in response text (same regex list as OpenRouter script)
- [ ] One CSV row per citation URL (match ai-citation-tool shape for easy diff)
- [ ] `--dry-run` prints plan only; **no stub fake URLs**
- [ ] Fail if `PERPLEXITY_API_KEY` missing (no silent stubs)
- [ ] Document commands in `seo-content-plan.md`
- [ ] Optional: GitHub Action in private repo on weekly cron
- [ ] Optional: OpenAI Responses API + web search (later)
- [ ] Update quarterly loop cadence in plan after Layer B ships

**Do not** copy Microsoft's repo into `raegan-ai` vendor folder. Port ~100-150 lines of provider logic only.

---

## Interpreting results (all layers)

| Pattern | Meaning | Action |
|---------|---------|--------|
| Nobody cites Raegan on category queries | Normal pre-launch | Directories, GSC indexing, `/vs/*` pages |
| Directories cited (TAAFT, Futurepedia) | Lists beat your site | Submit [`directory-submissions.md`](../directory-submissions.md) |
| Same competitor on 3+ queries | Default incumbent | Add `/vs/{name}/` or controlled listicle |
| Raegan branded only | Not in category graph | Directories + internal links |
| Raegan on category | Winning | Strengthen matching pillar FAQ |
| OpenRouter yes, live no | Parametric only | Ignore A; fix B/C gaps |
| Perplexity API cites directories not raegan.ai | Expected early | Submit directories; comparison pages |

---

## After audits: what to do (summary)

Full tree in [`seo-content-plan.md`](../seo-content-plan.md) section *Post-audit decision tree*.

**Order:** directories → `/vs/` pages → pillar FAQ tweaks → indexing → **then** blog/listicle only if gap persists.

**Blog is last**, not first. See *Blog and listicle policy* in the plan.

---

## External links

- [ai-citation-tool (Layer B reference)](https://github.com/ps0394/ai-citation-tool)
- [openrouter-mcp (Layer A interactive / benchmarks)](https://github.com/physics91/openrouter-mcp)
- [OpenRouter keys](https://openrouter.ai/keys)
- [Perplexity API](https://docs.perplexity.ai/)
- [llms.txt spec](https://llmstxt.org/)