Where Compass turns messy requests into executable work
A visual walkthrough of the current backend shape, how planning recognition changed the answerability boundary, and why issue #782 now matters.
Compass is not one open-ended agent. It is a typed pipeline: Planner drafts a shape, recognition can advise, CatalogResolver owns execution IDs, Executor reads rows, and Renderer/Quality surface evidence.
Natalie's core question exposed the first design tension: should catalog awareness happen before execution? PR #753 adds that advisory layer. PR #804 shows the next rule: selected guidance may shape planner validation, but raw prose must not become a second dispatcher.
Source baseline: origin/main refreshed at 28909809 after PR #804; PR #753 planning recognition, PR #804 peer guidance, and issue #782 guardrails reviewed.
Where does Compass decide a request is answerable?
The team already agrees at 30,000 feet: use AI for meaning, then deterministic code for truth. The hard part is deciding where meaning becomes executable authority.
1. Query logic
Can Compass understand the research plan? Example: "show starting salary for districts with highest FRPL share" is not just a salary lookup.
2. Entity resolution
Can Compass map user terms to governed catalog objects? Example: "starting salary", "Philadelphia", "observations", "FRPL".
3. Data coverage
Once the plan and entities are resolved, does the covered universe actually contain current reviewed rows and citations?
A bad answerability outcome can come from the planner choosing the wrong shape, the resolver missing the right entity, the executor not supporting the shape, or real coverage gaps. Those are different bugs.
The backend is a turn pipeline, not one agent graph
Each handoff has a typed contract. That contract is how Compass keeps the answer from becoming free-form model prose.
Important property
Executed data answers are rendered from `ResultSet` and `ValidationAuthority`, not composed freely by the LLM.
Important nuance
Recognition can show candidate labels early, but final catalog resolution still owns whether requested nouns become execution authority.
Compass has a small set of executable templates
Think of these as the current work orders the backend knows how to carry out safely. The full shape is the whole `QueryPlan`, not just the operation name.
lookup
Exact selected-district or scoped metric table.
Example: salary for Dallas, or all districts with metric X.
rank
Top, bottom, ordered, or sorted district rows.
Example: highest starting salary.
count
Denominator, threshold count, or categorical breakdown.
Example: how many districts offer X?
trend
Longitudinal or multi-year metric rows.
Example: five-year salary trend.
profile_lookup
NCES/profile facts for selected districts.
Example: enrollment or FRPL for Denver.
similarity
Peer discovery without a policy metric comparison.
Example: who are Denver's peers?
peer_comparison
Anchor plus deterministic peers, compared on a governed metric.
Example: Denver peers on max salary.
policy_guidance
NCTQ stances, rationales, and exemplars from the typed library.
This is a route, not a district-data operation.
Each layer should own exactly one kind of decision
This is the chain-of-custody model that keeps Compass from drifting into confident but ungrounded answers.
Resolution now has an advisory pass and an authoritative pass
PR #753 moved catalog awareness earlier, but kept the authority boundary intact: recognition can advise; execution still resolves.
When Compass misses, locate the miss before fixing it
The same user-visible failure can come from four different layers. Fixing the wrong layer adds more brittleness.
Shape miss
Planner picked an operation the executor cannot compile, or treated a multi-step research plan like a simple lookup.
Resolution miss
Metric, district, profile field, or peer phrase did not map to the governed catalog object even though the data exists.
Context miss
Follow-up turn lost prior result rows, prior metric, pending clarification, or state narrowing.
Coverage miss
The plan and entities are right, but reviewed rows are missing, stale, not applicable, or outside the Pathfinder.
Before adding a special case, classify the failure by layer and by authority: selected guidance and catalog aliases are safer; raw-message code that rewrites execution behavior is the issue #782 escape-hatch risk.
The conservative architecture is worth preserving
The current design has real strengths. The problem is not that the system is typed. The problem is where we put the next layer of meaning.
- Typed `PlannerTurn` and `QueryPlan` contracts.
- Planning recognition as an advisory-only early warning layer.
- Catalog authority for IDs and approved fields.
- Bounded candidate-only adjudication.
- Deterministic renderer over validated `ResultSet` artifacts.
- Persisted turn snapshots and resolution reports for forensic review.
- Do not solve every failure with phrase heuristics.
- Do not let the model invent metric IDs or unsupported research operations.
- Do not collapse catalog ambiguity into a confident answer.
- Do not accept a user constraint and merely disclose later that it was not enforced.
- Do not judge readiness from aggregate score alone.
The shape system is stronger, but boundary drift is still the risk
Issue #782 reframes the risk: not wild hallucination, but second paths that can answer, repair, soften, or score outside the clean chain.
Mode-dependent recognition
Planning recognition exists, but behavior depends on `COMPASS_PLANNING_RECOGNITION_MODE`: off, observe, intercept, or finalize.
Validator boundary
Selected guidance metadata can require typed planner behavior. Validator code should not re-read the raw user message to dispatch below the planner.
Shadow catalog authority
Catalog aliases, unsupported concepts, and contextual defaults should stay in reviewed catalog data, with startup/smoke checks for expected launch rows.
Accepted but unenforced constraints
A user filter should be enforced before rendering or turned into a clarification/refusal, not answered with a later caveat.
Soft launch evidence
Non-blocking verdicts are fine for chat UX, but controlled sweeps must show trace gaps, verdict-write failures, and skipped rows explicitly.
Shape logic spread
Planner prompts, selected guidance, recognition, validators, executor predicates, catalog aliases, and tests still encode pieces of shape knowledge.
The register now separates retired debt from relapse points
The deck is aligned only if it names what PR #804 retired and what still needs guardrails before Compass can close issue #782.
`launch_rescue.py` is removed on current main. Do not recreate it; route replacement behavior through planner guidance, catalog, execution, or refusal.
`answerability_recovery.py` is removed on current main. Future repairs need a typed owner and scenario evidence.
Selected guidance metadata and typed `PlannerTurn` fields are allowed. Raw `frame.message` checks below snippet selection are not.
Aliases, blockers, and defaults belong in `compass.catalog_aliases` / governed seed and migration data, not hidden Python constants.
Unsupported concepts should be governed product data, not scattered constants.
Safe adjudication still fails if the right governed candidate never appears.
`metric_value_filter_not_applied` is the symptom class: accepted filters must be enforced or refused.
Verdict, trace, envelope, and denominator gaps must show up in launch evidence.
`direct` and `policy_guidance` need guards against bypassing governed data routes.
Safe paths add advisory context or candidates. Risky paths rewrite the user request, choose execution behavior, hide missing catalog authority, or pretend an unenforced constraint was applied.
After PR #804, the question is guardrails and coverage
The next move is not simply "more AI or more code." It is to keep selected guidance narrow, move recognition gaps into governed catalog data, and block raw-message relapse.
Guidance can require a typed operation such as `peer_comparison`; it should not become hidden execution authority.
Do not reintroduce launch rescue or answerability recovery under a new name when a prompt fails.
Catalog aliases, unsupported concepts, and defaults should live in reviewed data, not hidden Python constants.
Document supported request shapes and every remaining workaround, then classify each as keep, move, repair, or delete.
Start from issue #782: keep planning recognition advisory, let selected guidance validate typed shape only, use final CatalogResolver execution as the authority boundary, and require scenario evidence for every repair.
Decisions to make before the next implementation wave
These are product/architecture choices, not just code cleanup tasks.
Which recognition mode is live?
Are we evaluating off, observe, intercept, or finalize behavior when we judge answerability?
What counts as a relapse?
Is a path advisory candidate broadening or selected-guidance validation, or can it change execution behavior by rereading raw prose?
What moves to catalog?
Which aliases, unsupported concepts, and governed defaults should become reviewed data instead of Python constants?
What belongs in typed repair?
If Compass needs a second pass, what typed inputs, authority limits, and tests prevent it from becoming launch rescue again?
Which constraints hard-fail?
If Compass accepts a filter or condition, should failure to enforce it always clarify or refuse before rendering?
What proof closes #782?
Which traces, scenario families, and tests show that a workaround was removed without losing the intended behavior?
What guard blocks relapse?
Which tests or lint checks prevent the next urgent bug fix from adding raw-message dispatch again?
Turn issue #782 into measurable guardrail work
Use the issue as the shared artifact. Link every PR, scenario, debug trace, and reviewer note until each remaining risk has an owner and a disposition.
Remove escape hatches without weakening the chain of custody.
- Mark launch rescue and answerability recovery as retired by current main.
- Document remaining relapse points in one register.
- Classify each as keep, move to catalog, move to typed repair, or delete/refuse.
- Record which layer currently owns the behavior and which layer should own it.
- Keep the safe-vs-risky distinction explicit: advisory guidance is not execution authority.
- Link PRs as attempts with evidence, not as closure by themselves.
Move behavior into governed layers one family at a time.
- Run affected scenarios whenever selected guidance, catalog aliases, or typed execution support changes.
- Move aliases, unsupported concepts, and contextual defaults into governed catalog data.
- Do not recreate deleted rescue families under new helper names.
- Make accepted filters enforced-or-refused, never disclosed as not applied.
- Make trace/verdict gaps visible in controlled sweeps and add guards against new raw-message dispatch.
We now do X. We changed Y files/layers. This fixes prompts like Z. Still open: A/B/C. Current verification: scenario/debug links and exact pass/partial status.