Is SubQ Snap worth paying for?

1. What is actually special about SubQ?

SubQ Snap Preview is not a conversational model that writes an analysis. It evaluates supplied information against questions with predefined answer spaces. Its Decisions API uses POST /v1/systemone and supports:

• Noul: a yes/no judgment represented by a number between 0 and 1.
• Choice: select a supplied category, with a distribution and confidence.
• Score: probability-weighted position on 2–10 ordered rubric levels. A score can fall between levels.

Multiple questions can share one evidence state. Snap supports text and up to four images. The vendor documents a 384K-token supported context and $0.05 per million input tokens, with output free. Its launch article attributes its efficiency to Subquadratic Sparse Attention and scoring allowed answers rather than generating a free-form response.

The practical differentiator is economics and interface, not verified market insight. A fast, cheap classifier can replace repetitive routing work otherwise assigned to a generative model. It cannot explain a rating in unrestricted prose. To inspect “why,” we supplied a menu of evidence limitations; the resulting reason code is a classification of our options, not a discovered rationale or access to internal reasoning.

Vendor benchmark figures are vendor-reported general decision evaluations. They do not demonstrate trading alpha. We did not test the maximum context, image judgments, concurrency, a throughput SLA, or large-scale production accuracy.

2. Experiment design: compare, do not just agree

The first two rounds explored class rankings, current picks, evidence attribution, ordering effects, synthetic rule gates and a local opinion. The third round asked a sharper question: does SubQ add something beyond our own analysis?

1. We recorded the assistant’s candidate ratings, asset-class ratings and explanations before the new comparative calls. File fingerprints were recorded in every call ledger.
2. SubQ rated the same eight public candidates under the same 0–4 research-value rubric. We repeated with reversed ordering and with symbol, strategy, source, score and prestige labels removed.
3. We authored 24 evidence/claim cases covering cohort transfer, missing rows, nested freshness, signal versus fill time, source independence and insufficient information. Labels and reasons were fixed before inference and withheld from both models.
4. SubQ classified those cases three times: original order, reversed order and paraphrased question instructions. Local Qwen2.5-14B classified the same cases once.
5. SubQ classified the main blocker for each candidate, then rated forward-strategy research value by asset class.

This is not a fully blind independent benchmark. The assistant had seen earlier SubQ answers, authored the rubric and reference labels, and helped select cases. Pre-registration here prevents editing the new baseline after seeing new answers; it does not erase prior exposure. Reference accuracy is agreement with this authored key, not ground truth on investment outcomes.

The rubric was: 0 no defensible lead; 1 weak lead with major limitations; 2 tentative identifiable research lead; 3 strong corroborated research lead; 4 documented clean net edge with prospective/holdout support. These are research priorities, not expected returns, win probabilities or allocation weights.

3. Our ratings versus SubQ’s ratings

The mean absolute distance from our ratings was 0.314 points out of 4. This is agreement, not an accuracy metric: no realised future outcomes label these research scores. The largest variant spread was 0.456 points for IEF. Ratings 3 and 4 were not reached.

The disagreement is informative, but not proof of unique insight. Our baseline prioritised the concrete XRP history-attribution lead. SubQ favoured gold and rated IEF more generously. In a prior prerequisite-first task SubQ itself chose XRP, whereas broad shortlist prompts chose gold. The evaluation objective and answer menu matter.

“Why” as an evidence code

SubQ selected history_scope as the main blocker for seven candidates and negative_exact_history for SPY. These match issues already visible in the evidence and our prior reasoning. The model did not produce a novel causal explanation, discover a missing dataset, or establish future profitability.

Earlier prerequisitive tests rejected documented clean, cost-adjusted same-strategy/same-symbol history for all eight candidates, even when score labels were removed. That is a stronger operational result than an uncertain “best pick” ranking.

4. Semantic accuracy: useful, but a revealing failure

The local model also withheld support from one narrowly supported uncertainty statement, contributing to its binary score. Separating unsupported from not assessable is partly dependent on label definitions; the binary support/non-support result is therefore included rather than hiding that ambiguity.

The error that matters most

Evidence: Commodity is absent from the 14-day and 48-hour class summaries.
Claim: no Commodity trades occurred in those windows.

Our reference: unsupported. Missing summary rows cannot prove zero activity. SubQ nevertheless selected supported in the original and paraphrased-question variants, with low reported confidence of 0.418 and 0.305. In reversed order it chose not assessable, not our exact reference label but safely avoiding support. Local Qwen also supported the invalid claim.

This is directly relevant to our real Commodity recency gap. The prompt explicitly said unknown is not zero. Passing simple rule fixtures did not prevent this semantic mistake. A reviewer who knows this data contract was sufficient to reject it without any model call.

The original SubQ variant also called a completely unevidenced “probably passes all gates” claim unsupported rather than our not assessable. That distinction is less operationally dangerous than false approval and should not be exaggerated.

What this test does not prove

• The three SubQ runs reuse the same 24 cases: they are not 72 independent examples.
• The 3–4 extra matching answers versus this local configuration are not statistically established superiority.
• We did not measure the assistant’s accuracy independently: the assistant wrote the reference key. Its 24 reference answers are not an honest “100% benchmark score.”
• A single local model, one prompt and one run are not a comparison against all other AIs or the best achievable local configuration.
• Neither classifier was evaluated against prospective trade outcomes. Model confidence remains uncalibrated.

5. Forward-trade performance by asset class

The table uses the public verdict’s intrabar_truth fields, with independently recorded research ratings and a fresh SubQ class-rating call. The aggregate intrabar cohort’s lookback is unspecified. These metrics are published evidence, not a newly revalidated clean-trade calculation. Raw recent-window PF comes from a different population and must not be merged into intrabar results.

Recent windows reveal why a single score is inadequate

• Equity: raw 14-day PF 0.630 on 306 decisive rows; raw 48-hour PF 0.146 on 64, WR 17.19%. Investigate losing conditions, not a “high WR” story.
• Crypto: raw 14-day PF 0.957 on 2,048 decisive rows; 48-hour PF 0.752 on 296. Duplicate groups and source concentration are explicitly flagged.
• Futures: raw 48-hour PF 5.215 on 36 versus 14-day PF 0.519 on 237. A conditional lead, not persistent class profitability.
• ETF: discovery net PF 1.8488 on 14 differs from intrabar PF 0 on 28; raw 14-day PF 0.599 on 199. Cohort differences require reconciliation.
• Commodity: both recent summaries lack its class row. Recent performance is unknown from those feeds, not zero.

SubQ and our baseline agree that Commodity is the strongest relative class investigation. They do not establish that Commodity will deliver the highest future profit. No supplied class clears the documented directional money-ready requirements.

6. Latest picks, history attribution and database checks

The dashboard snapshot generated at 19:57 UTC contained 23 visible active entries, zero in the main Smart Picks list, zero Extreme Conviction entries and one in a separate older Smart feed. Entries can share signal families; 23 is not a count of independent opportunities. The feed’s own timestamp was 17:57 UTC and over three hours old at initial inspection.

Largest positive active marks included XLK +0.15%, XLF +0.12% and EWY +0.10%. Their displayed strategy forward trade counts were zero. Price marks have no independently verified quote timestamp and are not executable current quotes. The comparison does not chase these winners.

The XRP lead is real as a research question, not a proven current strategy. Public XRP SHORT pooled intrabar history is n=57 / PF 1.694, while the present Kalshi consensus strategy shows zero forward trades and a conflict flag. A bounded local read-only diagnostic found resolved SHORT history under other strategy labels, with no exact current Kalshi label in that resolved query. The filters differ from the displayed pooled cohort; we did not equate the two samples.

Read-only database checks also examined recent visible-symbol history, three large closed winners and a 10,000-row latest-ID coverage diagnostic by asset class. All queries used read-only transactions and rollback. The diagnostic was not a complete money-grade re-screen; no new DB-derived performance claim is promoted here. No private DB export was sent to an external AI or attached to this article.

During input validation we caught omitted alternative public timestamps: seven entries had imprecise entry times, but all had precise public signal metadata. A counted corrective call preserved the earlier choices. Signal time is not necessarily fill time. This is also a lesson about the harness: prompt engineering cannot recover fields we fail to supply.

7. Exact calls, tokens, cost and latency

All 20 inference calls returned HTTP 200, and all 328 answers were validated for answer keys, legal choices, type and probability bounds. The final round’s eight-call cap was enforced by a persistent attempt ledger; there were no automatic SubQ retries or failed SubQ inference calls.

API accounting also includes one authenticated model listing, one unauthenticated listing and one public metadata probe that returned route-not-found: 23 total assistant HTTP requests to api.subq.ai, of which 20 were inference. The user’s own terminal requests are excluded; documentation visits are not inference calls.

Final-round SubQ latency was 0.280–0.629 seconds per request, totalling 3.955 seconds. The local semantic comparator took 29.4 seconds, with 1,154 prompt tokens and 240 generated tokens. This is not a controlled architecture speed comparison: the models, hardware, warm state, payload format and generated output differ, and the local server may be shared.

Across the investigation, two local Qwen generations succeeded. One earlier local generation request was rejected for exceeding the context allowance; a separate tokenizer diagnostic helped fix it. No extra cloud provider key was needed.

Costs are token-only estimates at the published rate, not verified account debits or inclusive operational cost. Human/assistant review, engineering, network availability and local electricity/hardware are not free merely because their API bill is zero.

Projected benefit at volume

At 6,000 input tokens per classification request, the published price implies approximately $0.0003 per request, $0.30 per 1,000 requests, or $9 per 30,000 monthly requests. A 20,000-token request costs approximately $0.001. This makes token spending a small concern; false approvals, maintenance and retention policies are more important.

Real monetary benefit depends on whether it replaces expensive review or causes new overhead. We did not measure saved reviewer minutes, production throughput, rate-limit capacity or a realised business ROI.

8. Is our own analysis sufficient?

For the decisions we need today, mostly yes. Our analysis already identified missing strategy attribution, misleading current winners, poor recent class performance, nested staleness and the difference between research priority and capital readiness. SubQ did not supply a new validated profitable strategy.

Its demonstrated addition was a cheap, fast structured pass that matched more reference labels than this one local configuration and could route many repetitive evidence checks. It is not a replacement for data-contract knowledge: the assistant’s explicit missing-row reasoning was better than either model’s original answer.

Practical conclusion: do not buy a belief in “special trading intelligence.” At the observed token price, a small semantic shadow test is affordable. Whether it is worth keeping depends on reducing real review work without silently approving invalid claims.

9. Concrete next investigations and acceptance tests

1. XRP SHORT attribution: separate exact strategy, symbol/direction, source family and regime. Do not transplant pooled PF into Kalshi. Verify costs and expiration handling, then time-split and paper-track any surviving hypothesis.
2. Commodity/gold reconciliation: explain the absent recent class rows and identify the population behind displayed history. A missing row must be routed as unavailable evidence, never zero trades.
3. Equity loss diagnosis: preregister entry-condition hypotheses before examining held-out outcomes. Avoid choosing thresholds to fit the same bad window.
4. Futures turnaround: match recent winners to intrabar outcomes and source identity. Require persistence outside the selected 48-hour window.
5. SubQ evaluation: build a reviewer-labelled, blinded holdout set of real claims and aliases, not more obvious synthetic gates. Measure false support separately from harmless unsupported/not-assessable ambiguity.
6. Operational comparison: tune a compact local prompt and a deterministic metadata/contract baseline; repeat model runs under comparable payloads. Measure review time saved, latency, cost and abstentions.

Before any deployment as a semantic reviewer, define a maximum acceptable false-approval rate, assess a statistically meaningful holdout, and enforce explicit human escalation for unscoped history. No confidence threshold is approved from these 24 cases.

A better prompt contract

STATE: timestamped public evidence; exact history scope;
       separate intrabar / raw / net cohorts; missing fields; conflicts.

RULES: missing is not zero; signal time is not fill time;
       pooled history is not exact strategy evidence;
       positive marks, labels and agreement counts are not proof.

QUESTIONS:
  Is exact, cost-adjusted forward history documented? [yes/no]
  Which prerequisite is missing? [bounded categories]
  Is this narrowly worded claim supported? [support/invalid/unknown]
  Which investigation comes first? [candidate IDs/abstain]

APPLICATION:
  Deterministic contracts stay authoritative.
  Escalate invalid scope and disagreements.
  Never interpret confidence as a trade's profit probability.

10. Privacy, limitations and reproducibility

Credentials were read locally for their intended authentication and were not printed, committed or published. External prompts used public audit evidence or synthetic fixtures. DB diagnostics remained local and read-only. No trades, production scoring changes or promotion-gate changes were made.

SubQ’s retention documentation says indefinite retention is the default, with organisation-level zero retention available by request. Per-request store:false is documented for Chat/Responses, not Decisions; we did not assume it applies to Snap. That is another reason not to send private datasets casually.

The downloadable public evidence bundle contains the frozen assistant ratings and reasons, authored reference labels, model comparison results and per-call ledgers. It contains no keys or private DB records. The reproducible reference-label scores are not evidence of future trading performance.

Download the public experiment evidence and call ledger (JSON)

Observed limitations include prior assistant exposure, authored-case bias, an ambiguous three-way label boundary, repeated cases, a single local comparator, order/label/task sensitivity, unknown aggregate lookback, separate cohort populations, unverified confidence calibration, snapshot rather than executable prices, and no prospective financial validation.

11. Sources

1. SubQ Decisions API contract.
2. SubQ model availability and published pricing.
3. Vendor launch, architecture and benchmark claims (not independently replicated here).
4. SubQ retention policy.
5. Public dashboard payload, generated October 12, 19:57 UTC.
6. Official directional verdict and intrabar fields, generated 19:55 UTC.
7. Discovery/net PF registry, generated 19:57 UTC; explicitly not promotion-eligible.
8. 14-day raw recent cohort and 48-hour raw cohort, generated 20:05 UTC.
9. Frozen comparison evidence and inference ledgers. Live feeds can advance after publication; use this snapshot bundle for the reported numbers.