Keshav
Research21 registered entries

Research

Two programs. I write down what an experiment is supposed to show before it runs, and the result stays here whichever way it goes.

4 refuted · 3 supported · 1 superseded · 1 inconclusive · 3 mixed · 3 not run · 3 fail · 3 pass

FRONTIER

A research program of Ostrium Labs · 12 entries

Does deciding which attempt runs next get more independently verified results out of a fixed compute budget?

A deterministic Rust control plane that runs research attempts under a fixed compute budget and accepts a result only through a verifier receipt. Allocation policies — round-robin, Thompson sampling with and without hints, greedy priors — compete for the same budget; every campaign is logged to a hash-chained event log and can be replayed bit for bit.

State Simulation and deterministic-task pilots complete; no model-backed study has run

Source on GitHub ↗Case study

  1. exp-000Refuted2026-10-26

    Posterior allocation in simulation

    Called first
    With heterogeneous success rates, Thompson sampling with hints solves more problems than round-robin at the same budget.
    Setup
    Simulated verifier, 50 problems, six regime × budget cells, 200 seeds each; 24,000 runs. Paired bootstrap, 95% intervals.
    Against
    Round-robin allocation
    Result
    Five of six cells favour Thompson (R5-scarce +12.95). One does not: R2-moderate −0.19. The preregistered rule required every cell.
    Limits
    Simulated verifier. The headline values reproduce only with the Windows build's platform libm; later errata re-ran it with a portable libm and no frozen verdict changed.
    results/exp-000/README.md
  2. exp-001Refuted2026-10-26

    Why it lost: misleading hints, correlation, verifier noise

    Called first
    Mechanism M — misleading hints — explains the R2 loss.
    Setup
    40,200 simulated runs varying hint quality, problem correlation and verifier false-positive rate.
    Against
    Round-robin allocation
    Result
    The loss replicates (−0.26). Mechanism M is refuted, though confounded. At a 2% verifier false-positive rate, 8–27% of accepted results are false depending on regime.
    Limits
    Mixed scorecard; the mechanism test is confounded with hint optimism, which exp-002 isolates.
    results/exp-001/README.md
  3. exp-002Supported2026-10-26

    Optimistic hints on dead problems

    Called first
    Mechanism O: optimistic hints send attempts to problems that cannot be solved.
    Setup
    12,600 simulated runs sweeping hint optimism λ from 0 to 1.
    Against
    Round-robin allocation
    Result
    Thompson spends 46.9 more attempts on dead problems; the difference moves from −0.26 to +0.56 as λ goes 0 → 1.
    Limits
    Simulation only; identifies a mechanism, not a real-task effect.
    results/exp-002/README.md
  4. exp-003Superseded2026-10-26

    First real-task pilot

    Called first
    Adaptive allocation solves more real (non-LLM) tasks than round-robin.
    Setup
    34 deterministic DSL tasks (26 live), catalog order, 2,100 campaigns at budgets 60 / 240 / 480.
    Against
    Round-robin allocation
    Result
    +1.02 / +0.62 / +0.28. Reported as supported, then withdrawn: tasks ran in catalog order, which leaks into every policy.
    Limits
    Order confound — superseded by exp-004.
    results/exp-003/README.md
  5. exp-004Inconclusive2026-10-26

    Real tasks, order shuffled

    Called first
    The exp-003 effect survives when task order is shuffled per seed.
    Setup
    Same task set, order shuffled per seed, 100 seeds per budget.
    Against
    Round-robin allocation
    Result
    Positive at budget 60 (+1.86); not distinguishable from zero at 240 (+0.19, interval crosses 0); hint-free Thompson −0.12. FIFO order alone moves results by −9.03.
    Limits
    Effect only at the smallest budget; the hint carries most of it.
    results/exp-004/README.md
  6. exp-005Refuted2026-10-26

    How weak is the verifier?

    Called first
    The frozen acceptance rule separates suites by false-acceptance rate.
    Setup
    Offline study: 491 known-wrong programs run against each candidate test suite.
    Against
    Every other suite
    Result
    The 40-case ±20 suite accepts 1.83% (9 programs); every other suite accepts none — a floor effect, so the frozen rule cannot be tested here.
    Limits
    Floor effect: the comparison has no resolution.
    results/exp-005/study.json
  7. exp-006Mixed2026-10-27

    The price of a strong verifier

    Called first
    A stronger verifier suite removes false solves without costing yield.
    Setup
    v2 suite with 199 checks (0.995 CU per check against 0.215), 2,100 runs.
    Against
    The weaker suite, under each policy
    Result
    Zero false solves in 2,100 runs — and fewer true solves: round-robin loses 3.93 / 1.88 / 1.06 by budget. Hinted Thompson still leads round-robin by +3.41 at budget 60.
    Limits
    Costs are modelled compute units, not measured time.
    results/exp-006/README.md
  8. exp-007Supported2026-10-27

    Tasks with dependencies

    Called first
    Adaptive allocation helps more when problems depend on each other.
    Setup
    Catalog v3: 24 dependency problems (19 live), 1,800 campaigns.
    Against
    Round-robin allocation
    Result
    +4.44 at budget 120 and +4.45 at 480, shrinking to +0.48 at 1,920. A greedy prior loses 1.92 at the largest budget.
    Limits
    A pilot on a constructed task family; not a sealed benchmark.
    results/exp-007/evaluation.json
  9. exp-008Supported2026-10-27

    Fail-fast charging

    Called first
    Charging a failed verification only up to its first failing case recovers the yield the strong suite cost.
    Setup
    3,900 campaigns under a fail-fast charge model.
    Against
    Full charging
    Result
    Round-robin recovers 71% / 93% / 96% of the lost yield; dependency tasks +5.07 at budget 120. All six registered checks pass — in the charge model.
    Limits
    The saving is modelled. exp-009 tests whether it is physical.
    results/exp-008/evaluation.json
  10. exp-009Refuted2026-10-27

    Is the modelled saving real?

    Called first
    Fail-fast verification saves wall time in proportion to the modelled charge.
    Setup
    1,257 candidates × 2 modes × 3 repetitions, timed; four campaigns of 700.
    Against
    Full verification, timed
    Result
    6.4% measured against 92.9% modelled. About 92% of a verification's cost is fixed interpreter overhead (≈120.5 ms + 0.05 ms per case). Verdicts were identical.
    Limits
    One host (i5-1135G7, 8 threads); campaign-level timing was confounded by host load and is reported as uninformative.
    results/exp-009/README.md
  11. stage-2Not run

    First model-backed study

    Called first
    H1 holds with language-model workers behind the sandbox on a held-out suite.
    Setup
    Phase 0 only: replay infrastructure tested against a fake transport.
    Against
    Round-robin allocation
    Result
    Not run. No provider call has been made; it needs a model, a provider and a spending cap.
    docs/research/stage2-plan.md
  12. H2 / H3Not run

    Hybrid latent/text reasoning (HybridAgent)

    Called first
    Hybrid latent/text reasoning changes verified yield per unit of compute.
    Setup
    Adapter exists; its generate() reports unavailable when it cannot run.
    Against
    A matched text-only model (not yet built)
    Result
    Not run and blocked: needs the 8B checkpoint reproduced and a GPU training budget. Both candidate comparisons are confounded as designed.
    Limits
    Nothing about HybridAgent has been validated.
    python/frontier_py/workers/hybrid_agent.py
Next
  1. Re-run the allocation comparisons under the measured cost model (fixed overhead + per-case cost)
  2. Measure a warm-interpreter verifier
  3. Stage 2 — the first model-backed study, once a provider, model and spending cap are chosen
Reproduce
cargo test --workspace
python -m pytest python -q
bash scripts/qualify.sh

Temporal Compute

Ostrium Temporal Compute · 9 entries

What computation is still useful when its value depends on when it finishes?

A deterministic Rust simulator for jobs whose value depends on completion time, and a Python harness that drives a real renderer. Generation I asked which waiting job should run next and lost to EDF; it is retired. Generation II asks, given the intermediate state, which unit of computation is worth doing next — allocating render samples by marginal value.

State Generation I complete and negative; Generation II's primary experiment is preregistered and not run

Source on GitHub ↗Case study

  1. TC-0Mixed2026-10-06

    Seven toy fixtures

    Called first
    Explicit completion-time utility improves mission utility over deadline-only scheduling.
    Setup
    7 fixtures × 5 policies, run twice with byte-for-byte trace replay.
    Against
    FIFO, fixed priority, EDF
    Result
    Utility density wins the mixed fixture (4,370 against EDF 353); EDF wins deadline pressure (160 against 100); density loses its own trap fixture by 100 points.
    Limits
    Semantic evidence only — it refutes 'the greedy policies always beat EDF', and proves nothing at scale.
    results/README.md
  2. TC-S1Fail2026-10-06

    Temporal utility against EDF, at scale

    Called first
    The temporal-utility scheduler earns at least 5% more utility than the strongest conventional baseline.
    Setup
    Protocol frozen before the run. 200 synthetic Earth-observation scenarios × 48 tasks (9,600 jobs), 15,882 runs; five registered gates.
    Against
    FIFO, fixed priority, EDF, EDF + priority
    Result
    524,509 against EDF's 552,525 — 5.07% worse. Wins/ties/losses 15/15/170. All 12 sensitivity variants negative. Failed all five gates. The utility-density variant came within 0.35% of EDF.
    Limits
    No simulation parameter is a measured mission quantity. Temporal utility let 1,203 cloud jobs expire unstarted, against EDF's 37.
    docs/tc-s1/results.md
  3. TC-R1EFail2026-10-07

    Renderer qualification

    Called first
    A production renderer can meet five accounting and estimator requirements.
    Setup
    Mitsuba 3.9.1 and PBRT-v4 evaluated against the contract.
    Against
    —
    Result
    Qualification blocked.
    Limits
    Infrastructure gate, not a scientific result.
    docs/tc-r1/renderer-qualification.md
  4. TC-R1E2Fail2026-10-07

    PBRT accounting contract

    Called first
    PBRT's sample accounting can be proven complete on native fixtures.
    Setup
    7 scenes × 1,024 samples.
    Against
    —
    Result
    Fixtures pass; the complete-envelope proof is not established. Fails to qualify.
    docs/tc-r1/tc-r1e2-result.md
  5. TC-R1E3Pass2026-10-07

    Transmission and triangle coverage

    Called first
    The extended fixtures close the accounting gap.
    Setup
    Native fixtures in CI; 1,952 transmission events.
    Against
    —
    Result
    Pass. PBRT-v4 on the CPU is selected as the kernel.
    docs/tc-r1/tc-r1e3-result.md
  6. Native validationPass2026-10-08

    Bridge, streams and failure containment

    Called first
    The bridge, paired streams, ray accounting and failure containment hold on a bounded matrix.
    Setup
    580 of 800 commands; 547 batches; 470,577 rays.
    Against
    —
    Result
    Pass on the bounded matrix; the per-sample maximum is 13.
    Limits
    Says nothing about baseline competence, policy quality or time benefit at primary scale.
    docs/tc-r1/native-validation-result.md
  7. Offline gate v1Mixed2026-10-12

    Are the baselines competent?

    Called first
    The comparators are competent and the controller is distinguishable from them.
    Setup
    Synthetic fields without the renderer, 3 seeds.
    Against
    Uniform sampling, MC-UCB (β = 1), variance-guided
    Result
    Neither conventional baseline is competent (MC-UCB drifts 0.36–0.67 against a 0.05 limit). The controller differs from variance-guided only through its recent-improvement term.
    Limits
    Led to an amendment of the baselines before any primary run.
    docs/tc-r1/offline-gate-result.md
  8. Offline gate v2Pass2026-10-12

    Amended baselines

    Called first
    The amended comparators are competent.
    Setup
    Same design, new seeds, amendment 0004.
    Against
    Amended MC-UCB and variance-guided
    Result
    Go: drift 0. But the cost lever is 0.098, under the 0.10 threshold — the controller is distinct only through a noisy term.
    Limits
    An open question remains whether a controller that differs mainly through noise is a meaningful test of the hypothesis.
    docs/tc-r1/offline-gate-v2-result.md
  9. TC-R1Not run

    Marginal-value sample allocation (primary)

    Called first
    A state-aware marginal-value controller allocates rendering work more efficiently than uniform sampling and a strong adaptive baseline.
    Setup
    Preregistered: 10 scenes × 8 seeds. Pass requires ≥ 10% relative-MSE reduction and ≥ 5% time benefit.
    Against
    Uniform, amended MC-UCB, amended variance-guided
    Result
    Not run. The next-step review found the controller collapses to variance-guided under equal cost, and projects 1,800–4,200 single-worker hours per pass.
    Limits
    No outcomes, references or freeze tag exist yet.
    docs/tc-r1/preregistration.md
Next
  1. Deterministic charged-ray cost and a common time cap
  2. A persistent renderer worker and a bounded local measurement
  3. Host selection, then the execution freeze — before any primary run
Reproduce
cargo test --workspace --all-features
python scripts/tc_s1.py
python -m unittest discover -s experiments/tc-r1 -p 'test_*.py'