You are working on a research idea from the Modal Research Exchange: DPPO divergence-masked loss vs GRPO ratio-clipped loss, Qwen3.5-2B GSM8K LoRA RL

DPPO swaps PPO/GRPO ratio clipping for a trust region based on policy divergence; we compare GSM8K test accuracy against GRPO clipping at equal rollouts and updates.

Goal: reproduce this on Spindle, Modal's Tinker-compatible training service, using the stock tinker SDK or
tinker-cookbook pointed at TINKER_BASE_URL. Report real metrics only. If it cannot run on Spindle, say why.

Selected experiment version: exp_e6b0965af3
The saved experiment spec below defines the hypothesis, methods and success criterion.
Launch mode: redo
Purpose: Repeat the selected saved experiment version to check reproducibility. This repetition tests that version's hypothesis and methods.

Source: https://modal.enterprise.slack.com/archives/C0BMATL1S4F/p1792778791187569

Triage: DPPO replaces GRPO's ratio clipping with a divergence-based token mask. It is client-side loss code via forward_backward_custom in the cookbook RL loop on GSM8K. No fork is needed.

Experiment spec:
{
  "title": "DPPO divergence-masked loss vs GRPO ratio-clipped loss, Qwen3.5-2B GSM8K LoRA RL",
  "hypothesis": "On Qwen3.5-2B LoRA RL on GSM8K with 2 policy updates per rollout batch, the DPPO loss reaches higher greedy exact-match accuracy on the 1319-item GSM8K test set than the GRPO clipped-ratio loss. The lower bound of the paired 95% CI on DPPO minus GRPO must be above 0.",
  "method": "The requested Qwen3.5-0.8B is not served, so we use the closest served small model, Qwen/Qwen3.5-2B.\n\nBoth arms run the same client-side RL loop: the tinker-cookbook math_rl GSM8K env, renderer, prompts, reward and data order. Each step samples groups_per_batch=32 GSM8K train prompts with group_size=8 rollouts at temperature 1.0 and max_tokens=512. Advantages are group-mean-centered (GRPO).\n\nThe rollout batch is split into 2 minibatches. That gives 2 optim_steps per batch, so the policy drifts from the sampling policy and the trust region can act. With one update per fresh batch, neither mask nor clip fires. Both losses are computed in one forked custom loss path through forward_backward_custom, using the sampler logprobs as p_old.\n\nThe arms differ in exactly one switch:\n- Baseline (GRPO): token loss -min(r*A, clip(r, 1-0.2, 1+0.2)*A).\n- Treatment (DPPO, arXiv 2602.04879): token loss -r*A, with tokens masked out when the divergence between the current and sampling policy exceeds the threshold delta in the direction the advantage pushes. The divergence is estimated from the token probabilities, e.g. |p - p_old|, rather than the single-sample ratio.\n\nTriage only received a truncated digest of the paper. The exact divergence estimator, delta and masking direction must be pinned from the paper's released code before any training. Until then the correctness preflight is unresolved and no full sweep runs.\n\nTraining: LoRA rank 32, AdamW lr 2e-5, 50 rollout steps (100 optim steps) per arm, seed 0. Both arms start from the same base weights.\n\nEvaluation: the final checkpoint of each arm is scored with greedy decoding on all 1319 GSM8K test items, using the cookbook's numeric answer grader.",
  "base_model": "Qwen/Qwen3.5-2B",
  "lora_rank": 32,
  "training_type": "rl",
  "dataset": {
    "name": "gsm8k",
    "hf_path": "openai/gsm8k",
    "split": "main/train",
    "n_examples": 1600
  },
  "optimizer": {
    "name": "adam",
    "lr": 2e-05
  },
  "steps": 100,
  "batch_size": 256,
  "max_seq_len": 1024,
  "eval": {
    "benchmark": "GSM8K test (openai/gsm8k main/test)",
    "metric": "greedy exact-match accuracy",
    "n_examples": 1319,
    "aggregation": null
  },
  "baseline": "GRPO clipped-ratio loss (eps=0.2) through the same custom loss path. Identical rollouts per step, 2 minibatch updates per batch, lr, rank, seed, data order and eval.",
  "experiment_features": {
    "custom_loss": true,
    "custom_optimizer": false,
    "paired_comparison": true,
    "paired_training_arms": true,
    "checkpoint_restore": false,
    "warmup_adaptation": false,
    "generated_training_data": false,
    "learned_generator": false,
    "heldout_loss": false,
    "adaptive_selection": false
  },
  "correctness_preflight": {
    "required": true,
    "status": "unresolved: DPPO reference code revision and exact divergence/threshold not yet pinned (paper digest failed); no full sweep until resolved",
    "paths": [
      {
        "id": "grpo_dppo_custom_loss",
        "kind": "loss",
        "baseline_revision": "UNRESOLVED: pin tinker-cookbook commit whose stock ppo/importance-sampling loss defines GRPO clipping",
        "reference_revision": "UNRESOLVED: pin commit of the official DPPO implementation for arXiv 2602.04879",
        "tolerances": {
          "baseline_loss": {
            "atol": 1e-05,
            "rtol": 0.0001
          },
          "baseline_gradient": {
            "atol": 1e-05,
            "rtol": 0.001,
            "min_cosine": 0.9999
          },
          "reference_loss": {
            "atol": 1e-05,
            "rtol": 0.0001
          },
          "reference_gradient": {
            "atol": 1e-05,
            "rtol": 0.001,
            "min_cosine": 0.9999
          }
        }
      }
    ]
  },
  "activation_gate": {
    "applicable": true,
    "reason": "DPPO only differs from GRPO on tokens whose divergence exceeds the trust region. A 5-step pilot of the treatment arm, with 2 updates per batch, must show that the mask fires on a nontrivial but not dominant share of tokens and that the policy actually drifts. Executed config values must match the plan.",
    "checks": [
      {
        "metric": "dppo_masked_token_fraction",
        "measures": "fraction of loss tokens masked by the DPPO divergence rule on the second minibatch update, mean over pilot steps",
        "min": 0.001,
        "max": 0.3
      },
      {
        "metric": "policy_drift_abs_p_q90",
        "measures": "90th percentile of |p_current - p_sampler| over loss tokens at the second update",
        "min": 0.005
      },
      {
        "metric": "grpo_clip_fraction",
        "measures": "fraction of tokens clipped by the eps=0.2 ratio clip in the baseline loss on the same pilot batches (confirms the regime is off-policy for the baseline too)",
        "min": 0.0005
      },
      {
        "metric": "executed_updates_per_batch",
        "measures": "optim_step calls per rollout batch as logged",
        "min": 2,
        "max": 2
      },
      {
        "metric": "executed_base_model",
        "measures": "model name reported at client creation",
        "one_of": [
          "Qwen/Qwen3.5-2B"
        ]
      }
    ]
  },
  "restoration_preflight": {
    "applicable": true,
    "reason": "Both arms are LoRA-trained from the same base. Before the first update, verify identical initial state: compare the logprobs on a fixed tokenized GSM8K batch across both freshly created clients, which must be bitwise or within 1e-6. Also verify the active optimizer is AdamW with the planned config after creation. Server identity cannot be introspected and stays unverified.",
    "fixed_batch_required": true,
    "groups": [
      {
        "arms": [
          {
            "name": "grpo_seed0",
            "optimizer": "adam",
            "optimizer_config": {
              "learning_rate": 2e-05,
              "beta1": 0.9,
              "beta2": 0.95,
              "eps": 1e-08
            }
          },
          {
            "name": "dppo_seed0",
            "optimizer": "adam",
            "optimizer_config": {
              "learning_rate": 2e-05,
              "beta1": 0.9,
              "beta2": 0.95,
              "eps": 1e-08
            }
          }
        ]
      }
    ]
  },
  "success_criterion": "On the 1319 GSM8K test items, the paired 95% Hoeffding CI for DPPO accuracy minus GRPO accuracy has a lower bound above 0.0, roughly a 7.5-point gap at this n. A smaller positive gap is reported as inconclusive, not validated.",
  "decision_rule": {
    "version": 1,
    "estimand": "Difference in greedy GSM8K test exact-match accuracy, DPPO minus GRPO, for these fixed final checkpoints of Qwen3.5-2B LoRA; larger is better",
    "unit": "fixed_checkpoint_items",
    "seeds": [
      0
    ],
    "alpha": 0.05,
    "method": "paired_hoeffding",
    "criteria": [
      {
        "id": "primary",
        "metric": "accuracy",
        "weights": {
          "treatment": 1,
          "baseline": -1
        },
        "margin": 0.0,
        "bounds": [
          0,
          1
        ]
      }
    ]
  },
  "primary_endpoint": "gsm8k_test_greedy_exact_match_accuracy@final_checkpoint (1319 items)",
  "arms": [
    {
      "name": "grpo_seed0",
      "loss": "grpo_clip_eps0.2",
      "seed": 0,
      "lr": 2e-05,
      "updates_per_batch": 2
    },
    {
      "name": "dppo_seed0",
      "loss": "dppo_divergence_mask",
      "seed": 0,
      "lr": 2e-05,
      "updates_per_batch": 2
    }
  ],
  "feasibility_required": true,
  "feasibility": null,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 120
  },
  "generation_audit": {
    "required": false,
    "answer_comparison": "numeric_exact",
    "min_sample_size": 100,
    "max_label_errors": 0,
    "max_eval_overlaps": 0
  },
  "command": "Custom tinker SDK loop; the loss function is the custom part. Sketch:\n\nimport os, tinker\nsc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-2B', rank=32)\n\n# Rollouts: reuse tinker_cookbook.recipes.math_rl GSM8K env and renderer.\n# group_size=8, groups_per_batch=32, max_tokens=512, seed=0.\nfor step in range(50):\n    sampler = tc.save_weights_and_get_sampling_client(f's{step}')\n    batch = rollout(sampler, gsm8k_train_prompts[step], group_size=8)  # stores sampler logprobs as old_lp\n    adv = group_mean_center(batch.rewards)\n    for mb in split(batch, 2):\n        tc.forward_backward_custom(mb.data, make_loss(LOSS, old_lp=mb.old_lp, adv=mb.adv)).result()\n        tc.optim_step(tinker.AdamParams(learning_rate=2e-5)).result()\n\n# LOSS='grpo': -min(r*A, clip(r, 0.8, 1.2)*A)\n# LOSS='dppo': -(r*A) * keep_mask(div(p, p_old) <= delta), with div and delta pinned from the paper code\n# Final eval: greedy decoding on openai/gsm8k test (1319 items), numeric exact match.",
  "feasibility_check": {
    "status": "unknown",
    "problems": [
      "Measured feasibility not recorded"
    ],
    "wall_seconds": null,
    "gpu_hours": null,
    "known_gpu_hours": 0,
    "phases": {}
  }
}

How to work:
- Run the experiment on Spindle with the stock tinker SDK (or tinker-cookbook) pointed at $TINKER_BASE_URL.
  Every request needs the headers Modal-Key: $MODAL_PROXY_TOKEN_ID and Modal-Secret: $MODAL_PROXY_TOKEN_SECRET,
  plus TINKER_API_KEY. These are already set as env vars in this session: call Spindle directly from here. Don't
  search the repo for them or wrap your client in a Modal app to get them. Never print their values.
- Any Modal app you do need (a Spindle fork, a GPU eval) goes in the environment research-exchange-runs, named with
  run_21b1618aa7. Never deploy or run anything in the research-exchange environment (it is production), never mount the
  volume research-exchange-data, and never create Modal environments.
- Keep the run small and inside the spec's budget. Real numbers only; never estimate or invent a metric.
- Before any arm trains, run the spec's preflight probe (or, when it has none, create a training client with each
  arm's exact optimizer and config and take one optim_step on one example). If the shared Spindle rejects an arm's
  optimizer or option (for example an HTTP 400 at model creation), that arm needs a fork: fork as described below, or
  stop with status "failed" and outcome_reason "incomplete_execution". Never swap in a different optimizer silently.
- Compare tuned arms: if the spec gives per-arm learning-rate grids, run each arm's grid under the same budget and
  report the selected LR per arm. A selected LR at the edge of its grid makes the verdict "inconclusive".
- Before a paired comparison, run the restoration preflight in the saved spec. Save initialization_provenance
  (version 1) with one entry per named arm: initialization (pinned base_checkpoint, measured adapter_sha256,
  source=server_response or local_hash, evidence_ref, before_first_update=true), restore_count, optimizer_checks,
  and initial_output. All arms/replays in an initialization group must load the same initial checkpoint/adapter.
  A matching seed, step-0 loss, or base accuracy does not establish matched initialization (LoRA B may be zero).
  optimizer_checks must cover creation (after_restore=0) and EVERY restore (1..restore_count), before_next_update=true,
  actual optimizer family and actual_config from a server_response plus evidence_ref. Compare against each arm's
  requested optimizer/config. Creating from saved state can silently fall back to AdamW: creating TriMuon first
  then loading weights is a workaround to verify, not proof. Stop on mismatch. Do not invent SDK inspection methods;
  if no server response exposes the active optimizer, use source=declared and actual=null: this stays unverified.
  On a fork, add a minimal observation of the concrete server optimizer after restoration; retain its code revision
  and response. Check a create-save-restore fixed-gradient update against the uninterrupted client before a sweep.
  initial_output records available=true, batch_sha256 (token IDs/targets/masks), output_sha256 (all full-precision
  logits/logprobs, not an averaged loss), evidence_ref and before_first_update=true; compare across the paired group.
  If capture is unavailable, record available=false with a reason. For exact replay, require it in the spec.
  Retain the evidence before cleanup. Publish the provenance while running; full training waits for verified checks.
- Initialization proof may use measured adapter_sha256, or an immutable FULL initialization checkpoint with
  source=server_response, checkpoint_uri, checkpoint_immutable=true, checkpoint_scope=full_initialization,
  loaded_checkpoint_uri (from the actual successful load response), load_succeeded=true, evidence_ref and
  before_first_update=true. Keep base_checkpoint in both proof types. A declared URI or base-model-only load does
  not prove matching adapters. Compare all available hashes and URI identities; mixed proofs need a shared identity.
  A lone resumed training arm may form one group only when checkpoint_restore=true, paired_training_arms=false,
  and it records at least one actual restore. Exclude frozen, untrained baselines from optimizer restoration groups.
- If the spec's base_model is not served on Spindle, use the closest served model of the same family
  (e.g. Qwen/Qwen3.5-2B for a smaller Qwen3.5) and say so in the summary; a missing model is not a reason to stop.
- If the shared Spindle lacks something the experiment needs (an optimizer, loss, or feature), you may fork
  Spindle (https://github.com/modal-projects/spindle), add it, and deploy your fork following its README as a
  throwaway Modal app named spindle-fork-run_21b1618aa7 in the research-exchange-runs environment. Call your fork with
  Modal-Key: $RUNS_PROXY_TOKEN_ID and Modal-Secret: $RUNS_PROXY_TOKEN_SECRET (the shared Spindle token gets 401 on
  research-exchange-runs); they're also set as env vars here. Run every arm on the fork so the comparison is fair. Never push to, redeploy, reconfigure, or stop
  the shared Spindle deployment, and never open PRs or issues on the Spindle repo; if the change looks worth
  upstreaming, propose it in your report. Include the fork's diff and app name in your report, and when you are
  done stop spindle-fork-run_21b1618aa7 and every app your fork deployed (Spindle deploys separate trainer, lora and
  inference apps; give them names containing run_21b1618aa7), with `modal app stop -e research-exchange-runs <name>`,
  then check `modal app list -e research-exchange-runs` shows none of yours still deployed. Stop nothing else.
- Spindle errors like "trainer capacity exhausted" or 429/503 are usually temporary because other runs share it:
  retry with backoff (e.g. 1, 2, 4 minutes) and update your structured output while you wait.
  If no comparison training step has run yet and it is still refused after about 10 minutes, don't keep waiting:
  release the Spindle clients this session created where the SDK lets you (yours only), then finish with
  "status": "failed", "verdict": "inconclusive", "outcome_reason": "incomplete_execution" and "capacity_block".
  The exchange may re-queue the run and start a fresh session later, within the spec's wall-time budget. If
  comparison steps already ran, keep retrying for up to 30 minutes so finished work isn't lost, then stop and
  report "capacity_block" with the real step count.
- Release probe and setup clients you no longer need before starting the paired arms.
- Clean up after yourself before you finish: stop every Modal app, sandbox, or other resource you started for
  this run (by exact name), and nothing else.

Use the structured output tool to keep your output updated with exactly this JSON shape, and set "status" to
"succeeded" or "failed" when you are done:
{"status": "running" | "succeeded" | "failed",
 "summary": "<= 2 plain sentences: what ran and what it showed",
 "metrics": {"<name>": <number>, ..., "arms": {"<arm>": {"<name>": <number>}}},
 "completed_arms": ["<arm>"],
 "verdict": "supported" | "refuted" | "inconclusive",
 "outcome_reason": "supported" | "refuted" | "tie" | "mixed_criteria" | "insufficient_evidence" | "incomplete_execution",
 "capacity_block": {"error": "<the exact Spindle error>", "comparison_steps_completed": <int>,
                    "released_clients": true | false}}
Include "capacity_block" only when Spindle capacity stopped the run.
A finished result must include a valid "verdict". Finishing execution alone does not support the hypothesis.
Update it while you work, not only at the end: post the base eval as soon as you have it, and each arm's metrics
under "arms" with the arm added to "completed_arms" as soon as that arm finishes. Keep every number you posted
earlier in later updates. The research exchange saves each update, so if the session ends early the finished arms
still count as partial results. Leave "verdict" out until you are done.
Set "verdict" when you are done: "supported" or "refuted" only if every arm ran and the comparison judged the
success criterion; "inconclusive" for a tie, mixed criteria, insufficient evidence, or incomplete comparison.
Use "succeeded" when execution completed, including ties and mixed scientific outcomes; "failed" when execution
stopped unsuccessfully. Optional "outcome_reason" describes the science separately from execution status:
"supported"/"refuted" agree with the corresponding verdict; the other reasons require "inconclusive".
"tie" means the completed comparison falls in the spec's tie band; "mixed_criteria" means some success criteria
pass and others fail; "insufficient_evidence" means the evidence cannot settle the hypothesis (for example,
uncertainty is too large); "incomplete_execution" means required execution or evaluation did not finish.
Report the reason only with a final result. Never infer a tie or support just because execution succeeded.

Changes from the spec: the output also has
 "deviations": [{"field": "model" | "objective" | "effective_batch" | "steps" | "dataset" | "evaluator" | "endpoint",
                 "original": "<what the spec says>", "effective": "<what you run instead>", "reason": "<why>"}],
 "original_hypothesis_tested": true | false,
 "tested_instead": "<the claim this run actually tests, when original_hypothesis_tested is false>"
Use [] when you run the spec as written. effective_batch is sequences per optimizer step after accumulation.
Before you train with any change (a substituted model, a different loss or RL objective, a smaller batch, another
dataset or split, a different grader or eval set, a different decision metric or step), add it to "deviations" with
status "running" first, then continue. A change you only report at the end is flagged as unrecorded. If the change
means the run no longer tests the spec's hypothesis, set "original_hypothesis_tested": false and fill
"tested_instead". Changing the objective (e.g. GRPO to REINFORCE) or the endpoint always counts as not testing it:
the exchange then records the hypothesis as inconclusive and keeps your verdict for the changed experiment.


Correctness preflight is required before ANY full sweep using a new/custom/forked loss or optimizer, even
if discovered after launch. First reproduce the stock baseline THROUGH the new path, with treatment disabled;
then compare the enabled implementation to the pinned treatment reference. Use the same fixed inputs, weights,
optimizer state, masks, normalization, precision and seeds. Loss paths need both loss AND full gradient errors;
optimizer paths need update errors. Similar gradient norms alone do not prove parity. Check packed versus
unpacked layouts on meaningful fixed inputs, including stale inputs when relevant. Reset state independently
for each LR probe, measure update norms (per factor when applicable), and verify the prespecified LR scaling.
STOP before full training/sweeps if any check is missing or fails. If the spec has no accepted preflight plan
for a discovered custom path, report it as required/unverified and request a spec revision; do not pick a loose
tolerance after observing errors. Activation and checkpoint provenance are separate gates.
Add correctness_preflight alongside status/summary/metrics/verdict in structured output (or RESULT):
{"required": true, "paths": [{"id": "<spec path id>", "fixed_input": {"sha256": "<64 hex>",
"description": "<retrievable fixed batch/factors/gradients and reset optimizer-state fixture>"},
"baseline_revision": "<spec baseline revision>", "reference_revision": "<spec reference revision>",
"implementation_revision": "<tested code commit or content hash>", "precision": "<actual dtype/modes>",
"normalization": "<sum/mean, token mask/count and update/rank scale>",
"errors": {"<each required baseline_* and reference_* check>": {"max_abs_error": <number>,
"reference_max_abs": <number>, "relative_l2_error": <number or null for zero reference>,
"cosine_similarity": <number or null for zero vector>}},
"lr_scaling": [{"learning_rate": <number>, "update_norm": <measured number>}, ...]}]}.
For stock-only execution use {"required": false, "paths": []}. Keep probe scripts and inputs as report artifacts.
max_abs_error is max(abs(actual-reference)); relative_l2_error is norm(actual-reference)/norm(reference).
Acceptance requires max_abs_error <= atol + rtol*reference_max_abs and any prespecified min_cosine.
LR scaling checks the relative change of update_norm/LR against the first probe. These are measured errors,
not self-declared pass flags. Small standard-library helpers parity_errors and lr_scaling_error live in
research_exchange.correctness; `python -m research_exchange.correctness spec.json preflight.json` validates
evidence and exits nonzero unless passed/not_applicable. Gate full-sweep launch on that exit code when available.
The Exchange validates/persists reported checks and withholds definitive judgments when required checks fail
or are missing. It cannot prevent autonomous training calls or attest reported remote measurements.

Before full training, follow the spec's activation_gate: measure every prespecified diagnostic in a short pilot
using the exact full-run model, loss path, dtype, lag and updates per batch. Stop before full training when a
required diagnostic is missing or outside its accepted range; do not start full training. Do not change ranges after measuring.
Stopping full training does not end the investigation. First use saved evidence to distinguish a code or
instrumentation defect, a wrong reference/variant, and an inactive treatment regime. Check deterministic inputs
that must activate and must not activate, including gradients and boundary decisions. Preserve the actual loss
inputs and masks, with hashes. For policy constraints, record sampler checkpoint/version and immutable rollout
logprobs, current logprobs before each update, token identities, advantage signs, and loss-token denominators.
Log signed and absolute drift tails (p90/p99/max), threshold exceedance counts and removed-gradient counts;
zero-advantage and padding tokens must not count as treatment engagement. Distinguish optimizer calls, disjoint
minibatches, epochs over reused data, and the number of intervening updates since sampling.
Continue offline diagnosis after a gate failure. Calibration training may continue only under a recorded,
source-justified diagnostic plan with fixed settings, selection rule, update and wall-time caps within the
remaining authorized budget. Keep calibration data separate from held-out outcome evaluation. If the accepted
spec has no such plan, retain a concrete revised spec/deviation and cheapest next diagnostic; do not execute
changed training settings under the old spec. Never lower a threshold or relax a gate merely to pass it.
In the RESULT summary, state the verified or suspected cause, evidence gaps, fixes and cheapest next diagnostic.
Include activation_calibration in RESULT: {"values": {"<metric>": <measured number or string>}, "note": "<pilot
length and regime>"}. Preserve it separately from scalar metrics. The server validates reported calibration,
not remote execution; a missing or failing required check withholds a definitive scientific judgment.

Freeze the evaluator before scoring any arm. Report an evaluation object alongside metrics:
{ "version": 1, "evaluators": { "<evaluator_id>": {
  "code_sha256": "<sha256 of the executed grader/evaluation code and dependency versions>",
  "prompt_template": "<exact template, tokenizer/chat template and rendering settings>",
  "decoding": {"temperature": 0, "max_tokens": 512, "stop": []},
  "parser": "<exact answer extraction/normalization code or immutable version>",
  "items": {"ids": ["<stable IDs binding each actual input and reference>"]}
}}, "scores": [{"arm": "baseline", "metric": "accuracy", "value": 0.8,
  "evaluator_id": "<evaluator_id>", "seed": 0, "predictions_ref": "predictions.jsonl"}],
"comparisons": [{"metric": "accuracy", "arms": ["baseline", "treatment"]}] }
evaluator_id is hashlib.sha256(json.dumps(provenance, sort_keys=True, separators=(",", ":"),
ensure_ascii=True, allow_nan=False).encode()).hexdigest(). Include seed per score when applicable.
For large item lists use items {"manifest_ref": "items.jsonl", "manifest_sha256": "<sha256>"}
instead of ids; references are evidence file names. Include evaluator_id, metric, arm and item_id in
each prediction record. Keep the scored model/checkpoint separate from evaluator identity.
Any grader, parser, prompt, decoding or item change requires re-evaluating EVERY compared arm/seed
and the base model with the new evaluator, including already completed arms. Until then report
inconclusive. Keep old scores identified with their original evaluator; never relabel them.

When spec.decision_rule is present, report decision {version:1, observations:[
 {item_id:"<stable ID>",arm:"treatment",seed:0,metric:"accuracy",value:1,
  evaluator_id:"<frozen evaluator hash>"}, ...]} for every arm, item and scheduled seed.
For large evaluations, avoid putting all rows in structured output. Attach a UTF-8 JSONL
file with those same scored rows (or a JSON array / object with observations), retain it
via evidence.files as kind metrics, then report only decision {version:1,
observations_file:{name:"scored-observations.jsonl",sha256:"<sha256 of exact file bytes>"}}.
Use exactly one of observations or observations_file. The reference is a retained file
name and hash, never a path or URL. The server reads only this run's verified immutable
evidence copy, capped at 10 MiB and 100000 observations; missing/corrupt files stay
unverified. Upload a new immutable file name/hash for each update. Keep selection
metadata in the decision object. No scores, item IDs, seeds or evaluator IDs are inferred.
Values are finite measured outcomes in the endpoint's original units; never clip or
normalize them. For bounded accuracy, include failures as zero, not just successful items.
Report unrounded per-arm/seed score means so they reproduce the paired observations.
Record evaluation provenance and per-arm/seed scores with explicit items.ids (hashed
manifest-only item lists are not yet resolved by the decision validator). Persist the raw
predictions through evidence too. The server recomputes paired effects and uncertainty;
it does not trust an agent-supplied interval or parse prose. Never add a seed after seeing
the gap or treat test items/repeated evals as independent training seeds. For adaptive
selection, observations must use only confirmation items, and decision.selection must
contain frozen_before_confirmation:true. For kind weight_edit also report
budgets:{<arm>:{values_touched:N,proposals:N}} for selected and random-control arms.
Report supported/refuted only when every
criterion's confidence interval settles the prespecified margin; otherwise inconclusive.


Retain review evidence in structured output under "evidence":
{"version":1,"files":[{"name":"run.py","kind":"script","content":"<exact executed script>"},
 {"name":"effective-config.json","kind":"effective_config","content":"<effective JSON config>"}],
 "checkpoints":[{"arm":"baseline","step":100,"uri":"tinker://training-id/weights/checkpoint-name"}],
 "per_arm_metrics":{"baseline":{"accuracy":0.5}}}
Include files for the executed script, effective config, code revision (kind code_revision) and any patch
(kind code_patch), per-item predictions (kind predictions), and per-arm metrics. Include generated_tasks
when training on generated data. Use JSONL predictions with item_id, arm, prediction and reference in each row;
retain seed, checkpoint, metric and evaluator_id metadata when available. Invalid outputs must remain rows.
For larger files attach them in Devin and set source_url to the HTTPS /attachments/... URL instead of content;
never use a session page URL, VM path or arbitrary external URL. Optional sha256 must match the original bytes.
The web server copies accepted files into this run's storage and exposes reviewer downloads. Post evidence as
stages finish, before cleanup. File names are immutable: use a new name for each stage/checkpoint/update.
Each file is capped at 10 MiB, with 64 files / 40 MiB total per run. Never include .env, credentials, auth headers,
secret values or credential-bearing URLs. Read credentials from environment variables in scripts. Checkpoint
URIs are references, not copied weights; omit credentials and signed URL query strings. Missing evidence is
shown as unknown or partial, never as verified. Do not invent an artifact or checkpoint that does not exist.


Feasibility is a measured projection, separate from scientific metrics and actual GPU usage.
After resolving the served model, run a representative timing pilot before the full comparison.
Publish feasibility in structured output before continuing; repeat it after changing model, arms,
context, batch, steps, dataset size or evaluation count. Include fork/setup time, training AND
held-out evaluation, generation AND filtering samples, measured yield, and a recovery reserve.
Never scale timing by parameter count or assume concurrency is free. Measure the actual schedule.
Preserve the primary endpoint when downsizing: drop optional secondary evals, extra seeds or LR
points first, then shorten training with an explicit deviation; never replace held-out eval by
train loss. If the primary comparison cannot fit, stop and report inconclusive.
Use this additional structured field (null means no measurements yet):
"feasibility": {"config": <configuration used for the timing pilot>,
"effective_config": <current configuration including base_model, arms, steps, batch_size,
max_seq_len/max_tokens, n_eval; update this immediately on a change>, "primary_endpoint": <spec.primary_endpoint or spec.eval.metric>,
"source": <pilot log/artifact reference>, "measurements": {
"setup": {"seconds": <measured>, "units": 1, "planned_units": 1, "samples": 0,
"gpu_count": null, "allocation_source": null},
"training": {"seconds": <pilot duration>, "units": <arm-steps timed>,
"planned_units": <all arm-steps including reruns>, "samples": <examples/rollouts timed>,
"gpu_count": null, "allocation_source": null},
"evaluation": {"seconds": <pilot duration>, "units": <eval items/draws timed>,
"planned_units": <all eval items/draws across checkpoints and arms>, "samples": <items timed>,
"gpu_count": null, "allocation_source": null}},
"generation": null,
"recovery": {"seconds": <reserved wall seconds>, "gpu_count": null, "allocation_source": null}}.
For generated tasks replace generation:null by {"attempted": <pilot candidates>, "accepted":
<kept pilot candidates>, "target_accepted": <dataset size>, "filter_samples_per_candidate": <count>}
and add generation and filter measurements (units=candidates and filter rollouts respectively).
Include an audit measurement for independent task re-solves when the shared generation_audit
applicability is enforced. generated_training_data=true overrides a legacy required=false flag;
unresolved new feature declarations need resolution and cannot skip generation/filter/audit sizing.
Record GPU counts only from explicit allocation evidence, including all trainer/inference GPUs
allocated for the timed phase; null for shared Spindle when allocation is unknown. CPU-only
phases may explicitly use 0. Keep projected wall seconds, projected GPU-hours and observed usage
separate. Do not invent timings or GPU-hours to fill this contract.


Resolve applicability with experiment_features.check_applicability(spec,"generation_audit").
An explicit generated_training_data=true requires the audit. False can establish non-applicability when
no explicit contract requires the check; missing feature evidence stays unknown. For training on generated
tasks, complete generation_audit BEFORE training. Running generated Python only checks execution; it does not verify labels. Save all candidates as JSONL in evidence.files kind generated_tasks:
item_id (unique string), prompt, answer, disposition (accepted/rejected/execution_error). Retain generated code
and execution errors when available. Save the FULL frozen evaluation set as JSONL (item_id,prompt), kind
generated_tasks; this is for overlap checking, never training. Freeze both files before sampling.
Report generation_audit with version=1, stage="before_training", dataset_file, evaluation_file, audit_file
(a new immutable .json name), sample_seed (integer), counts={generated,accepted,rejected,execution_errors},
samples=[{item_id,independent_answer,resolver_id,method:"independent_resolve",reasoning,blind_to_label:true}],
label_error_count, overlap_count. Select min(accepted_count, spec.generation_audit.min_sample_size or 100)
accepted tasks deterministically with research_exchange.generation_audit.select_sample(tasks,size,seed),
which sorts by SHA256 of the JSON [seed,item_id]. Re-solve those problems from their TEXT without seeing
the stored answer or generated code (human or a separate solver invocation); record solver identity and
worked solution. Never reuse the generator's Python as independent evidence. Report observed disagreements
and the sample denominator; answer agreement with a declared solution does not establish semantic truth.
New specs must explicitly set generation_audit.answer_comparison to numeric_exact for strict scalar
numbers/fractions (zero tolerance), or categorical_exact for nonempty string labels. Categorical comparison
is exact and case-sensitive, with no whitespace normalization or synonym mapping. Choose from the label
contract, never the experiment/dataset name; missing or unsupported strategies remain unverified.
Use answer_agreement(label, independent_answer, strategy) with that declared strategy, and evaluation_overlap
for NFKC/casefold/whitespace-normalized exact prompt overlap against the full held-out set. This does not
detect paraphrases or semantic leakage. Report duplicate and overlap counts, and persist the dataset and
audit manifest through evidence before cleanup. Failed/missing audits block a definitive scientific verdict.
Call generation_audit.validate(report, spec.generation_audit, tasks=tasks, evaluation=evaluation,
require_comparison=True) and require
status="passed" before the first optimizer update. Do not invent passing re-solve evidence to get past this check.
If any sampled label is wrong or overlaps evaluation, repair/reject affected tasks and freeze NEW dataset
and audit file versions, then repeat the audit before training. Do not train on known bad labels as-is.

Instructions from whoever started this run (these override the defaults above):
User explicitly requested re-trigger after the DPPO zero-mask investigation. This launch is a BOUNDED DIAGNOSTIC continuation, not a full outcome comparison. These requester instructions override the original 50-step/two-minibatch training schedule and test evaluation. Keep the original failed pilot and accepted spec intact. Before any training, persist revised-diagnostic-spec.json, effective-config.json, and explicit deviations describing the schedule, aggregation and scope changes below; do not pretend the original hypothesis was tested.

Read the committed diagnosis and patch in modal-projects/research-exchange@c469bca5b89f1142da1cdf6e1a64d5adbe4874ce, docs/dppo-zero-mask/diagnosis.md, losses-instrumentation.patch, and scripts/audit_dppo_evidence.py. Prior run run_19badd842c evidence is available at /api/runs/run_19badd842c/evidence on this exchange. Verify manifest SHA256 and sizes. Independent offline replay passed source-gradient/mask parity; do not rely only on the previous executor's report.

First recover and archive the prior code/rx.py through authorized session access if available; inspect sampler capture time, token alignment, loss mask, and immutable old logprobs. Prior script was not downloadable. If unavailable, archive a new explicit loop and label it a replacement, not audited old code. Reuse downloadable old losses as the reference starting point, apply the retained INSTRUMENTATION-only patch, and independently rerun fixture/forced cases, both signs, recovery directions, boundaries, tails, zero advantages, padding, ratio cap, PPO clip diagnostics and gradients. Zero-advantage/padding tokens cannot count as treatment engagement. This patch changes no loss gradient. Never write a production volume or change credentials.

DIAGNOSTIC SETTINGS FIXED BEFORE MEASUREMENT: Qwen/Qwen3.5-2B, rank32; DPPO Binary-TV delta=.15, C=10 (explicit released-code variant, not paper C=5), PPO epsilon=.2 counterfactual diagnostics; original AdamW lr2e-5 beta1=.9 beta2=.95 eps1e-8 weight_decay0 grad_clip0; original renderer, temperature1, max512 and group-centered advantages. Restore immutable shared full initial state with observed successful load; do not start from trained pilot weights. One batch of 32 GSM8K TRAIN groups x8 responses, split into eight minibatches of FOUR COMPLETE GROUPS each, TWO epochs over that SAME frozen batch. Max16 optimizer updates total, no more sampling after that batch. Freeze and hash tokens, sampler logprobs, advantages and masks; assert hashes before every reuse. Sampler checkpoint/version must be recorded; no old-logprob refresh.

HARD BUDGET: at most12 wall minutes for this diagnostic including setup, sampling and training; never exceed the original experiment's2 GPU-hour cap or any lower remaining allocation. Check remaining budget before service work. No budget/concurrency increase, model/LR/threshold sweep, full training, GSM8K test evaluation or held-out outcome selection. If setup exhausts12min, persist evidence and stop. Only this single diagnostic is authorized in this launch.

Before each loss/update, persist actual current and sampler logprobs, token IDs, masks, advantages and actual dtype. Save actual decisions and removed nonzero-gradient counts, both all-loss-token and nonzero-advantage denominators, signed/absolute p90/p99/max, threshold exceedances, probability bins and ratio-cap hits. Fixed sentinel tokens from this SAME training batch: compare trainer logprobs at initialization and after updates1/2/4/8/16 to separate optimizer drift from initial trainer/sampler mismatch. Compute counterfactual PPO on same tensors; this is not a trained baseline arm.

Fixed analysis: assess ALL second-epoch loss calls pooled by loss-token counts: DPPO masked fraction .001 to .3, PPO counterfactual clip fraction>=.0005, absolute probability drift pooled q90>=.005, plus measured nonzero removed-gradient evidence for each advantage sign if that sign occurs. Declare aggregation/schedule as revisions BEFORE execution; they differ from the original second-minibatch unweighted mean. Preserve the original gate report separately; do not make original gate pass by overwriting it. Do not select the best step, lower delta, relax criteria, or automatically launch another diagnostic/full training. At16updates/12minutes stop regardless of activation. If inactive, explain using actual tails and signs; distinguish code/instrumentation defect, wrong reference/variant, legitimate inactive regime and missing evidence. If active, preserve evidence and provide the frozen revised comparison spec as the next action, without executing full comparison.

Final report must state verified/suspected cause, exact fixes, tests, limitations and cheapest next action; scientific verdict remains inconclusive for the untested accuracy hypothesis. Archive ALL executed scripts (use inline content/run evidence export if attachment retrieval fails), revised spec, sources/revisions, token fixtures, per-update data, hashes, patch, checkpoints and metrics. Do not end with an opaque activation-failed badge or silently mark an unperformed hypothesis supported/refuted.