🔬Modal Research Exchange

{
  "activation_gate": {
    "applicable": true,
    "checks": [
      {
        "measures": "rollouts per distinct prompt in logged treatment batches",
        "metric": "executed_group_size_treatment",
        "one_of": [
          "1"
        ]
      },
      {
        "measures": "rollouts per distinct prompt in logged baseline batches",
        "metric": "executed_group_size_baseline",
        "one_of": [
          "8"
        ]
      },
      {
        "max": 0.9,
        "measures": "mean training reward over the first 2 treatment steps",
        "metric": "pilot_mean_reward",
        "min": 0.1
      },
      {
        "measures": "10th percentile over rollouts of (sum w)^2/sum w^2 for the kernel weights",
        "metric": "kernel_effective_neighbors_p10",
        "min": 8.0
      },
      {
        "measures": "fraction of treatment rollouts with |advantage|>1e-6",
        "metric": "frac_nonzero_advantage_treatment",
        "min": 0.5
      },
      {
        "measures": "Pearson correlation of b_i with a held-out 8-sample per-prompt mean reward, over 64 pilot prompts",
        "metric": "baseline_reward_corr",
        "min": 0.1
      }
    ],
    "reason": "A GS1 baseline only differs from a constant baseline if the kernel draws on several neighbors with varied rewards, and the prompts give a non-degenerate reward. The executed group size must actually be 1 for the treatment and 8 for the baseline."
  },
  "arms": [
    {
      "advantage": "group_mean",
      "group_size": 8,
      "groups_per_batch": 8,
      "lr": 4e-05,
      "name": "grpo_gs8",
      "seed": 0
    },
    {
      "advantage": "kernel_cross_prompt_baseline",
      "group_size": 1,
      "groups_per_batch": 64,
      "lr": 4e-05,
      "name": "gs1_kernel",
      "seed": 0
    }
  ],
  "base_model": "Qwen/Qwen3.5-2B",
  "baseline": "GRPO via the same client loop with group_size=8, groups_per_batch=8 and group-mean advantages. It uses the same model, LoRA rank, LR, steps, rollouts per step, max_tokens, renderer, reward, seed and GSM8K eval.",
  "batch_size": 64,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 120
  },
  "command": "Custom client loop (python rx_gs1.py --arm {grpo_gs8|gs1_kernel}); the only custom part is the compute_advantages function.\n\nSetup:\n  import os, tinker\n  sc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\n  tc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-2B', rank=32, seed=0)\n  Use the tinker_cookbook math_rl gsm8k env builder for prompts, renderer and reward.\n\nEach of 40 steps:\n  1. sampler = tc.save_weights_and_get_sampling_client(...).\n  2. Draw prompts from a fixed seed-0 shuffled order: GRPO takes 8 prompts x 8 samples, GS1 takes 64 prompts x 1 sample. max_tokens=512, temperature=1.0.\n  3. Grade with the env reward.\n  4. Compute advantages: GRPO uses r minus group mean. GS1 uses r_i minus a Gaussian-kernel-weighted mean over a 256-rollout window excluding i, with phi = (log prompt length, sampler mean prompt-token logprob) and bandwidth 0.5.\n  5. Build Datums with sampling logprobs and advantages.\n  6. tc.forward_backward(datums, loss_fn='importance_sampling').\n  7. tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).\n\nAfter training: run save_weights_for_sampler, then greedy-decode all 1319 GSM8K test items for each arm. Write per-item correctness to /tmp/rx-gs1/{arm}/eval.jsonl, and score the paired Hoeffding rule from those files.\n\nActivation-failure plan:\n- (a) Offline first: unit-test compute_advantages in float64 on synthetic rewards. Check forced activation (heterogeneous phi gives nonconstant b_i) and forced nonactivation (huge bandwidth gives b_i equal to the window mean, i.e. a batch-mean REINFORCE baseline). Run gradient checks of the new loop with the treatment disabled against stock cookbook GRPO advantages on one fixed batch.\n- (b) A 2-step treatment pilot logs the gate metrics.\n- (c) If kernel_effective_neighbors fails, the single pre-registered calibration is bandwidth in {0.5, 1.0}, chosen by the gate on pilot training prompts only, never on GSM8K test. The cap is 2 pilot steps and 10 wall minutes.\n- (d) Keep the original failed pilot logs. Any other setting change requires a recorded revised spec before execution. Gates are never relaxed.\n\nNo full run happens while correctness_preflight is unresolved.",
  "correctness_preflight": {
    "paths": [
      {
        "baseline_revision": "UNRESOLVED: pin the tinker-cookbook commit SHA whose rl loop / math_rl group-mean advantage is the stock baseline",
        "id": "gs1_kernel_advantage_loop",
        "kind": "loss",
        "reference_revision": "UNRESOLVED: pin a commit of an independently written float64 numpy reference of the kernel baseline (or the Trajectory reference if released)",
        "tolerances": {
          "baseline_gradient": {
            "atol": 1e-05,
            "min_cosine": 0.99999,
            "rtol": 0.0001
          },
          "baseline_loss": {
            "atol": 1e-06,
            "rtol": 1e-05
          },
          "reference_gradient": {
            "atol": 1e-05,
            "min_cosine": 0.99999,
            "rtol": 0.0001
          },
          "reference_loss": {
            "atol": 1e-06,
            "rtol": 1e-05
          }
        }
      }
    ],
    "required": true,
    "status": "unresolved: the source was not fetched and its kernel is unreleased, so there is no trusted treatment reference; pin revisions and an independent reference before any full sweep"
  },
  "dataset": {
    "hf_path": "openai/gsm8k",
    "n_examples": 2560,
    "name": "gsm8k",
    "split": "main/train"
  },
  "decision_rule": {
    "alpha": 0.05,
    "criteria": [
      {
        "bounds": [
          0,
          1
        ],
        "id": "primary",
        "margin": -0.03,
        "metric": "accuracy",
        "weights": {
          "baseline": -1,
          "treatment": 1
        }
      }
    ],
    "estimand": "Paired difference in GSM8K test exact-match accuracy (GS1 kernel minus GRPO) for these fixed seed-0 final checkpoints; larger is better",
    "method": "paired_hoeffding",
    "seeds": [
      0
    ],
    "unit": "fixed_checkpoint_items",
    "version": 1
  },
  "eval": {
    "aggregation": null,
    "benchmark": "GSM8K test (openai/gsm8k main/test)",
    "metric": "exact-match accuracy, greedy, final checkpoint",
    "n_examples": 1319
  },
  "experiment_features": {
    "adaptive_selection": false,
    "checkpoint_restore": false,
    "custom_loss": true,
    "custom_optimizer": false,
    "generated_training_data": false,
    "heldout_loss": false,
    "learned_generator": false,
    "paired_comparison": true,
    "paired_training_arms": true,
    "warmup_adaptation": false
  },
  "feasibility": {
    "config": null,
    "effective_config": {
      "arms": [
        "grpo_gs8",
        "gs1_kernel"
      ],
      "base_model": "Qwen/Qwen3.5-2B",
      "batch_size": 64,
      "max_seq_len": 1024,
      "max_tokens": 512,
      "n_eval": 1319,
      "steps": 40
    },
    "generation": null,
    "measurements": null,
    "primary_endpoint": "gsm8k_test_exact_match_accuracy_final_checkpoint",
    "recovery": null,
    "source": null
  },
  "feasibility_check": {
    "generation_applicability": {
      "check": "generation_audit",
      "enforce": false,
      "features": {
        "generated_training_data": false
      },
      "reasons": [
        "spec.generation_audit opts out and no feature it covers is true"
      ],
      "status": "not_applicable"
    },
    "generation_counts": null,
    "gpu_hours": null,
    "known_gpu_hours": 0,
    "phases": {},
    "problems": [
      "Effective configuration not recorded",
      "Pilot evidence source not recorded",
      "Representative training timing not recorded",
      "Representative setup timing not recorded",
      "Representative evaluation timing not recorded",
      "Positive recovery reserve not recorded"
    ],
    "status": "unknown",
    "wall_seconds": null
  },
  "feasibility_required": true,
  "generation_audit": {
    "answer_comparison": "numeric_exact",
    "max_eval_overlaps": 0,
    "max_label_errors": 0,
    "min_sample_size": 100,
    "required": false
  },
  "hypothesis": "At 64 rollouts per step for 40 steps on Qwen3.5-2B, GS1 RL with a kernel-weighted cross-prompt baseline is noninferior to GRPO (group_size=8) on GSM8K test exact-match accuracy. The noninferiority margin is 3 points.",
  "lora_rank": 32,
  "max_seq_len": 1024,
  "method": "Both arms train a rank-32 LoRA on Qwen/Qwen3.5-2B from the same base. They use one client-side RL loop built on the tinker-cookbook rl loop, with the stock importance_sampling loss and AdamW. The prompts come from GSM8K train with the stock math_rl renderer and the verifiable exact-answer reward. Each step uses 64 rollouts, max_tokens=512 and temperature 1.0.\n\nThe arms differ only in the advantage estimator:\n- GRPO baseline: 8 prompts x 8 rollouts. Advantage = reward minus group mean, matching the cookbook's default.\n- GS1 treatment: 64 distinct prompts x 1 rollout. Advantage = r_i minus b_i. Here b_i is a kernel-weighted mean of rewards from other prompts in a sliding window of the last 256 rollouts. The window covers the current batch excluding i, plus the previous steps.\n\nThe kernel is fixed before running: a Gaussian kernel exp(-||phi_i - phi_j||^2 / (2*0.5^2)) on a 2-d standardized prompt feature phi = (log prompt-token length, sampling-policy mean token logprob of the prompt). The prompt logprob is computed by the sampling client before rollout. If kernel mass is below 1e-3, the estimator falls back to the window mean.\n\nThe Trajectory post could not be fetched and does not release its kernel, so this kernel is our own pre-registered choice and not the paper's exact method. Kernel hillclimbing, FlashREINFORCE, the trained-critic comparison and cross-task transfer are out of scope for this first node. Each is a follow-up spec.\n\nThe shared Spindle is enough; no fork is needed. Evaluation is greedy decoding on the full GSM8K test split (1319 items), scored by exact match, for the final checkpoint of each arm.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "preflight": {
    "probe": "For each arm, create the training client with adam, rank=32 and seed=0. Run one forward_backward with loss_fn='importance_sampling' on one sampled GSM8K example carrying that arm's computed advantage, then one optim_step(tinker.AdamParams(learning_rate=4e-5)). Confirm both succeed and the loss is finite. Then run the offline advantage unit tests and the 2-step activation pilot.",
    "riskiest_assumption": "The custom GS1 advantage path is correct and engages. Specifically, the new loop reproduces stock GRPO advantages when the treatment is disabled, and the kernel baseline gives a nonconstant, reward-predictive b_i. Otherwise a GS1-vs-GRPO gap reflects bugs or a degenerate batch-mean baseline, not the method.",
    "tuning": "Both arms use the same optimizer (AdamW) and a fixed LR of 4e-5 taken from cookbook math_rl defaults for small Qwen models. There is no per-arm grid, because the hypothesis concerns the advantage estimator at a fixed recipe. An LR-sensitivity follow-up would give each arm a 3-value grid selected on a GSM8K train holdout, with the grid-edge rule applied."
  },
  "primary_endpoint": "gsm8k_test_exact_match_accuracy_final_checkpoint",
  "restoration_preflight": {
    "applicable": true,
    "fixed_batch_required": true,
    "groups": [
      {
        "arms": [
          {
            "name": "grpo_gs8",
            "optimizer": "adam",
            "optimizer_config": {
              "beta1": 0.9,
              "beta2": 0.95,
              "eps": 1e-08,
              "lora_rank": 32,
              "lr": 4e-05,
              "seed": 0
            }
          },
          {
            "name": "gs1_kernel",
            "optimizer": "adam",
            "optimizer_config": {
              "beta1": 0.9,
              "beta2": 0.95,
              "eps": 1e-08,
              "lora_rank": 32,
              "lr": 4e-05,
              "seed": 0
            }
          }
        ]
      }
    ],
    "reason": "Both trained arms start from the same base LoRA initialization within seed 0."
  },
  "spec_preflight": {
    "findings": [],
    "status": "passed"
  },
  "steps": 40,
  "success_criterion": "GS1 minus GRPO GSM8K test exact-match accuracy, as a lower 95% Hoeffding bound over 1319 paired items, must exceed -0.03 (noninferiority margin of 3 points). If the interval straddles -0.03, the pilot is inconclusive and is not counted as a win.",
  "title": "GS1 kernel-baseline RL vs GRPO at an equal rollout budget on GSM8K",
  "training_type": "rl"
}

[
  {
    "arm": "grpo_gs8",
    "step": 40,
    "uri": "tinker://6bdf8c9f-d8e5-4ab6-b329-c8bf4c870ad6:train:0/sampler_weights/full_grpo_gs8_final"
  },
  {
    "arm": "gs1_kernel",
    "step": 40,
    "uri": "tinker://9d5a9202-7e11-4148-a5de-408a86b54120:train:0/sampler_weights/full_gs1_kernel_final"
  },
  {
    "arm": "shared_init",
    "step": 0,
    "uri": "tinker://68826b4d3bd91c3487d3433a3cd5252e/weights/rx_gs1_shared_init"
  }
]

{
  "base": {
    "accuracy": 0.714177407126611
  },
  "grpo_gs8": {
    "accuracy": 0.7285822592873389
  },
  "gs1_kernel": {
    "accuracy": 0.7589082638362395
  }
}

base_gsm8k_test_accuracy

0.7142

paired_effect_gs1_minus_grpo

0.03033

hoeffding_lower_95

-0.03707

hoeffding_upper_95

0.09772

items_gs1_only_correct

152

items_grpo_only_correct

112

• checkpoint · grpo_gs8 · step 40 · tinker://6bdf8c9f-d8e5-4ab6-b329-c8bf4c870ad6:train:0/sampler_weights/full_grpo_gs8_final
• checkpoint · gs1_kernel · step 40 · tinker://9d5a9202-7e11-4148-a5de-408a86b54120:train:0/sampler_weights/full_gs1_kernel_final
• checkpoint · shared_init · step 0 · tinker://68826b4d3bd91c3487d3433a3cd5252e/weights/rx_gs1_shared_init
• effective_config · effective-config.json
• code_revision · code-revision.json
• metrics · decision-summary.json

1. Nov 4, 18:52 UTC · succeeded · done: grpo_gs8, gs1_kernel · Both 40-step arms (GRPO 8x8 vs GS1 kernel-baseline 64x1, 64 rollouts/step, Qwen/Qwen3.5-2B rank-32 LoRA, AdamW lr 4e-5) trained from one shared initial LoRA state on shared Spindle and were greedily …
2. Nov 4, 18:37 UTC · running · Correctness preflight and activation gates passed (gates at the pre-registered calibration bandwidth 1.0 after bandwidth 0.5 failed kernel_effective_neighbors_p10); both 40-step arms are training con…
3. Nov 4, 18:31 UTC · running · Correctness preflight passed (stock GRPO reproduced through new loop; kernel impl matches independent float64 reference). Base Qwen3.5-2B GSM8K test greedy accuracy 0.7142; 2-step pilot at bandwidth …
4. Nov 4, 18:26 UTC · running · Preflight in progress for GS1 kernel-baseline vs GRPO on Qwen/Qwen3.5-2B on shared Spindle; offline float64 advantage tests passed.

correctness_preflight

{'required': True, 'paths': [{'id': 'gs1_kernel_advantage_loop', 'fixed_input': {'sha256': '27693d34e5c60fc9201a8218c318c9fb525d4362fd05e0b84ca16892809c0b29', 'description': 'Baseline: GRPO batch 0 (8 prompts x 8 rollouts, base weights, shuffled GSM8K train seed 0), datums through the new loop vs stock cookbook remove_constant_reward_groups + compute_advantages + assemble_training_data; current logprobs from tc.forward on the restored shared init (no optimizer state). Treatment fixed batch sha256 f31eafa7a4a8820ffbaa3cc464f2a6cfbdb27239ce42970db6f2130b0c0b180f (64 prompts x 1, step-0 window); features/rewards/b saved in /tmp/rx-gs1/preflight/fixture.npz'}, 'baseline_revision': 'tinker-cookbook f79fdb2ba4cd967ef2807ba43097c52985f9f17f (rl/data_processing.compute_advantages)', 'reference_revision': 'reference_kernel.py sha256 64df0cec7653f563bfe0aabee36620d616c12f64959bc4a12861491b60f7c937 (independent pure-Python float64 loops)', 'implementation_revision': 'common.py sha256 b083632d18b80a640017565202c56ecf3163da937381aa0ebeb76207d81be394', 'precision': 'advantages float64 numpy; server forward logprobs float32; loss/grad recomputed in float64', 'normalization': 'importance_sampling loss = -sum over action tokens of exp(lp-lp_sampled)*A (sum, no token-count normalization); gradient = dLoss/dlogprob per action token, concatenated over datums; prompt tokens excluded', 'errors': {'baseline_loss': {'max_abs_error': 9.8452167094365e-06, 'reference_max_abs': 1465.3696715088377, 'relative_l2_error': 6.718589104754188e-09, 'cosine_similarity': 1.0}, 'baseline_gradient': {'max_abs_error': 4.810152027800996e-07, 'reference_max_abs': 1.0900486795400675, 'relative_l2_error': 9.442298521549862e-08, 'cosine_similarity': 0.9999999999999954}, 'reference_loss': {'max_abs_error': 0.0, 'reference_max_abs': 3046.5462369418633, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}, 'reference_gradient': {'max_abs_error': 0.0, 'reference_max_abs': 1.2409016033993248, 'relative_l2_error': 0.0, 'cosine_similarity': 0.9999999999999999}}, 'lr_scaling': []}]}

correctness_validation

{'status': 'unverified', 'required': True, 'checks': [{'path': 'gs1_kernel_advantage_loop', 'check': 'baseline_loss', 'status': 'passed', 'tolerance': {'atol': 1e-06, 'rtol': 1e-05}, 'measured': {'max_abs_error': 9.8452167094365e-06, 'reference_max_abs': 1465.3696715088377, 'relative_l2_error': 6.718589104754188e-09, 'cosine_similarity': 1.0}}, {'path': 'gs1_kernel_advantage_loop', 'check': 'baseline_gradient', 'status': 'passed', 'tolerance': {'atol': 1e-05, 'rtol': 0.0001, 'min_cosine': 0.99999}, 'measured': {'max_abs_error': 4.810152027800996e-07, 'reference_max_abs': 1.0900486795400675, 'relative_l2_error': 9.442298521549862e-08, 'cosine_similarity': 0.9999999999999954}}, {'path': 'gs1_kernel_advantage_loop', 'check': 'reference_loss', 'status': 'passed', 'tolerance': {'atol': 1e-06, 'rtol': 1e-05}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 3046.5462369418633, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, {'path': 'gs1_kernel_advantage_loop', 'check': 'reference_gradient', 'status': 'passed', 'tolerance': {'atol': 1e-05, 'rtol': 0.0001, 'min_cosine': 0.99999}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 1.2409016033993248, 'relative_l2_error': 0.0, 'cosine_similarity': 0.9999999999999999}}], 'reason': 'gs1_kernel_advantage_loop: matching reference_revision; gs1_kernel_advantage_loop: matching baseline_revision', 'evidence_source': 'executor report; remote execution is not attested', 'applicability': 'required', 'reported_verdict': 'refuted'}

decision

{'version': 1, 'observations_file': {'name': 'scored-observations.jsonl', 'sha256': '1cb63f2e14ee5b6c10b796f569f11dbe7f5a2b72e2387db257f1845b3008576b'}}

decision_validation

{'status': 'unverified', 'issues': ['Observation file name/hash is not retained for this run.'], 'criteria': [], 'outcome_reason': 'insufficient_evidence', 'applicability': {'check': 'decision_rule', 'status': 'required', 'enforce': True, 'reasons': ['experiment_features: paired_comparison is true'], 'features': {'paired_comparison': True, 'adaptive_selection': False}}, 'reported_verdict': 'refuted'}

evaluation

{'version': 1, 'evaluators': {'3180ae93de6fc3ec2e276b113d523b701ce8381c3112332664587db3ca857f13': {'code_sha256': 'sha256 over common.py, cookbook math_env.py, math_grading.py, renderers/qwen3_5.py and dep versions (tinker 0.37.0, tinker-cookbook f79fdb2); see evaluator.py', 'prompt_template': "tinker-cookbook@f79fdb2 MathEnv: renderer qwen3_5_disable_thinking (Qwen/Qwen3.5-2B tokenizer), standard_fewshot_prefix + question + ' Write your answer in \\boxed{} format.', build_generation_prompt", 'decoding': {'temperature': 0, 'max_tokens': 512, 'stop': ['renderer stop sequences (<|im_end|>)']}, 'parser': 'renderer.parse_response -> get_text_content -> MathEnv.check_answer (extract_boxed + sympy grade_answer, math_grading.py @ f79fdb2)', 'items': {'ids': 'gsm8k_test_0 .. gsm8k_test_1318 (openai/gsm8k main test, dataset order)'}}}, 'scores': [{'arm': 'base', 'metric': 'accuracy', 'value': 0.714177407126611, 'evaluator_id': '3180ae93de6fc3ec2e276b113d523b701ce8381c3112332664587db3ca857f13', 'seed': 0, 'predictions_ref': 'base/eval.jsonl'}, {'arm': 'grpo_gs8', 'metric': 'accuracy', 'value': 0.7285822592873389, 'evaluator_id': '3180ae93de6fc3ec2e276b113d523b701ce8381c3112332664587db3ca857f13', 'seed': 0, 'predictions_ref': 'grpo_gs8/eval.jsonl'}, {'arm': 'gs1_kernel', 'metric': 'accuracy', 'value': 0.7589082638362395, 'evaluator_id': '3180ae93de6fc3ec2e276b113d523b701ce8381c3112332664587db3ca857f13', 'seed': 0, 'predictions_ref': 'gs1_kernel/eval.jsonl'}], 'comparisons': [{'metric': 'accuracy', 'arms': ['grpo_gs8', 'gs1_kernel']}]}

evaluator_validation

{'status': 'invalid', 'issues': ['Evaluator 3180ae93de6fc3ec2e276b113d523b701ce8381c3112332664587db3ca857f13: item IDs or a hashed item manifest are required.', 'Evaluator 3180ae93de6fc3ec2e276b113d523b701ce8381c3112332664587db3ca857f13: code_sha256 is required.', 'Evaluator 3180ae93de6fc3ec2e276b113d523b701ce8381c3112332664587db3ca857f13: identity does not match provenance hash.'], 'comparisons': [{'metric': 'accuracy', 'status': 'comparable', 'arms': ['base', 'grpo_gs8', 'gs1_kernel'], 'evaluator_ids': ['3180ae93de6fc3ec2e276b113d523b701ce8381c3112332664587db3ca857f13'], 'missing_arms': []}]}

feasibility

{'status': 'unknown', 'problems': ['Re-projection required: base_model changed or was not recorded', 'Re-projection required: lora_rank changed or was not recorded', 'Re-projection required: dataset changed or was not recorded', 'Re-projection required: steps changed or was not recorded', 'Re-projection required: batch_size changed or was not recorded', 'Re-projection required: max_seq_len changed or was not recorded', 'Re-projection required: max_tokens changed or was not recorded', 'Re-projection required: arms changed or was not recorded', 'Re-projection required: n_examples changed or was not recorded', 'Re-projection required: n_eval changed or was not recorded'], 'wall_seconds': None, 'gpu_hours': None, 'known_gpu_hours': 0, 'phases': {'setup': {'wall_seconds': 30.0, 'planned_units': 1, 'planned_samples': 0.0, 'gpu_hours': None}, 'training': {'wall_seconds': 2960.0, 'planned_units': 80, 'planned_samples': 5120.0, 'gpu_hours': None}, 'evaluation': {'wall_seconds': 78.0, 'planned_units': 3957, 'planned_samples': 3957.0, 'gpu_hours': None}, 'recovery': {'wall_seconds': 1800, 'gpu_hours': None}}, 'generation_counts': None, 'generation_applicability': {'check': 'generation_audit', 'status': 'not_applicable', 'enforce': False, 'reasons': ['spec.generation_audit opts out and no feature it covers is true'], 'features': {'generated_training_data': False}}, 'record': {'config': {'pilot': '2 gs1 steps + 1 grpo step, 64 rollouts/step, max_tokens 512; full GSM8K test base eval'}, 'effective_config': {'base_model': 'Qwen/Qwen3.5-2B', 'arms': ['grpo_gs8', 'gs1_kernel'], 'steps': 40, 'batch_size': 64, 'max_tokens': 512, 'max_seq_len': 1024, 'n_eval': 1319, 'renderer': 'qwen3_5_disable_thinking', 'bandwidth': 1.0}, 'primary_endpoint': 'gsm8k_test_exact_match_accuracy_final_checkpoint', 'source': '/tmp/rx-gs1/pilot_bw1.0, pilot_bw0.5_FAILED_ess, base/result.json', 'measurements': {'setup': {'seconds': 30, 'units': 1, 'planned_units': 1, 'samples': 0, 'gpu_count': None, 'allocation_source': None}, 'training': {'seconds': 74, 'units': 2, 'planned_units': 80, 'samples': 128, 'gpu_count': None, 'allocation_source': None}, 'evaluation': {'seconds': 26.0, 'units': 1319, 'planned_units': 3957, 'samples': 1319, 'gpu_count': None, 'allocation_source': None}}, 'generation': None, 'recovery': {'seconds': 1800, 'gpu_count': None, 'allocation_source': None}}, 'recorded_at': 1793818345.2719748}

feasibility_reported_verdict

refuted

feasibility_required

True

generation_audit_validation

{'status': 'not_applicable', 'reasons': ['spec.generation_audit opts out and no feature it covers is true'], 'applicability': {'check': 'generation_audit', 'status': 'not_applicable', 'enforce': False, 'reasons': ['spec.generation_audit opts out and no feature it covers is true'], 'features': {'generated_training_data': False}}}

initialization_provenance

{'version': 1, 'arms': {'grpo_gs8': {'initialization': {'base_checkpoint': 'Qwen/Qwen3.5-2B', 'checkpoint_uri': 'tinker://68826b4d3bd91c3487d3433a3cd5252e/weights/rx_gs1_shared_init', 'checkpoint_immutable': True, 'checkpoint_scope': 'full_initialization', 'loaded_checkpoint_uri': 'tinker://68826b4d3bd91c3487d3433a3cd5252e/weights/rx_gs1_shared_init', 'load_succeeded': True, 'adapter_sha256': None, 'source': 'server_response', 'evidence_ref': 'full/grpo_gs8/init.json', 'before_first_update': True, 'note': 'per-model seed rejected by Spindle; save_weights_external returns 404, so adapter bytes could not be hashed'}, 'restore_count': 1, 'optimizer_checks': [{'after_restore': 1, 'before_next_update': True, 'requested': {'family': 'adamw', 'lr': 4e-05, 'beta1': 0.9, 'beta2': 0.95, 'eps': 1e-08}, 'actual_family': 'adamw', 'actual_config': {'type': 'adamw'}, 'source': 'server_response', 'evidence_ref': 'full/grpo_gs8/init.json get_info.optimizer_config'}], 'initial_output': {'available': True, 'batch_sha256': 'ac29bdd156bb2eadeb594d90d224b21e7fe872c1fead938bd6e044edfd3dfc45', 'output_sha256': 'c4c5fce62c860e40dafd529d82274b4ef15f3bac860ee57feb6b47ea741f67c2', 'evidence_ref': 'full/grpo_gs8/init.json', 'before_first_update': True}}, 'gs1_kernel': {'initialization': {'base_checkpoint': 'Qwen/Qwen3.5-2B', 'checkpoint_uri': 'tinker://68826b4d3bd91c3487d3433a3cd5252e/weights/rx_gs1_shared_init', 'checkpoint_immutable': True, 'checkpoint_scope': 'full_initialization', 'loaded_checkpoint_uri': 'tinker://68826b4d3bd91c3487d3433a3cd5252e/weights/rx_gs1_shared_init', 'load_succeeded': True, 'adapter_sha256': None, 'source': 'server_response', 'evidence_ref': 'full/gs1_kernel/init.json', 'before_first_update': True, 'note': 'per-model seed rejected by Spindle; save_weights_external returns 404, so adapter bytes could not be hashed'}, 'restore_count': 1, 'optimizer_checks': [{'after_restore': 1, 'before_next_update': True, 'requested': {'family': 'adamw', 'lr': 4e-05, 'beta1': 0.9, 'beta2': 0.95, 'eps': 1e-08}, 'actual_family': 'adamw', 'actual_config': {'type': 'adamw'}, 'source': 'server_response', 'evidence_ref': 'full/gs1_kernel/init.json get_info.optimizer_config'}], 'initial_output': {'available': True, 'batch_sha256': 'ac29bdd156bb2eadeb594d90d224b21e7fe872c1fead938bd6e044edfd3dfc45', 'output_sha256': 'c4c5fce62c860e40dafd529d82274b4ef15f3bac860ee57feb6b47ea741f67c2', 'evidence_ref': 'full/gs1_kernel/init.json', 'before_first_update': True}}}, 'probe': {'grpo_gs8': {'loss': -36.20917892456055, 'finite': True, 'optim_step': 'update_successful'}, 'gs1_kernel': {'loss': -64.30147552490234, 'finite': True, 'optim_step': 'update_successful'}}}

last_reply

07d81be394\",\"reference_kernel.py\":\"64df0cec7653f563bfe0aabee36620d616c12f64959bc4a12861491b60f7c937\",\"offline_tests.py\":\"8fd017226d217e712ec0e7dfced8e9ad4e4a72d672a82889e99e46727f67d87b\",\"preflight.py\":\"0d7a118dfc4b0eef0c2d6af98fd5518ba97e4a4165db2b8315cef1efaf0f2702\",\"evaluator.py\":\"4b9d1464b694ca625f0393e9473513e75e273291f58feae0a917a024beae0c1f\",\"score.py\":\"d1a09c4e46ff29bc0e167c33b557a7d820f93e174426323d921a8589516c7cbc\",\"base_eval.py\":\"af99383c1e7b677985e5dd730fa7a764bfc4ccb7e1480137dcd6dc54d8787262\"},\"predictions_sha256\":{\"base/eval.jsonl\":\"12aa2ae9cfe90775b10c85c4619eb814ae3d7ebc5e1357ed274181383261d92e\",\"grpo_gs8/eval.jsonl\":\"f06fc564f2f60bfa8d1d156d6f00afbd2a5df3fb6c8e8abf09e6aa6ba1ebbcff\",\"gs1_kernel/eval.jsonl\":\"a3fd59ac6b4a34f4751b927a02d5a9cf91e1abf016fb1bfde460bfa3b51d8b6e\"},\"note\":\"full script and prediction files too large to inline; hashes only\"}"}, {"name": "decision-summary.json", "kind": "metrics", "content": "{\"n\":1319,\"effect\":0.030326004548900682,\"lower\":-0.03707154383198885,\"upper\":0.09772355292979021,\"halfwidth_one_sided\":0.06739754838088953,\"acc\":{\"grpo_gs8\":0.7285822592873389,\"gs1_kernel\":0.7589082638362395,\"base\":0.714177407126611},\"wins_gs1\":152,\"wins_grpo\":112,\"supported\":false,\"observations_sha256\":\"1cb63f2e14ee5b6c10b796f569f11dbe7f5a2b72e2387db257f1845b3008576b\"}"} ], "checkpoints": [ {"arm": "grpo_gs8", "step": 40, "uri": "tinker://6bdf8c9f-d8e5-4ab6-b329-c8bf4c870ad6:train:0/sampler_weights/full_grpo_gs8_final"}, {"arm": "gs1_kernel", "step": 40, "uri": "tinker://9d5a9202-7e11-4148-a5de-408a86b54120:train:0/sampler_weights/full_gs1_kernel_final"}, {"arm": "shared_init", "step": 0, "uri": "tinker://68826b4d3bd91c3487d3433a3cd5252e/weights/rx_gs1_shared_init"} ], "per_arm_metrics": {"base": {"accuracy": 0.714177407126611}, "grpo_gs8": {"accuracy": 0.7285822592873389}, "gs1_kernel": {"accuracy": 0.7589082638362395}} } } ```

launch

{'experiment_id': 'exp_32150a771c', 'context_status': 'snapshotted', 'superseded': False, 'mode': 'ordinary', 'purpose': 'Execute the selected saved experiment version.'}

progress_updated_at

1793817454.119873

reported_verdict

refuted

restoration_validation

{'status': 'unverified', 'required': True, 'checks': [{'check': 'grpo_gs8', 'status': 'unverified', 'reason': 'grpo_gs8: missing or duplicate arm evidence.'}, {'check': 'gs1_kernel', 'status': 'unverified', 'reason': 'gs1_kernel: missing or duplicate arm evidence.'}], 'reason': 'grpo_gs8: missing or duplicate arm evidence.', 'applicability': 'required'}

slack_started

True

started_at

1793816221.766266

verdict_report

{'effect': 0.030326004548900682, 'margin': -0.03, 'lower': -0.03707154383198885, 'upper': 0.09772355292979021, 'seeds': 1, 'confidence': 'weak', 'reasoning': 'GS1 beat GRPO by 3.0 points (0.7589 vs 0.7286), but the one-sided 95% paired Hoeffding lower bound over 1319 items is -0.0371, below the -0.03 margin. Under the pre-registered rule, an interval straddling the margin is not a win, so the result is recorded as refuted. Confidence is weak: one training seed, a wide Hoeffding interval (half-width 0.067), the treatment ran at the pre-registered calibration bandwidth 1.0 because bandwidth 0.5 failed the ESS gate, and Spindle rejected per-model seeds, so matched initialization rests on a shared restored checkpoint rather than a measured adapter hash.'}