🔬Modal Research Exchange

{
  "activation_gate": {
    "applicable": true,
    "checks": [
      {
        "measures": "rollouts per distinct prompt in logged treatment batches",
        "metric": "executed_group_size_treatment",
        "one_of": [
          "1"
        ]
      },
      {
        "measures": "rollouts per distinct prompt in logged baseline batches",
        "metric": "executed_group_size_baseline",
        "one_of": [
          "8"
        ]
      },
      {
        "max": 0.9,
        "measures": "mean training reward over the first 2 treatment steps",
        "metric": "pilot_mean_reward",
        "min": 0.1
      },
      {
        "measures": "10th percentile over rollouts of (sum w)^2/sum w^2 for the kernel weights",
        "metric": "kernel_effective_neighbors_p10",
        "min": 8.0
      },
      {
        "measures": "fraction of treatment rollouts with |advantage|>1e-6",
        "metric": "frac_nonzero_advantage_treatment",
        "min": 0.5
      },
      {
        "measures": "Pearson correlation of b_i with a held-out 8-sample per-prompt mean reward, over 64 pilot prompts",
        "metric": "baseline_reward_corr",
        "min": 0.1
      }
    ],
    "reason": "A GS1 baseline only differs from a constant baseline if the kernel draws on several neighbors with varied rewards, and the prompts give a non-degenerate reward. The executed group size must actually be 1 for the treatment and 8 for the baseline."
  },
  "arms": [
    {
      "advantage": "group_mean",
      "group_size": 8,
      "groups_per_batch": 8,
      "lr": 4e-05,
      "name": "grpo_gs8",
      "seed": 0
    },
    {
      "advantage": "kernel_cross_prompt_baseline",
      "group_size": 1,
      "groups_per_batch": 64,
      "lr": 4e-05,
      "name": "gs1_kernel",
      "seed": 0
    }
  ],
  "base_model": "Qwen/Qwen3.5-2B",
  "baseline": "GRPO via the same client loop with group_size=8, groups_per_batch=8 and group-mean advantages. It uses the same model, LoRA rank, LR, steps, rollouts per step, max_tokens, renderer, reward, seed and GSM8K eval.",
  "batch_size": 64,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 120
  },
  "command": "Custom client loop (python rx_gs1.py --arm {grpo_gs8|gs1_kernel}); the only custom part is the compute_advantages function.\n\nSetup:\n  import os, tinker\n  sc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\n  tc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-2B', rank=32, seed=0)\n  Use the tinker_cookbook math_rl gsm8k env builder for prompts, renderer and reward.\n\nEach of 40 steps:\n  1. sampler = tc.save_weights_and_get_sampling_client(...).\n  2. Draw prompts from a fixed seed-0 shuffled order: GRPO takes 8 prompts x 8 samples, GS1 takes 64 prompts x 1 sample. max_tokens=512, temperature=1.0.\n  3. Grade with the env reward.\n  4. Compute advantages: GRPO uses r minus group mean. GS1 uses r_i minus a Gaussian-kernel-weighted mean over a 256-rollout window excluding i, with phi = (log prompt length, sampler mean prompt-token logprob) and bandwidth 0.5.\n  5. Build Datums with sampling logprobs and advantages.\n  6. tc.forward_backward(datums, loss_fn='importance_sampling').\n  7. tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).\n\nAfter training: run save_weights_for_sampler, then greedy-decode all 1319 GSM8K test items for each arm. Write per-item correctness to /tmp/rx-gs1/{arm}/eval.jsonl, and score the paired Hoeffding rule from those files.\n\nActivation-failure plan:\n- (a) Offline first: unit-test compute_advantages in float64 on synthetic rewards. Check forced activation (heterogeneous phi gives nonconstant b_i) and forced nonactivation (huge bandwidth gives b_i equal to the window mean, i.e. a batch-mean REINFORCE baseline). Run gradient checks of the new loop with the treatment disabled against stock cookbook GRPO advantages on one fixed batch.\n- (b) A 2-step treatment pilot logs the gate metrics.\n- (c) If kernel_effective_neighbors fails, the single pre-registered calibration is bandwidth in {0.5, 1.0}, chosen by the gate on pilot training prompts only, never on GSM8K test. The cap is 2 pilot steps and 10 wall minutes.\n- (d) Keep the original failed pilot logs. Any other setting change requires a recorded revised spec before execution. Gates are never relaxed.\n\nNo full run happens while correctness_preflight is unresolved.",
  "correctness_preflight": {
    "paths": [
      {
        "baseline_revision": "UNRESOLVED: pin the tinker-cookbook commit SHA whose rl loop / math_rl group-mean advantage is the stock baseline",
        "id": "gs1_kernel_advantage_loop",
        "kind": "loss",
        "reference_revision": "UNRESOLVED: pin a commit of an independently written float64 numpy reference of the kernel baseline (or the Trajectory reference if released)",
        "tolerances": {
          "baseline_gradient": {
            "atol": 1e-05,
            "min_cosine": 0.99999,
            "rtol": 0.0001
          },
          "baseline_loss": {
            "atol": 1e-06,
            "rtol": 1e-05
          },
          "reference_gradient": {
            "atol": 1e-05,
            "min_cosine": 0.99999,
            "rtol": 0.0001
          },
          "reference_loss": {
            "atol": 1e-06,
            "rtol": 1e-05
          }
        }
      }
    ],
    "required": true,
    "status": "unresolved: the source was not fetched and its kernel is unreleased, so there is no trusted treatment reference; pin revisions and an independent reference before any full sweep"
  },
  "dataset": {
    "hf_path": "openai/gsm8k",
    "n_examples": 2560,
    "name": "gsm8k",
    "split": "main/train"
  },
  "decision_rule": {
    "alpha": 0.05,
    "criteria": [
      {
        "bounds": [
          0,
          1
        ],
        "id": "primary",
        "margin": -0.03,
        "metric": "accuracy",
        "weights": {
          "baseline": -1,
          "treatment": 1
        }
      }
    ],
    "estimand": "Paired difference in GSM8K test exact-match accuracy (GS1 kernel minus GRPO) for these fixed seed-0 final checkpoints; larger is better",
    "method": "paired_hoeffding",
    "seeds": [
      0
    ],
    "unit": "fixed_checkpoint_items",
    "version": 1
  },
  "eval": {
    "aggregation": null,
    "benchmark": "GSM8K test (openai/gsm8k main/test)",
    "metric": "exact-match accuracy, greedy, final checkpoint",
    "n_examples": 1319
  },
  "experiment_features": {
    "adaptive_selection": false,
    "checkpoint_restore": false,
    "custom_loss": true,
    "custom_optimizer": false,
    "generated_training_data": false,
    "heldout_loss": false,
    "learned_generator": false,
    "paired_comparison": true,
    "paired_training_arms": true,
    "warmup_adaptation": false
  },
  "feasibility": {
    "config": null,
    "effective_config": {
      "arms": [
        "grpo_gs8",
        "gs1_kernel"
      ],
      "base_model": "Qwen/Qwen3.5-2B",
      "batch_size": 64,
      "max_seq_len": 1024,
      "max_tokens": 512,
      "n_eval": 1319,
      "steps": 40
    },
    "generation": null,
    "measurements": null,
    "primary_endpoint": "gsm8k_test_exact_match_accuracy_final_checkpoint",
    "recovery": null,
    "source": null
  },
  "feasibility_check": {
    "generation_applicability": {
      "check": "generation_audit",
      "enforce": false,
      "features": {
        "generated_training_data": false
      },
      "reasons": [
        "spec.generation_audit opts out and no feature it covers is true"
      ],
      "status": "not_applicable"
    },
    "generation_counts": null,
    "gpu_hours": null,
    "known_gpu_hours": 0,
    "phases": {},
    "problems": [
      "Effective configuration not recorded",
      "Pilot evidence source not recorded",
      "Representative training timing not recorded",
      "Representative setup timing not recorded",
      "Representative evaluation timing not recorded",
      "Positive recovery reserve not recorded"
    ],
    "status": "unknown",
    "wall_seconds": null
  },
  "feasibility_required": true,
  "generation_audit": {
    "answer_comparison": "numeric_exact",
    "max_eval_overlaps": 0,
    "max_label_errors": 0,
    "min_sample_size": 100,
    "required": false
  },
  "hypothesis": "At 64 rollouts per step for 40 steps on Qwen3.5-2B, GS1 RL with a kernel-weighted cross-prompt baseline is noninferior to GRPO (group_size=8) on GSM8K test exact-match accuracy. The noninferiority margin is 3 points.",
  "lora_rank": 32,
  "max_seq_len": 1024,
  "method": "Both arms train a rank-32 LoRA on Qwen/Qwen3.5-2B from the same base. They use one client-side RL loop built on the tinker-cookbook rl loop, with the stock importance_sampling loss and AdamW. The prompts come from GSM8K train with the stock math_rl renderer and the verifiable exact-answer reward. Each step uses 64 rollouts, max_tokens=512 and temperature 1.0.\n\nThe arms differ only in the advantage estimator:\n- GRPO baseline: 8 prompts x 8 rollouts. Advantage = reward minus group mean, matching the cookbook's default.\n- GS1 treatment: 64 distinct prompts x 1 rollout. Advantage = r_i minus b_i. Here b_i is a kernel-weighted mean of rewards from other prompts in a sliding window of the last 256 rollouts. The window covers the current batch excluding i, plus the previous steps.\n\nThe kernel is fixed before running: a Gaussian kernel exp(-||phi_i - phi_j||^2 / (2*0.5^2)) on a 2-d standardized prompt feature phi = (log prompt-token length, sampling-policy mean token logprob of the prompt). The prompt logprob is computed by the sampling client before rollout. If kernel mass is below 1e-3, the estimator falls back to the window mean.\n\nThe Trajectory post could not be fetched and does not release its kernel, so this kernel is our own pre-registered choice and not the paper's exact method. Kernel hillclimbing, FlashREINFORCE, the trained-critic comparison and cross-task transfer are out of scope for this first node. Each is a follow-up spec.\n\nThe shared Spindle is enough; no fork is needed. Evaluation is greedy decoding on the full GSM8K test split (1319 items), scored by exact match, for the final checkpoint of each arm.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "preflight": {
    "probe": "For each arm, create the training client with adam, rank=32 and seed=0. Run one forward_backward with loss_fn='importance_sampling' on one sampled GSM8K example carrying that arm's computed advantage, then one optim_step(tinker.AdamParams(learning_rate=4e-5)). Confirm both succeed and the loss is finite. Then run the offline advantage unit tests and the 2-step activation pilot.",
    "riskiest_assumption": "The custom GS1 advantage path is correct and engages. Specifically, the new loop reproduces stock GRPO advantages when the treatment is disabled, and the kernel baseline gives a nonconstant, reward-predictive b_i. Otherwise a GS1-vs-GRPO gap reflects bugs or a degenerate batch-mean baseline, not the method.",
    "tuning": "Both arms use the same optimizer (AdamW) and a fixed LR of 4e-5 taken from cookbook math_rl defaults for small Qwen models. There is no per-arm grid, because the hypothesis concerns the advantage estimator at a fixed recipe. An LR-sensitivity follow-up would give each arm a 3-value grid selected on a GSM8K train holdout, with the grid-edge rule applied."
  },
  "primary_endpoint": "gsm8k_test_exact_match_accuracy_final_checkpoint",
  "restoration_preflight": {
    "applicable": true,
    "fixed_batch_required": true,
    "groups": [
      {
        "arms": [
          {
            "name": "grpo_gs8",
            "optimizer": "adam",
            "optimizer_config": {
              "beta1": 0.9,
              "beta2": 0.95,
              "eps": 1e-08,
              "lora_rank": 32,
              "lr": 4e-05,
              "seed": 0
            }
          },
          {
            "name": "gs1_kernel",
            "optimizer": "adam",
            "optimizer_config": {
              "beta1": 0.9,
              "beta2": 0.95,
              "eps": 1e-08,
              "lora_rank": 32,
              "lr": 4e-05,
              "seed": 0
            }
          }
        ]
      }
    ],
    "reason": "Both trained arms start from the same base LoRA initialization within seed 0."
  },
  "spec_preflight": {
    "findings": [],
    "status": "passed"
  },
  "steps": 40,
  "success_criterion": "GS1 minus GRPO GSM8K test exact-match accuracy, as a lower 95% Hoeffding bound over 1319 paired items, must exceed -0.03 (noninferiority margin of 3 points). If the interval straddles -0.03, the pilot is inconclusive and is not counted as a win.",
  "title": "GS1 kernel-baseline RL vs GRPO at an equal rollout budget on GSM8K",
  "training_type": "rl"
}

[
  {
    "arm": "grpo_gs8",
    "step": 40,
    "uri": "tinker://13a563a8fcb5108149fcf830eaa613b0:train:0/sampler_weights/grpo_gs8-final"
  },
  {
    "arm": "gs1_kernel",
    "step": 40,
    "uri": "tinker://e3180b48c9f5a53786775fab2b71413d:train:0/sampler_weights/gs1_kernel-final"
  }
]

{
  "base": {
    "accuracy": 0.7020470053070508
  },
  "grpo_gs8": {
    "accuracy": 0.7536012130401819
  },
  "gs1_kernel": {
    "accuracy": 0.7467778620166793
  }
}

base_gsm8k_test_accuracy

0.702

effect_gs1_minus_grpo

-0.006823

hoeffding_lower_one_sided_95

-0.07422

n_items

1,319

n_gs1_only_correct

114

n_grpo_only_correct

123

• checkpoint · grpo_gs8 · step 40 · tinker://13a563a8fcb5108149fcf830eaa613b0:train:0/sampler_weights/grpo_gs8-final
• checkpoint · gs1_kernel · step 40 · tinker://e3180b48c9f5a53786775fab2b71413d:train:0/sampler_weights/gs1_kernel-final
• effective_config · effective-config.json
• code_revision · code-revision.txt

1. Nov 4, 19:10 UTC · succeeded · done: grpo_gs8, gs1_kernel · Two Qwen3.5-2B rank-32 LoRA arms trained for 40 steps at 64 rollouts per step on shared Spindle from the same saved initial adapter: GRPO 8x8 and GS1 64x1 with a Gaussian-kernel cross-prompt baseline…
2. Nov 4, 19:07 UTC · running · done: grpo_gs8 · GRPO arm (8x8, 40 steps) finished: GSM8K test accuracy 0.7536 vs base 0.7020; GS1 kernel arm finishing.
3. Nov 4, 18:44 UTC · running · Correctness preflight passed; activation gate passed at the pre-registered calibration bandwidth 1.0 (bandwidth 0.5 failed ESS p10 7.97<8). Base Qwen3.5-2B GSM8K test accuracy 0.7020; both 40-step ar…
4. Nov 4, 18:41 UTC · running · Correctness preflight passed (new loop reproduces stock GRPO datums/loss exactly; kernel path matches an independent float64 reference). Activation pilot at bandwidth 0.5 failed kernel_effective_neig…
5. Nov 4, 18:36 UTC · running · Preflight (initialization provenance, correctness preflight, arm probes) running on shared Spindle with Qwen/Qwen3.5-2B.

correctness_preflight

{'required': True, 'paths': [{'id': 'gs1_kernel_advantage_loop', 'fixed_input': {'sha256': 'e6cc1032207fafc779b9ac68119003efd2fbb09531f27c65b42a31635f68cc5f', 'description': 'preflight/fixed_inputs.json: 8x8 GRPO batch (rows 5000-5007) and 64x1 GS1 batch (rows 6000-6063) sampled at initial weights, with phi, synthetic 192-rollout history, rewards, tokens; datum sha 4330a4f0... (baseline) and f1cceea5... (treatment)'}, 'baseline_revision': 'tinker-cookbook f79fdb2ba4cd967ef2807ba43097c52985f9f17f (rl/data_processing.compute_advantages + assemble_training_data)', 'reference_revision': 'ref_kernel.py sha256 c0f04f304657f41d07a1ff1b6f310254338e1a6d7284fccad411c5eb7535121a (independent pure-python float64 loops, own datum construction)', 'implementation_revision': 'rx_gs1.py d24fc60d28da38150a4d4f1740d8fd1cc93836807461f6e2f6ffbdb080fb946a; kernel.py da5bd473e2ff6df119b5be841808851536c671d40f083de2236be722448aa70e', 'precision': 'advantages float64 computed, stored float32 in Datum; server forward bf16 model logprobs; loss/gradient recomputed in float64', 'normalization': 'importance_sampling loss sum over action tokens (loss:sum); gradient is d loss / d token-logprob = -exp(lp-q)*A over all tokens (not parameter gradients, which Spindle does not expose)', 'errors': {'baseline_loss': {'max_abs_error': 0.0, 'reference_max_abs': 688.5188732537263, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}, 'baseline_gradient': {'max_abs_error': 0.0, 'reference_max_abs': 1.1173313793593103, 'relative_l2_error': 0.0, 'cosine_similarity': 0.9999999999999999}, 'reference_loss': {'max_abs_error': 0.0, 'reference_max_abs': 2638.182763421831, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}, 'reference_gradient': {'max_abs_error': 0.0, 'reference_max_abs': 1.1467986929886362, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, 'lr_scaling': []}]}

correctness_validation

{'status': 'unverified', 'required': True, 'checks': [{'path': 'gs1_kernel_advantage_loop', 'check': 'baseline_loss', 'status': 'passed', 'tolerance': {'atol': 1e-06, 'rtol': 1e-05}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 688.5188732537263, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, {'path': 'gs1_kernel_advantage_loop', 'check': 'baseline_gradient', 'status': 'passed', 'tolerance': {'atol': 1e-05, 'rtol': 0.0001, 'min_cosine': 0.99999}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 1.1173313793593103, 'relative_l2_error': 0.0, 'cosine_similarity': 0.9999999999999999}}, {'path': 'gs1_kernel_advantage_loop', 'check': 'reference_loss', 'status': 'passed', 'tolerance': {'atol': 1e-06, 'rtol': 1e-05}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 2638.182763421831, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, {'path': 'gs1_kernel_advantage_loop', 'check': 'reference_gradient', 'status': 'passed', 'tolerance': {'atol': 1e-05, 'rtol': 0.0001, 'min_cosine': 0.99999}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 1.1467986929886362, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}], 'reason': 'gs1_kernel_advantage_loop: matching reference_revision; gs1_kernel_advantage_loop: matching baseline_revision', 'evidence_source': 'executor report; remote execution is not attested', 'applicability': 'required', 'reported_verdict': 'refuted'}

decision

{'version': 1, 'observations_file': {'name': 'scored-observations.jsonl', 'sha256': 'd36411b5b1ea0afd02a673cc4522280db626212211b332e88dacb6e6ed37f530'}}

decision_validation

{'status': 'unverified', 'issues': ['Observation file name/hash is not retained for this run.'], 'criteria': [], 'outcome_reason': 'insufficient_evidence', 'applicability': {'check': 'decision_rule', 'status': 'required', 'enforce': True, 'reasons': ['experiment_features: paired_comparison is true'], 'features': {'paired_comparison': True, 'adaptive_selection': False}}, 'reported_verdict': 'refuted'}

evaluation

{'version': 1, 'evaluators': {'b33b1807cbc389be6b35f2826324e18d1bac0edf0b2d4245d9e24742b7bb0d16': {'code_sha256': 'see evaluator.json (eval_gsm8k.py + cookbook math_grading/math_env + dep versions)', 'prompt_template': "tinker-cookbook f79fdb2b MathEnv standard few-shot prefix + question + ' Write your answer in \\boxed{} format.'; renderer qwen3_5_disable_thinking", 'decoding': {'temperature': 0, 'max_tokens': 512, 'stop': ['<|im_end|>']}, 'parser': 'last \\boxed{} via extract_boxed; normalize; float equality else string equality', 'items': {'ids': 'gsm8k-test-0000..gsm8k-test-1318 (full openai/gsm8k main test, in order)'}}}, 'scores': [{'arm': 'base', 'metric': 'accuracy', 'value': 0.7020470053070508, 'evaluator_id': 'b33b1807cbc389be6b35f2826324e18d1bac0edf0b2d4245d9e24742b7bb0d16', 'seed': 0, 'predictions_ref': 'predictions.jsonl'}, {'arm': 'grpo_gs8', 'metric': 'accuracy', 'value': 0.7536012130401819, 'evaluator_id': 'b33b1807cbc389be6b35f2826324e18d1bac0edf0b2d4245d9e24742b7bb0d16', 'seed': 0, 'predictions_ref': 'predictions.jsonl'}, {'arm': 'gs1_kernel', 'metric': 'accuracy', 'value': 0.7467778620166793, 'evaluator_id': 'b33b1807cbc389be6b35f2826324e18d1bac0edf0b2d4245d9e24742b7bb0d16', 'seed': 0, 'predictions_ref': 'predictions.jsonl'}], 'comparisons': [{'metric': 'accuracy', 'arms': ['grpo_gs8', 'gs1_kernel']}]}

evaluator_validation

{'status': 'invalid', 'issues': ['Evaluator b33b1807cbc389be6b35f2826324e18d1bac0edf0b2d4245d9e24742b7bb0d16: item IDs or a hashed item manifest are required.', 'Evaluator b33b1807cbc389be6b35f2826324e18d1bac0edf0b2d4245d9e24742b7bb0d16: code_sha256 is required.', 'Evaluator b33b1807cbc389be6b35f2826324e18d1bac0edf0b2d4245d9e24742b7bb0d16: identity does not match provenance hash.'], 'comparisons': [{'metric': 'accuracy', 'status': 'comparable', 'arms': ['base', 'grpo_gs8', 'gs1_kernel'], 'evaluator_ids': ['b33b1807cbc389be6b35f2826324e18d1bac0edf0b2d4245d9e24742b7bb0d16'], 'missing_arms': []}]}

feasibility

{'status': 'unknown', 'problems': ['Re-projection required: lora_rank changed or was not recorded', 'Re-projection required: dataset changed or was not recorded', 'Re-projection required: steps changed or was not recorded', 'Re-projection required: max_seq_len changed or was not recorded', 'Re-projection required: arms changed or was not recorded', 'Re-projection required: n_examples changed or was not recorded', 'Re-projection required: n_eval changed or was not recorded'], 'wall_seconds': None, 'gpu_hours': None, 'known_gpu_hours': 0, 'phases': {'setup': {'wall_seconds': 27.6, 'planned_units': 1, 'planned_samples': 0.0, 'gpu_hours': None}, 'training': {'wall_seconds': 2479.2, 'planned_units': 80, 'planned_samples': 5120.0, 'gpu_hours': None}, 'evaluation': {'wall_seconds': 55.83, 'planned_units': 3957, 'planned_samples': 3957.0, 'gpu_hours': None}, 'recovery': {'wall_seconds': 1200, 'gpu_hours': None}}, 'generation_counts': None, 'generation_applicability': {'check': 'generation_audit', 'status': 'not_applicable', 'enforce': False, 'reasons': ['spec.generation_audit opts out and no feature it covers is true'], 'features': {'generated_training_data': False}}, 'record': {'config': {'base_model': 'Qwen/Qwen3.5-2B', 'arm': 'gs1_kernel', 'steps': 2, 'batch_size': 64, 'max_tokens': 512}, 'effective_config': {'base_model': 'Qwen/Qwen3.5-2B', 'arms': ['grpo_gs8', 'gs1_kernel'], 'steps': 40, 'batch_size': 64, 'max_seq_len': 1024, 'max_tokens': 512, 'n_eval': 1319}, 'primary_endpoint': 'gsm8k_test_exact_match_accuracy_final_checkpoint', 'source': 'pilot_bw1.log, preflight.log, eval_base.log', 'measurements': {'setup': {'seconds': 27.6, 'units': 1, 'planned_units': 1, 'samples': 0, 'gpu_count': None, 'allocation_source': None}, 'training': {'seconds': 61.98, 'units': 2, 'planned_units': 80, 'samples': 128, 'gpu_count': None, 'allocation_source': None}, 'evaluation': {'seconds': 18.61, 'units': 1319, 'planned_units': 3957, 'samples': 1319, 'gpu_count': None, 'allocation_source': None}}, 'generation': None, 'recovery': {'seconds': 1200, 'gpu_count': None, 'allocation_source': None}}, 'recorded_at': 1793817847.071463}

feasibility_reported_verdict

refuted

feasibility_required

True

generation_audit_validation

{'status': 'not_applicable', 'reasons': ['spec.generation_audit opts out and no feature it covers is true'], 'applicability': {'check': 'generation_audit', 'status': 'not_applicable', 'enforce': False, 'reasons': ['spec.generation_audit opts out and no feature it covers is true'], 'features': {'generated_training_data': False}}}

initialization_provenance

{'version': 1, 'arms': {'grpo_gs8': {'initialization': {'base_checkpoint': 'Qwen/Qwen3.5-2B', 'source': 'server_response', 'checkpoint_uri': 'tinker://97af5cc4420906ad0572b9a4b349d326/weights/rx-gs1-init', 'checkpoint_immutable': True, 'checkpoint_scope': 'full_initialization', 'loaded_checkpoint_uri': 'tinker://97af5cc4420906ad0572b9a4b349d326/weights/rx-gs1-init', 'load_succeeded': True, 'adapter_sha256': None, 'evidence_ref': 'grpo_gs8/init_provenance.json', 'before_first_update': True}, 'restore_count': 1, 'optimizer_checks': [{'after_restore': 0, 'before_next_update': True, 'source': 'server_response', 'actual_family': 'adamw', 'actual_config': {'type': 'adamw'}, 'requested': {'family': 'adam', 'learning_rate': 4e-05, 'beta1': 0.9, 'beta2': 0.95, 'eps': 1e-08}, 'evidence_ref': 'grpo_gs8/init_provenance.json get_info_at_creation'}, {'after_restore': 1, 'before_next_update': True, 'source': 'server_response', 'actual_family': 'adamw', 'actual_config': {'type': 'adamw'}, 'evidence_ref': 'grpo_gs8/init_provenance.json get_info_after_restore'}], 'initial_output': {'available': True, 'batch_sha256': '768759f25252529e24d65d1496ff0df3f282ea28f6c038f24676fef0c1465849', 'output_sha256': '5b3eaba4df53efa57a673f368a149eb731dd87f14bb86f5969f03985134e7144', 'evidence_ref': 'grpo_gs8/init_provenance.json', 'before_first_update': True}}, 'gs1_kernel': {'initialization': {'base_checkpoint': 'Qwen/Qwen3.5-2B', 'source': 'server_response', 'checkpoint_uri': 'tinker://97af5cc4420906ad0572b9a4b349d326/weights/rx-gs1-init', 'checkpoint_immutable': True, 'checkpoint_scope': 'full_initialization', 'loaded_checkpoint_uri': 'tinker://97af5cc4420906ad0572b9a4b349d326/weights/rx-gs1-init', 'load_succeeded': True, 'adapter_sha256': None, 'evidence_ref': 'gs1_kernel/init_provenance.json', 'before_first_update': True}, 'restore_count': 1, 'optimizer_checks': [{'after_restore': 0, 'before_next_update': True, 'source': 'server_response', 'actual_family': 'adamw', 'actual_config': {'type': 'adamw'}, 'requested': {'family': 'adam', 'learning_rate': 4e-05, 'beta1': 0.9, 'beta2': 0.95, 'eps': 1e-08}, 'evidence_ref': 'gs1_kernel/init_provenance.json get_info_at_creation'}, {'after_restore': 1, 'before_next_update': True, 'source': 'server_response', 'actual_family': 'adamw', 'actual_config': {'type': 'adamw'}, 'evidence_ref': 'gs1_kernel/init_provenance.json get_info_after_restore'}], 'initial_output': {'available': True, 'batch_sha256': '768759f25252529e24d65d1496ff0df3f282ea28f6c038f24676fef0c1465849', 'output_sha256': '5b3eaba4df53efa57a673f368a149eb731dd87f14bb86f5969f03985134e7144', 'evidence_ref': 'gs1_kernel/init_provenance.json', 'before_first_update': True}}}}

launch

{'experiment_id': 'exp_32150a771c', 'context_status': 'snapshotted', 'superseded': False, 'mode': 'ordinary', 'purpose': 'Execute the selected saved experiment version.'}

progress_updated_at

1793819401.3912942

reported_verdict

refuted

restoration_validation

{'status': 'unverified', 'required': True, 'checks': [{'check': 'grpo_gs8', 'status': 'unverified', 'reason': 'grpo_gs8: missing or duplicate arm evidence.'}, {'check': 'gs1_kernel', 'status': 'unverified', 'reason': 'gs1_kernel: missing or duplicate arm evidence.'}], 'reason': 'grpo_gs8: missing or duplicate arm evidence.', 'applicability': 'required'}

slack_started

True

started_at

1793816968.7824461

verdict_report

{'effect': -0.006823351023502654, 'margin': -0.03, 'lower': -0.08161271360462294, 'upper': 0.06796601155761764, 'seeds': 1, 'confidence': 'weak', 'reasoning': "GS1-kernel scored 0.68 points below GRPO (0.7468 vs 0.7536; 114 vs 123 discordant items). The paired Hoeffding lower bound is -0.074, below the -0.03 margin, so the pre-registered criterion is not met and the verdict is refuted. This is weak evidence: at n=1319 the Hoeffding half-width (~0.067) is wider than the margin, so even an effect of 0 could not have passed. Other limits: one seed, kernel bandwidth recalibrated from 0.5 to 1.0 under the spec's pre-registered rule, and the disable-thinking renderer."}