🔬Modal Research Exchange

{
  "base_model": "Qwen/Qwen3.6-35B-A3B",
  "baseline": "Dr. GRPO, with mean-centered advantages and no std normalization. It runs at lag 0 and lag 8 with identical model, LoRA rank, lr, data order, rollout budget, PPO clip and eval as TailRL.",
  "batch_size": 32,
  "budget": {
    "gpu_hours": 32.0,
    "max_wall_minutes": 600
  },
  "command": "Custom tinker SDK loop reusing the sec-search-rl rollout and tool code. Custom parts are the advantage function and the lagged sampler selection.\n\nimport os, tinker\nsc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = sc.create_lora_training_client(base_model='Qwen/Qwen3.6-35B-A3B', rank=32)\nckpts = []\nfor t in range(24):\n    ckpts.append(tc.save_weights_for_sampler(name=f'u{t}').result().path)\n    samp = sc.create_sampling_client(model_path=ckpts[max(0, t - LAG)])\n    # roll out 32 queries x 8 trajectories with search/grep/read/curate tools, max 20 turns, temperature 1, keeping sampler logprobs\n    # reward = F4 of curated set (-0.2 if empty)\n    # A = tailrl_advantages(r) if ADV == 'tailrl' else r - r.mean()\n    # drop constant-reward groups\n    tc.forward_backward(datums, loss_fn='ppo').result()  # clip ratio [0.8, 1.2], sampler logprobs as behavior logprobs\n    tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).result()\n\nRun for ADV in {tailrl, drgrpo} x LAG in {0, 8}, seed 0.",
  "dataset": {
    "hf_path": "pat-jj/harness-1-train-data",
    "n_examples": 1024,
    "name": "Harness-1 SEC multi-clue search (sec_1024 train; sec_1024 dev + sec_test for eval)",
    "split": "train"
  },
  "eval": {
    "benchmark": "SEC search held-out (sec_1024 dev 64 + sec_test 64)",
    "metric": "mean F4 of curated set (also precision, recall, best-of-2), temperature 1, 2 draws per query",
    "n_examples": 128
  },
  "hypothesis": "At lag 8, PPO clipping removes at least 1.5x more of TailRL's |A|-weighted update mass than Dr. GRPO's, and TailRL's lag-0 to lag-8 F4 drop exceeds Dr. GRPO's by at least 0.05.",
  "lora_rank": 32,
  "max_seq_len": 16384,
  "method": "Linked sources failed to digest, so this design follows the refine notes from the paper v2, the official advantages.py and the lilo codegolf port.\n\nBudget forces a scale-down. The refined plan was about 30 runs at about 70 GPU-hours each on gpt-oss-20b. 32 GPU-hours buys only a 2x2 pilot:\n- Objectives: TailRL vs Dr. GRPO.\n- Lags: 0 vs 8.\n- One seed per cell.\n- Model: Qwen/Qwen3.6-35B-A3B. It has 3B active parameters, has already run on Spindle (step5), and needs no gpt-oss expert-LoRA workaround.\n\nTask: modal-projects/sec-search-rl (Harness-1). Smaller settings than the repro:\n- Up to 20 turns and 16k trajectory tokens.\n- Reward is F4 of the curated set, -0.2 if empty, with no format bonus.\n\nTraining: 32 queries x 8 rollouts per update, 24 updates, LoRA r32, AdamW, matched init, data order and seed.\n\nOnly two things vary across the four runs:\n- Advantage (the one thing that differs between baseline and treatment):\n  - TailRL: lilo codegolf/reward.py tailrl_advantages, centered and with no std normalization. Unit-test it against the official TailRL advantages.py first.\n  - Dr. GRPO: r - mean(r).\n  - Constant-reward groups are dropped for both.\n- Lag L, the condition:\n  - Each update saves a sampler checkpoint.\n  - Rollouts come from the checkpoint L updates old, and behavior logprobs are taken from that sampler.\n  - The update uses PPO-clipped IS [0.8,1.2], one update per batch, no KL.\n\nGates before reading results:\n- Lag-0 sampler/trainer KL sits near the 0.002-0.003 floor, with clip rate near 1%.\n- Lag-8 KL is at least 3x the lag-0 KL. If not, rerun at L=16.\n\nPrimary metrics are per-step staleness metrics, logged for updates 9-24:\n- IS-ratio p95/p99.\n- Raw clip rate.\n- Clip-active |A| mass fraction, bucketed by within-group reward rank and advantage sign.\n- KL, entropy, and curated-doc count (watching for over-curation).\n\nEval every 8 updates on 128 held-out queries (sec_1024 dev 64 + sec_test 64) x 2 sampled draws, reporting F4, precision, recall and best-of-2.\n\nWith one seed, the F4 difference-of-differences is reported as a pilot effect size. It sets seed counts for a follow-up and is not a decisive F4 test.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "steps": 24,
  "success_criterion": "Validated: TailRL's lag-8 clip-active |A| mass fraction (mean over updates 9-24) is at least 1.5x Dr. GRPO's, and TailRL's lag-8 F4 drop exceeds Dr. GRPO's by at least 0.05. Measured seed spread is about 0.09 and eval-draw noise about 0.02, so a single-seed F4 gap is only directional. Refuted: the mass ratio is below 1.2 and the F4 drop difference is within \u00b10.05. Anything in between triggers the multi-seed follow-up. Results are void if the lag-8 KL gate (at least 3x the lag-0 KL) fails.",
  "title": "TailRL vs Dr. GRPO under lag 0 and lag 8 on SEC search, a pilot that fits 32 GPU-hours",
  "training_type": "rl"
}

base_eval_f4

0.1882

base_eval_precision

0.2589

base_eval_recall

0.2568

base_eval_best_of_2_f4

0.3128

base_eval_n_curated

2.637

base_eval_empty_rate

0.3242

partial_lag8_mass_frac_ratio_u9_11

1.027

partial_lag8_mass_frac_ratio

1.02

1. Nov 4, 21:23 UTC · running · done: tailrl_L8 · tailrl_L8 finished 24 updates (eval F4 0.261 vs base 0.188); lag-8 KL (~0.0041) is only ~1.1x the lag-0 KL (~0.0036), so the spec's lag-8 KL gate is failing, and an offline drift probe shows checkpoi…
2. Nov 4, 20:38 UTC · running · done: tailrl_L8 · tailrl_L8 finished 24 updates (eval F4 0.261 vs base 0.188); lag-8 KL (~0.0041) is only ~1.1x the lag-0 KL (~0.0036), so the spec's lag-8 KL gate is failing, and an offline drift probe shows checkpoi…
3. Nov 4, 19:32 UTC · running · Four arms (TailRL/Dr.GRPO x lag 0/8, 8x8 batch) are training on Spindle; early lag-8 updates show KL ~0.0041 vs lag-0 ~0.0038, so the spec's lag-8 KL gate (>=3x) is trending to fail and the spec's co…
4. Nov 4, 18:34 UTC · running · Timing/activation pilot done (base eval F4 0.188 on 128x2 held-out items); full 4-arm run (TailRL/Dr.GRPO x lag 0/8) launched at a reduced batch of 8 queries x 8 rollouts because measured Spindle for…
5. Nov 4, 17:59 UTC · running · Setup done: local BM25 SEC harness built (Harness-1 public corpus), TailRL advantage port unit-tested against official TailRL code (max abs err 1.8e-15), shared init LoRA state created on Spindle; ti…

correctness_preflight

{'required': True, 'paths': [{'id': 'advantage_tailrl_and_drgrpo', 'fixed_input': {'sha256': 'edc5f9caa4039525c50d5c9e5dff9ce6296e4dd12182cc3e2a5b6a6c96f3b921', 'description': 'adv_fixture.json: 2000 reward groups of size 8 (continuous, tied, -0.2 empty penalty, single-success)'}, 'baseline_revision': 'Zanette-Labs/TailRL@5682c6ac reinforce_baseline_advantage', 'reference_revision': 'Zanette-Labs/TailRL@5682c6ac code_opt/advantages.py tailrl_advantage', 'implementation_revision': 'spindle@dbf84ac examples/codeforces-codegolf/codegolf/reward.py tailrl_advantages (verbatim copy)', 'precision': 'float64', 'normalization': 'per-group, mean-centred, N-scaled (TailRL), no std normalization', 'errors': {'reference_tailrl': {'max_abs_error': 1.7763568394002505e-15, 'reference_max_abs': 8.394356796360896, 'relative_l2_error': 1.0480901573599502e-16, 'cosine_similarity': 1.0}, 'baseline_drgrpo': {'max_abs_error': 2.220446049250313e-16, 'reference_max_abs': 1.049294599545112, 'relative_l2_error': 1.135070817677618e-16, 'cosine_similarity': 1.0}}, 'lr_scaling': []}]}

correctness_validation

{'status': 'unverified', 'required': True, 'checks': [], 'reason': 'Correctness preflight was not recorded.', 'evidence_source': 'executor report; remote execution is not attested', 'applicability': 'unknown', 'reported_verdict': 'inconclusive'}

ended_reason

message unknown

feasibility

{'status': 'unknown', 'problems': ['Re-projection required: lora_rank changed or was not recorded', 'Re-projection required: dataset changed or was not recorded', 'Re-projection required: steps changed or was not recorded', 'Re-projection required: batch_size changed or was not recorded', 'Re-projection required: max_seq_len changed or was not recorded', 'Re-projection required: arms changed or was not recorded', 'Re-projection required: n_examples changed or was not recorded', 'Re-projection required: n_eval changed or was not recorded', 'Primary endpoint changed or was not recorded'], 'wall_seconds': None, 'gpu_hours': None, 'known_gpu_hours': 0, 'phases': {'setup': {'wall_seconds': 210.5, 'planned_units': 1, 'planned_samples': 0.0, 'gpu_hours': None}, 'training': {'wall_seconds': 84038.40000000001, 'planned_units': 144, 'planned_samples': 36864.0, 'gpu_hours': None}, 'evaluation': {'wall_seconds': 4980.5, 'planned_units': 1792, 'planned_samples': 1792.0, 'gpu_hours': None}, 'recovery': {'wall_seconds': 3600, 'gpu_hours': None}}, 'generation_counts': None, 'generation_applicability': {'check': 'generation_audit', 'status': 'unknown', 'enforce': False, 'reasons': ['spec has no experiment_features: generated_training_data'], 'features': {'generated_training_data': None}}, 'record': {'config': {'base_model': 'Qwen/Qwen3.6-35B-A3B', 'batch': '32x8', 'pilot': '1 rollout batch + base eval concurrently, 1 forward_backward of 256 trajectories'}, 'effective_config': {'base_model': 'Qwen/Qwen3.6-35B-A3B', 'arms': ['tailrl_L0', 'drgrpo_L0', 'tailrl_L8', 'drgrpo_L8', 'tailrl_L16', 'drgrpo_L16'], 'steps': 24, 'batch_size': '8 queries x 8 rollouts', 'max_seq_len': 16384, 'max_turns': 20, 'n_eval': 128, 'eval_draws': 2, 'eval_points': [0, 24]}, 'primary_endpoint': 'lag-8 clip-active |A| mass fraction ratio (TailRL/Dr.GRPO, updates 9-24) and F4 drop difference', 'source': 'out/pilot2_timing.log, run.log', 'measurements': {'setup': {'seconds': 210.5, 'units': 1, 'planned_units': 1, 'samples': 0, 'gpu_count': None, 'allocation_source': None}, 'training': {'seconds': 583.6, 'units': 1, 'planned_units': 144, 'samples': 256, 'gpu_count': None, 'allocation_source': None}, 'evaluation': {'seconds': 711.5, 'units': 256, 'planned_units': 1792, 'samples': 256, 'gpu_count': None, 'allocation_source': None}}, 'generation': None, 'recovery': {'seconds': 3600, 'gpu_count': None, 'allocation_source': None}}, 'recorded_at': 1793820769.123115}

feasibility_required

True

launch

{'experiment_id': 'exp_9c3284c970', 'context_status': 'legacy_unknown', 'superseded': False, 'mode': 'ordinary', 'purpose': 'Execute the selected saved experiment version.'}

progress_updated_at

1793824684.1669283

reported_outcome_reason

incomplete_execution

slack_started

True

started_at

1793812865.768047