🔬Modal Research Exchange

{
  "base_model": "Qwen/Qwen3.6-35B-A3B",
  "baseline": "Dr. GRPO, with mean-centered advantages and no std normalization. It runs at lag 0 and lag 8 with identical model, LoRA rank, lr, data order, rollout budget, PPO clip and eval as TailRL.",
  "batch_size": 32,
  "budget": {
    "gpu_hours": 32.0,
    "max_wall_minutes": 600
  },
  "command": "Custom tinker SDK loop reusing the sec-search-rl rollout and tool code. Custom parts are the advantage function and the lagged sampler selection.\n\nimport os, tinker\nsc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = sc.create_lora_training_client(base_model='Qwen/Qwen3.6-35B-A3B', rank=32)\nckpts = []\nfor t in range(24):\n    ckpts.append(tc.save_weights_for_sampler(name=f'u{t}').result().path)\n    samp = sc.create_sampling_client(model_path=ckpts[max(0, t - LAG)])\n    # roll out 32 queries x 8 trajectories with search/grep/read/curate tools, max 20 turns, temperature 1, keeping sampler logprobs\n    # reward = F4 of curated set (-0.2 if empty)\n    # A = tailrl_advantages(r) if ADV == 'tailrl' else r - r.mean()\n    # drop constant-reward groups\n    tc.forward_backward(datums, loss_fn='ppo').result()  # clip ratio [0.8, 1.2], sampler logprobs as behavior logprobs\n    tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).result()\n\nRun for ADV in {tailrl, drgrpo} x LAG in {0, 8}, seed 0.",
  "dataset": {
    "hf_path": "pat-jj/harness-1-train-data",
    "n_examples": 1024,
    "name": "Harness-1 SEC multi-clue search (sec_1024 train; sec_1024 dev + sec_test for eval)",
    "split": "train"
  },
  "eval": {
    "benchmark": "SEC search held-out (sec_1024 dev 64 + sec_test 64)",
    "metric": "mean F4 of curated set (also precision, recall, best-of-2), temperature 1, 2 draws per query",
    "n_examples": 128
  },
  "hypothesis": "At lag 8, PPO clipping removes at least 1.5x more of TailRL's |A|-weighted update mass than Dr. GRPO's, and TailRL's lag-0 to lag-8 F4 drop exceeds Dr. GRPO's by at least 0.05.",
  "lora_rank": 32,
  "max_seq_len": 16384,
  "method": "Linked sources failed to digest, so this design follows the refine notes from the paper v2, the official advantages.py and the lilo codegolf port.\n\nBudget forces a scale-down. The refined plan was about 30 runs at about 70 GPU-hours each on gpt-oss-20b. 32 GPU-hours buys only a 2x2 pilot:\n- Objectives: TailRL vs Dr. GRPO.\n- Lags: 0 vs 8.\n- One seed per cell.\n- Model: Qwen/Qwen3.6-35B-A3B. It has 3B active parameters, has already run on Spindle (step5), and needs no gpt-oss expert-LoRA workaround.\n\nTask: modal-projects/sec-search-rl (Harness-1). Smaller settings than the repro:\n- Up to 20 turns and 16k trajectory tokens.\n- Reward is F4 of the curated set, -0.2 if empty, with no format bonus.\n\nTraining: 32 queries x 8 rollouts per update, 24 updates, LoRA r32, AdamW, matched init, data order and seed.\n\nOnly two things vary across the four runs:\n- Advantage (the one thing that differs between baseline and treatment):\n  - TailRL: lilo codegolf/reward.py tailrl_advantages, centered and with no std normalization. Unit-test it against the official TailRL advantages.py first.\n  - Dr. GRPO: r - mean(r).\n  - Constant-reward groups are dropped for both.\n- Lag L, the condition:\n  - Each update saves a sampler checkpoint.\n  - Rollouts come from the checkpoint L updates old, and behavior logprobs are taken from that sampler.\n  - The update uses PPO-clipped IS [0.8,1.2], one update per batch, no KL.\n\nGates before reading results:\n- Lag-0 sampler/trainer KL sits near the 0.002-0.003 floor, with clip rate near 1%.\n- Lag-8 KL is at least 3x the lag-0 KL. If not, rerun at L=16.\n\nPrimary metrics are per-step staleness metrics, logged for updates 9-24:\n- IS-ratio p95/p99.\n- Raw clip rate.\n- Clip-active |A| mass fraction, bucketed by within-group reward rank and advantage sign.\n- KL, entropy, and curated-doc count (watching for over-curation).\n\nEval every 8 updates on 128 held-out queries (sec_1024 dev 64 + sec_test 64) x 2 sampled draws, reporting F4, precision, recall and best-of-2.\n\nWith one seed, the F4 difference-of-differences is reported as a pilot effect size. It sets seed counts for a follow-up and is not a decisive F4 test.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "steps": 24,
  "success_criterion": "Validated: TailRL's lag-8 clip-active |A| mass fraction (mean over updates 9-24) is at least 1.5x Dr. GRPO's, and TailRL's lag-8 F4 drop exceeds Dr. GRPO's by at least 0.05. Measured seed spread is about 0.09 and eval-draw noise about 0.02, so a single-seed F4 gap is only directional. Refuted: the mass ratio is below 1.2 and the F4 drop difference is within \u00b10.05. Anything in between triggers the multi-seed follow-up. Results are void if the lag-8 KL gate (at least 3x the lag-0 KL) fails.",
  "title": "TailRL vs Dr. GRPO under lag 0 and lag 8 on SEC search, a pilot that fits 32 GPU-hours",
  "training_type": "rl"
}

base_eval_f4

0.136

base_eval_precision

0.1743

base_eval_recall

0.1362

base_eval_best_of_2_f4

0.1874

pilot_lag0_kl_k3_update1

0.003938

pilot_lag0_kl_k3_update2

0.004128

pilot_lag0_clip_rate_raw_update1

0.04026

pilot_lag0_clip_rate_raw_update2

0.03946

1. Nov 4, 20:48 UTC · running · 2x2 run in progress: lag-0 arms at update 9/24 and lag-8 arms at 18/24. Held-out F4 rose from 0.136 (base) to 0.22-0.23 at lag-0 u8, so far the lag-8 KL is only about 1.1-1.2x the lag-0 floor, and th…
2. Nov 4, 20:12 UTC · running · 2x2 run in progress: lag-0 arms at update 9/24 and lag-8 arms at 18/24. Held-out F4 rose from 0.136 (base) to 0.22-0.23 at lag-0 u8, so far the lag-8 KL is only about 1.1-1.2x the lag-0 floor, and th…
3. Nov 4, 19:06 UTC · running · 2x2 run in progress on shared Spindle. Lag-0 arms are at update 5/24 and lag-8 arms at 11/24. Early lag-8 sampler/trainer KL is only about 1.3-1.5x the lag-0 floor, so the spec's L=16 fallback arms w…
4. Nov 4, 18:11 UTC · running · Full 2x2 run launched on shared Spindle (Qwen3.6-35B-A3B, LoRA r32): TailRL vs Dr. GRPO at lag 0 and lag 8, 24 updates of 16 queries x 8 rollouts; base eval F4 = 0.136.
5. Nov 4, 17:10 UTC · running · Setting up SEC search harness (BM25 index over sec_1_4 built locally), TailRL unit test passed; waiting on Spindle trainer startup for Qwen3.6-35B-A3B LoRA.

correctness_preflight

{'required': True, 'paths': [{'id': 'tailrl_advantage', 'fixed_input': {'sha256': '4e98279407fcca20c7c9c787b55fcfcfc9b37fc866eb14d36bf4caa455732ad0', 'description': '2000 groups of 8 rewards (uniform, F4-like with -0.2 empties and ties, categorical, gaussian), numpy RandomState(0)'}, 'baseline_revision': 'Zanette-Labs/TailRL@5682c6a reinforce_baseline_advantage', 'reference_revision': 'Zanette-Labs/TailRL@5682c6a experiments/code_optimization/code_opt/advantages.py tailrl_advantage(scale_by_group_size=True)', 'implementation_revision': 'advantages.py (numpy port)', 'precision': 'float64', 'normalization': 'per-group, x G scaling, mean-centred, no std; constant groups dropped by caller', 'errors': {'reference_tailrl': {'max_abs_error': 1.7763568394002505e-15, 'reference_max_abs': 21.602965521673823, 'relative_l2_error': 6.88393888216291e-17, 'cosine_similarity': 0.9999999999999999}, 'baseline_drgrpo': {'max_abs_error': 4.440892098500626e-16, 'reference_max_abs': 3.1369444304518392, 'relative_l2_error': 7.423207618167053e-17, 'cosine_similarity': 1.0}}, 'lr_scaling': []}]}

correctness_validation

{'status': 'unverified', 'required': True, 'checks': [], 'reason': 'Correctness preflight was not recorded.', 'evidence_source': 'executor report; remote execution is not attested', 'applicability': 'unknown', 'reported_verdict': 'inconclusive'}

ended_reason

message unknown

feasibility

{'status': 'unknown', 'problems': ['Re-projection required: lora_rank changed or was not recorded', 'Re-projection required: dataset changed or was not recorded', 'Re-projection required: steps changed or was not recorded', 'Re-projection required: batch_size changed or was not recorded', 'Re-projection required: arms changed or was not recorded', 'Re-projection required: n_examples changed or was not recorded', 'Primary endpoint changed or was not recorded'], 'wall_seconds': None, 'gpu_hours': None, 'known_gpu_hours': 0, 'phases': {'setup': {'wall_seconds': 258.0, 'planned_units': 1, 'planned_samples': 0.0, 'gpu_hours': None}, 'training': {'wall_seconds': 91968.0, 'planned_units': 128, 'planned_samples': 32768.0, 'gpu_hours': None}, 'evaluation': {'wall_seconds': 2673.0, 'planned_units': 2816, 'planned_samples': 1408.0, 'gpu_hours': None}, 'recovery': {'wall_seconds': 3600, 'gpu_hours': None}}, 'generation_counts': None, 'generation_applicability': {'check': 'generation_audit', 'status': 'unknown', 'enforce': False, 'reasons': ['spec has no experiment_features: generated_training_data'], 'features': {'generated_training_data': None}}, 'record': {'config': {'base_model': 'Qwen/Qwen3.6-35B-A3B', 'arms': ['tailrl_lag0'], 'steps': 2, 'batch_size': 32, 'group': 8, 'max_seq_len': 16384, 'n_eval': 128}, 'effective_config': {'base_model': 'Qwen/Qwen3.6-35B-A3B', 'arms': ['tailrl_lag0', 'tailrl_lag8', 'drgrpo_lag0', 'drgrpo_lag8', 'tailrl_lag16 (fallback)', 'drgrpo_lag16 (fallback)'], 'steps': 24, 'batch_size': 16, 'group': 8, 'max_seq_len': 16384, 'max_turns': 20, 'n_eval': 128, 'eval_draws': 2}, 'primary_endpoint': 'lag-8 clip-active |A| mass fraction ratio (TailRL/DrGRPO, updates 9-24) and F4 drop difference', 'source': 'out_pilot/tailrl_lag0_metrics.jsonl, base_pilot.log, out/*_metrics.jsonl', 'measurements': {'setup': {'seconds': 258, 'units': 1, 'planned_units': 1, 'samples': 0, 'gpu_count': None, 'allocation_source': None}, 'training': {'seconds': 1437, 'units': 2, 'planned_units': 128, 'samples': 512, 'gpu_count': None, 'allocation_source': None}, 'evaluation': {'seconds': 243, 'units': 256, 'planned_units': 2816, 'samples': 128, 'gpu_count': None, 'allocation_source': None}}, 'generation': None, 'recovery': {'seconds': 3600, 'gpu_count': None, 'allocation_source': None}}, 'recorded_at': 1793819216.289172}

feasibility_required

True

launch

{'experiment_id': 'exp_9c3284c970', 'context_status': 'legacy_unknown', 'superseded': False, 'mode': 'ordinary', 'purpose': 'Execute the selected saved experiment version.'}

progress_updated_at

1793823159.2228441

reported_outcome_reason

incomplete_execution

slack_started

True

started_at

1793810870.7985597