🔬Modal Research Exchange

{
  "base_model": "Qwen/Qwen3.6-35B-A3B",
  "baseline": "Dr. GRPO, with mean-centered advantages and no std normalization. It runs at lag 0 and lag 8 with identical model, LoRA rank, lr, data order, rollout budget, PPO clip and eval as TailRL.",
  "batch_size": 32,
  "budget": {
    "gpu_hours": 32.0,
    "max_wall_minutes": 600
  },
  "command": "Custom tinker SDK loop reusing the sec-search-rl rollout and tool code. Custom parts are the advantage function and the lagged sampler selection.\n\nimport os, tinker\nsc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = sc.create_lora_training_client(base_model='Qwen/Qwen3.6-35B-A3B', rank=32)\nckpts = []\nfor t in range(24):\n    ckpts.append(tc.save_weights_for_sampler(name=f'u{t}').result().path)\n    samp = sc.create_sampling_client(model_path=ckpts[max(0, t - LAG)])\n    # roll out 32 queries x 8 trajectories with search/grep/read/curate tools, max 20 turns, temperature 1, keeping sampler logprobs\n    # reward = F4 of curated set (-0.2 if empty)\n    # A = tailrl_advantages(r) if ADV == 'tailrl' else r - r.mean()\n    # drop constant-reward groups\n    tc.forward_backward(datums, loss_fn='ppo').result()  # clip ratio [0.8, 1.2], sampler logprobs as behavior logprobs\n    tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).result()\n\nRun for ADV in {tailrl, drgrpo} x LAG in {0, 8}, seed 0.",
  "dataset": {
    "hf_path": "pat-jj/harness-1-train-data",
    "n_examples": 1024,
    "name": "Harness-1 SEC multi-clue search (sec_1024 train; sec_1024 dev + sec_test for eval)",
    "split": "train"
  },
  "eval": {
    "benchmark": "SEC search held-out (sec_1024 dev 64 + sec_test 64)",
    "metric": "mean F4 of curated set (also precision, recall, best-of-2), temperature 1, 2 draws per query",
    "n_examples": 128
  },
  "hypothesis": "At lag 8, PPO clipping removes at least 1.5x more of TailRL's |A|-weighted update mass than Dr. GRPO's, and TailRL's lag-0 to lag-8 F4 drop exceeds Dr. GRPO's by at least 0.05.",
  "lora_rank": 32,
  "max_seq_len": 16384,
  "method": "Linked sources failed to digest, so this design follows the refine notes from the paper v2, the official advantages.py and the lilo codegolf port.\n\nBudget forces a scale-down. The refined plan was about 30 runs at about 70 GPU-hours each on gpt-oss-20b. 32 GPU-hours buys only a 2x2 pilot:\n- Objectives: TailRL vs Dr. GRPO.\n- Lags: 0 vs 8.\n- One seed per cell.\n- Model: Qwen/Qwen3.6-35B-A3B. It has 3B active parameters, has already run on Spindle (step5), and needs no gpt-oss expert-LoRA workaround.\n\nTask: modal-projects/sec-search-rl (Harness-1). Smaller settings than the repro:\n- Up to 20 turns and 16k trajectory tokens.\n- Reward is F4 of the curated set, -0.2 if empty, with no format bonus.\n\nTraining: 32 queries x 8 rollouts per update, 24 updates, LoRA r32, AdamW, matched init, data order and seed.\n\nOnly two things vary across the four runs:\n- Advantage (the one thing that differs between baseline and treatment):\n  - TailRL: lilo codegolf/reward.py tailrl_advantages, centered and with no std normalization. Unit-test it against the official TailRL advantages.py first.\n  - Dr. GRPO: r - mean(r).\n  - Constant-reward groups are dropped for both.\n- Lag L, the condition:\n  - Each update saves a sampler checkpoint.\n  - Rollouts come from the checkpoint L updates old, and behavior logprobs are taken from that sampler.\n  - The update uses PPO-clipped IS [0.8,1.2], one update per batch, no KL.\n\nGates before reading results:\n- Lag-0 sampler/trainer KL sits near the 0.002-0.003 floor, with clip rate near 1%.\n- Lag-8 KL is at least 3x the lag-0 KL. If not, rerun at L=16.\n\nPrimary metrics are per-step staleness metrics, logged for updates 9-24:\n- IS-ratio p95/p99.\n- Raw clip rate.\n- Clip-active |A| mass fraction, bucketed by within-group reward rank and advantage sign.\n- KL, entropy, and curated-doc count (watching for over-curation).\n\nEval every 8 updates on 128 held-out queries (sec_1024 dev 64 + sec_test 64) x 2 sampled draws, reporting F4, precision, recall and best-of-2.\n\nWith one seed, the F4 difference-of-differences is reported as a pilot effect size. It sets seed counts for a follow-up and is not a decisive F4 test.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "steps": 24,
  "success_criterion": "Validated: TailRL's lag-8 clip-active |A| mass fraction (mean over updates 9-24) is at least 1.5x Dr. GRPO's, and TailRL's lag-8 F4 drop exceeds Dr. GRPO's by at least 0.05. Measured seed spread is about 0.09 and eval-draw noise about 0.02, so a single-seed F4 gap is only directional. Refuted: the mass ratio is below 1.2 and the F4 drop difference is within \u00b10.05. Anything in between triggers the multi-seed follow-up. Results are void if the lag-8 KL gate (at least 3x the lag-0 KL) fails.",
  "title": "TailRL vs Dr. GRPO under lag 0 and lag 8 on SEC search, a pilot that fits 32 GPU-hours",
  "training_type": "rl"
}

tailrl_unit_test_max_abs_diff

1.776e-15

tailrl_lag0_kl_v2_u9_24

0.00237

drgrpo_lag0_kl_v2_u9_24

0.002209

tailrl_lag8_kl_v2_u9_24

0.003216

drgrpo_lag8_kl_v2_u9_24

0.002725

kl_gate_ratio_tailrl

1.357

kl_gate_ratio_drgrpo

1.233

tailrl_lag0_clip_rate_u9_24

0.01189

drgrpo_lag0_clip_rate_u9_24

0.0115

tailrl_lag8_clip_rate_u9_24

0.01438

drgrpo_lag8_clip_rate_u9_24

0.01437

tailrl_lag0_clip_absA_massfrac_u9_24

0.01187

drgrpo_lag0_clip_absA_massfrac_u9_24

0.01117

tailrl_lag8_clip_absA_massfrac_u9_24

0.01337

drgrpo_lag8_clip_absA_massfrac_u9_24

0.01297

lag8_massfrac_ratio_tailrl_over_drgrpo

1.031

tailrl_lag8_ratio_p95_u9_24

1.073

tailrl_lag8_ratio_p99_u9_24

1.24

drgrpo_lag8_ratio_p95_u9_24

1.074

drgrpo_lag8_ratio_p99_u9_24

1.232

tailrl_lag0_entropy_u9_24

0.3125

drgrpo_lag0_entropy_u9_24

0.2986

tailrl_lag8_entropy_u9_24

0.3329

drgrpo_lag8_entropy_u9_24

0.346

tailrl_lag0_curated_docs_u9_24

2.985

drgrpo_lag0_curated_docs_u9_24

3.358

tailrl_lag8_curated_docs_u9_24

0.6653

drgrpo_lag8_curated_docs_u9_24

0.7038

tailrl_lag0_eval_f4_u24

0.2791

drgrpo_lag0_eval_f4_u24

0.2929

tailrl_lag8_eval_f4_u24

0.06856

drgrpo_lag8_eval_f4_u24

0.09641

tailrl_lag0_eval_precision_u24

0.3341

drgrpo_lag0_eval_precision_u24

0.2904

tailrl_lag8_eval_precision_u24

0.09593

drgrpo_lag8_eval_precision_u24

0.1601

tailrl_lag0_eval_recall_u24

0.2823

drgrpo_lag0_eval_recall_u24

0.3026

tailrl_lag8_eval_recall_u24

0.06886

drgrpo_lag8_eval_recall_u24

0.09509

tailrl_lag0_eval_best_of_2_f4_u24

0.366

drgrpo_lag0_eval_best_of_2_f4_u24

0.3649

tailrl_lag8_eval_best_of_2_f4_u24

0.1217

drgrpo_lag8_eval_best_of_2_f4_u24

0.1494

tailrl_f4_drop_lag0_to_lag8

0.2105

drgrpo_f4_drop_lag0_to_lag8

0.1965

f4_drop_diff_tailrl_minus_drgrpo

0.01405

1. Nov 4, 01:58 UTC · succeeded · All four arms (TailRL and Dr. GRPO at lag 0 and lag 8; Qwen3.6-35B-A3B LoRA r32, 24 updates of 32x8 on shared Spindle; the per-model seed was not set because Spindle rejected it) ran to completion. L…
2. Nov 3, 22:03 UTC · running · 2x2 pilot running on shared Spindle: lag-0 arms at update 5/24 and lag-8 arms at 7/24 (~11 min/update), with lag-0 KL 0.0020-0.0027 and clip rate 1.1-1.4% so far.
3. Nov 3, 21:15 UTC · running · Restarted the 2x2 pilot (TailRL/Dr. GRPO x lag 0/8) on shared Spindle with per-update state saves after the first attempt lost its trainers; lag-16 arms dropped because ~25 min/update (77% of rollout…
4. Nov 3, 20:36 UTC · running · First attempt lost all four Spindle trainers (410 model_lost) at updates 7-9 and sampler checkpoints could not restore training; restarted all arms (lag 0/8/16, since early lag-8 KL was only ~1.2x la…
5. Nov 3, 20:19 UTC · running · All four arms reached updates 7-9 on shared Spindle before Spindle terminated their trainer engines (410 model_lost); retrying trainer creation with backoff to resume from saved sampler checkpoints.
6. Nov 3, 20:12 UTC · running · Four arms (TailRL/Dr. GRPO x lag 0/8) are training on shared Spindle with Qwen/Qwen3.6-35B-A3B; early lag-8 KL is close to lag-0 KL, so lag-16 arms are queued to start when the lag-8 arms finish.

feasibility

{'status': 'unknown', 'problems': ['Measured feasibility not recorded'], 'wall_seconds': None, 'gpu_hours': None, 'known_gpu_hours': 0, 'phases': {}, 'record': None, 'recorded_at': 1793738777.3298795}

reported_verdict

inconclusive