🔬Modal Research Exchange

{
  "base_model": "Qwen/Qwen3.6-35B-A3B",
  "baseline": "Dr. GRPO, with mean-centered advantages and no std normalization. It runs at lag 0 and lag 8 with identical model, LoRA rank, lr, data order, rollout budget, PPO clip and eval as TailRL.",
  "batch_size": 32,
  "budget": {
    "gpu_hours": 32.0,
    "max_wall_minutes": 600
  },
  "command": "Custom tinker SDK loop reusing the sec-search-rl rollout and tool code. Custom parts are the advantage function and the lagged sampler selection.\n\nimport os, tinker\nsc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = sc.create_lora_training_client(base_model='Qwen/Qwen3.6-35B-A3B', rank=32)\nckpts = []\nfor t in range(24):\n    ckpts.append(tc.save_weights_for_sampler(name=f'u{t}').result().path)\n    samp = sc.create_sampling_client(model_path=ckpts[max(0, t - LAG)])\n    # roll out 32 queries x 8 trajectories with search/grep/read/curate tools, max 20 turns, temperature 1, keeping sampler logprobs\n    # reward = F4 of curated set (-0.2 if empty)\n    # A = tailrl_advantages(r) if ADV == 'tailrl' else r - r.mean()\n    # drop constant-reward groups\n    tc.forward_backward(datums, loss_fn='ppo').result()  # clip ratio [0.8, 1.2], sampler logprobs as behavior logprobs\n    tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).result()\n\nRun for ADV in {tailrl, drgrpo} x LAG in {0, 8}, seed 0.",
  "dataset": {
    "hf_path": "pat-jj/harness-1-train-data",
    "n_examples": 1024,
    "name": "Harness-1 SEC multi-clue search (sec_1024 train; sec_1024 dev + sec_test for eval)",
    "split": "train"
  },
  "eval": {
    "benchmark": "SEC search held-out (sec_1024 dev 64 + sec_test 64)",
    "metric": "mean F4 of curated set (also precision, recall, best-of-2), temperature 1, 2 draws per query",
    "n_examples": 128
  },
  "hypothesis": "At lag 8, PPO clipping removes at least 1.5x more of TailRL's |A|-weighted update mass than Dr. GRPO's, and TailRL's lag-0 to lag-8 F4 drop exceeds Dr. GRPO's by at least 0.05.",
  "lora_rank": 32,
  "max_seq_len": 16384,
  "method": "Linked sources failed to digest, so this design follows the refine notes from the paper v2, the official advantages.py and the lilo codegolf port.\n\nBudget forces a scale-down. The refined plan was about 30 runs at about 70 GPU-hours each on gpt-oss-20b. 32 GPU-hours buys only a 2x2 pilot:\n- Objectives: TailRL vs Dr. GRPO.\n- Lags: 0 vs 8.\n- One seed per cell.\n- Model: Qwen/Qwen3.6-35B-A3B. It has 3B active parameters, has already run on Spindle (step5), and needs no gpt-oss expert-LoRA workaround.\n\nTask: modal-projects/sec-search-rl (Harness-1). Smaller settings than the repro:\n- Up to 20 turns and 16k trajectory tokens.\n- Reward is F4 of the curated set, -0.2 if empty, with no format bonus.\n\nTraining: 32 queries x 8 rollouts per update, 24 updates, LoRA r32, AdamW, matched init, data order and seed.\n\nOnly two things vary across the four runs:\n- Advantage (the one thing that differs between baseline and treatment):\n  - TailRL: lilo codegolf/reward.py tailrl_advantages, centered and with no std normalization. Unit-test it against the official TailRL advantages.py first.\n  - Dr. GRPO: r - mean(r).\n  - Constant-reward groups are dropped for both.\n- Lag L, the condition:\n  - Each update saves a sampler checkpoint.\n  - Rollouts come from the checkpoint L updates old, and behavior logprobs are taken from that sampler.\n  - The update uses PPO-clipped IS [0.8,1.2], one update per batch, no KL.\n\nGates before reading results:\n- Lag-0 sampler/trainer KL sits near the 0.002-0.003 floor, with clip rate near 1%.\n- Lag-8 KL is at least 3x the lag-0 KL. If not, rerun at L=16.\n\nPrimary metrics are per-step staleness metrics, logged for updates 9-24:\n- IS-ratio p95/p99.\n- Raw clip rate.\n- Clip-active |A| mass fraction, bucketed by within-group reward rank and advantage sign.\n- KL, entropy, and curated-doc count (watching for over-curation).\n\nEval every 8 updates on 128 held-out queries (sec_1024 dev 64 + sec_test 64) x 2 sampled draws, reporting F4, precision, recall and best-of-2.\n\nWith one seed, the F4 difference-of-differences is reported as a pilot effect size. It sets seed counts for a follow-up and is not a decisive F4 test.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "steps": 24,
  "success_criterion": "Validated: TailRL's lag-8 clip-active |A| mass fraction (mean over updates 9-24) is at least 1.5x Dr. GRPO's, and TailRL's lag-8 F4 drop exceeds Dr. GRPO's by at least 0.05. Measured seed spread is about 0.09 and eval-draw noise about 0.02, so a single-seed F4 gap is only directional. Refuted: the mass ratio is below 1.2 and the F4 drop difference is within \u00b10.05. Anything in between triggers the multi-seed follow-up. Results are void if the lag-8 KL gate (at least 3x the lag-0 KL) fails.",
  "title": "TailRL vs Dr. GRPO under lag 0 and lag 8 on SEC search, a pilot that fits 32 GPU-hours",
  "training_type": "rl"
}

ended_reason

message failed

feasibility

{'status': 'unknown', 'problems': ['Measured feasibility not recorded'], 'wall_seconds': None, 'gpu_hours': None, 'known_gpu_hours': 0, 'phases': {}, 'record': None, 'recorded_at': 1793806034.3956628}

feasibility_required

True

launch

{'experiment_id': 'exp_9c3284c970', 'context_status': 'legacy_unknown', 'superseded': False, 'mode': 'ordinary', 'purpose': 'Execute the selected saved experiment version.'}

reported_outcome_reason

incomplete_execution

slack_started

True

started_at

1793806015.176826