🔬Modal Research Exchange

{
  "base_model": "Qwen/Qwen3.5-4B",
  "baseline": "Dr. GRPO (mean-centered, no std) advantages under the identical lag-0/lag-8 schedule, model, data order, clip range, lr and eval. The comparison is TailRL's lag-8 minus lag-0 F4 change against Dr. GRPO's.",
  "batch_size": 128,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 180
  },
  "command": "# Custom tinker SDK script: the advantage fn and the lagged sampler are custom; everything else is stock.\nimport os, tinker, collections\nfrom sec_search_rl.env import SecSearchEnv  # modal-projects/sec-search-rl, max_turns=12, max_tokens=8192\nfrom lilo_codegolf.reward import tailrl_advantages  # unit-tested vs Zanette-Labs advantages.py@5682c6a\nADV = os.environ['ADV']; LAG = int(os.environ['LAG'])  # ADV in {tailrl, drgrpo}, LAG in {0, 8}\nsc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-4B', rank=32)\nckpts = collections.deque([tc.save_weights_and_get_sampling_client(name='step0')], maxlen=LAG+1)\nfor step in range(24):\n    sampler = ckpts[0]  # checkpoint LAG updates old (or step0 while warming up)\n    groups = SecSearchEnv.rollout(sampler, n_queries=16, group_size=8, seed=step, temperature=1.0)  # keeps behavior logprobs\n    groups = [g for g in groups if len(set(g.rewards)) > 1]\n    for g in groups:\n        g.adv = tailrl_advantages(g.rewards) if ADV == 'tailrl' else [r - sum(g.rewards)/len(g.rewards) for r in g.rewards]\n    data = SecSearchEnv.to_datums(groups)  # target_tokens, sampler logprobs, per-token advantages\n    out = tc.forward_backward(data, loss_fn='ppo', loss_fn_config={'clip_low_threshold': 0.8, 'clip_high_threshold': 1.2}).result()\n    SecSearchEnv.log_staleness(out, data, groups)  # KL, IS p95/p99, clip rate, clip-active |A| mass by rank/sign, entropy, curated docs\n    tc.optim_step(tinker.AdamParams(learning_rate=2e-5)).result()\n    ckpts.append(tc.save_weights_and_get_sampling_client(name=f'step{step+1}'))\nSecSearchEnv.evaluate(tc.save_weights_and_get_sampling_client(name='final'), split='sec_test', draws=2, log_path=f'/tmp/rx-tailrl-lag/{ADV}-L{LAG}')",
  "dataset": {
    "hf_path": "pat-jj/harness-1-train-data",
    "n_examples": 1024,
    "name": "Harness-1 SEC multi-clue search questions (sec_1024 train, sec_test eval)",
    "split": "train"
  },
  "eval": {
    "benchmark": "sec-search-rl sec_test (held-out SEC multi-clue questions)",
    "metric": "mean F4 of curated set over 2 sampled draws (temp 1.0), plus precision/recall and clip-active |A| mass at lag 8",
    "n_examples": 64
  },
  "hypothesis": "At rollout lag 8 under PPO-clipped IS, two things hold. First, TailRL loses at least 1.5x more of its |A|-weighted advantage mass to clipping than Dr. GRPO. Second, TailRL's lag-induced drop in held-out F4 exceeds Dr. GRPO's by at least 0.05.",
  "lora_rank": 32,
  "max_seq_len": 8192,
  "method": "Pilot, scaled down from the 30-run plan to fit the 2 GPU-hour cap. Task: the modal-projects/sec-search-rl (Harness-1) agent with BM25/grep/read/curate tools over the SEC chunk corpus. Reward is F-beta (beta=4) of the curated set, or -0.2 if the set is empty. Turns are capped at 12 and trajectories at 8192 tokens (the original uses 40 turns and 30,720 tokens).\n\nModel: Qwen/Qwen3.5-4B LoRA r32. The validated recipe uses openai/gpt-oss-20b, which is served, but it needs the PR #16 expert-LoRA workaround and about 5-7 h per run. So gpt-oss-20b is the follow-up once this pilot shows that lag actually bites.\n\nDesign is a 2x2: advantage estimator {TailRL, Dr. GRPO} x lag L {0, 8}, 1 seed per cell. All cells share the init, data order seed, sampler temperature 1.0, lr, PPO clip [0.8, 1.2], one update per batch, and no KL penalty. The decision under test is the advantage function. Lag is the stress condition.\n- TailRL uses lilo codegolf/reward.py tailrl_advantages: sort rewards ascending, w_i = N*sum_{j<=i}(r(j)-r(j-1))/(N-j+1), A = w - mean(w), no std division. It must be unit-tested against Zanette-Labs/TailRL advantages.py at commit 5682c6a before any run.\n- Dr. GRPO uses A = r - mean_group(r).\n- Constant-reward groups are dropped in both arms.\n- Lag L means each batch is sampled with the sampler checkpoint saved L updates earlier. The behavior logprobs from that sampler feed the PPO ratio.\n\nPer-step logging:\n- sampler/learner KL\n- IS-ratio p95/p99 by turn index\n- raw clip rate\n- clip-active |A| mass, bucketed by within-group reward rank and advantage sign\n- entropy\n- curated-doc count, as an over-curation check\n\nGates:\n- Lag-0 KL must sit near the measured floor (about 0.002-0.003) with clipping about 1%.\n- Lag-8 KL must be at least 3x the lag-0 KL. If not, the pilot is inconclusive and the next step reruns at L=16.\n\nEval: at step 0 and the final step, on sec_test (64 queries) x 2 sampled draws, reporting F4, precision, recall and best-of-2.\n\nScore-centering arms, std-GRPO and the 4+ seed gpt-oss-20b scale-up are deferred to follow-up specs. They should only run if this pilot passes its gates.",
  "optimizer": {
    "lr": 2e-05,
    "name": "adam"
  },
  "steps": 24,
  "success_criterion": "Gates first: lag-8 KL must be at least 3x lag-0 KL, otherwise the result is inconclusive. Validated if TailRL's clip-active |A| mass fraction at lag 8 is at least 1.5x Dr. GRPO's, averaged over the last 12 updates. Its F4 drop (lag0 minus lag8) must also exceed Dr. GRPO's by at least 0.05, which clears the measured eval-draw noise of about \u00b10.02. Refuted if the mass ratio is below 1.2 and the difference-of-differences is within \u00b10.05. Anything else is inconclusive and escalates to the multi-seed gpt-oss-20b plan.",
  "title": "Lag-8 versus lag-0 pilot for TailRL and Dr. GRPO on the sec-search-rl agent",
  "training_type": "rl"
}

{
  "base_model": "Qwen/Qwen3.5-0.8B",
  "baseline": "GRPO (std-normalized) advantages under identical lag-0 and lag-4 settings: same seeds, data order, clip range, LR, group size and eval. The comparison is the lag-4 minus lag-0 change for each objective.",
  "batch_size": 256,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 120
  },
  "command": "# custom SDK loop (custom parts: advantage fn, stale-sampler queue, clipped-IS loss + metrics); run for adv in {maxrl,grpo} x lag in {0,4} x seed in {0,1}\nimport os, collections, tinker, torch\nsc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-0.8B', rank=32)\ndef adv(r, kind):  # r: tensor[8] of 0/1\n    k, N = r.sum(), len(r)\n    if kind == 'maxrl': return torch.zeros_like(r) if k == 0 else torch.where(r > 0, N / k - 1, -1.0)\n    return (r - r.mean()) / (r.std() + 1e-6)\nsamplers = collections.deque(maxlen=LAG + 1)\nfor step in range(60):\n    samplers.append(tc.save_weights_and_get_sampling_client(name=f's{step}'))\n    behav = samplers[0]  # lag-LAG checkpoint (fresh when LAG=0)\n    groups = rollout(behav, gsm8k_batch(step, seed), group_size=8, n_groups=32, max_tokens=512, temperature=1.0)  # keep behavior logprobs; reward = exact match, format_coef=0\n    data = build_datums(groups, lambda r: adv(r, ADV))\n    def loss(data, logprobs):  # PPO-clipped IS: ratio = exp(cur - behav), clip [0.8,1.2]; logs ratio p95, KL, clip frac, |A|-weighted clip frac by k-bucket\n        return clipped_is_loss(data, logprobs, lo=0.8, hi=1.2)\n    tc.forward_backward_custom(data, loss).result()\n    tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).result()\nevaluate_pass_at_k(tc.save_weights_and_get_sampling_client(name='final'), split='test', n=500, k=8, temperature=1.0)",
  "dataset": {
    "hf_path": "openai/gsm8k",
    "n_examples": 7473,
    "name": "GSM8K",
    "split": "train (config main); eval on test"
  },
  "eval": {
    "benchmark": "GSM8K test (first 500 problems)",
    "metric": "Primary: |A|-weighted clipped fraction at lag 4, by k-bucket. Secondary: sampled pass@1 and pass@8 (temp 1, 8 samples, unbiased)",
    "n_examples": 500
  },
  "hypothesis": "At 4-step rollout lag with PPO clip [0.8,1.2], TailRL/MaxRL loses at least 1.5x the |A|-weighted clipped advantage mass that GRPO loses, concentrated in groups with k<=2 successes of 8. Its lag-induced pass@8 drop is also larger than GRPO's.",
  "lora_rank": 32,
  "max_seq_len": 1024,
  "method": "Train Qwen3.5-0.8B LoRA rank 32 on GSM8K train with binary reward: exact-match correctness, format_coef=0, so TailRL equals MaxRL exactly. Use group_size 8 and 32 groups per batch. The only objective difference is the per-group advantage. TailRL/MaxRL uses A = N/k - 1 for successes, -1 for failures, and 0 when k=0. GRPO uses A = (r - p)/(std + eps). Both are group-mean centered. The lag factor is fresh rollouts (lag 0) vs rollouts sampled from the sampler checkpoint saved 4 optimizer steps earlier (lag 4). Behavior logprobs come from the stale sampler. The loss is the PPO-clipped IS objective with clip [0.8,1.2], no KL, one update per batch. The full 2x2 (objective x lag) runs with 2 seeds per cell. Primary readouts are logged every step and bucketed by k in {1,2,3-5,6-7}: IS-ratio p95 and spread, approx KL(behavior||current), raw clip fraction, and |A|-weighted clipped fraction. Secondary readout is sampled pass@1 and pass@8 at the end (temperature 1, 8 samples, unbiased estimator). Sanity checks: run a step-0 base eval. At lag 0, confirm the IS ratio is about 1 and the clip fraction is about 0. Confirm lag 4 actually raises KL and clipping before reading accuracy. Triage could not fetch the paper, site or repo. The estimator follows the refine notes citing the official advantages.py (commit 5682c6a) and modal-projects/lilo codegolf/reward.py.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "steps": 60,
  "success_criterion": "Validated if, averaged over steps 20-60 and 2 seeds, TailRL's lag-4 |A|-weighted clipped fraction is >=1.5x GRPO's, the gap exceeds 2x the seed-to-seed range, and the excess sits in the k<=2 buckets. Accuracy is supporting evidence only: TailRL's lag-induced pass@8 drop exceeds GRPO's by >=3 pts. Refuted if the ratio is <1.2x or within seed noise. A 500-problem accuracy diff-of-diffs is underpowered, so it alone cannot validate.",
  "title": "Lag-0 vs lag-4 staleness damage for TailRL/MaxRL vs GRPO advantages",
  "training_type": "rl"
}

{
  "base_model": "Qwen/Qwen3.5-0.8B",
  "baseline": "Each objective at lag 0 (fresh rollouts) under identical settings. The comparison is the drop delta_TailRL = acc(TailRL,lag0) - acc(TailRL,lag4) versus delta_GRPO = acc(GRPO,lag0) - acc(GRPO,lag4).",
  "batch_size": 64,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 120
  },
  "command": "Custom tinker SDK script, one run per arm via OBJ={grpo,tailrl} LAG={0,4}:\n\nsc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-0.8B', rank=32)\nhist = [tc.save_weights_and_get_sampling_client(name='s0')]\n\nfor step in range(60):\n    samp = hist[max(0, len(hist) - 1 - LAG)]\n    groups = sample(samp, gsm8k_batch(step), group_size=8, groups_per_batch=8, max_tokens=512)\n    r = reward(groups)\n    adv = (grpo_adv if OBJ == 'grpo' else tailrl_adv)(r - r.mean(axis=1, keepdims=True))\n    tc.forward_backward(build_datums(groups, adv, sampler_logprobs), loss_fn='importance_sampling').result()\n    tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).result()\n    hist.append(tc.save_weights_and_get_sampling_client(name=f's{step+1}'))\n\nThen evaluate the final sampler greedily on 500 openai/gsm8k test problems.",
  "dataset": {
    "hf_path": "openai/gsm8k",
    "n_examples": 7473,
    "name": "GSM8K",
    "split": "train"
  },
  "eval": {
    "benchmark": "GSM8K test",
    "metric": "greedy exact-match accuracy",
    "n_examples": 500
  },
  "hypothesis": "Training on rollouts 4 steps stale lowers TailRL's GSM8K test accuracy at least 3 points more than it lowers GRPO's, versus fresh rollouts.",
  "lora_rank": 32,
  "max_seq_len": 1024,
  "method": "Qwen3.5-0.8B with a LoRA adapter is trained by RL on GSM8K train with a binary exact-match reward. The run uses a short tinker SDK loop instead of the math_rl recipe because two pieces are custom.\n\nCustom piece 1, advantages: computed client-side. GRPO uses reward minus group mean. TailRL uses its own tail-focused advantage on the same group-mean-centered scores. Triage had no source for TailRL's exact definition, so the submitter must supply that function before launch.\n\nCustom piece 2, staleness: the stale arm samples rollouts from the sampler weights saved k=4 optimizer steps earlier. The fresh arm (k=0) samples from the current weights. Both arms train with the stock importance_sampling loss using sampler logprobs.\n\nThe design is 2x2: objective {GRPO, TailRL} by lag {0, 4}. Everything else is identical across arms: seed, data order, renderer, hyperparameters. The decision under test is the staleness effect within each objective, so each lag-4 run is compared to its own lag-0 run.\n\nMetrics logged per step:\n- mean KL between sampler and learner logprobs\n- importance-ratio mean and max\n- fraction of zero-advantage groups\n- train reward\n\nGSM8K stands in for the agentic task in the original request.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "steps": 60,
  "success_criterion": "Validated if delta_TailRL - delta_GRPO >= 3.0 points on 500 GSM8K test problems. The single-arm standard error is about 2 points, so a 3-point gap in deltas is treated as the minimum signal. Failed if the gap is below 1 point. Anything in between is inconclusive and warrants a second seed.",
  "title": "Staleness penalty on GSM8K for TailRL versus GRPO, group-mean centered",
  "training_type": "rl"
}

{
  "base_model": "Qwen/Qwen3.5-0.8B",
  "baseline": "GRPO advantage (centered reward / group std), with identical sampling, lag schedule, IS clipping, LR, seed and eval. Each objective's lag-4 run is also compared against its own lag-0 run.",
  "batch_size": 128,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 120
  },
  "command": "python rx_tailrl_lag.py --adv {grpo,tailrl} --lag {0,4}  # custom tinker SDK script:\nimport os, tinker; from collections import deque\nsc = tinker.ServiceClient(base_url=os.environ[\"TINKER_BASE_URL\"])\ntc = sc.create_lora_training_client(base_model=\"Qwen/Qwen3.5-0.8B\", rank=32)\nbuf = deque(maxlen=LAG+1); buf.append(tc.save_weights_and_get_sampling_client(name=\"s0\"))\nfor step in range(60):\n    sampler = buf[0]  # weights from LAG steps ago\n    groups = sample_gsm8k_groups(sampler, n_groups=16, group_size=8, max_tokens=512)  # cookbook math_rl env\n    r = center(groups.rewards)  # score centering, both arms\n    adv = grpo_adv(r) if ADV=='grpo' else tail_adv(r, q=0.75)  # CUSTOM\n    log_staleness_metrics(tc, groups)  # IS ratio, clip frac, KL, probe-grad cosine\n    tc.forward_backward_custom(build_data(groups, adv), clipped_is_pg_loss)  # CUSTOM loss\n    tc.optim_step(tinker.AdamParams(learning_rate=2e-5))\n    buf.append(tc.save_weights_and_get_sampling_client(name=f\"s{step+1}\"))\neval_gsm8k(tc.save_weights_and_get_sampling_client(name=\"final\"), n=500)",
  "dataset": {
    "hf_path": "openai/gsm8k",
    "n_examples": 7473,
    "name": "GSM8K",
    "split": "main/train"
  },
  "eval": {
    "benchmark": "GSM8K test",
    "metric": "exact-match accuracy, greedy decoding",
    "n_examples": 500
  },
  "hypothesis": "Going from on-policy to 4-step-stale rollouts lowers GSM8K test accuracy for TailRL by at least 3 points more than it lowers it for GRPO.",
  "lora_rank": 32,
  "max_seq_len": 1024,
  "method": "Use the tinker SDK RL loop with the GSM8K env from tinker_cookbook.recipes.math_rl and Qwen/Qwen3.5-0.8B LoRA rank 32. Each step samples 16 groups of 8 rollouts. In the stale condition, rollouts come from the sampler weights saved 4 optimizer steps earlier, kept in a ring buffer of save_weights_for_sampler checkpoints; the on-policy condition uses lag 0. Rewards are score-centered per group (reward minus group mean) for both objectives.\n\nThe only thing that differs between arms is the advantage estimator, which is custom client-side code:\n- GRPO: the centered reward divided by the group std.\n- TailRL: a tail-weighted advantage that only credits rollouts above the group's 75th-percentile reward. The exact form is our assumption, because no TailRL source was provided; it should be confirmed with the submitter before running.\n\nThe loss is an importance-sampled policy gradient with a PPO-style ratio clip of 0.2, implemented through forward_backward_custom.\n\nStaleness metrics are logged every step, computing learner logprobs on the stale rollouts:\n- mean and p95 of the per-token importance ratio\n- clip fraction\n- approximate KL between the sampler policy and the learner policy\n- cosine similarity between the stale-rollout update direction and an on-policy probe gradient on 2 fresh groups\n\nThere are 4 runs (2 objectives x lag 0 and lag 4). The quantity compared is each objective's accuracy drop from lag 0 to lag 4.",
  "optimizer": {
    "lr": 2e-05,
    "name": "adam"
  },
  "steps": 60,
  "success_criterion": "Validated if the TailRL accuracy drop (lag0 minus lag4) exceeds the GRPO drop by at least 3 points on 500 GSM8K test examples, about 2x the roughly 1.5-pt seed noise, and TailRL's mean clip fraction at lag 4 is at least 1.5x GRPO's. Failed if the difference in drops is under 1 point.",
  "title": "Staleness sensitivity of TailRL vs GRPO advantages under 4-step rollout lag",
  "training_type": "rl"
}