🔬Modal Research Exchange

{
  "base_model": "Qwen/Qwen3.5-0.8B",
  "baseline": "GRPO (std-normalized) advantages under identical lag-0 and lag-4 settings: same seeds, data order, clip range, LR, group size and eval. The comparison is the lag-4 minus lag-0 change for each objective.",
  "batch_size": 256,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 120
  },
  "command": "# custom SDK loop (custom parts: advantage fn, stale-sampler queue, clipped-IS loss + metrics); run for adv in {maxrl,grpo} x lag in {0,4} x seed in {0,1}\nimport os, collections, tinker, torch\nsc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-0.8B', rank=32)\ndef adv(r, kind):  # r: tensor[8] of 0/1\n    k, N = r.sum(), len(r)\n    if kind == 'maxrl': return torch.zeros_like(r) if k == 0 else torch.where(r > 0, N / k - 1, -1.0)\n    return (r - r.mean()) / (r.std() + 1e-6)\nsamplers = collections.deque(maxlen=LAG + 1)\nfor step in range(60):\n    samplers.append(tc.save_weights_and_get_sampling_client(name=f's{step}'))\n    behav = samplers[0]  # lag-LAG checkpoint (fresh when LAG=0)\n    groups = rollout(behav, gsm8k_batch(step, seed), group_size=8, n_groups=32, max_tokens=512, temperature=1.0)  # keep behavior logprobs; reward = exact match, format_coef=0\n    data = build_datums(groups, lambda r: adv(r, ADV))\n    def loss(data, logprobs):  # PPO-clipped IS: ratio = exp(cur - behav), clip [0.8,1.2]; logs ratio p95, KL, clip frac, |A|-weighted clip frac by k-bucket\n        return clipped_is_loss(data, logprobs, lo=0.8, hi=1.2)\n    tc.forward_backward_custom(data, loss).result()\n    tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).result()\nevaluate_pass_at_k(tc.save_weights_and_get_sampling_client(name='final'), split='test', n=500, k=8, temperature=1.0)",
  "dataset": {
    "hf_path": "openai/gsm8k",
    "n_examples": 7473,
    "name": "GSM8K",
    "split": "train (config main); eval on test"
  },
  "eval": {
    "benchmark": "GSM8K test (first 500 problems)",
    "metric": "Primary: |A|-weighted clipped fraction at lag 4, by k-bucket. Secondary: sampled pass@1 and pass@8 (temp 1, 8 samples, unbiased)",
    "n_examples": 500
  },
  "hypothesis": "At 4-step rollout lag with PPO clip [0.8,1.2], TailRL/MaxRL loses at least 1.5x the |A|-weighted clipped advantage mass that GRPO loses, concentrated in groups with k<=2 successes of 8. Its lag-induced pass@8 drop is also larger than GRPO's.",
  "lora_rank": 32,
  "max_seq_len": 1024,
  "method": "Train Qwen3.5-0.8B LoRA rank 32 on GSM8K train with binary reward: exact-match correctness, format_coef=0, so TailRL equals MaxRL exactly. Use group_size 8 and 32 groups per batch. The only objective difference is the per-group advantage. TailRL/MaxRL uses A = N/k - 1 for successes, -1 for failures, and 0 when k=0. GRPO uses A = (r - p)/(std + eps). Both are group-mean centered. The lag factor is fresh rollouts (lag 0) vs rollouts sampled from the sampler checkpoint saved 4 optimizer steps earlier (lag 4). Behavior logprobs come from the stale sampler. The loss is the PPO-clipped IS objective with clip [0.8,1.2], no KL, one update per batch. The full 2x2 (objective x lag) runs with 2 seeds per cell. Primary readouts are logged every step and bucketed by k in {1,2,3-5,6-7}: IS-ratio p95 and spread, approx KL(behavior||current), raw clip fraction, and |A|-weighted clipped fraction. Secondary readout is sampled pass@1 and pass@8 at the end (temperature 1, 8 samples, unbiased estimator). Sanity checks: run a step-0 base eval. At lag 0, confirm the IS ratio is about 1 and the clip fraction is about 0. Confirm lag 4 actually raises KL and clipping before reading accuracy. Triage could not fetch the paper, site or repo. The estimator follows the refine notes citing the official advantages.py (commit 5682c6a) and modal-projects/lilo codegolf/reward.py.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "steps": 60,
  "success_criterion": "Validated if, averaged over steps 20-60 and 2 seeds, TailRL's lag-4 |A|-weighted clipped fraction is >=1.5x GRPO's, the gap exceeds 2x the seed-to-seed range, and the excess sits in the k<=2 buckets. Accuracy is supporting evidence only: TailRL's lag-induced pass@8 drop exceeds GRPO's by >=3 pts. Refuted if the ratio is <1.2x or within seed noise. A 500-problem accuracy diff-of-diffs is underpowered, so it alone cannot validate.",
  "title": "Lag-0 vs lag-4 staleness damage for TailRL/MaxRL vs GRPO advantages",
  "training_type": "rl"
}

base_pass1

0.5623

base_pass8

0.874

maxrl_lag4_absA_clip_frac

0.001411

grpo_lag4_absA_clip_frac

0.001748

maxrl_over_grpo_lag4_absA_clip_ratio

0.8072

maxrl_lag4_absA_clip_seed_range

8e-06

grpo_lag4_absA_clip_seed_range

0.0001324

maxrl_lag0_absA_clip_frac

0.000582

grpo_lag0_absA_clip_frac

0.0006826

maxrl_lag4_k1_absA_clip_frac

0.001348

maxrl_lag4_k2_absA_clip_frac

0.001344

grpo_lag4_k1_absA_clip_frac

0.001594

grpo_lag4_k2_absA_clip_frac

0.00158

maxrl_lag4_clipped_mass_share_k_le2

0.0005308

grpo_lag4_clipped_mass_share_k_le2

0.0003282

maxrl_lag0_kl

0.0002816

maxrl_lag4_kl

0.0004554

grpo_lag0_kl

0.0002626

grpo_lag4_kl

0.0005141

maxrl_lag0_ratio_p95

1.019

grpo_lag0_ratio_p95

1.014

maxrl_lag4_ratio_p95

1.029

grpo_lag4_ratio_p95

1.032

maxrl_lag0_pass1

0.7728

maxrl_lag4_pass1

0.7272

grpo_lag0_pass1

0.8209

grpo_lag4_pass1

0.7268

maxrl_lag0_pass8

0.95

maxrl_lag4_pass8

0.943

grpo_lag0_pass8

0.95

grpo_lag4_pass8

0.927

maxrl_lag_pass8_drop_pts

0.7

grpo_lag_pass8_drop_pts

2.3

steps_per_arm

60

arms_completed

8

wall_minutes

41.7