🔬Modal Research Exchange

{
  "base_model": "Qwen/Qwen3.5-0.8B",
  "baseline": "GRPO (std-normalized) advantages under identical lag-0 and lag-4 settings: same seeds, data order, clip range, LR, group size and eval. The comparison is the lag-4 minus lag-0 change for each objective.",
  "batch_size": 256,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 120
  },
  "command": "# custom SDK loop (custom parts: advantage fn, stale-sampler queue, clipped-IS loss + metrics); run for adv in {maxrl,grpo} x lag in {0,4} x seed in {0,1}\nimport os, collections, tinker, torch\nsc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\ntc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-0.8B', rank=32)\ndef adv(r, kind):  # r: tensor[8] of 0/1\n    k, N = r.sum(), len(r)\n    if kind == 'maxrl': return torch.zeros_like(r) if k == 0 else torch.where(r > 0, N / k - 1, -1.0)\n    return (r - r.mean()) / (r.std() + 1e-6)\nsamplers = collections.deque(maxlen=LAG + 1)\nfor step in range(60):\n    samplers.append(tc.save_weights_and_get_sampling_client(name=f's{step}'))\n    behav = samplers[0]  # lag-LAG checkpoint (fresh when LAG=0)\n    groups = rollout(behav, gsm8k_batch(step, seed), group_size=8, n_groups=32, max_tokens=512, temperature=1.0)  # keep behavior logprobs; reward = exact match, format_coef=0\n    data = build_datums(groups, lambda r: adv(r, ADV))\n    def loss(data, logprobs):  # PPO-clipped IS: ratio = exp(cur - behav), clip [0.8,1.2]; logs ratio p95, KL, clip frac, |A|-weighted clip frac by k-bucket\n        return clipped_is_loss(data, logprobs, lo=0.8, hi=1.2)\n    tc.forward_backward_custom(data, loss).result()\n    tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).result()\nevaluate_pass_at_k(tc.save_weights_and_get_sampling_client(name='final'), split='test', n=500, k=8, temperature=1.0)",
  "dataset": {
    "hf_path": "openai/gsm8k",
    "n_examples": 7473,
    "name": "GSM8K",
    "split": "train (config main); eval on test"
  },
  "eval": {
    "benchmark": "GSM8K test (first 500 problems)",
    "metric": "Primary: |A|-weighted clipped fraction at lag 4, by k-bucket. Secondary: sampled pass@1 and pass@8 (temp 1, 8 samples, unbiased)",
    "n_examples": 500
  },
  "hypothesis": "At 4-step rollout lag with PPO clip [0.8,1.2], TailRL/MaxRL loses at least 1.5x the |A|-weighted clipped advantage mass that GRPO loses, concentrated in groups with k<=2 successes of 8. Its lag-induced pass@8 drop is also larger than GRPO's.",
  "lora_rank": 32,
  "max_seq_len": 1024,
  "method": "Train Qwen3.5-0.8B LoRA rank 32 on GSM8K train with binary reward: exact-match correctness, format_coef=0, so TailRL equals MaxRL exactly. Use group_size 8 and 32 groups per batch. The only objective difference is the per-group advantage. TailRL/MaxRL uses A = N/k - 1 for successes, -1 for failures, and 0 when k=0. GRPO uses A = (r - p)/(std + eps). Both are group-mean centered. The lag factor is fresh rollouts (lag 0) vs rollouts sampled from the sampler checkpoint saved 4 optimizer steps earlier (lag 4). Behavior logprobs come from the stale sampler. The loss is the PPO-clipped IS objective with clip [0.8,1.2], no KL, one update per batch. The full 2x2 (objective x lag) runs with 2 seeds per cell. Primary readouts are logged every step and bucketed by k in {1,2,3-5,6-7}: IS-ratio p95 and spread, approx KL(behavior||current), raw clip fraction, and |A|-weighted clipped fraction. Secondary readout is sampled pass@1 and pass@8 at the end (temperature 1, 8 samples, unbiased estimator). Sanity checks: run a step-0 base eval. At lag 0, confirm the IS ratio is about 1 and the clip fraction is about 0. Confirm lag 4 actually raises KL and clipping before reading accuracy. Triage could not fetch the paper, site or repo. The estimator follows the refine notes citing the official advantages.py (commit 5682c6a) and modal-projects/lilo codegolf/reward.py.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "steps": 60,
  "success_criterion": "Validated if, averaged over steps 20-60 and 2 seeds, TailRL's lag-4 |A|-weighted clipped fraction is >=1.5x GRPO's, the gap exceeds 2x the seed-to-seed range, and the excess sits in the k<=2 buckets. Accuracy is supporting evidence only: TailRL's lag-induced pass@8 drop exceeds GRPO's by >=3 pts. Refuted if the ratio is <1.2x or within seed noise. A 500-problem accuracy diff-of-diffs is underpowered, so it alone cannot validate.",
  "title": "Lag-0 vs lag-4 staleness damage for TailRL/MaxRL vs GRPO advantages",
  "training_type": "rl"
}