🔬Modal Research Exchange

@peyton this was the og paper https://arxiv.org/pdf/2012.02096, formalizes auto environment design as a zero sum game between teacher + student policies, since then there have been a bunch of variants/improvements from the same author michael dennis -- https://arxiv.org/pdf/2203.01302, https://proceedings.neurips.cc/paper_files/paper/2021/file/0e915db6326b6fb6a3c56546980a8c93-Paper.pdf

https://arxiv.org/abs/2601.18778 this is the closest i've seen to that idea in the llm space, where a teacher policy will be RL'ed based on how its produced tasks improve student scores on some eval dataset

{
  "adjacent_experiment": "",
  "reproducibility_reason": "Both teacher and student are open-model LoRAs driven through the tinker SDK. The teacher's reward is computed client-side from the student's probe-set logprobs, so the only custom code is the reward and loop.",
  "reproducible_on_spindle": true,
  "spec": {
    "base_model": "Qwen/Qwen3.5-4B",
    "baseline": "Same loop, seeds, prompts, student settings (lr 1e-4, rank 32, 150 SFT steps of 32 generated tasks) and eval, but the teacher LoRA is never updated, so the frozen teacher generates all tasks.",
    "batch_size": 32,
    "budget": {
      "gpu_hours": 2.0,
      "max_wall_minutes": 120
    },
    "command": "python ued_teacher.py  # custom loop: sc=tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL']); teacher=sc.create_lora_training_client(base_model='Qwen/Qwen3.5-4B', rank=32); student=sc.create_lora_training_client(base_model='Qwen/Qwen3.5-0.8B', rank=32); for step in range(150): tasks=teacher.save_weights_and_get_sampling_client('t').sample(...32 prompts...); nll0=probe_nll(student); student.forward_backward(sft_datums(tasks),'cross_entropy'); student.optim_step(tinker.AdamParams(learning_rate=1e-4)); r=nll0-probe_nll(student); adv=r-ema; teacher.forward_backward_custom(pg_datums(tasks,adv), is_loss); teacher.optim_step(tinker.AdamParams(learning_rate=2e-5))  # baseline arm: TEACHER_FROZEN=1 skips teacher.optim_step; final eval via student.save_weights_and_get_sampling_client on gsm8k test",
    "dataset": {
      "hf_path": "openai/gsm8k",
      "n_examples": 7473,
      "name": "GSM8K (probe set, seed examples, eval)",
      "split": "train (128 probe + seed prompts) / test (eval)"
    },
    "eval": {
      "benchmark": "GSM8K test (student, greedy, zero-shot)",
      "metric": "exact-match accuracy on final number",
      "n_examples": 1319
    },
    "hypothesis": "A student trained on tasks from a teacher RL-trained on student probe-loss reduction beats a student trained on tasks from the same frozen teacher by at least 3 points of GSM8K test exact-match accuracy.",
    "lora_rank": 32,
    "max_seq_len": 1024,
    "method": "Two LoRA training clients are used. The teacher is Qwen/Qwen3.5-4B, rank 32, trained with RL. The student is Qwen/Qwen3.5-0.8B, rank 32, trained with SFT. Each outer step runs as follows. (1) The teacher samples 32 grade-school math problems, each with a worked solution ending in '#### <number>', from a fixed prompt seeded with 2 random GSM8K train examples. (2) The student takes one SFT step on those 32 problem/solution pairs. (3) Student mean NLL is recomputed on a fixed probe set of 128 GSM8K train examples via logprob calls, and the batch reward is (NLL before the step) minus (NLL after the step). Malformed generations get reward -1. (4) Each task gets advantage = batch reward minus an EMA baseline of past batch rewards, and the teacher takes one importance-sampling policy-gradient step through forward_backward_custom. Everything above is custom client-side loop code; only standard SDK calls are used. The baseline arm runs the identical loop with the teacher's optim_step skipped, so the teacher stays frozen. After 150 outer steps, each final student is evaluated on GSM8K test.",
    "optimizer": {
      "lr": 2e-05,
      "name": "adam"
    },
    "steps": 150,
    "success_criterion": "Validated if the RL-teacher student scores at least 3.0 points higher GSM8K test accuracy than the frozen-teacher student (standard error is about 1.3 points at n=1319) and the teacher's mean batch reward rises over training. Failed if the gap is under 1.5 points or negative.",
    "title": "RL teacher vs frozen teacher generating GSM8K-style SFT data for a 0.8B student",
    "training_type": "rl"
  },
  "summary": "A teacher model is rewarded by how much its generated math problems lower a student's loss on held-out GSM8K; we compare student GSM8K accuracy against a frozen teacher.",
  "tags": [
    "rl",
    "data",
    "paper",
    "curriculum"
  ],
  "title": "Does an RL-trained task generator beat a frozen one for student SFT?",
  "value_estimate": 3,
  "value_rationale": "A positive result would show Spindle users can learn a synthetic-data generator against their own eval. The batch-level reward is noisy, though, so a first run is more likely to settle feasibility than yield a product recommendation."
}