🔬Modal Research Exchange

<https://arxiv.org/abs/2601.20802|arxiv.org/abs/2601.20802>
how well does SDPO work on multi-turn support agent tasks (e.g. tau-bench) with a simple rubric-based LLM judge reward? is it better at improving on certain dimensions than others, e.g. invalid tool use, ungrounded claims, or clarifying user intent upfront given ambiguous circumstances?

{
  "adjacent_experiment": "",
  "reproducibility_reason": "SDPO maps to client-side token advantages from self-teacher logprobs under the stock importance-sampling loss in the Tinker RL loop. A frozen served model can act as user simulator and judge.",
  "reproducible_on_spindle": true,
  "sources": [
    {
      "brief": null,
      "error": "digest failed: ask_json failed after 2 attempts: no JSON object in response: '{\"title\":\"Reinforcement Learning via Self-Distillation\",\"claim\":\"Scalar outcome rewards in RLVR create a credit-assignment bottleneck. Self-Distillation Policy Optimization (SDPO) converts tokenized e'",
      "full_text": true,
      "kind": "paper",
      "title": "Reinforcement Learning via Self-Distillation",
      "url": "https://arxiv.org/abs/2601.20802"
    }
  ],
  "spec": {
    "activation_gate": {
      "applicable": true,
      "checks": [
        {
          "measures": "model name reported by the created training client",
          "metric": "executed_base_model",
          "one_of": [
            "Qwen/Qwen3.5-4B"
          ]
        },
        {
          "max": 1.0,
          "measures": "share of pilot rollouts where the judge returned a non-empty critique used as teacher context",
          "metric": "sdpo_frac_rollouts_with_nonempty_critique",
          "min": 0.5
        },
        {
          "max": 2.0,
          "measures": "mean |teacher_logprob - student_logprob| over agent loss tokens",
          "metric": "sdpo_mean_abs_token_advantage_nats",
          "min": 0.01
        },
        {
          "max": 10.0,
          "measures": "p99 of |teacher_logprob - student_logprob| over agent loss tokens, catching blow-ups from template mismatch",
          "metric": "sdpo_p99_abs_token_advantage_nats"
        },
        {
          "max": 1.0,
          "measures": "share of task groups whose judge scores are not all equal",
          "metric": "grpo_frac_groups_nonzero_reward_variance",
          "min": 0.25
        },
        {
          "max": 1.0,
          "measures": "share of judge calls returning valid 3-dimension JSON scores",
          "metric": "judge_parse_rate",
          "min": 0.95
        }
      ],
      "reason": "SDPO only acts when the critique-conditioned teacher assigns different logprobs than the student to the student's own tokens. GRPO only acts when judge scores vary within a group. A 2-step pilot of each arm checks both, plus the executed config."
    },
    "arms": [
      {
        "advantage": "judge scalar minus group mean",
        "lr": 4e-05,
        "name": "grpo_seed0",
        "seed": 0,
        "steps": 20
      },
      {
        "advantage": "self-teacher logprob minus student logprob, critique-conditioned",
        "lr": 4e-05,
        "name": "sdpo_seed0",
        "seed": 0,
        "steps": 20
      }
    ],
    "base_model": "Qwen/Qwen3.5-4B",
    "baseline": "GRPO with the judge's scalar mean-of-rubric reward and a group-mean baseline. It uses the same model, LoRA init, tasks, task order, group size, turn cap, LR, steps, simulator, judge and eval as the treatment.",
    "batch_size": 8,
    "budget": {
      "gpu_hours": 2.0,
      "max_wall_minutes": 240
    },
    "command": "Custom tinker SDK script (client-side custom part = advantage computation only). svc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL']); for arm in ['grpo','sdpo']: tc = svc.create_lora_training_client(base_model='Qwen/Qwen3.5-4B', rank=32, seed=0). sim = judge = svc.create_sampling_client(base_model='Qwen/Qwen3.5-9B'). Tasks come from the pinned tau-bench retail train split: the first 160 in fixed order, 8 per step, identical across arms. Per step: pol = tc.save_weights_and_get_sampling_client(); roll out 4 trajectories per task (max 10 turns, max_tokens 1024 per turn) against the tau-bench retail tools and the simulator. The judge returns JSON {invalid_tool_use, ungrounded_claims, clarification, critique}, and reward = mean of the three scores. GRPO adv = reward - group mean on every agent token. SDPO adv per token = pol.compute_logprobs(critique+best_demo+trajectory)[agent tokens] - sampled logprobs. Then tc.forward_backward(datums, loss_fn='importance_sampling'); tc.optim_step(tinker.AdamParams(learning_rate=4e-5, beta1=0.9, beta2=0.95, eps=1e-8)). Log activation-gate metrics every step. After step 20, run save_weights_for_sampler and evaluate on all 115 retail test tasks at temperature 0 with the tau-bench env reward; also log judge per-dimension scores. Activation-failure plan: first run offline forced checks. Empty critique must give near-zero SDPO advantage, and an injected wrong-answer critique must give nonzero advantage. Gradient checks follow per the correctness preflight. The 2-step pilot is kept as-is; if a gate fails, record a revised spec/deviation before changing the teacher template or turn cap. Gates are never relaxed. Update cap is 20 steps per arm; the wall cap is 240 min total.",
    "correctness_preflight": {
      "paths": [
        {
          "baseline_revision": "UNRESOLVED: pin tinker-cookbook commit SHA of stock rl importance_sampling + group-mean advantage",
          "id": "grpo_judge_advantage_via_custom_path",
          "kind": "loss",
          "reference_revision": "UNRESOLVED: same pinned tinker-cookbook SHA (forked baseline must reproduce stock exactly)",
          "tolerances": {
            "baseline_gradient": {
              "atol": 0.0001,
              "min_cosine": 0.999,
              "rtol": 0.01
            },
            "baseline_loss": {
              "atol": 0.0001,
              "rtol": 0.001
            },
            "reference_gradient": {
              "atol": 0.0001,
              "min_cosine": 0.999,
              "rtol": 0.01
            },
            "reference_loss": {
              "atol": 0.0001,
              "rtol": 0.001
            }
          }
        },
        {
          "baseline_revision": "UNRESOLVED: pinned tinker-cookbook SHA; with teacher context empty the SDPO path must reproduce stock on-policy-distillation zero advantage / stock loss",
          "id": "sdpo_self_teacher_advantage",
          "kind": "loss",
          "reference_revision": "UNRESOLVED: pinned commit of the SDPO authors' reference code for arXiv 2601.20802",
          "tolerances": {
            "baseline_gradient": {
              "atol": 0.0001,
              "min_cosine": 0.999,
              "rtol": 0.01
            },
            "baseline_loss": {
              "atol": 0.0001,
              "rtol": 0.001
            },
            "reference_gradient": {
              "atol": 0.001,
              "min_cosine": 0.995,
              "rtol": 0.02
            },
            "reference_loss": {
              "atol": 0.001,
              "rtol": 0.01
            }
          }
        }
      ],
      "reason": "SDPO is a custom advantage path. The paper digest failed, and no pinned SDPO reference implementation or exact teacher-context and loss form has been established. The required SHAs below must be filled before any training. Until then: no full sweep.",
      "required": true,
      "status": "unresolved"
    },
    "dataset": {
      "hf_path": null,
      "n_examples": 160,
      "name": "tau-bench retail (train tasks for RL, test tasks for eval)",
      "split": "train"
    },
    "decision_rule": {
      "alpha": 0.05,
      "criteria": [
        {
          "bounds": [
            0,
            1
          ],
          "id": "primary",
          "margin": 0.05,
          "metric": "pass_at_1",
          "weights": {
            "baseline": -1,
            "treatment": 1
          }
        }
      ],
      "estimand": "Difference in tau-bench retail test pass^1 between the SDPO-trained and GRPO-trained Qwen3.5-4B LoRA at step 20, conditional on these checkpoints; larger is better",
      "method": "paired_hoeffding",
      "seeds": [
        0
      ],
      "unit": "fixed_checkpoint_items",
      "version": 1
    },
    "eval": {
      "aggregation": null,
      "benchmark": "tau-bench retail test (pinned sierra-research/tau-bench release), user simulator = frozen Qwen3.5-9B, temperature 0",
      "metric": "pass^1 (fraction of tasks where the environment's final database-state reward = 1)",
      "n_examples": 115
    },
    "experiment_features": {
      "adaptive_selection": false,
      "checkpoint_restore": false,
      "custom_loss": true,
      "custom_optimizer": false,
      "generated_training_data": false,
      "heldout_loss": false,
      "learned_generator": false,
      "paired_comparison": true,
      "paired_training_arms": true,
      "warmup_adaptation": false
    },
    "feasibility_check": {
      "gpu_hours": null,
      "known_gpu_hours": 0,
      "phases": {},
      "problems": [
        "Measured feasibility not recorded"
      ],
      "status": "unknown",
      "wall_seconds": null
    },
    "feasibility_required": true,
    "generation_audit": {
      "answer_comparison": "numeric_exact",
      "max_eval_overlaps": 0,
      "max_label_errors": 0,
      "min_sample_size": 100,
      "required": false
    },
    "hypothesis": "At equal rollout budget, SDPO using the judge's written critique as teacher context beats GRPO using the same judge's scalar score on tau-bench retail test pass^1 by at least 5 points.",
    "lora_rank": 32,
    "max_seq_len": 16384,
    "method": "Policy: Qwen/Qwen3.5-4B with a LoRA adapter, acting as the tau-bench retail tool-calling agent. Frozen Qwen/Qwen3.5-9B base sampling clients serve as the tau-bench user simulator and as a rubric judge. The judge scores each finished trajectory 0-1 on three dimensions: invalid tool use, ungrounded claims, and clarifying ambiguous intent. It also writes a short critique. Both arms train from the same fresh LoRA init on the same task order and the same sampled groups per step: 8 train tasks x group 4, at most 10 agent turns. Baseline (GRPO): the advantage is the judge mean score minus the group mean, applied to all agent tokens, using the stock importance_sampling loss. Treatment (SDPO): the same model is re-prompted with the judge critique, plus the best same-group trajectory if its score is at least 0.8, inserted before the original turns. This makes it a self-teacher. Teacher logprobs come from compute_logprobs on the agent's own response tokens. The per-token advantage is teacher_logprob - student_logprob, the reverse-KL distillation used for on-policy distillation in tinker-cookbook, and goes through the same importance_sampling loss. The arms therefore differ only in how token advantages are computed. The SDPO paper's digest failed, so the exact SDPO formulation (teacher context template, any mixing with the outcome reward, clipping) must be confirmed from the paper and a pinned reference implementation before any training run. The correctness preflight is unresolved, so no full sweep runs until it is resolved. tau-bench is taken from the pinned sierra-research/tau-bench GitHub release, not from HF; its retail train split supplies training tasks, and its test split is used only for eval. Primary eval uses tau-bench's own database-state reward, not the judge, to avoid scoring the arms with the training reward. Per-dimension judge scores on the test trajectories are secondary and descriptive. No Spindle fork is needed.",
    "optimizer": {
      "lr": 4e-05,
      "name": "adam"
    },
    "preflight": {
      "probe": "Resolve the correctness preflight SHAs and run its loss and gradient checks first. Then create both arms' training clients exactly as configured (Qwen3.5-4B, rank 32, seed 0, Adam lr 4e-5, betas 0.9/0.95, eps 1e-8). Run one rollout of one train task, one judge call, and one teacher compute_logprobs. Then one forward_backward(importance_sampling) and one optim_step per arm. Confirm the fixed-batch logprobs match across the arms before the update.",
      "riskiest_assumption": "The self-teacher (same 4B model conditioned on a 9B judge's critique) produces a meaningful, stable token-level signal on long multi-turn tool-use trajectories within the context limit, and the SDPO form we implement matches the paper.",
      "tuning": "Both arms use the same optimizer (Adam) and a single fixed LR of 4e-5, the cookbook LoRA RL default, with no per-arm grid and no adaptive selection, because the budget does not allow one. A clear loss scale mismatch in the pilot (SDPO mean |adv| outside the gate) requires a recorded spec revision, not post-hoc tuning."
    },
    "primary_endpoint": "tau-bench retail test pass^1 (115 tasks, environment reward, step-20 checkpoints)",
    "restoration_preflight": {
      "applicable": true,
      "fixed_batch_required": true,
      "groups": [
        {
          "arms": [
            {
              "name": "grpo_seed0",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "learning_rate": 4e-05
              }
            },
            {
              "name": "sdpo_seed0",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "learning_rate": 4e-05
              }
            }
          ]
        }
      ],
      "reason": "Both arms train from a fresh LoRA on the same base. Initial identity must be shown by matching logprobs on a fixed tokenized batch before the first update, and the active Adam config must be verified after creation."
    },
    "spec_preflight": {
      "findings": [],
      "status": "passed"
    },
    "steps": 20,
    "success_criterion": "SDPO minus GRPO pass^1 on the 115 tau-bench retail test tasks exceeds 0.05. The lower bound of the paired Hoeffding 95% interval must be above 0.05. A point gain above 0.05 that fails the interval is reported as inconclusive. Per-dimension judge deltas are reported descriptively only.",
    "title": "SDPO vs GRPO on tau-bench retail with a rubric judge, Qwen3.5-4B LoRA",
    "training_type": "rl"
  },
  "summary": "SDPO turns judge critiques into a token-level self-distillation signal; we compare tau-bench retail pass rate and per-dimension judge scores against GRPO trained on the same judge reward.",
  "tags": [
    "rl",
    "paper",
    "agents",
    "eval",
    "custom-loss"
  ],
  "title": "Does SDPO beat GRPO on tau-bench retail with a judge reward?",
  "value_estimate": 3,
  "value_rationale": "Customers building support agents would care whether critique-driven self-distillation beats scalar-judge GRPO. A 2-GPU-hour single-seed pilot on tau-bench gives only a coarse signal, though. The paper digest failed, so triage saw only the link and a truncated claim."
}