🔬Modal Research Exchange

<https://arxiv.org/abs/2601.20802>
how well does SDPO work on multi-turn support agent tasks (e.g. tau-bench) with a simple rubric-based LLM judge reward? is it better at improving on certain dimensions than others, e.g. invalid tool use, ungrounded claims, or clarifying user intent upfront given ambiguous circumstances?

[micah] <@U0CE8HJ4BGX>

{
  "adjacent_experiment": "",
  "reproducibility_reason": "Token-level SDPO fits the tinker SDK RL loop: client-side per-token advantages from compute_logprobs under a feedback-conditioned prompt. Served open models act as frozen user simulator and rubric judge.",
  "reproducible_on_spindle": true,
  "sources": [
    {
      "brief": {
        "caveats": "- Scope gap: the paper has no multi-turn agent experiments, no tau-bench, and no rubric-judge reward. Every claim about the idea's setting is extrapolation.\n- Feedback quality dependence: the authors state that uninformative or misleading feedback may not help. LLM-judge rationales can be noisy, and the self-teacher could reinforce judge-pleasing but wrong behavior. Ungrounded-claim and clarification dimensions may be hard to localize to tokens.\n- Model-scale dependence: gains grow with model size and SDPO underperforms GRPO on Qwen2.5-1.5B. Small open models, or strongly quantized or low-rank adapters, may lose the self-teacher's retrospection ability.\n- Mixed results in Sec 3: SDPO is worse than GRPO on some cells (e.g. Olmo3 tool use 62.1 vs 65.0, Qwen3 Biology 56.8 vs 59.9 at 5h). The aggregate 70.2 vs 66.6 hides this. Hyperparameters were tuned per method on 5h accuracy. GRPO's tuned off-policy variant uses a different mini-batch setup than on-policy SDPO, so comparisons are not perfectly controlled.\n- Strong results are on single-turn reasoning and code. The LCB gain (48.8 vs 41.2) uses public tests as the training signal and private tests as validation. Feedback there is highly structured, since stack traces point at errors.\n- Test-time results are selection-biased: questions were retained only if some method solved them, there are only 28 questions, and SDPO is slower than baselines on several (Table 10 shows 0.5x to 0.8x on 8 of 19).\n- No LoRA experiments, no models above 8B (except the Qwen3 scaling sweep), and no wall-clock vs reward-hacking analysis. No ablation separates the effect of shorter outputs from the effect of credit assignment on accuracy.\n- The teacher is not frozen. An unregularized teacher diverges (Table 4), so EMA or trust-region regularization is required and adds hyperparameters.\n- Forgetting is reduced relative to GRPO but still present (holdout average 42.4 vs 43.5 base).\n- Text extraction is clean and full (appendices included). Figures are only described by captions, so exact curves are unavailable.",
        "claim": "Scalar outcome rewards in RLVR create a credit-assignment bottleneck. Many environments also return tokenized feedback (runtime errors, judge evaluations). SDPO conditions the current model on that feedback to form a \"self-teacher\" and distills its next-token predictions back into the policy. The paper claims this beats GRPO in sample efficiency and final accuracy, including when only scalar rewards exist and successful rollouts serve as the feedback.",
        "context": "Builds on RLVR/GRPO and its recent fixes (DAPO clip-higher, Dr.GRPO normalization, TIS). It also builds on on-policy distillation (Agarwal 2024; Thinking Machines blog), where the teacher is a stronger external model; SDPO swaps in the same model plus feedback. It is related to self-refine and Reflexion style in-context correction, which do not update weights, and to off-policy self-distillation on teacher outputs (ReST-style SFT), which the paper shows is worse (Table 5). It is also related to PRMs, which need a separate model, and to concurrent work on a similar objective (Shenfeld et al. \"Self-distillation enables continual learning\", Zhao et al. \"Self-distilled reasoner\", Song et al. text-feedback RL, Chen et al. retrospective in-context learning).\n\nRelation to the idea: the proposer asks whether SDPO works on multi-turn support agents (tau-bench) with a rubric-based LLM judge, and whether gains differ across dimensions such as invalid tool use, ungrounded claims and clarification. The paper's \"judge evaluations\" framing and its listed future work (agentic and beyond-verifiable settings) support the motivation. However, it never evaluates an LLM-judge reward, multi-turn interaction, or per-dimension breakdowns. The closest evidence is single-turn tool-call mapping on ToolAlpaca, where SDPO gave mixed results (+0.8 Qwen3, -2.9 Olmo3 at 5h).\n\nWhy Modal might care: SDPO is a near drop-in change to RLVR pipelines, with reported shorter generations (about 3x), less forgetting than GRPO, and better wall-clock sample efficiency. If it works with text judge feedback, rubric rationales become dense credit assignment for agent training. That fits a LoRA-on-Spindle service where rollout cost is the main expense.",
        "evidence": "All numbers are from the paper, with 3 seeds unless noted. Training was full-parameter in verl with vLLM. The paper states each run used one node with 4 GH200 GPUs and took about 6 hours including initialization and validation. The stated optimizer is AdamW.\n\n(1) Scalar-reward settings (Sec 3). Models were Qwen3-8B and Olmo3-7B-Instruct, non-thinking, max response length 8192. Tasks were SciKnowEval L3 Chemistry/Physics/Biology/Materials and ToolAlpaca tool use, scored as avg@16 with a train/test split. The baseline was an improved GRPO (asymmetric clip with eps-high 0.28, no std normalization, TIS correction). The headline is 70.2% vs 66.6% aggregate final accuracy for SDPO vs GRPO. Examples from Table 3:\n- Qwen3-8B Chemistry at 5h: SDPO 80.9, GRPO 74.5, on-policy GRPO 63.4.\n- Olmo3 Chemistry at 5h: SDPO 80.0, GRPO 56.7. SDPO reached the 5h GRPO accuracy in 50 min.\n- Tool use at 5h, Qwen3-8B: SDPO 68.5, GRPO 67.7, on-policy GRPO 65.7.\n- Tool use at 5h, Olmo3: SDPO 62.1, GRPO 65.0, so SDPO is worse here in Table 3. In Table 7 (per-task tuned) it is 62.5 vs 65.0, still worse.\n- Biology and Physics are mixed: Qwen3 Biology at 5h is SDPO 56.8 vs GRPO 59.9, and Qwen3 Physics at 5h is 75.6 vs 72.7.\nSDPO's responses were about 3.2x shorter on average (Table 8: 255.8 vs 820.8 tokens for Qwen3-8B, 343.9 vs 1095.4 for Olmo3), and 11x shorter on Olmo3 Chemistry.\n\n(2) Rich feedback on LiveCodeBench v6 (Sec 4). 131 questions, Qwen3-8B, validation accuracy averaged over 4 rollouts. Feedback is LeetCode-style, with public tests as the training signal and private tests for validation. SDPO reached 48.8% vs GRPO 41.2%, and matched GRPO's final accuracy with 4x fewer generations. Other baselines (Table 9) were GSPO 40.1, CISPO 41.2, and GRPO restricted to high-entropy tokens 37.8. Claude Sonnet 4 scored 40.5 and Opus 4 scored 39.7 on the same subset. The gain over GRPO grows with model size across the Qwen3 family (Fig 8). Qwen2.5-1.5B-Instruct shows SDPO underperforming GRPO (Fig 17). SDPO+GRPO is more robust at 0.6B but slightly worse than SDPO at 8B.\n\n(3) Ablations. Sequence-level < token-level < logit-level SDPO, but even sequence-level beats GRPO (Fig 10). In Table 6, environment output plus own solution works best (48.3 vs 39.9 for output only and 42.6 for own solution only). Including the student's own attempt y in the teacher prompt collapses diversity (30.0% same output, entropy 0.23).\n\n(4) Forgetting (Table 5, Qwen3-8B). Average holdout across IFEval, ArenaHard-v2 (hard prompt and creative writing) and MMLU-Pro is 42.4 for SDPO, 41.8 for GRPO, 41.4 for SFT on self-teacher outputs, and 43.5 for the base model. SDPO therefore forgets less than GRPO, but does not eliminate the drop.\n\n(5) Test-time self-distillation (Sec 5). 19 hard questions (pass@64<0.5) and 9 very hard (pass@64<0.03), selected because at least one method solved them. Question batch size is 1, 16 rollouts, up to 2750 generations, 5 seeds. On very hard questions discovery@2750 was SDPO 53.2% vs best-of-k 41.5% vs multi-turn 35.6%, with about 3x fewer generations to reach 22% discovery. On hard questions it was 78% for SDPO, versus 72.3% solved by best-of-k and 68.4% by multi-turn. Per-question speedups range from 0.5x to 13.6x (Table 10), and SDPO is slower on several questions.",
        "method": "For each prompt x, sample G rollouts from the policy and get feedback f_i for each. Feedback can be environment output (e.g. a LeetCode-style runtime error or failed test) and/or a successful sibling rollout from the same group used as a \"correct solution\". Build a reprompt (Table 2): the prompt, then \"Correct solution: ...\" if one exists, then \"The following is feedback from your unsuccessful earlier attempt: ...\", then \"Correctly solve the original question.\" The original response is appended as the assistant turn. Recompute the log-probs of the ORIGINAL response under this feedback-conditioned context, so no extra sampling is needed. Loss = sum_t KL(pi_theta(.|x,y_<t) || stopgrad(pi_theta(.|x,f,y_<t))). The gradient equals a logit-level policy gradient with advantage log(q_teacher/pi_student). It drops into a standard verl GRPO pipeline by swapping the advantages. Memory is reduced with top-K distillation (K=100 in Sec 3, K=20 in Secs 4 and 5) plus a tail term. Stability tricks: a regularized teacher (EMA with rate 0.05 in Sec 3 and 0.01 in Secs 4 and 5, or a trust-region interpolation with the initial teacher) and Jensen-Shannon divergence (Sec 3) or reverse-KL (Secs 4 and 5). An optional hybrid is lambda*A_GRPO + (1-lambda)*A_SDPO, with lambda=0.9 in Fig 11.",
        "reproduction": "Fit to Modal's setup: the idea (tau-bench with a rubric LLM judge) is the paper's own listed future work (\"agentic settings\", \"beyond verifiable rewards\"). The paper does NOT test tau-bench, multi-turn agents, or rubric-judge reward, so this would be an extension, not a replication.\n\nData and environment:\n- tau-bench retail/airline tasks with a user simulator and tool calls. The paper gives no template for multi-turn trajectories, so the design is up to us.\n- A rubric judge outputs per-dimension scores plus textual rationale. Dimensions would be invalid tool use, ungrounded claims, and upfront clarification, matching the idea text.\n- The judge text is the feedback f. Include a successful sibling rollout as \"correct solution\" when one exists (the paper's best config combined output plus own solution). Exclude the student's own attempt y from the teacher prompt.\n- Hold out tau-bench tasks for eval and report per-dimension judge scores plus the native tau-bench pass^k reward.\n\nSmallest experiment: a 3-arm comparison on one mid-size model, ideally Qwen3-8B-class or the largest Qwen3.5 Spindle supports. Arms: (a) GRPO on the scalar rubric score, (b) SDPO with the rubric text as feedback, (c) optionally SDPO+GRPO with lambda=0.9. Log per-dimension judge scores and tau-bench task success. The paper's evidence suggests base-model scale matters, so do not start with a small model.\n\nPaper hyperparameters to start from (full-parameter, not LoRA):\n- Sec 4 coding settings: LR 1e-6 constant, AdamW, wd 0.01, grad clip 1.0, 8 rollouts, 32 questions per batch, temperature 1.0, top-K=20, reverse-KL, EMA teacher rate 0.01, rollout IS clip 2.\n- Sec 3 settings: LR 1e-5, JS divergence, top-K=100, EMA rate 0.05, 10 warmup steps.\n- Max response 8192, prompt 2048, non-thinking mode.\n- LoRA typically wants a higher LR than full fine-tuning, so these LRs will not transfer directly.\n\nSpindle/LoRA fit:\n- The self-teacher must be the current policy, including the LoRA delta, conditioned on feedback. The teacher log-probs need a second forward pass over the same sampled tokens with a different context. This needs a Tinker-style API that can return log-probs (ideally top-K logprobs per position) for given tokens under the current weights, and a way to inject a custom per-token or per-logit loss. If Spindle only exposes scalar-advantage policy-gradient or importance_sampling losses, then sequence-level or token-level SDPO is feasible: advantage = teacher_logprob - student_logprob on the sampled token. Full logit-level SDPO (top-K plus tail) is not feasible unless custom losses are supported.\n- The paper finds logit > token > sequence, so a token-level port would be a weaker version than the paper's best.\n- The EMA teacher needs a second copy of the LoRA weights, or an interpolated teacher using the reference adapter state (trust-region, which needs reference log-probs). Frozen-initial-teacher is a cheap fallback that the paper reports works (Table 4: 48.8 best vs 50.6 trust-region).\n- Multi-turn: tokens come from multiple assistant turns interleaved with tool and user messages. Teacher re-scoring must mask non-assistant tokens and keep the feedback context consistent across turns. The paper gives no guidance, so this is a design choice. A per-trajectory feedback block prepended to the first turn is the simplest option.\n- Compute: the paper reports SDPO overhead over GRPO is small (Fig 5). Rollouts in long tau-bench dialogues will dominate.",
        "title": "Reinforcement Learning via Self-Distillation"
      },
      "error": null,
      "full_text": true,
      "kind": "paper",
      "title": "Reinforcement Learning via Self-Distillation",
      "url": "https://arxiv.org/abs/2601.20802"
    }
  ],
  "spec": {
    "activation_gate": {
      "applicable": true,
      "checks": [
        {
          "measures": "fraction of real assistant loss tokens with |teacher_logprob - student_logprob| > 0.1 nats",
          "metric": "sdpo_frac_tokens_abs_adv_gt_0.1",
          "min": 0.05
        },
        {
          "max": 3.0,
          "measures": "mean |teacher - student| logprob over assistant tokens, in nats",
          "metric": "sdpo_mean_abs_token_adv",
          "min": 0.01
        },
        {
          "measures": "fraction of trajectories whose teacher prompt has nonempty judge rationale",
          "metric": "sdpo_frac_trajectories_with_feedback",
          "min": 0.9
        },
        {
          "measures": "fraction of rollout groups with nonzero composite variance",
          "metric": "grpo_frac_groups_nonzero_adv",
          "min": 0.25
        },
        {
          "measures": "fraction of judge calls returning 3 parsable scores plus rationale",
          "metric": "judge_parse_success_rate",
          "min": 0.95
        },
        {
          "measures": "model name reported by the created training client",
          "metric": "executed_base_model",
          "one_of": [
            "Qwen/Qwen3.5-4B"
          ]
        }
      ],
      "reason": "SDPO acts only if the feedback-conditioned teacher assigns materially different logprobs to the sampled assistant tokens. GRPO acts only if groups have nonzero reward variance. Both are measured in a 2-step pilot, which also confirms the executed config."
    },
    "arms": [
      {
        "advantage": "group-mean-centered composite, no std norm",
        "lr": 4e-05,
        "name": "grpo",
        "seed": 0
      },
      {
        "advantage": "teacher_logprob - student_logprob under feedback-conditioned prompt",
        "lr": 4e-05,
        "name": "sdpo_token",
        "seed": 0
      }
    ],
    "base_model": "Qwen/Qwen3.5-4B",
    "baseline": "GRPO-style RL on the scalar composite rubric score, using a group-mean baseline and the stock importance_sampling loss. Same model, LoRA, lr, tasks, task order, rollout budget, simulator, judge and eval as SDPO.",
    "batch_size": 16,
    "budget": {
      "gpu_hours": 2.0,
      "max_wall_minutes": 120
    },
    "command": "Custom python script against the tinker SDK. Custom parts: the multi-turn tau-bench env wrapper, judge calls, and SDPO advantage computation. Setup: svc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL']). tc = svc.create_lora_training_client(base_model='Qwen/Qwen3.5-4B', rank=32). user_sim = svc.create_sampling_client(base_model='Qwen/Qwen3-8B'). judge = svc.create_sampling_client(base_model='openai/gpt-oss-120b'). Data: train_tasks = tau-bench retail train tasks, shuffled with seed 0, first 80; the identical list and order is used for both arms. Training loop, for step in range(20): (1) sc = tc.save_weights_and_get_sampling_client(f'{arm}-{step}'). (2) For 4 tasks x 4 rollouts, run up to 12 agent turns with sc, executing tool calls in the tau-bench retail env and drawing user turns from user_sim. Record assistant token ids and sampler logprobs, with mask=1 on assistant tokens only. (3) The judge returns JSON {invalid_tool_use, ungrounded_claims, upfront_clarification, rationale}, and composite = mean. (4) Advantages: if arm == 'grpo', adv = composite - group_mean, broadcast over masked tokens. If arm == 'sdpo_token', build the teacher context (feedback block + best sibling if composite >= 0.8 and not self, never the self attempt). Then t_lp = sc.compute_logprobs(teacher_context + original trajectory tokens), aligned on assistant tokens, and adv = clip(t_lp - sampler_lp, -5, 5) * mask. (5) Training step: tc.forward_backward(datums with target_tokens, logprobs=sampler_lp, advantages=adv, loss_fn='importance_sampling'), then tc.optim_step(tinker.AdamParams(learning_rate=4e-5)). Logging: per-step activation metrics and the judge parse rate. Eval: on test tasks 0-59, one greedy rollout per arm checkpoint, scored by the same judge and native tau-bench reward. Per-item results are saved for paired Hoeffding; per-dimension means are reported as secondary. Activation-failure plan: first run offline forced checks. Feed a teacher context with the correct sibling inserted versus an empty feedback block, confirm advantage magnitude changes, and check that gradients are nonzero only on masked assistant tokens. Then run a 2-step pilot per arm; if a gate fails, preserve the pilot logs. Only source-justified fallbacks are allowed, chosen by a fixed rule (first that passes) and recorded as a spec deviation before any full run: first add a frozen-initial-teacher context, then raise max turns to 16. Calibration is capped at 4 steps and 15 wall minutes, uses train tasks only, and never touches the eval tasks. Gates are never relaxed. Full runs only after correctness_preflight is resolved and feasibility is published from a timing pilot.",
    "correctness_preflight": {
      "paths": [
        {
          "baseline_revision": "tinker-cookbook commit hash pinned at run time (stock rl loop importance_sampling + group-mean advantages)",
          "id": "grpo_via_new_advantage_path",
          "kind": "loss",
          "reference_revision": "same pinned tinker-cookbook commit",
          "tolerances": {
            "baseline_gradient": {
              "atol": 1e-05,
              "min_cosine": 0.9999,
              "rtol": 0.0001
            },
            "baseline_loss": {
              "atol": 1e-06,
              "rtol": 1e-05
            },
            "reference_gradient": {
              "atol": 1e-05,
              "min_cosine": 0.9999,
              "rtol": 0.0001
            },
            "reference_loss": {
              "atol": 1e-06,
              "rtol": 1e-05
            }
          }
        },
        {
          "baseline_revision": "tinker-cookbook commit hash pinned at run time, with SDPO disabled reproducing stock GRPO advantages",
          "id": "sdpo_token_advantage",
          "kind": "loss",
          "reference_revision": "float64 reimplementation of token-level SDPO advantage (arXiv 2601.20802v1 eq. for token-level KL gradient), committed hash pinned at run time; official SDPO repo commit if available",
          "tolerances": {
            "baseline_gradient": {
              "atol": 1e-05,
              "min_cosine": 0.9999,
              "rtol": 0.0001
            },
            "baseline_loss": {
              "atol": 1e-06,
              "rtol": 1e-05
            },
            "reference_gradient": {
              "atol": 0.0001,
              "min_cosine": 0.999,
              "rtol": 0.001
            },
            "reference_loss": {
              "atol": 0.0001,
              "rtol": 0.001
            }
          }
        }
      ],
      "required": true,
      "status": "unresolved until revisions are pinned; no full run before resolution"
    },
    "dataset": {
      "hf_path": null,
      "n_examples": 80,
      "name": "tau-bench retail (sierra-research/tau-bench GitHub, pinned commit; train tasks for training, test tasks for eval)",
      "split": "train (training) / test (eval, first 60 tasks by id)"
    },
    "decision_rule": {
      "alpha": 0.05,
      "criteria": [
        {
          "bounds": [
            0,
            1
          ],
          "id": "primary",
          "margin": 0.03,
          "metric": "composite_rubric_score",
          "weights": {
            "baseline": -1,
            "treatment": 1
          }
        },
        {
          "bounds": [
            0,
            1
          ],
          "id": "reward_guard",
          "margin": -0.1,
          "metric": "tau_bench_task_reward",
          "weights": {
            "baseline": -1,
            "treatment": 1
          }
        }
      ],
      "estimand": "On held-out tau-bench retail tasks, conditional on the trained checkpoints: the difference in mean composite rubric score between token-level SDPO and GRPO (larger is better), with a noninferiority guard on task reward.",
      "method": "paired_hoeffding",
      "seeds": [
        0
      ],
      "unit": "fixed_checkpoint_items",
      "version": 1
    },
    "eval": {
      "aggregation": null,
      "benchmark": "tau-bench retail test tasks, 60 tasks, 1 greedy rollout each, frozen Qwen3-8B simulator, frozen gpt-oss-120b judge",
      "metric": "mean composite rubric score (0-1); guard: mean tau-bench task reward (0/1 DB-state match)",
      "n_examples": 60
    },
    "experiment_features": {
      "adaptive_selection": false,
      "checkpoint_restore": false,
      "custom_loss": true,
      "custom_optimizer": false,
      "generated_training_data": false,
      "heldout_loss": false,
      "learned_generator": false,
      "paired_comparison": true,
      "paired_training_arms": true,
      "warmup_adaptation": false
    },
    "feasibility": null,
    "feasibility_check": {
      "gpu_hours": null,
      "known_gpu_hours": 0,
      "phases": {},
      "problems": [
        "Measured feasibility not recorded"
      ],
      "status": "unknown",
      "wall_seconds": null
    },
    "feasibility_required": true,
    "generation_audit": {
      "answer_comparison": "categorical_exact",
      "max_eval_overlaps": 0,
      "max_label_errors": 0,
      "min_sample_size": 100,
      "required": false
    },
    "hypothesis": "With a fixed frozen judge, token-level SDPO beats GRPO by at least 0.03 on mean composite rubric score (0-1) over 60 held-out tau-bench retail tasks at the same rollout budget, without losing more than 0.10 tau-bench task reward.",
    "lora_rank": 32,
    "max_seq_len": 8192,
    "method": "This extends arXiv 2601.20802, which never tested multi-turn agents or judge rewards; it is not a replication. Both arms train a rank-32 LoRA on Qwen/Qwen3.5-4B, non-thinking, from the same base. Training uses tau-bench retail train tasks; held-out tasks come from the retail test split. Source is sierra-research/tau-bench on GitHub, pinned to a commit hash at run time. The 4B model is a scale-down forced by the 2 GPU-hour budget. The paper reports SDPO gains shrink at small scale, so a null result here is weak evidence against SDPO at 8B+. The user simulator is a frozen served Qwen/Qwen3-8B, substituting for tau-bench's default GPT-4o; it only samples and is never trained. The rubric judge is a frozen served openai/gpt-oss-120b. For each trajectory it returns three 0-1 scores and a short rationale per dimension. The dimensions are invalid_tool_use (inverted, so higher is better), ungrounded_claims (inverted) and upfront_clarification. Composite reward = mean of the three. Rollouts: each step takes 4 tasks x 4 rollouts, 16 trajectories, for 20 steps. Max 12 agent turns per episode, 512 tokens per turn, temperature 1.0. Each arm takes one optim_step per batch, so the policy is on-policy at sampling time. Baseline arm (GRPO): per-token advantage = composite minus the group mean, without std normalization (Dr.GRPO-style, as the paper's improved GRPO). It is broadcast over assistant tokens and passed to the stock importance_sampling loss. Treatment arm (token-level SDPO, the paper's middle variant; logit-level top-K is out of scope): the teacher prompt is the original system prompt, prepended with three things. First, a 'Correct solution:' block with the highest-composite sibling trajectory from the same group, if its composite >= 0.8 and it is not the trajectory itself. Second, 'The following is feedback from your unsuccessful earlier attempt:' plus the judge rationales. Third, 'Correctly solve the original question.' The student's own attempt is never included in the teacher prompt, following paper Table 6. The original multi-turn trajectory is then rescored with sampling_client.compute_logprobs under this context. Teacher = current policy at sampling time, stop-grad. There is no EMA: 20 steps is short, and this deviation is recorded. Per-token advantage = teacher_logprob - student_logprob on each sampled assistant token. It is clipped to [-5,5], with user, tool and system tokens masked to 0, and fed to the same importance_sampling loss. The two arms differ only in the advantage computation. Shared settings: AdamW (stock tinker.AdamParams) at lr 4e-5, renderer, tasks, task order, simulator, judge and eval. Per-dimension judge scores are logged as secondary, descriptive answers to the idea's question.",
    "optimizer": {
      "lr": 4e-05,
      "name": "adam"
    },
    "preflight": {
      "probe": "Create both arms' training clients (Qwen/Qwen3.5-4B, rank 32, adam via tinker.AdamParams lr 4e-5). Run one retail train episode and one judge call, compute SDPO teacher logprobs with compute_logprobs, and take one forward_backward plus optim_step per arm on that single trajectory. Time each phase and confirm the same fixed-batch logprobs from both fresh clients before the update.",
      "riskiest_assumption": "Three conditions must hold in 2 GPU-hours. A 4B policy must complete multi-turn retail episodes with parsable tool calls. The frozen judge's rationales must give non-trivial teacher-student logprob gaps. Episodes, simulator and judge calls must fit 2 arms x 20 steps x 16 trajectories plus 2 x 60 eval episodes.",
      "tuning": "Both arms use the same AdamW optimizer, so there is no per-arm LR grid. Both use a fixed lr of 4e-5, a standard LoRA RL learning rate, set a priori. No hyperparameter is selected from results. If the pilot shows divergence (mean |adv| above the gate or NaN), stop and record a deviation rather than tuning on eval tasks."
    },
    "primary_endpoint": "heldout_composite_rubric_score_mean",
    "restoration_preflight": {
      "applicable": true,
      "fixed_batch_required": true,
      "groups": [
        {
          "arms": [
            {
              "name": "grpo",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "learning_rate": 4e-05,
                "lora_rank": 32
              }
            },
            {
              "name": "sdpo_token",
              "optimizer": "adam",
              "optimizer_config": {
                "beta1": 0.9,
                "beta2": 0.95,
                "eps": 1e-08,
                "learning_rate": 4e-05,
                "lora_rank": 32
              }
            }
          ]
        }
      ],
      "reason": "Both arms train from the same base LoRA initialization. Initial-state identity is checked by comparing per-token logprobs on a fixed tokenized trajectory batch for both fresh clients before any update. Optimizer config is read back after creation. Server identity is unverified beyond this."
    },
    "spec_preflight": {
      "findings": [],
      "status": "passed"
    },
    "steps": 20,
    "success_criterion": "Validated if the 95% simultaneous paired Hoeffding interval for SDPO minus GRPO mean composite rubric score is entirely above 0.03 on 60 held-out tasks. The tau-bench reward contrast interval must also lie entirely above -0.10. Failed if the composite interval lies entirely below 0.03. Otherwise inconclusive, which is likely at n=60; that outcome motivates a larger run.",
    "title": "Token-level SDPO vs GRPO on tau-bench retail with a frozen rubric judge, Qwen3.5-4B LoRA",
    "training_type": "rl"
  },
  "summary": "SDPO turns judge feedback into per-token credit by rescoring rollouts under a feedback-conditioned prompt; we compare held-out rubric scores and tau-bench reward against GRPO.",
  "tags": [
    "rl",
    "eval",
    "paper",
    "agents",
    "lora"
  ],
  "title": "Does token-level SDPO beat GRPO on tau-bench retail with a rubric judge?",
  "value_estimate": 4,
  "value_rationale": "If rubric-judge rationales give denser credit than scalar GRPO for agent training, that is a near drop-in recipe change Spindle customers training support agents would adopt. The 4B, 2 GPU-hour scale makes a first result suggestive at best."
}