# Reference resolution and remaining contract mismatch

Paper: https://arxiv.org/html/2601.20802v2; code: https://github.com/lasgroup/SDPO/tree/7c457fc1b1f636ae794eb0362ba37d4743b06fbc

The saved method explicitly selects sampled-token reverse-KL advantages, not the authors YAML defaults (full-logit/top-100+tail distillation, alpha=0.5, EMA teacher). The CPU reference therefore invokes the authors implemented sampled branch with full_logit_distillation=False, alpha=1, is_clip=None, token-mean. A second diagnostic uses the authors is_clip=2 option. Neither variant was used for training. This is not a claim that sampled reverse-KL is the default SDPO recipe.

Let l=current student logprob, o=immutable sampled student logprob, t=teacher logprob, N=masked token count. Specified IS surrogate: L_IS=-sum(mask*exp(l-o)*detach(t-o))/N. Authors sampled reverse-KL surrogate: L_ref=sum(mask*detach(l-t)*l)/N. At l=o their parameter gradients coincide, but raw loss values need not. With stale inputs they can also have different gradients. This suggests a surrogate/reference acceptance-contract mismatch, not an inactive treatment or an SDPO-vs-GRPO outcome. All measured gradients cover the complete 5x11 toy policy parameter matrix, NOT Qwen model parameters.

The authors generic reprompt template appends solution/feedback and asks to correctly solve the original question. It is preserved in authors-actor.yaml. No tau-bench-specific chat rendering, empty-critique behavior, real feedback injection, reward mixing, demo threshold, or clipping choice has been certified for remote training. No unchanged settings were silently replaced.

Exact zero/zero gradients in the disabled-teacher control have null cosine (undefined). The validator leaves the mandatory min_cosine check unverified. No fabricated cosine=1 is supplied. Pinning the saved UNRESOLVED revision strings in an effective spec eliminates revision mismatch, but does not eliminate the measured loss failure or undefined-zero-cosine check.

Cheapest next diagnostic: revise only the correctness contract before spending GPU: explicitly select the sampled-token variant and an on-policy one-update lag contract; define whether to compare a source-justified scalar surrogate or a predeclared detached-value alignment, and define zero/zero gradient acceptance. Rerun the retained CPU fixture. Do not simply raise tolerances. If a stale-input regime must be equivalent, change/justify the objective explicitly and revalidate all gradients. Only then probe actual Spindle sampler/trainer logprob alignment, active Adam config and matched init, followed by the authorized timing and activation pilots. No revised training was executed.

## Paper appendix A (plain-text extraction, including math annotation text)

A.1 Gradient Estimators
In this seciton, we discuss two possible gradient estimators for the KL divergence between the current policy πθ​(y∣x)\pi_{\theta}(y\mid x) and the teacher policy qθ​(y∣x,f)q_{\theta}(y\mid x,f).
Per-token estimator.
Deriving the gradient of the SDPO loss as defined in Equation 1:
ℒtoken(θ):=𝔼y∼stopgrad(πθ(⋅∣x))[∑t=1TKL(πθ(⋅∣x,y<t)∥stopgrad(πθ(⋅∣x,f,y<t)))]\mathcal{L}_{\mathrm{token}}(\theta):=\mathbb{E}_{y\sim\mathrm{stopgrad}(\pi_{\theta}(\cdot\mid x))}\left[\sum_{t=1}^{T}\mathrm{KL}(\pi_{\theta}(\cdot\mid x,y_{<t})\|\mathrm{stopgrad}(\pi_{\theta}(\cdot\mid x,f,y_{<t})))\right]
(5)
leads to the following estimator (see a detailed proof in Section B.1), which corresponds to the sum of gradients of the KL divergence at each token:
∇ℒtoken(θ)=𝔼y∼πθ(⋅∣x)[∑t=1T𝔼y^t∼πθ(⋅∣x,y<t)[∇θlogπθ(y^t∣x,y<t)⋅logπθ​(y^t∣x,y<t)πθ​(y^t∣x,f,y<t)]].\boldsymbol{\nabla}\mathcal{L}_{\text{token}}(\theta)=\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}\left[\sum_{t=1}^{T}\mathbb{E}_{\hat{y}_{t}\sim\pi_{\theta}(\cdot\mid x,y_{<t})}\!\!\left[\boldsymbol{\nabla}_{\!\!\theta}\,\log\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})\cdot\log\frac{\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})}{\pi_{\theta}(\hat{y}_{t}\mid x,f,y_{<t})}\right]\right].
(6)
This corresponds to the estimator presented in Proposition 2.1.
This gradient estimator effectively assumes that the sampling distribution generating yy is fixed.
Sequence-level estimator.
An alternative self-distillation objective minimizes the sequence-level KL divergence between student and self-teacher, i.e.,
ℒseq(θ):=KL(πθ∥qθ)=𝔼y∼πθ(⋅∣x)[logπθ​(y∣x)qθ​(y∣x,f)]=∑t=1T𝔼st∼Πθ[KL(πθ(⋅∣st)∥qθ(⋅∣st,f))],\displaystyle\begin{split}\mathcal{L}_{\mathrm{seq}}(\theta):=\mathrm{KL}\left(\pi_{\theta}\|q_{\theta}\right)&=\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}\left[\log\frac{\pi_{\theta}(y\mid x)}{q_{\theta}(y\mid x,f)}\right]\\
&=\sum_{t=1}^{T}\mathbb{E}_{s_{t}\sim\Pi_{\theta}}\left[\mathrm{KL}\left(\pi_{\theta}(\cdot\mid s_{t})\|q_{\theta}(\cdot\mid s_{t},f)\right)\right],\end{split}
(7)
where st=(x,y<t)s_{t}=(x,y_{<t}) is the prefix (“state”) at step tt and Πθ\Pi_{\theta} denotes the prefix distribution under policy πθ\pi_{\theta}.
Estimating the gradient of this objective additionally takes into account how the choice of yty_{t} influences future states y>ty_{>t} (due to the additional dependence on Πθ\Pi_{\theta}).
Amini et al. (2025) show that the corresponding gradient estimator is given by
∇ℒseq(θ)=∇ℒtoken(θ)+𝔼y∼πθ(⋅∣x)[∑t=1TKL(πθ(⋅∣st)∥qθ(⋅∣st,f))∇θlogΠθ(st)].\boldsymbol{\nabla}\mathcal{L}_{\text{seq}}(\theta)=\boldsymbol{\nabla}\mathcal{L}_{\text{token}}(\theta)+\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}\left[\sum_{t=1}^{T}\mathrm{KL}\left(\pi_{\theta}(\cdot\mid s_{t})\|q_{\theta}(\cdot\mid s_{t},f)\right)\boldsymbol{\nabla}_{\!\!\theta}\,\log\Pi_{\theta}(s_{t})\right].
(8)
The additional term of the sequence-level gradient captures how prefixes influence the self-distillation divergence of future tokens.
We also experimented with this sequence-level gradient estimator but did not find measurable gains relative to its additional complexity.
A.2 Regularized teacher
In contrast to standard distillation, the teacher in SDPO changes throughout training. This bootstrapping enables the teacher to improve, but it may also lead to training instability.
To stabilize training, we seek to prevent the teacher qq from quickly diverging from the initial teacher qθref\smash{q_{\theta_{{\mathrm{ref}}}}}.
We can achieve this by placing an explicit trust-region constraint on qq (Schulman et al., 2015; Peng et al., 2019), that is:
∑tKL(q(yt∣x,f,y<t)∥qθref(yt∣x,f,y<t))≤ϵ,ϵ>0.\sum_{t}\mathrm{KL}\left(q(y_{t}\mid x,f,y_{<t})\|q_{\theta_{{\mathrm{ref}}}}(y_{t}\mid x,f,y_{<t})\right)\leq\epsilon,\quad\epsilon>0.
(9)
This trust-region can be implemented in two ways:
1.
Explicit trust-region: We can define the teacher as the policy closest to qθq_{\theta} while satisfying the trust-region constraint.
This teacher can be expressed as
q⁡(yt∣x,f,y<t)∝exp⁡((1−α)​log⁡qθref​(yt∣x,f,y<t)+α​log⁡qθ​(yt∣x,f,y<t)),q(y_{t}\mid x,f,y_{<t})\propto\exp\!\big((1-\alpha)\log q_{\theta_{{\mathrm{ref}}}}(y_{t}\mid x,f,y_{<t})+\alpha\log q_{\theta}(y_{t}\mid x,f,y_{<t})\big),
(10)
with α∈(0,1)\alpha\in(0,1) the inverse Lagrange multiplier for the trust-region constraint.
We include a full derivation in Section B.2.
We can plug this explicitly constrained teacher directly into the SDPO objective.
2.
Exponential moving average (EMA): Alternatively, we can stabilize the teacher’s parameters directly; parameterizing qθ′q_{\theta^{\prime}} by θ′\theta^{\prime} and updating as θ′←(1−α)​θ′+α​θ\theta^{\prime}\leftarrow(1-\alpha)\theta^{\prime}+\alpha\theta with α∈(0,1)\alpha\in(0,1).
Note that each implementation has a different practical advantage:
The EMA teacher requires additional GPU memory for θ′\theta^{\prime} yet does not introduce any runtime overhead.
In contrast, the trust-region teacher requires an additional log-prob computation with qθref\smash{q_{\theta_{{\mathrm{ref}}}}} yet does not require additional GPU memory if θref{\theta_{{\mathrm{ref}}}} is used for explicit KL regularization.
A.3 Approximate Logit Distillation
To save GPU memory, we perform distillation only on the top-KK tokens predicted by the student:
ℒSDPO​(θ)\displaystyle\mathcal{L}_{\mathrm{SDPO}}(\theta)
=∑t=1TKL(πθ(⋅∣x,y<t)∥stopgrad(qθ(⋅∣x,f,y<t)))\displaystyle=\sum_{t=1}^{T}\mathrm{KL}(\pi_{\theta}(\cdot\mid x,y_{<t})\|\mathrm{stopgrad}(q_{\theta}(\cdot\mid x,f,y_{<t})))
≈∑t=1T∑y^t∈topK​(πθ)πθ​(y^t∣x,y<t)⋅log⁡πθ​(y^t∣x,y<t)stopgrad⁡(qθ​(y^t∣x,f,y<t))+(1−∑y^t∈topK​(πθ)πθ​(y^t∣x,y<t))⋅log⁡1−∑y^t∈topK​(πθ)πθ​(y^t∣x,y<t)stopgrad⁡(1−∑y^t∈topK​(πθ)qθ​(y^t∣x,f,y<t))⏟tail\displaystyle\approx\begin{multlined}\sum_{t=1}^{T}\sum_{\hat{y}_{t}\in\mathrm{top}_{K}(\pi_{\theta})}\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})\cdot\log\frac{\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})}{\mathrm{stopgrad}(q_{\theta}(\hat{y}_{t}\mid x,f,y_{<t}))}\\
+\underbrace{\Big(1-\textstyle\sum_{\hat{y}_{t}\in\mathrm{top}_{K}(\pi_{\theta})}\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})\Big)\cdot\log\frac{1-\textstyle\sum_{\hat{y}_{t}\in\mathrm{top}_{K}(\pi_{\theta})}\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})}{\mathrm{stopgrad}\Big(1-\textstyle\sum_{\hat{y}_{t}\in\mathrm{top}_{K}(\pi_{\theta})}q_{\theta}(\hat{y}_{t}\mid x,f,y_{<t})\Big)}}_{\text{tail}}\end{multlined}
Here, the top-KK is with respect to student.
Without top-KK distillation, we would have to keep two copies of logits in memory: one for teacher and student each.
Top-KK distillation avoids virtually any memory overhead without impacting performance significantly, since most tokens of the vocabulary are not informative at a given time.
A.4 Off-Policy Training: Generalization to Logit-Level Losses
PPO-style clipping (Schulman et al., 2017) with truncated importance sampling (Yao et al., 2025), clip-higher (Yu et al., 2025), fixed length normalization (Liu et al., 2025b):
ℒtoken(θ):=−1∑i=1G|yi|∑i=1G∑t=1|yi|min(wi,tTIS,ρ)min(wi,tAi,t,clip(wi,t,1−εlow,1+εhigh)Ai,t),\mathcal{L}_{\mathrm{token}}(\theta):=-{\color[rgb]{1,0.5,0}\frac{1}{\sum_{i=1}^{G}|y_{i}|}}\sum_{i=1}^{G}\sum_{t=1}^{|y_{i}|}{\color[rgb]{0.9492,0.3281,0.3555}\min\left(w^{\mathrm{TIS}}_{i,t},\rho\right)}\min\left(w_{i,t}A_{i,t},\text{clip}(w_{i,t},1-\varepsilon_{\text{low}},1+{\color[rgb]{0.3477,0.7344,0.168}\varepsilon_{\text{high}}})A_{i,t}\right),
(13)
with wi,t:=πθ​(yi,t∣x,yi,<t)πθold​(yi,t∣x,yi,<t)w_{i,t}:=\frac{\pi_{\theta}(y_{i,t}\mid x,y_{i,<t})}{\pi_{\theta_{{\mathrm{old}}}}(y_{i,t}\mid x,y_{i,<t})}, wi,tTIS:=πθold​(yi,t∣x,yi,<t)πθoldrollout​(yi,t∣x,yi,<t)w^{\mathrm{TIS}}_{i,t}:=\frac{\pi_{\theta_{{\mathrm{old}}}}(y_{i,t}\mid x,y_{i,<t})}{\pi_{\theta_{{\mathrm{old}}}}^{\mathrm{rollout}}(y_{i,t}\mid x,y_{i,<t})}, and Ai,tA_{i,t} denotes the per-token advantage.
We extend this to a logit-level loss:
ℒlogit(θ):=−1∑i=1G|yi|∑i=1G∑t=1|yi|∑y^i,tmin(πθold(y^i,t∣x,yi,<t),ρπθoldrollout(y^i,t∣x,yi,<t))min⁡(wi,t​(y^i,t)​Ai,t​(y^i,t),clip​(wi,t​(y^i,t),1−εlow,1+εhigh)​Ai,t​(y^i,t)),\begin{multlined}\mathcal{L}_{\mathrm{logit}}(\theta):=-{\color[rgb]{1,0.5,0}\frac{1}{\sum_{i=1}^{G}|y_{i}|}}\sum_{i=1}^{G}\sum_{t=1}^{|y_{i}|}{\color[rgb]{0.168,0.3125,0.668}\sum_{\hat{y}_{i,t}}}\ {\color[rgb]{0.9492,0.3281,0.3555}\min\left(\pi_{\theta_{{\mathrm{old}}}}(\hat{y}_{i,t}\mid x,y_{i,<t}),\rho\pi_{\theta_{{\mathrm{old}}}}^{\mathrm{rollout}}(\hat{y}_{i,t}\mid x,y_{i,<t})\right)}\\
\min\left(w_{i,t}(\hat{y}_{i,t})A_{i,t}(\hat{y}_{i,t}),\text{clip}(w_{i,t}(\hat{y}_{i,t}),1-\varepsilon_{\text{low}},1+{\color[rgb]{0.3477,0.7344,0.168}\varepsilon_{\text{high}}})A_{i,t}(\hat{y}_{i,t})\right),\end{multlined}
(14)
where y^i,t\hat{y}_{i,t} sums over all possible tokens at position tt for rollout ii (or the KK most likely under πθold\pi_{\theta_{{\mathrm{old}}}}, cf. Section A.3).
The TIS changes since we explicitly weight each logit by its probability under πθold\pi_{\theta_{{\mathrm{old}}}} rather than relying on a Monte Carlo estimate of the expectation over next-token predictions.
Here, Ai,t​(y^i,t)A_{i,t}(\hat{y}_{i,t}) is a per-logit advantage.
In our experiments for SDPO, we apply the TIS term on a token-level rather than logit-level.
Appendix