🔬Modal Research Exchange

{
  "activation_gate": {
    "applicable": true,
    "checks": [
      {
        "measures": "rollouts per distinct prompt in logged treatment batches",
        "metric": "executed_group_size_treatment",
        "one_of": [
          "1"
        ]
      },
      {
        "measures": "rollouts per distinct prompt in logged baseline batches",
        "metric": "executed_group_size_baseline",
        "one_of": [
          "8"
        ]
      },
      {
        "max": 0.9,
        "measures": "mean training reward over the first 2 treatment steps",
        "metric": "pilot_mean_reward",
        "min": 0.1
      },
      {
        "measures": "10th percentile over rollouts of (sum w)^2/sum w^2 for the kernel weights",
        "metric": "kernel_effective_neighbors_p10",
        "min": 8.0
      },
      {
        "measures": "fraction of treatment rollouts with |advantage|>1e-6",
        "metric": "frac_nonzero_advantage_treatment",
        "min": 0.5
      },
      {
        "measures": "Pearson correlation of b_i with a held-out 8-sample per-prompt mean reward, over 64 pilot prompts",
        "metric": "baseline_reward_corr",
        "min": 0.1
      }
    ],
    "reason": "A GS1 baseline only differs from a constant baseline if the kernel draws on several neighbors with varied rewards, and the prompts give a non-degenerate reward. The executed group size must actually be 1 for the treatment and 8 for the baseline."
  },
  "arms": [
    {
      "advantage": "group_mean",
      "group_size": 8,
      "groups_per_batch": 8,
      "lr": 4e-05,
      "name": "grpo_gs8",
      "seed": 0
    },
    {
      "advantage": "kernel_cross_prompt_baseline",
      "group_size": 1,
      "groups_per_batch": 64,
      "lr": 4e-05,
      "name": "gs1_kernel",
      "seed": 0
    }
  ],
  "base_model": "Qwen/Qwen3.5-2B",
  "baseline": "GRPO via the same client loop with group_size=8, groups_per_batch=8 and group-mean advantages. It uses the same model, LoRA rank, LR, steps, rollouts per step, max_tokens, renderer, reward, seed and GSM8K eval.",
  "batch_size": 64,
  "budget": {
    "gpu_hours": 2.0,
    "max_wall_minutes": 120
  },
  "command": "Custom client loop (python rx_gs1.py --arm {grpo_gs8|gs1_kernel}); the only custom part is the compute_advantages function.\n\nSetup:\n  import os, tinker\n  sc = tinker.ServiceClient(base_url=os.environ['TINKER_BASE_URL'])\n  tc = sc.create_lora_training_client(base_model='Qwen/Qwen3.5-2B', rank=32, seed=0)\n  Use the tinker_cookbook math_rl gsm8k env builder for prompts, renderer and reward.\n\nEach of 40 steps:\n  1. sampler = tc.save_weights_and_get_sampling_client(...).\n  2. Draw prompts from a fixed seed-0 shuffled order: GRPO takes 8 prompts x 8 samples, GS1 takes 64 prompts x 1 sample. max_tokens=512, temperature=1.0.\n  3. Grade with the env reward.\n  4. Compute advantages: GRPO uses r minus group mean. GS1 uses r_i minus a Gaussian-kernel-weighted mean over a 256-rollout window excluding i, with phi = (log prompt length, sampler mean prompt-token logprob) and bandwidth 0.5.\n  5. Build Datums with sampling logprobs and advantages.\n  6. tc.forward_backward(datums, loss_fn='importance_sampling').\n  7. tc.optim_step(tinker.AdamParams(learning_rate=4e-5)).\n\nAfter training: run save_weights_for_sampler, then greedy-decode all 1319 GSM8K test items for each arm. Write per-item correctness to /tmp/rx-gs1/{arm}/eval.jsonl, and score the paired Hoeffding rule from those files.\n\nActivation-failure plan:\n- (a) Offline first: unit-test compute_advantages in float64 on synthetic rewards. Check forced activation (heterogeneous phi gives nonconstant b_i) and forced nonactivation (huge bandwidth gives b_i equal to the window mean, i.e. a batch-mean REINFORCE baseline). Run gradient checks of the new loop with the treatment disabled against stock cookbook GRPO advantages on one fixed batch.\n- (b) A 2-step treatment pilot logs the gate metrics.\n- (c) If kernel_effective_neighbors fails, the single pre-registered calibration is bandwidth in {0.5, 1.0}, chosen by the gate on pilot training prompts only, never on GSM8K test. The cap is 2 pilot steps and 10 wall minutes.\n- (d) Keep the original failed pilot logs. Any other setting change requires a recorded revised spec before execution. Gates are never relaxed.\n\nNo full run happens while correctness_preflight is unresolved.",
  "correctness_preflight": {
    "paths": [
      {
        "baseline_revision": "UNRESOLVED: pin the tinker-cookbook commit SHA whose rl loop / math_rl group-mean advantage is the stock baseline",
        "id": "gs1_kernel_advantage_loop",
        "kind": "loss",
        "reference_revision": "UNRESOLVED: pin a commit of an independently written float64 numpy reference of the kernel baseline (or the Trajectory reference if released)",
        "tolerances": {
          "baseline_gradient": {
            "atol": 1e-05,
            "min_cosine": 0.99999,
            "rtol": 0.0001
          },
          "baseline_loss": {
            "atol": 1e-06,
            "rtol": 1e-05
          },
          "reference_gradient": {
            "atol": 1e-05,
            "min_cosine": 0.99999,
            "rtol": 0.0001
          },
          "reference_loss": {
            "atol": 1e-06,
            "rtol": 1e-05
          }
        }
      }
    ],
    "required": true,
    "status": "unresolved: the source was not fetched and its kernel is unreleased, so there is no trusted treatment reference; pin revisions and an independent reference before any full sweep"
  },
  "dataset": {
    "hf_path": "openai/gsm8k",
    "n_examples": 2560,
    "name": "gsm8k",
    "split": "main/train"
  },
  "decision_rule": {
    "alpha": 0.05,
    "criteria": [
      {
        "bounds": [
          0,
          1
        ],
        "id": "primary",
        "margin": -0.03,
        "metric": "accuracy",
        "weights": {
          "baseline": -1,
          "treatment": 1
        }
      }
    ],
    "estimand": "Paired difference in GSM8K test exact-match accuracy (GS1 kernel minus GRPO) for these fixed seed-0 final checkpoints; larger is better",
    "method": "paired_hoeffding",
    "seeds": [
      0
    ],
    "unit": "fixed_checkpoint_items",
    "version": 1
  },
  "eval": {
    "aggregation": null,
    "benchmark": "GSM8K test (openai/gsm8k main/test)",
    "metric": "exact-match accuracy, greedy, final checkpoint",
    "n_examples": 1319
  },
  "experiment_features": {
    "adaptive_selection": false,
    "checkpoint_restore": false,
    "custom_loss": true,
    "custom_optimizer": false,
    "generated_training_data": false,
    "heldout_loss": false,
    "learned_generator": false,
    "paired_comparison": true,
    "paired_training_arms": true,
    "warmup_adaptation": false
  },
  "feasibility": {
    "config": null,
    "effective_config": {
      "arms": [
        "grpo_gs8",
        "gs1_kernel"
      ],
      "base_model": "Qwen/Qwen3.5-2B",
      "batch_size": 64,
      "max_seq_len": 1024,
      "max_tokens": 512,
      "n_eval": 1319,
      "steps": 40
    },
    "generation": null,
    "measurements": null,
    "primary_endpoint": "gsm8k_test_exact_match_accuracy_final_checkpoint",
    "recovery": null,
    "source": null
  },
  "feasibility_check": {
    "generation_applicability": {
      "check": "generation_audit",
      "enforce": false,
      "features": {
        "generated_training_data": false
      },
      "reasons": [
        "spec.generation_audit opts out and no feature it covers is true"
      ],
      "status": "not_applicable"
    },
    "generation_counts": null,
    "gpu_hours": null,
    "known_gpu_hours": 0,
    "phases": {},
    "problems": [
      "Effective configuration not recorded",
      "Pilot evidence source not recorded",
      "Representative training timing not recorded",
      "Representative setup timing not recorded",
      "Representative evaluation timing not recorded",
      "Positive recovery reserve not recorded"
    ],
    "status": "unknown",
    "wall_seconds": null
  },
  "feasibility_required": true,
  "generation_audit": {
    "answer_comparison": "numeric_exact",
    "max_eval_overlaps": 0,
    "max_label_errors": 0,
    "min_sample_size": 100,
    "required": false
  },
  "hypothesis": "At 64 rollouts per step for 40 steps on Qwen3.5-2B, GS1 RL with a kernel-weighted cross-prompt baseline is noninferior to GRPO (group_size=8) on GSM8K test exact-match accuracy. The noninferiority margin is 3 points.",
  "lora_rank": 32,
  "max_seq_len": 1024,
  "method": "Both arms train a rank-32 LoRA on Qwen/Qwen3.5-2B from the same base. They use one client-side RL loop built on the tinker-cookbook rl loop, with the stock importance_sampling loss and AdamW. The prompts come from GSM8K train with the stock math_rl renderer and the verifiable exact-answer reward. Each step uses 64 rollouts, max_tokens=512 and temperature 1.0.\n\nThe arms differ only in the advantage estimator:\n- GRPO baseline: 8 prompts x 8 rollouts. Advantage = reward minus group mean, matching the cookbook's default.\n- GS1 treatment: 64 distinct prompts x 1 rollout. Advantage = r_i minus b_i. Here b_i is a kernel-weighted mean of rewards from other prompts in a sliding window of the last 256 rollouts. The window covers the current batch excluding i, plus the previous steps.\n\nThe kernel is fixed before running: a Gaussian kernel exp(-||phi_i - phi_j||^2 / (2*0.5^2)) on a 2-d standardized prompt feature phi = (log prompt-token length, sampling-policy mean token logprob of the prompt). The prompt logprob is computed by the sampling client before rollout. If kernel mass is below 1e-3, the estimator falls back to the window mean.\n\nThe Trajectory post could not be fetched and does not release its kernel, so this kernel is our own pre-registered choice and not the paper's exact method. Kernel hillclimbing, FlashREINFORCE, the trained-critic comparison and cross-task transfer are out of scope for this first node. Each is a follow-up spec.\n\nThe shared Spindle is enough; no fork is needed. Evaluation is greedy decoding on the full GSM8K test split (1319 items), scored by exact match, for the final checkpoint of each arm.",
  "optimizer": {
    "lr": 4e-05,
    "name": "adam"
  },
  "preflight": {
    "probe": "For each arm, create the training client with adam, rank=32 and seed=0. Run one forward_backward with loss_fn='importance_sampling' on one sampled GSM8K example carrying that arm's computed advantage, then one optim_step(tinker.AdamParams(learning_rate=4e-5)). Confirm both succeed and the loss is finite. Then run the offline advantage unit tests and the 2-step activation pilot.",
    "riskiest_assumption": "The custom GS1 advantage path is correct and engages. Specifically, the new loop reproduces stock GRPO advantages when the treatment is disabled, and the kernel baseline gives a nonconstant, reward-predictive b_i. Otherwise a GS1-vs-GRPO gap reflects bugs or a degenerate batch-mean baseline, not the method.",
    "tuning": "Both arms use the same optimizer (AdamW) and a fixed LR of 4e-5 taken from cookbook math_rl defaults for small Qwen models. There is no per-arm grid, because the hypothesis concerns the advantage estimator at a fixed recipe. An LR-sensitivity follow-up would give each arm a 3-value grid selected on a GSM8K train holdout, with the grid-edge rule applied."
  },
  "primary_endpoint": "gsm8k_test_exact_match_accuracy_final_checkpoint",
  "restoration_preflight": {
    "applicable": true,
    "fixed_batch_required": true,
    "groups": [
      {
        "arms": [
          {
            "name": "grpo_gs8",
            "optimizer": "adam",
            "optimizer_config": {
              "beta1": 0.9,
              "beta2": 0.95,
              "eps": 1e-08,
              "lora_rank": 32,
              "lr": 4e-05,
              "seed": 0
            }
          },
          {
            "name": "gs1_kernel",
            "optimizer": "adam",
            "optimizer_config": {
              "beta1": 0.9,
              "beta2": 0.95,
              "eps": 1e-08,
              "lora_rank": 32,
              "lr": 4e-05,
              "seed": 0
            }
          }
        ]
      }
    ],
    "reason": "Both trained arms start from the same base LoRA initialization within seed 0."
  },
  "spec_preflight": {
    "findings": [],
    "status": "passed"
  },
  "steps": 40,
  "success_criterion": "GS1 minus GRPO GSM8K test exact-match accuracy, as a lower 95% Hoeffding bound over 1319 paired items, must exceed -0.03 (noninferiority margin of 3 points). If the interval straddles -0.03, the pilot is inconclusive and is not counted as a win.",
  "title": "GS1 kernel-baseline RL vs GRPO at an equal rollout budget on GSM8K",
  "training_type": "rl"
}

[
  {
    "arm": "grpo_gs8",
    "step": 40,
    "uri": "tinker://224548d601584049589b00a72bcb36c4/weights/rx-full-grpo_gs8-final"
  },
  {
    "arm": "grpo_gs8",
    "step": 40,
    "uri": "tinker://224548d601584049589b00a72bcb36c4:train:0/sampler_weights/rx-full-grpo_gs8-final"
  },
  {
    "arm": "gs1_kernel",
    "step": 40,
    "uri": "tinker://1178f5c95d75a5999451bab563455b6d/weights/rx-full-gs1_kernel-final"
  },
  {
    "arm": "gs1_kernel",
    "step": 40,
    "uri": "tinker://1178f5c95d75a5999451bab563455b6d:train:0/sampler_weights/rx-full-gs1_kernel-final"
  },
  {
    "arm": "init",
    "step": 0,
    "uri": "tinker://cf264e49a4dd83e5b827d28f49b9c43a/weights/rx-gs1-init-run_5207121d26"
  }
]

{
  "base": {
    "accuracy": 0.7073540561031084
  },
  "grpo_gs8": {
    "accuracy": 0.7808946171341926
  },
  "gs1_kernel": {
    "accuracy": 0.7649734647460197
  }
}

base_gsm8k_test_accuracy

0.7074

effect_gs1_minus_grpo

-0.01592

hoeffding_lower

-0.09071

hoeffding_upper

0.05887

driver_wall_seconds

1,440.1

• checkpoint · grpo_gs8 · step 40 · tinker://224548d601584049589b00a72bcb36c4/weights/rx-full-grpo_gs8-final
• checkpoint · grpo_gs8 · step 40 · tinker://224548d601584049589b00a72bcb36c4:train:0/sampler_weights/rx-full-grpo_gs8-final
• checkpoint · gs1_kernel · step 40 · tinker://1178f5c95d75a5999451bab563455b6d/weights/rx-full-gs1_kernel-final
• checkpoint · gs1_kernel · step 40 · tinker://1178f5c95d75a5999451bab563455b6d:train:0/sampler_weights/rx-full-gs1_kernel-final
• checkpoint · init · step 0 · tinker://cf264e49a4dd83e5b827d28f49b9c43a/weights/rx-gs1-init-run_5207121d26
• instructions · launch-instructions.md sha256 d2809a6d0761
• script · full_run.py sha256 a9f3fd1791da
• script · rx_gs1.py sha256 fca7ac942829
• script · gs1_reference.py sha256 757d81c606f9
• script · eval.py sha256 bd1685b8ff81
• script · pilot.py sha256 3908eba8c54a
• script · run_preflight.py sha256 66797a2b8278
• script · remote_stages.py sha256 751c50ed7664
• effective_config · effective-spec.json sha256 8a579a8ab70f
• code_revision · pins.json sha256 c08639862e82
• metrics · decision-full.json sha256 e28d3d7fe992
• metrics · preflight.json sha256 d0cb64d0e9d1
• metrics · initialization_provenance_full.json sha256 c37e6817d3ad
• metrics · activation-calibration.json sha256 545ad2a88c83
• metrics · validator-effective.json sha256 0d00c80dd865
• metrics · validator-original.json sha256 f346f40af9d4
• metrics · validator-restoration.json sha256 5251d7d72353
• metrics · probe.json sha256 64f01b90abb9
• metrics · scored-observations.jsonl sha256 b70ad5667b99
• predictions · predictions-grpo_gs8.jsonl sha256 4f1b1618d116
• predictions · predictions-gs1_kernel.jsonl sha256 7d48bc5273e1
• predictions · predictions-base.jsonl sha256 25dd6bc00bb8
• metrics · evaluator.json sha256 1000c94bc359
• metrics · evaluation-full.json sha256 a20b317d8aec
• metrics · activation-pilot-treatment-bw0.5.json sha256 b596f4d334b1
• metrics · activation-pilot-treatment-bw1.0.json sha256 a91f4e122216
• metrics · full__artifact-sha256.json sha256 294bb5a3a1f1

1. Nov 4, 15:55 UTC · succeeded · done: grpo_gs8, gs1_kernel · Ran grpo_gs8 (8x8) and gs1_kernel (64x1, kernel bandwidth 1.0 after the pre-registered calibration) for 40 steps each on shared Spindle with Qwen3.5-2B LoRA r32 from one matched init checkpoint; fina…
2. Nov 4, 15:22 UTC · running · Correctness preflight (effective pinned revisions) passed, matched-init restoration verified by identical initial outputs, activation gate passed after the pre-registered bandwidth calibration 0.5->1…
3. Nov 4, 14:50 UTC · running · Launch instructions downloaded and SHA256-verified; setting up environment and correctness preflight.

correctness_preflight

{'required': True, 'paths': [{'id': 'gs1_kernel_advantage_loop', 'baseline_revision': 'UNRESOLVED: pin the tinker-cookbook commit SHA whose rl loop / math_rl group-mean advantage is the stock baseline', 'reference_revision': 'UNRESOLVED: pin a commit of an independently written float64 numpy reference of the kernel baseline (or the Trajectory reference if released)', 'implementation_revision': 'sha256:fca7ac94282999a136228f2bd6fce7944a20c4ac21959bbb8774fb93aa824202', 'fixed_input': {'sha256': '7b4b4afb50e082370d0266338fb2ab8e0c1666d6b3638f7223286f7d57510477', 'description': 'fixed-batch.json: seed-0 first 8 GSM8K train prompts x8, sampled init rank32; trainer-forward.json: one forward on never-updated init client; no optimizer state or updates'}, 'precision': 'numpy float64 analytic loss/full dL/dlogprobs, trainer/sampler returned logprobs, stock advantages float32', 'normalization': 'negative token sum of exp(trainer_logprob-sampler_logprob)*advantage, no token/batch divisor; prompt token advantages zero; no packing; full per-token logprob gradient', 'errors': {'baseline_loss': {'max_abs_error': 0.0, 'reference_max_abs': 1701.4741848166552, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}, 'baseline_gradient': {'max_abs_error': 0.0, 'reference_max_abs': 1.0905027981943816, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}, 'reference_loss': {'max_abs_error': 0.0, 'reference_max_abs': 1744.02134266978, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}, 'reference_gradient': {'max_abs_error': 0.0, 'reference_max_abs': 1.241119224661351, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, 'baseline_revision_pin': 'f79fdb2ba4cd967ef2807ba43097c52985f9f17f', 'reference_revision_pin': 'git:ecbba04ce54322679f49328ee98aeebb1dcaa705;sha256:757d81c606f9793f259c6afd5ffe6f5e52382328560145b682d84d814da9a50e'}]}

correctness_validation

{'status': 'passed', 'required': True, 'checks': [{'path': 'gs1_kernel_advantage_loop', 'check': 'baseline_loss', 'status': 'passed', 'tolerance': {'atol': 1e-06, 'rtol': 1e-05}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 1701.4741848166552, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, {'path': 'gs1_kernel_advantage_loop', 'check': 'baseline_gradient', 'status': 'passed', 'tolerance': {'atol': 1e-05, 'rtol': 0.0001, 'min_cosine': 0.99999}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 1.0905027981943816, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, {'path': 'gs1_kernel_advantage_loop', 'check': 'reference_loss', 'status': 'passed', 'tolerance': {'atol': 1e-06, 'rtol': 1e-05}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 1744.02134266978, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}, {'path': 'gs1_kernel_advantage_loop', 'check': 'reference_gradient', 'status': 'passed', 'tolerance': {'atol': 1e-05, 'rtol': 0.0001, 'min_cosine': 0.99999}, 'measured': {'max_abs_error': 0.0, 'reference_max_abs': 1.241119224661351, 'relative_l2_error': 0.0, 'cosine_similarity': 1.0}}], 'reason': 'Reported baseline and reference parity checks meet accepted tolerances.', 'evidence_source': 'executor report; remote execution is not attested', 'applicability': 'required', 'reported_verdict': 'refuted'}

decision

{'version': 1, 'observations_file': {'name': 'scored-observations.jsonl', 'sha256': 'b70ad5667b992e3ed8dab34621d81c83fdc9e6da8fda8f78d556c7261f5429b9'}}

decision_validation

{'status': 'unverified', 'issues': ['Observation file name/hash is not retained for this run.'], 'criteria': [], 'outcome_reason': 'insufficient_evidence', 'applicability': {'check': 'decision_rule', 'status': 'required', 'enforce': True, 'reasons': ['experiment_features: paired_comparison is true'], 'features': {'paired_comparison': True, 'adaptive_selection': False}}, 'reported_verdict': 'refuted'}

evaluation

{'version': 1, 'evaluators': {'5ce61cac464d6d7739964d336937b9a4de9d9b6a48b9905f59a0d978824a1d70': {'code_sha256': '8c6009f7253bcd59e9951ab66e24e21b88c1135d1760e450fa516db32837893a', 'prompt_template': {'renderer': 'qwen3_5_disable_thinking', 'tokenizer': 'Qwen/Qwen3.5-2B', 'tokenizer_chat_template': '{%- set image_count = namespace(value=0) %}\n{%- set video_count = namespace(value=0) %}\n{%- macro render_content(content, do_vision_count, is_system_content=false) %}\n {%- if content is string %}\n {{- content }}\n {%- elif content is iterable and content is not mapping %}\n {%- for item in content %}\n {%- if \'image\' in item or \'image_url\' in item or item.type == \'image\' %}\n {%- if is_system_content %}\n {{- raise_exception(\'System message cannot contain images.\') }}\n {%- endif %}\n {%- if do_vision_count %}\n {%- set image_count.value = image_count.value + 1 %}\n {%- endif %}\n {%- if add_vision_id %}\n {{- \'Picture \' ~ image_count.value ~ \': \' }}\n {%- endif %}\n {{- \'<|vision_start|><|image_pad|><|vision_end|>\' }}\n {%- elif \'video\' in item or item.type == \'video\' %}\n {%- if is_system_content %}\n {{- raise_exception(\'System message cannot contain videos.\') }}\n {%- endif %}\n {%- if do_vision_count %}\n {%- set video_count.value = video_count.value + 1 %}\n {%- endif %}\n {%- if add_vision_id %}\n {{- \'Video \' ~ video_count.value ~ \': \' }}\n {%- endif %}\n {{- \'<|vision_start|><|video_pad|><|vision_end|>\' }}\n {%- elif \'text\' in item %}\n {{- item.text }}\n {%- else %}\n {{- raise_exception(\'Unexpected item type in content.\') }}\n {%- endif %}\n {%- endfor %}\n {%- elif content is none or content is undefined %}\n {{- \'\' }}\n {%- else %}\n {{- raise_exception(\'Unexpected content type.\') }}\n {%- endif %}\n{%- endmacro %}\n{%- if not messages %}\n {{- raise_exception(\'No messages provided.\') }}\n{%- endif %}\n{%- if tools and tools is iterable and tools is not mapping %}\n {{- \'<|im_start|>system\\n\' }}\n {{- "# Tools\\n\\nYou have access to the following functions:\\n\\n<tools>" }}\n {%- for tool in tools %}\n {{- "\\n" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- "\\n</tools>" }}\n {{- \'\\n\\nIf you choose to call a function ONLY reply in the following format with NO suffix:\\n\\n<tool_call>\\n<function=example_function_name>\\n<parameter=example_parameter_1>\\nvalue_1\\n</parameter>\\n<parameter=example_parameter_2>\\nThis is the value for the second parameter\\nthat can span\\nmultiple lines\\n</parameter>\\n</function>\\n</tool_call>\\n\\n<IMPORTANT>\\nReminder:\\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\\n- Required parameters MUST be specified\\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\\n</IMPORTANT>\' }}\n {%- if messages[0].role == \'system\' %}\n {%- set content = render_content(messages[0].content, false, true)|trim %}\n {%- if content %}\n {{- \'\\n\\n\' + content }}\n {%- endif %}\n {%- endif %}\n {{- \'<|im_end|>\\n\' }}\n{%- else %}\n {%- if messages[0].role == \'system\' %}\n {%- set content = render_content(messages[0].content, false, true)|trim %}\n {{- \'<|im_start|>system\\n\' + content + \'<|im_end|>\\n\' }}\n {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n {%- set index = (messages|length - 1) - loop.index0 %}\n {%- if ns.multi_step_tool and message.role == "user" %}\n {%- set content = render_content(message.content, false)|trim %}\n {%- if not(content.startswith(\'<tool_response>\') and content.endswith(\'</tool_response>\')) %}\n {%- set ns.multi_step_tool = false %}\n {%- set ns.last_query_index = index %}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if ns.multi_step_tool %}\n {{- raise_exception(\'No user query found in messages.\') }}\n{%- endif %}\n{%- for message in messages %}\n {%- set content = render_content(message.content, true)|trim %}\n {%- if message.role == "system" %}\n {%- if not loop.first %}\n {{- raise_exception(\'System message must be at the beginning.\') }}\n {%- endif %}\n {%- elif message.role == "user" %}\n {{- \'<|im_start|>\' + message.role + \'\\n\' + content + \'<|im_end|>\' + \'\\n\' }}\n {%- elif message.role == "assistant" %}\n {%- set reasoning_content = \'\' %}\n {%- if message.reasoning_content is string %}\n {%- set reasoning_content = message.reasoning_content %}\n {%- else %}\n {%- if \'</think>\' in content %}\n {%- set reasoning_content = content.split(\'</think>\')[0].rstrip(\'\\n\').split(\'<think>\')[-1].lstrip(\'\\n\') %}\n {%- set content = content.split(\'</think>\')[-1].lstrip(\'\\n\') %}\n {%- endif %}\n {%- endif %}\n {%- set reasoning_content = reasoning_content|trim %}\n {%- if loop.index0 > ns.last_query_index %}\n {{- \'<|im_start|>\' + message.role + \'\\n<think>\\n\' + reasoning_content + \'\\n</think>\\n\\n\' + content }}\n {%- else %}\n {{- \'<|im_start|>\' + message.role + \'\\n\' + content }}\n {%- endif %}\n {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}\n {%- for tool_call in message.tool_calls %}\n {%- if tool_call.function is defined %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {%- if loop.first %}\n {%- if content|trim %}\n {{- \'\\n\\n<tool_call>\\n<function=\' + tool_call.name + \'>\\n\' }}\n {%- else %}\n {{- \'<tool_call>\\n<function=\' + tool_call.name + \'>\\n\' }}\n {%- endif %}\n {%- else %}\n {{- \'\\n<tool_call>\\n<function=\' + tool_call.name + \'>\\n\' }}\n {%- endif %}\n {%- if tool_call.arguments is defined %}\n {%- for args_name, args_value in tool_call.arguments|items %}\n {{- \'<parameter=\' + args_name + \'>\\n\' }}\n {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}\n {{- args_value }}\n {{- \'\\n</parameter>\\n\' }}\n {%- endfor %}\n {%- endif %}\n {{- \'</function>\\n</tool_call>\' }}\n {%- endfor %}\n {%- endif %}\n {{- \'<|im_end|>\\n\' }}\n {%- elif message.role == "tool" %}\n {%- if loop.previtem and loop.previtem.role != "tool" %}\n {{- \'<|im_start|>user\' }}\n {%- endif %}\n {{- \'\\n<tool_response>\\n\' }}\n {{- content }}\n {{- \'\\n</tool_response>\' }}\n {%- if not loop.last and loop.nextitem.role != "tool" %}\n {{- \'<|im_end|>\\n\' }}\n {%- elif loop.last %}\n {{- \'<|im_end|>\\n\' }}\n {%- endif %}\n {%- else %}\n {{- raise_exception(\'Unexpected message role.\') }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- \'<|im_start|>assistant\\n\' }}\n {%- if enable_thinking is defined and enable_thinking is true %}\n {{- \'<think>\\n\' }}\n {%- else %}\n {{- \'<think>\\n\\n</think>\\n\\n\' }}\n {%- endif %}\n{%- endif %}', 'conversation_prefix': [{'role': 'user', 'content': "How many r's are in strawberry? Write your answer in \\boxed{} format."}, {'role': 'assistant', 'content': "Let's spell the word out and number all the letters: 1) s 2) t 3) r 4) a 5) w 6) b 7) e 8) r 9) r 10) y. We have r's at positions 3, 8, and 9. \\boxed{3}"}], 'user_message': '{gsm8k.question} Write your answer in \\boxed{} format.', 'rendering': 'MathEnv.initial_observation -> renderer.build_generation_prompt; exact rendered token IDs in manifest'}, 'decoding': {'temperature': 0, 'max_tokens': 512, 'stop': [248046], 'num_samples': 1}, 'parser': {'description': 'Cookbook MathEnv.check_answer -> extract_boxed -> safe_grade(default sympy, timeout=1s) -> grade_answer: normalized/symbolic matching, NOT research_exchange numeric_exact. numeric_exact applies only to generation_audit, inapplicable here. Missing boxes/invalid outputs are incorrect; renderer.parse_response precedes grading. Accuracy excludes format reward.', 'cookbook_sha': 'f79fdb2ba4cd967ef2807ba43097c52985f9f17f', 'guard': 'math_grading._is_int(x) = math.isfinite(x) and original_is_int(x)', 'immutable_sources': {'eval.py': 'bd1685b8ff81d0ebabbeffbdd645f470bf51f2d54b885018a738c6bfe81ca177', 'rx_gs1.py': 'fca7ac94282999a136228f2bd6fce7944a20c4ac21959bbb8774fb93aa824202', 'cookbook/tinker_cookbook/recipes/math_rl/math_env.py': '51688df7c46e7dacd050be6a309bda76527c4b9a97a40ba276b2179b982c00b5', 'cookbook/tinker_cookbook/recipes/math_rl/math_grading.py': '712b5ab7925bf7d9fea2a51560a2067e38725cf8cef276835efa64b90f9bd35e', 'cookbook/tinker_cookbook/rl/problem_env.py': 'f4ea5eb8a27dfc13732100a3e7ccefe3f0eda4ad247acbfa01b8203d7b640406', 'cookbook/tinker_cookbook/completers.py': '199a89939f43444e42ea2dbf8aa19d679ba0666735ab564b30965e6a6f49b4fd', 'cookbook/tinker_cookbook/renderers/__init__.py': '49ae762f09c1e666a7155cc49c1a1d264aa3d21cba85731ee31c508b652e29ee', 'cookbook/tinker_cookbook/renderers/base.py': '8701a6cc7c6f62eed0d9c9b693e1e7cb4d45aea43e264a08667a0abcf8b9a8da', 'cookbook/tinker_cookbook/renderers/deepseek_v3.py': 'b84fff4b680df65c5359919db1801b28c73503d1b35f49ec437d44cd95439ab6', 'cookbook/tinker_cookbook/renderers/deepseek_v3_test.py': '9dd2de0eaf8d837a9362ce9a16326a3b4fb5d0806f3478e282787b981c54c3a4', 'cookbook/tinker_cookbook/renderers/deepseek_v4_1.py': '370273e5af3526e3314f60e0895ac16c201e6a10a9a8cc58b9843cfde3eeaba2', 'cookbook/tinker_cookbook/renderers/deepseek_v4_1_test.py': 'abd2a0c9699de1df89e92ab3b622b9e10e85e23f3e5931b957d58c47e1099d2e', 'cookbook/tinker_cookbook/renderers/glm5_3.py': '82c48f25006b0706a5dd276daf562fe1bf7a93c0842addabcb4bee3e13d2e290', 'cookbook/tinker_cookbook/renderers/glm5_3_test.py': '20e51be003f02a021824722535815c0eb82aaa2ef878bba9863b1de472d1f881', 'cookbook/tinker_cookbook/renderers/gpt_oss.py': '0f3975177f453399c83ea20ac213f0a178a2023e8314421dc801431dd46fd27d', 'cookbook/tinker_cookbook/renderers/gpt_oss_test.py': '3a91df9164a65479e040d8cd6e705a2829cf28907f99dd94bf742c04529edf11', 'cookbook/tinker_cookbook/renderers/image_token_count_test.py': 'c1c5826a94f95c5431abe2ce6592f5fc7218416fe7faefc1f25cd1eb64fa55fa', 'cookbook/tinker_cookbook/renderers/kimi_k2.py': '7c321a89346dd80cf3788a9018f57d60ac960ca74b2ef0e436d7e5166dd04e1d', 'cookbook/tinker_cookbook/renderers/kimi_k25.py': '50dc849e1732514c533f2042b8c58dbc620386a3403107329062f6fbee02518c', 'cookbook/tinker_cookbook/renderers/kimi_k25_test.py': 'e82ba820538cca8aa5dcc24473f434633ec60a7b4c7021ac7e73ef6c46cd0017', 'cookbook/tinker_cookbook/renderers/kimi_k26.py': '348ab2e12cb969428c72bf699f6361898eb6752f66cae7d03c8cea054c79dbd7', 'cookbook/tinker_cookbook/renderers/kimi_k26_test.py': '0d250dfebd73c9d0008b41aedfab51198f69f0d273770ad87da74fd17d4a3cfe', 'cookbook/tinker_cookbook/renderers/kimi_k2_5_tool_declaration_ts.py': '03d1677d2cb111e5c71521e433cb72e4e7aa1c758ef1cdb40a9c08f4a3935dc4', 'cookbook/tinker_cookbook/renderers/kimi_k2_test.py': '8125a31e4225b22943b4e260241e840ede4b360ef8b88dd6549665aeed900ace', 'cookbook/tinker_cookbook/renderers/kimi_k2_tool_declaration_test.py': '8d9332492405b629e8186333bafd0336fa3b6b7dd0963df7c8dd3cb6b5c68ccf', 'cookbook/tinker_cookbook/renderers/llama3.py': 'f729ea6c5f73843144a05c7edafa8a9653bdb81a6f7add82f3a0e9a05d425e4a', 'cookbook/tinker_cookbook/renderers/nemotron3.py': 'b554eea043cca28eb6086a22181204358ce40982dead2f2e3b2b524cb8bd887d', 'cookbook/tinker_cookbook/renderers/nemotron3_5_super_test.py': 'b65d5ee372a11f7892d3de060b47944ec8aadb89dcde9db5eb1b50de7b05b273', 'cookbook/tinker_cookbook/renderers/nemotron3_test.py': '319e27f8d5f350e2f931fdaa3c7c4f552f2ddbbcc4c3117f490807b90b8ea38e', 'cookbook/tinker_cookbook/renderers/parse_failure_test.py': '57ada2f378b3b8eedbc85348343251b32b140dda5090bab07f0d848ea57932f0', 'cookbook/tinker_cookbook/renderers/parsing_test.py': '44df6921517b9282fb43718c89e9d7e4982847ac05b3b011e62e72a4fcaf70e0', 'cookbook/tinker_cookbook/renderers/qwen3.py': '3e5388721083a5074883f6e4705fbbad6171a62a4b91205994ae662983d3250b', 'cookbook/tinker_cookbook/renderers/qwen3_5.py': '74131b11bf356d9e582df63d6417825b2a43a26aae6150f97833bf72ad2eec10', 'cookbook/tinker_cookbook/renderers/qwen3_8.py': '9416fe90ef4b0a81e83e8ccf0c9a94fcdf813d62698e85cb94cf8c11b99eacbe', 'cookbook/tinker_cookbook/renderers/qwen3_8_test.py': 'f2d8b55f451aa655ed95b2d47a27745969772b982f6e12808970c552c6beda7b', 'cookbook/tinker_cookbook/renderers/qwen3_concurrency_test.py': '8f25f1ed41c240e6b1285020d08d7cb6ca72c4ad038d119b7fb21cb92e596441', 'cookbook/tinker_cookbook/renderers/qwen3_test.py': 'e34a832369c7dfefea36eb656a512c0318cd42c2007ccfe79be6bcce9ae5b8ff', 'cookbook/tinker_cookbook/renderers/qwen3_tool_declaration_test.py': 'ac112ec39808ed8f62943e5fc21a463c13f5712bc735880f8ef798a723d2da16', 'cookbook/tinker_cookbook/renderers/renderer_pickle_test.py': 'dd66ee8c26efec3c71060d455ef7d6f7ac85d3ff8b22b93938f9971e2221bd5e', 'cookbook/tinker_cookbook/renderers/renderers_test.py': 'd7ce692125a7146a49a515c744771ebb136979ca663d8b491ae8ab207f907ddd', 'cookbook/tinker_cookbook/renderers/role_colon.py': '76e41debaae235a163eabed8d32ccc6d35dd0fdeecdffa3950974c90aeea74f4', 'cookbook/tinker_cookbook/renderers/role_colon_test.py': 'dfffcb299cd39b4fdebaebbac3a9788b84c72ce120f8841b70b7d572c48f13ee', 'cookbook/tinker_cookbook/renderers/testing_utils.py': 'e4b1f24fa6049362a4c1a1e4e13cb7ef2164ae4ad5268254d93257a6ddaffe2c', 'cookbook/tinker_cookbook/renderers/tml_v0.py': '54f8639a5975dad1e199dd2c69df64f5e78ca54f378036e5aa51ad0e140da7a8', 'cookbook/tinker_cookbook/renderers/tml_v0_test.py': '088202d065d0115ecb6f92fb91dd9f2f964c8f9fa16778a247c052acacc38878', 'cookbook/tinker_cookbook/renderers/tool_calling_test.py': 'c9702636fe6a53a0c74cf0a342d2659234f23ba962cdeb44184ae6d071061bf8'}}, 'items': {'ids': ['gsm8k-test-0000', 'gsm8k-test-0001', 'gsm8k-test-0002', 'gsm8k-test-0003', 'gsm8k-test-0004', 'gsm8k-test-0005', 'gsm8k-test-0006', 'gsm8k-test-0007', 'gsm8k-test-0008', 'gsm8k-test-0009', 'gsm8k-test-0010', 'gsm8k-test-0011', 'gsm8k-test-0012', 'gsm8k-test-0013', 'gsm8k-test-0014', 'gsm8k-test-0015', 'gsm8k-test-0016', 'gsm8k-test-0017', 'gsm8k-test-0018', 'gsm8k-test-0019', 'gsm8k-test-0020', 'gsm8k-test-0021', 'gsm8k-test-0022', 'gsm8k-test-0023', 'gsm8k-test-0024', 'gsm8k-test-0025', 'gsm8k-test-0026', 'gsm8k-test-0027', 'gsm8k-test-0028', 'gsm8k-test-0029', 'gsm8k-test-0030', 'gsm8k-test-0031', 'gsm8k-test-0032', 'gsm8k-test-0033', 'gsm8k-test-0034', 'gsm8k-test-0035', 'gsm8k-test-0036', 'gsm8k-test-0037', 'gsm8k-test-0038', 'gsm8k-test-0039', 'gsm8k-test-0040', 'gsm8k-test-0041', 'gsm8k-test-0042', 'gsm8k-test-0043', 'gsm8k-test-0044', 'gsm8k-test-0045', 'gsm8k-test-0046', 'gsm8k-test-0047', 'gsm8k-test-0048', 'gsm8k-test-0049', 'gsm8k-test-0050', 'gsm8k-test-0051', 'gsm8k-test-0052', 'gsm8k-test-0053', 'gsm8k-test-0054', 'gsm8k-test-0055', 'gsm8k-test-0056', 'gsm8k-test-0057', 'gsm8k-test-0058', 'gsm8k-test-0059', 'gsm8k-test-0060', 'gsm8k-test-0061', 'gsm8k-test-0062', 'gsm8k-test-0063', 'gsm8k-test-0064', 'gsm8k-test-0065', 'gsm8k-test-0066', 'gsm8k-test-0067', 'gsm8k-test-0068', 'gsm8k-test-0069', 'gsm8k-test-0070', 'gsm8k-test-0071', 'gsm8k-test-0072', 'gsm8k-test-0073', 'gsm8k-test-0074', 'gsm8k-test-0075', 'gsm8k-test-0076', 'gsm8k-test-0077', 'gsm8k-test-0078', 'gsm8k-test-0079', 'gsm8k-test-0080', 'gsm8k-test-0081', 'gsm8k-test-0082', 'gsm8k-test-0083', 'gsm8k-test-0084', 'gsm8k-test-0085', 'gsm8k-test-0086', 'gsm8k-test-0087', 'gsm8k-test-0088', 'gsm8k-test-0089', 'gsm8k-test-0090', 'gsm8k-test-0091', 'gsm8k-test-0092', 'gsm8k-test-0093', 'gsm8k-test-0094', 'gsm8k-test-0095', 'gsm8k-test-0096', 'gsm8k-test-0097', 'gsm8k-test-0098', 'gsm8k-test-0099', 'gsm8k-test-0100', 'gsm8k-test-0101', 'gsm8k-test-0102', 'gsm8k-test-0103', 'gsm8k-test-0104', 'gsm8k-test-0105', 'gsm8k-test-0106', 'gsm8k-test-0107', 'gsm8k-test-0108', 'gsm8k-test-0109', 'gsm8k-test-0110', 'gsm8k-test-0111', 'gsm8k-test-0112', 'gsm8k-test-0113', 'gsm8k-test-0114', 'gsm8k-test-0115', 'gsm8k-test-0116', 'gsm8k-test-0117', 'gsm8k-test-0118', 'gsm8k-test-0119', 'gsm8k-test-0120', 'gsm8k-test-0121', 'gsm8k-test-0122', 'gsm8k-test-0123', 'gsm8k-test-0124', 'gsm8k-test-0125', 'gsm8k-test-0126', 'gsm8k-test-0127', 'gsm8k-test-0128', 'gsm8k-test-0129', 'gsm8k-test-0130', 'gsm8k-test-0131', 'gsm8k-test-0132', 'gsm8k-test-0133', 'gsm8k-test-0134', 'gsm8k-test-0135', 'gsm8k-test-0136', 'gsm8k-test-0137', 'gsm8k-test-0138', 'gsm8k-test-0139', 'gsm8k-test-0140', 'gsm8k-test-0141', 'gsm8k-test-0142', 'gsm8k-test-0143', 'gsm8k-test-0144', 'gsm8k-test-0145', 'gsm8k-test-0146', 'gsm8k-test-0147', 'gsm8k-test-0148', 'gsm8k-test-0149', 'gsm8k-test-0150', 'gsm8k-test-0151', 'gsm8k-test-0152', 'gsm8k-test-0153', 'gsm8k-test-0154', 'gsm8k-test-0155', 'gsm8k-test-0156', 'gsm8k-test-0157', 'gsm8k-test-0158', 'gsm8k-test-0159', 'gsm8k-test-0160', 'gsm8k-test-0161', 'gsm8k-test-0162', 'gsm8k-test-0163', 'gsm8k-test-0164', 'gsm8k-test-0165', 'gsm8k-test-0166', 'gsm8k-test-0167', 'gsm8k-test-0168', 'gsm8k-test-0169', 'gsm8k-test-0170', 'gsm8k-test-0171', 'gsm8k-test-0172', 'gsm8k-test-0173', 'gsm8k-test-0174', 'gsm8k-test-0175', 'gsm8k-test-0176', 'gsm8k-test-0177', 'gsm8k-test-0178', 'gsm8k-test-0179', 'gsm8k-test-0180', 'gsm8k-test-0181', 'gsm8k-test-0182', 'gsm8k-test-0183', 'gsm8k-test-0184', 'gsm8k-test-0185', 'gsm8k-test-0186', 'gsm8k-test-0187', 'gsm8k-test-0188', 'gsm8k-test-0189', 'gsm8k-test-0190', 'gsm8k-test-0191', 'gsm8k-test-0192', 'gsm8k-test-0193', 'gsm8k-test-0194', 'gsm8k-test-0195', 'gsm8k-test-0196', 'gsm8k-test-0197', 'gsm8k-test-0198', 'gsm8k-test-0199', 'gsm8k-test-0200', 'gsm8k-test-0201', 'gsm8k-test-0202', 'gsm8k-test-0203', 'gsm8k-test-0204', 'gsm8k-test-0205', 'gsm8k-test-0206', 'gsm8k-test-0207', 'gsm8k-test-0208', 'gsm8k-test-0209', 'gsm8k-test-0210', 'gsm8k-test-0211', 'gsm8k-test-0212', 'gsm8k-test-0213', 'gsm8k-test-0214', 'gsm8k-test-0215', 'gsm8k-test-0216', 'gsm8k-test-0217', 'gsm8k-test-0218', 'gsm8k-test-0219', 'gsm8k-test-0220', 'gsm8k-test-0221', 'gsm8k-test-0222', 'gsm8k-test-0223', 'gsm8k-test-0224', 'gsm8k-test-0225', 'gsm8k-test-0226', 'gsm8k-test-0227', 'gsm8k-test-0228', 'gsm8k-test-0229', 'gsm8k-test-0230', 'gsm8k-test-0231', 'gsm8k-test-0232', 'gsm8k-test-0233', 'gsm8k-test-0234', 'gsm8k-test-0235', 'gsm8k-test-0236', 'gsm8k-test-0237', 'gsm8k-test-0238', 'gsm8k-test-0239', 'gsm8k-test-0240', 'gsm8k-test-0241', 'gsm8k-test-0242', 'gsm8k-test-0243', 'gsm8k-test-0244', 'gsm8k-test-0245', 'gsm8k-test-0246', 'gsm8k-test-0247', 'gsm8k-test-0248', 'gsm8k-test-0249', 'gsm8k-test-0250', 'gsm8k-test-0251', 'gsm8k-test-0252', 'gsm8k-test-0253', 'gsm8k-test-0254', 'gsm8k-test-0255', 'gsm8k-test-0256', 'gsm8k-test-0257', 'gsm8k-test-0258', 'gsm8k-test-0259', 'gsm8k-test-0260', 'gsm8k-test-0261', 'gsm8k-test-0262', 'gsm8k-test-0263', 'gsm8k-test-0264', 'gsm8k-test-0265', 'gsm8k-test-0266', 'gsm8k-test-0267', 'gsm8k-test-0268', 'gsm8k-test-0269', 'gsm8k-test-0270', 'gsm8k-test-0271', 'gsm8k-test-0272', 'gsm8k-test-0273', 'gsm8k-test-0274', 'gsm8k-test-0275', 'gsm8k-test-0276', 'gsm8k-test-0277', 'gsm8k-test-0278', 'gsm8k-test-0279', 'gsm8k-test-0280', 'gsm8k-test-0281', 'gsm8k-test-0282', 'gsm8k-test-0283', 'gsm8k-test-0284', 'gsm8k-test-0285', 'gsm8k-test-0286', 'gsm8k-test-0287', 'gsm8k-test-0288', 'gsm8k-test-0289', 'gsm8k-test-0290', 'gsm8k-test-0291', 'gsm8k-test-0292', 'gsm8k-test-0293', 'gsm8k-test-0294', 'gsm8k-test-0295', 'gsm8k-test-0296', 'gsm8k-test-0297', 'gsm8k-test-0298', 'gsm8k-test-0299', 'gsm8k-test-0300', 'gsm8k-test-0301', 'gsm8k-test-0302', 'gsm8k-test-0303', 'gsm8k-test-0304', 'gsm8k-test-0305', 'gsm8k-test-0306', 'gsm8k-test-0307', 'gsm8k-test-0308', 'gsm8k-test-0309', 'gsm8k-test-0310', 'gsm8k-test-0311', 'gsm8k-test-0312', 'gsm8k-test-0313', 'gsm8k-test-0314', 'gsm8k-test-0315', 'gsm8k-test-0316', 'gsm8k-test-0317', 'gsm8k-test-0318', 'gsm8k-test-0319', 'gsm8k-test-0320', 'gsm8k-test-0321', 'gsm8k-test-0322', 'gsm8k-test-0323', 'gsm8k-test-0324', 'gsm8k-test-0325', 'gsm8k-test-0326', 'gsm8k-test-0327', 'gsm8k-test-0328', 'gsm8k-test-0329', 'gsm8k-test-0330', 'gsm8k-test-0331', 'gsm8k-test-0332', 'gsm8k-test-0333', 'gsm8k-test-0334', 'gsm8k-test-0335', 'gsm8k-test-0336', 'gsm8k-test-0337', 'gsm8k-test-0338', 'gsm8k-test-0339', 'gsm8k-test-0340', 'gsm8k-test-0341', 'gsm8k-test-0342', 'gsm8k-test-0343', 'gsm8k-test-0344', 'gsm8k-test-0345', 'gsm8k-test-0346', 'gsm8k-test-0347', 'gsm8k-test-0348', 'gsm8k-test-0349', 'gsm8k-test-0350', 'gsm8k-test-0351', 'gsm8k-test-0352', 'gsm8k-test-0353', 'gsm8k-test-0354', 'gsm8k-test-0355', 'gsm8k-test-0356', 'gsm8k-test-0357', 'gsm8k-test-0358', 'gsm8k-test-0359', 'gsm8k-test-0360', 'gsm8k-test-0361', 'gsm8k-test-0362', 'gsm8k-test-0363', 'gsm8k-test-0364', 'gsm8k-test-0365', 'gsm8k-test-0366', 'gsm8k-test-0367', 'gsm8k-test-0368', 'gsm8k-test-0369', 'gsm8k-test-0370', 'gsm8k-test-0371', 'gsm8k-test-0372', 'gsm8k-test-0373', 'gsm8k-test-0374', 'gsm8k-test-0375', 'gsm8k-test-0376', 'gsm8k-test-0377', 'gsm8k-test-0378', 'gsm8k-test-0379', 'gsm8k-test-0380', 'gsm8k-test-0381', 'gsm8k-test-0382', 'gsm8k-test-0383', 'gsm8k-test-0384', 'gsm8k-test-0385', 'gsm8k-test-0386', 'gsm8k-test-0387', 'gsm8k-test-0388', 'gsm8k-test-0389', 'gsm8k-test-0390', 'gsm8k-test-0391', 'gsm8k-test-0392', 'gsm8k-test-0393', 'gsm8k-test-0394', 'gsm8k-test-0395', 'gsm8k-test-0396', 'gsm8k-test-0397', 'gsm8k-test-0398', 'gsm8k-test-0399', 'gsm8k-test-0400', 'gsm8k-test-0401', 'gsm8k-test-0402', 'gsm8k-test-0403', 'gsm8k-test-0404', 'gsm8k-test-0405', 'gsm8k-test-0406', 'gsm8k-test-0407', 'gsm8k-test-0408', 'gsm8k-test-0409', 'gsm8k-test-0410', 'gsm8k-test-0411', 'gsm8k-test-0412', 'gsm8k-test-0413', 'gsm8k-test-0414', 'gsm8k-test-0415', 'gsm8k-test-0416', 'gsm8k-test-0417', 'gsm8k-test-0418', 'gsm8k-test-0419', 'gsm8k-test-0420', 'gsm8k-test-0421', 'gsm8k-test-0422', 'gsm8k-test-0423', 'gsm8k-test-0424', 'gsm8k-test-0425', 'gsm8k-test-0426', 'gsm8k-test-0427', 'gsm8k-test-0428', 'gsm8k-test-0429', 'gsm8k-test-0430', 'gsm8k-test-0431', 'gsm8k-test-0432', 'gsm8k-test-0433', 'gsm8k-test-0434', 'gsm8k-test-0435', 'gsm8k-test-0436', 'gsm8k-test-0437', 'gsm8k-test-0438', 'gsm8k-test-0439', 'gsm8k-test-0440', 'gsm8k-test-0441', 'gsm8k-test-0442', 'gsm8k-test-0443', 'gsm8k-test-0444', 'gsm8k-test-0445', 'gsm8k-test-0446', 'gsm8k-test-0447', 'gsm8k-test-0448', 'gsm8k-test-0449', 'gsm8k-test-0450', 'gsm8k-test-0451', 'gsm8k-test-0452', 'gsm8k-test-0453', 'gsm8k-test-0454', 'gsm8k-test-0455', 'gsm8k-test-0456', 'gsm8k-test-0457', 'gsm8k-test-0458', 'gsm8k-test-0459', 'gsm8k-test-0460', 'gsm8k-test-0461', 'gsm8k-test-0462', 'gsm8k-test-0463', 'gsm8k-test-0464', 'gsm8k-test-0465', 'gsm8k-test-0466', 'gsm8k-test-0467', 'gsm8k-test-0468', 'gsm8k-test-0469', 'gsm8k-test-0470', 'gsm8k-test-0471', 'gsm8k-test-0472', 'gsm8k-test-0473', 'gsm8k-test-0474', 'gsm8k-test-0475', 'gsm8k-test-0476', 'gsm8k-test-0477', 'gsm8k-test-0478', 'gsm8k-test-0479', 'gsm8k-test-0480', 'gsm8k-test-0481', 'gsm8k-test-0482', 'gsm8k-test-0483', 'gsm8k-test-0484', 'gsm8k-test-0485', 'gsm8k-test-0486', 'gsm8k-test-0487', 'gsm8k-test-0488', 'gsm8k-test-0489', 'gsm8k-test-0490', 'gsm8k-test-0491', 'gsm8k-test-0492', 'gsm8k-test-0493', 'gsm8k-test-0494', 'gsm8k-test-0495', 'gsm8k-test-0496', 'gsm8k-test-0497', 'gsm8k-test-0498', 'gsm8k-test-0499', 'gsm8k-test-0500', 'gsm8k-test-0501', 'gsm8k-test-0502', 'gsm8k-test-0503', 'gsm8k-test-0504', 'gsm8k-test-0505', 'gsm8k-test-0506', 'gsm8k-test-0507', 'gsm8k-test-0508', 'gsm8k-test-0509', 'gsm8k-test-0510', 'gsm8k-test-0511', 'gsm8k-test-0512', 'gsm8k-test-0513', 'gsm8k-test-0514', 'gsm8k-test-0515', 'gsm8k-test-0516', 'gsm8k-test-0517', 'gsm8k-test-0518', 'gsm8k-test-0519', 'gsm8k-test-0520', 'gsm8k-test-0521', 'gsm8k-test-0522', 'gsm8k-test-0523', 'gsm8k-test-0524', 'gsm8k-test-0525', 'gsm8k-test-0526', 'gsm8k-test-0527', 'gsm8k-test-0528', 'gsm8k-test-0529', 'gsm8k-test-0530', 'gsm8k-test-0531', 'gsm8k-test-0532', 'gsm8k-test-0533', 'gsm8k-test-0534', 'gsm8k-test-0535', 'gsm8k-test-0536', 'gsm8k-test-0537', 'gsm8k-test-0538', 'gsm8k-test-0539', 'gsm8k-test-0540', 'gsm8k-test-0541', 'gsm8k-test-0542', 'gsm8k-test-0543', 'gsm8k-test-0544', 'gsm8k-test-0545', 'gsm8k-test-0546', 'gsm8k-test-0547', 'gsm8k-test-0548', 'gsm8k-test-0549', 'gsm8k-test-0550', 'gsm8k-test-0551', 'gsm8k-test-0552', 'gsm8k-test-0553', 'gsm8k-test-0554', 'gsm8k-test-0555', 'gsm8k-test-0556', 'gsm8k-test-0557', 'gsm8k-test-0558', 'gsm8k-test-0559', 'gsm8k-test-0560', 'gsm8k-test-0561', 'gsm8k-test-0562', 'gsm8k-test-0563', 'gsm8k-test-0564', 'gsm8k-test-0565', 'gsm8k-test-0566', 'gsm8k-test-0567', 'gsm8k-test-0568', 'gsm8k-test-0569', 'gsm8k-test-0570', 'gsm8k-test-0571', 'gsm8k-test-0572', 'gsm8k-test-0573', 'gsm8k-test-0574', 'gsm8k-test-0575', 'gsm8k-test-0576', 'gsm8k-test-0577', 'gsm8k-test-0578', 'gsm8k-test-0579', 'gsm8k-test-0580', 'gsm8k-test-0581', 'gsm8k-test-0582', 'gsm8k-test-0583', 'gsm8k-test-0584', 'gsm8k-test-0585', 'gsm8k-test-0586', 'gsm8k-test-0587', 'gsm8k-test-0588', 'gsm8k-test-0589', 'gsm8k-test-0590', 'gsm8k-test-0591', 'gsm8k-test-0592', 'gsm8k-test-0593', 'gsm8k-test-0594', 'gsm8k-test-0595', 'gsm8k-test-0596', 'gsm8k-test-0597', 'gsm8k-test-0598', 'gsm8k-test-0599', 'gsm8k-test-0600', 'gsm8k-test-0601', 'gsm8k-test-0602', 'gsm8k-test-0603', 'gsm8k-test-0604', 'gsm8k-test-0605', 'gsm8k-test-0606', 'gsm8k-test-0607', 'gsm8k-test-0608', 'gsm8k-test-0609', 'gsm8k-test-0610', 'gsm8k-test-0611', 'gsm8k-test-0612', 'gsm8k-test-0613', 'gsm8k-test-0614', 'gsm8k-test-0615', 'gsm8k-test-0616', 'gsm8k-test-0617', 'gsm8k-test-0618', 'gsm8k-test-0619', 'gsm8k-test-0620', 'gsm8k-test-0621', 'gsm8k-test-0622', 'gsm8k-test-0623', 'gsm8k-test-0624', 'gsm8k-test-0625', 'gsm8k-test-0626', 'gsm8k-test-0627', 'gsm8k-test-0628', 'gsm8k-test-0629', 'gsm8k-test-0630', 'gsm8k-test-0631', 'gsm8k-test-0632', 'gsm8k-test-0633', 'gsm8k-test-0634', 'gsm8k-test-0635', 'gsm8k-test-0636', 'gsm8k-test-0637', 'gsm8k-test-0638', 'gsm8k-test-0639', 'gsm8k-test-0640', 'gsm8k-test-0641', 'gsm8k-test-0642', 'gsm8k-test-0643', 'gsm8k-test-0644', 'gsm8k-test-0645', 'gsm8k-test-0646', 'gsm8k-test-0647', 'gsm8k-test-0648', 'gsm8k-test-0649', 'gsm8k-test-0650', 'gsm8k-test-0651', 'gsm8k-test-0652', 'gsm8k-test-0653', 'gsm8k-test-0654', 'gsm8k-test-0655', 'gsm8k-test-0656', 'gsm8k-test-0657', 'gsm8k-test-0658', 'gsm8k-test-0659', 'gsm8k-test-0660', 'gsm8k-test-0661', 'gsm8k-test-0662', 'gsm8k-test-0663', 'gsm8k-test-0664', 'gsm8k-test-0665', 'gsm8k-test-0666', 'gsm8k-test-0667', 'gsm8k-test-0668', 'gsm8k-test-0669', 'gsm8k-test-0670', 'gsm8k-test-0671', 'gsm8k-test-0672', 'gsm8k-test-0673', 'gsm8k-test-0674', 'gsm8k-test-0675', 'gsm8k-test-0676', 'gsm8k-test-0677', 'gsm8k-test-0678', 'gsm8k-test-0679', 'gsm8k-test-0680', 'gsm8k-test-0681', 'gsm8k-test-0682', 'gsm8k-test-0683', 'gsm8k-test-0684', 'gsm8k-test-0685', 'gsm8k-test-0686', 'gsm8k-test-0687', 'gsm8k-test-0688', 'gsm8k-test-0689', 'gsm8k-test-0690', 'gsm8k-test-0691', 'gsm8k-test-0692', 'gsm8k-test-0693', 'gsm8k-test-0694', 'gsm8k-test-0695', 'gsm8k-test-0696', 'gsm8k-test-0697', 'gsm8k-test-0698', 'gsm8k-test-0699', 'gsm8k-test-0700', 'gsm8k-test-0701', 'gsm8k-test-0702', 'gsm8k-test-0703', 'gsm8k-test-0704', 'gsm8k-test-0705', 'gsm8k-test-0706', 'gsm8k-test-0707', 'gsm8k-test-0708', 'gsm8k-test-0709', 'gsm8k-test-0710', 'gsm8k-test-0711', 'gsm8k-test-0712', 'gsm8k-test-0713', 'gsm8k-test-0714', 'gsm8k-test-0715', 'gsm8k-test-0716', 'gsm8k-test-0717', 'gsm8k-test-0718', 'gsm8k-test-0719', 'gsm8k-test-0720', 'gsm8k-test-0721', 'gsm8k-test-0722', 'gsm8k-test-0723', 'gsm8k-test-0724', 'gsm8k-test-0725', 'gsm8k-test-0726', 'gsm8k-test-0727', 'gsm8k-test-0728', 'gsm8k-test-0729', 'gsm8k-test-0730', 'gsm8k-test-0731', 'gsm8k-test-0732', 'gsm8k-test-0733', 'gsm8k-test-0734', 'gsm8k-test-0735', 'gsm8k-test-0736', 'gsm8k-test-0737', 'gsm8k-test-0738', 'gsm8k-test-0739', 'gsm8k-test-0740', 'gsm8k-test-0741', 'gsm8k-test-0742', 'gsm8k-test-0743', 'gsm8k-test-0744', 'gsm8k-test-0745', 'gsm8k-test-0746', 'gsm8k-test-0747', 'gsm8k-test-0748', 'gsm8k-test-0749', 'gsm8k-test-0750', 'gsm8k-test-0751', 'gsm8k-test-0752', 'gsm8k-test-0753', 'gsm8k-test-0754', 'gsm8k-test-0755', 'gsm8k-test-0756', 'gsm8k-test-0757', 'gsm8k-test-0758', 'gsm8k-test-0759', 'gsm8k-test-0760', 'gsm8k-test-0761', 'gsm8k-test-0762', 'gsm8k-test-0763', 'gsm8k-test-0764', 'gsm8k-test-0765', 'gsm8k-test-0766', 'gsm8k-test-0767', 'gsm8k-test-0768', 'gsm8k-test-0769', 'gsm8k-test-0770', 'gsm8k-test-0771', 'gsm8k-test-0772', 'gsm8k-test-0773', 'gsm8k-test-0774', 'gsm8k-test-0775', 'gsm8k-test-0776', 'gsm8k-test-0777', 'gsm8k-test-0778', 'gsm8k-test-0779', 'gsm8k-test-0780', 'gsm8k-test-0781', 'gsm8k-test-0782', 'gsm8k-test-0783', 'gsm8k-test-0784', 'gsm8k-test-0785', 'gsm8k-test-0786', 'gsm8k-test-0787', 'gsm8k-test-0788', 'gsm8k-test-0789', 'gsm8k-test-0790', 'gsm8k-test-0791', 'gsm8k-test-0792', 'gsm8k-test-0793', 'gsm8k-test-0794', 'gsm8k-test-0795', 'gsm8k-test-0796', 'gsm8k-test-0797', 'gsm8k-test-0798', 'gsm8k-test-0799', 'gsm8k-test-0800', 'gsm8k-test-0801', 'gsm8k-test-0802', 'gsm8k-test-0803', 'gsm8k-test-0804', 'gsm8k-test-0805', 'gsm8k-test-0806', 'gsm8k-test-0807', 'gsm8k-test-0808', 'gsm8k-test-0809', 'gsm8k-test-0810', 'gsm8k-test-0811', 'gsm8k-test-0812', 'gsm8k-test-0813', 'gsm8k-test-0814', 'gsm8k-test-0815', 'gsm8k-test-0816', 'gsm8k-test-0817', 'gsm8k-test-0818', 'gsm8k-test-0819', 'gsm8k-test-0820', 'gsm8k-test-0821', 'gsm8k-test-0822', 'gsm8k-test-0823', 'gsm8k-test-0824', 'gsm8k-test-0825', 'gsm8k-test-0826', 'gsm8k-test-0827', 'gsm8k-test-0828', 'gsm8k-test-0829', 'gsm8k-test-0830', 'gsm8k-test-0831', 'gsm8k-test-0832', 'gsm8k-test-0833', 'gsm8k-test-0834', 'gsm8k-test-0835', 'gsm8k-test-0836', 'gsm8k-test-0837', 'gsm8k-test-0838', 'gsm8k-test-0839', 'gsm8k-test-0840', 'gsm8k-test-0841', 'gsm8k-test-0842', 'gsm8k-test-0843', 'gsm8k-test-0844', 'gsm8k-test-0845', 'gsm8k-test-0846', 'gsm8k-test-0847', 'gsm8k-test-0848', 'gsm8k-test-0849', 'gsm8k-test-0850', 'gsm8k-test-0851', 'gsm8k-test-0852', 'gsm8k-test-0853', 'gsm8k-test-0854', 'gsm8k-test-0855', 'gsm8k-test-0856', 'gsm8k-test-0857', 'gsm8k-test-0858', 'gsm8k-test-0859', 'gsm8k-test-0860', 'gsm8k-test-0861', 'gsm8k-test-0862', 'gsm8k-test-0863', 'gsm8k-test-0864', 'gsm8k-test-0865', 'gsm8k-test-0866', 'gsm8k-test-0867', 'gsm8k-test-0868', 'gsm8k-test-0869', 'gsm8k-test-0870', 'gsm8k-test-0871', 'gsm8k-test-0872', 'gsm8k-test-0873', 'gsm8k-test-0874', 'gsm8k-test-0875', 'gsm8k-test-0876', 'gsm8k-test-0877', 'gsm8k-test-0878', 'gsm8k-test-0879', 'gsm8k-test-0880', 'gsm8k-test-0881', 'gsm8k-test-0882', 'gsm8k-test-0883', 'gsm8k-test-0884', 'gsm8k-test-0885', 'gsm8k-test-0886', 'gsm8k-test-0887', 'gsm8k-test-0888', 'gsm8k-test-0889', 'gsm8k-test-0890', 'gsm8k-test-0891', 'gsm8k-test-0892', 'gsm8k-test-0893', 'gsm8k-test-0894', 'gsm8k-test-0895', 'gsm8k-test-0896', 'gsm8k-test-0897', 'gsm8k-test-0898', 'gsm8k-test-0899', 'gsm8k-test-0900', 'gsm8k-test-0901', 'gsm8k-test-0902', 'gsm8k-test-0903', 'gsm8k-test-0904', 'gsm8k-test-0905', 'gsm8k-test-0906', 'gsm8k-test-0907', 'gsm8k-test-0908', 'gsm8k-test-0909', 'gsm8k-test-0910', 'gsm8k-test-0911', 'gsm8k-test-0912', 'gsm8k-test-0913', 'gsm8k-test-0914', 'gsm8k-test-0915', 'gsm8k-test-0916', 'gsm8k-test-0917', 'gsm8k-test-0918', 'gsm8k-test-0919', 'gsm8k-test-0920', 'gsm8k-test-0921', 'gsm8k-test-0922', 'gsm8k-test-0923', 'gsm8k-test-0924', 'gsm8k-test-0925', 'gsm8k-test-0926', 'gsm8k-test-0927', 'gsm8k-test-0928', 'gsm8k-test-0929', 'gsm8k-test-0930', 'gsm8k-test-0931', 'gsm8k-test-0932', 'gsm8k-test-0933', 'gsm8k-test-0934', 'gsm8k-test-0935', 'gsm8k-test-0936', 'gsm8k-test-0937', 'gsm8k-test-0938', 'gsm8k-test-0939', 'gsm8k-test-0940', 'gsm8k-test-0941', 'gsm8k-test-0942', 'gsm8k-test-0943', 'gsm8k-test-0944', 'gsm8k-test-0945', 'gsm8k-test-0946', 'gsm8k-test-0947', 'gsm8k-test-0948', 'gsm8k-test-0949', 'gsm8k-test-0950', 'gsm8k-test-0951', 'gsm8k-test-0952', 'gsm8k-test-0953', 'gsm8k-test-0954', 'gsm8k-test-0955', 'gsm8k-test-0956', 'gsm8k-test-0957', 'gsm8k-test-0958', 'gsm8k-test-0959', 'gsm8k-test-0960', 'gsm8k-test-0961', 'gsm8k-test-0962', 'gsm8k-test-0963', 'gsm8k-test-0964', 'gsm8k-test-0965', 'gsm8k-test-0966', 'gsm8k-test-0967', 'gsm8k-test-0968', 'gsm8k-test-0969', 'gsm8k-test-0970', 'gsm8k-test-0971', 'gsm8k-test-0972', 'gsm8k-test-0973', 'gsm8k-test-0974', 'gsm8k-test-0975', 'gsm8k-test-0976', 'gsm8k-test-0977', 'gsm8k-test-0978', 'gsm8k-test-0979', 'gsm8k-test-0980', 'gsm8k-test-0981', 'gsm8k-test-0982', 'gsm8k-test-0983', 'gsm8k-test-0984', 'gsm8k-test-0985', 'gsm8k-test-0986', 'gsm8k-test-0987', 'gsm8k-test-0988', 'gsm8k-test-0989', 'gsm8k-test-0990', 'gsm8k-test-0991', 'gsm8k-test-0992', 'gsm8k-test-0993', 'gsm8k-test-0994', 'gsm8k-test-0995', 'gsm8k-test-0996', 'gsm8k-test-0997', 'gsm8k-test-0998', 'gsm8k-test-0999', 'gsm8k-test-1000', 'gsm8k-test-1001', 'gsm8k-test-1002', 'gsm8k-test-1003', 'gsm8k-test-1004', 'gsm8k-test-1005', 'gsm8k-test-1006', 'gsm8k-test-1007', 'gsm8k-test-1008', 'gsm8k-test-1009', 'gsm8k-test-1010', 'gsm8k-test-1011', 'gsm8k-test-1012', 'gsm8k-test-1013', 'gsm8k-test-1014', 'gsm8k-test-1015', 'gsm8k-test-1016', 'gsm8k-test-1017', 'gsm8k-test-1018', 'gsm8k-test-1019', 'gsm8k-test-1020', 'gsm8k-test-1021', 'gsm8k-test-1022', 'gsm8k-test-1023', 'gsm8k-test-1024', 'gsm8k-test-1025', 'gsm8k-test-1026', 'gsm8k-test-1027', 'gsm8k-test-1028', 'gsm8k-test-1029', 'gsm8k-test-1030', 'gsm8k-test-1031', 'gsm8k-test-1032', 'gsm8k-test-1033', 'gsm8k-test-1034', 'gsm8k-test-1035', 'gsm8k-test-1036', 'gsm8k-test-1037', 'gsm8k-test-1038', 'gsm8k-test-1039', 'gsm8k-test-1040', 'gsm8k-test-1041', 'gsm8k-test-1042', 'gsm8k-test-1043', 'gsm8k-test-1044', 'gsm8k-test-1045', 'gsm8k-test-1046', 'gsm8k-test-1047', 'gsm8k-test-1048', 'gsm8k-test-1049', 'gsm8k-test-1050', 'gsm8k-test-1051', 'gsm8k-test-1052', 'gsm8k-test-1053', 'gsm8k-test-1054', 'gsm8k-test-1055', 'gsm8k-test-1056', 'gsm8k-test-1057', 'gsm8k-test-1058', 'gsm8k-test-1059', 'gsm8k-test-1060', 'gsm8k-test-1061', 'gsm8k-test-1062', 'gsm8k-test-1063', 'gsm8k-test-1064', 'gsm8k-test-1065', 'gsm8k-test-1066', 'gsm8k-test-1067', 'gsm8k-test-1068', 'gsm8k-test-1069', 'gsm8k-test-1070', 'gsm8k-test-1071', 'gsm8k-test-1072', 'gsm8k-test-1073', 'gsm8k-test-1074', 'gsm8k-test-1075', 'gsm8k-test-1076', 'gsm8k-test-1077', 'gsm8k-test-1078', 'gsm8k-test-1079', 'gsm8k-test-1080', 'gsm8k-test-1081', 'gsm8k-test-1082', 'gsm8k-test-1083', 'gsm8k-test-1084', 'gsm8k-test-1085', 'gsm8k-test-1086', 'gsm8k-test-1087', 'gsm8k-test-1088', 'gsm8k-test-1089', 'gsm8k-test-1090', 'gsm8k-test-1091', 'gsm8k-test-1092', 'gsm8k-test-1093', 'gsm8k-test-1094', 'gsm8k-test-1095', 'gsm8k-test-1096', 'gsm8k-test-1097', 'gsm8k-test-1098', 'gsm8k-test-1099', 'gsm8k-test-1100', 'gsm8k-test-1101', 'gsm8k-test-1102', 'gsm8k-test-1103', 'gsm8k-test-1104', 'gsm8k-test-1105', 'gsm8k-test-1106', 'gsm8k-test-1107', 'gsm8k-test-1108', 'gsm8k-test-1109', 'gsm8k-test-1110', 'gsm8k-test-1111', 'gsm8k-test-1112', 'gsm8k-test-1113', 'gsm8k-test-1114', 'gsm8k-test-1115', 'gsm8k-test-1116', 'gsm8k-test-1117', 'gsm8k-test-1118', 'gsm8k-test-1119', 'gsm8k-test-1120', 'gsm8k-test-1121', 'gsm8k-test-1122', 'gsm8k-test-1123', 'gsm8k-test-1124', 'gsm8k-test-1125', 'gsm8k-test-1126', 'gsm8k-test-1127', 'gsm8k-test-1128', 'gsm8k-test-1129', 'gsm8k-test-1130', 'gsm8k-test-1131', 'gsm8k-test-1132', 'gsm8k-test-1133', 'gsm8k-test-1134', 'gsm8k-test-1135', 'gsm8k-test-1136', 'gsm8k-test-1137', 'gsm8k-test-1138', 'gsm8k-test-1139', 'gsm8k-test-1140', 'gsm8k-test-1141', 'gsm8k-test-1142', 'gsm8k-test-1143', 'gsm8k-test-1144', 'gsm8k-test-1145', 'gsm8k-test-1146', 'gsm8k-test-1147', 'gsm8k-test-1148', 'gsm8k-test-1149', 'gsm8k-test-1150', 'gsm8k-test-1151', 'gsm8k-test-1152', 'gsm8k-test-1153', 'gsm8k-test-1154', 'gsm8k-test-1155', 'gsm8k-test-1156', 'gsm8k-test-1157', 'gsm8k-test-1158', 'gsm8k-test-1159', 'gsm8k-test-1160', 'gsm8k-test-1161', 'gsm8k-test-1162', 'gsm8k-test-1163', 'gsm8k-test-1164', 'gsm8k-test-1165', 'gsm8k-test-1166', 'gsm8k-test-1167', 'gsm8k-test-1168', 'gsm8k-test-1169', 'gsm8k-test-1170', 'gsm8k-test-1171', 'gsm8k-test-1172', 'gsm8k-test-1173', 'gsm8k-test-1174', 'gsm8k-test-1175', 'gsm8k-test-1176', 'gsm8k-test-1177', 'gsm8k-test-1178', 'gsm8k-test-1179', 'gsm8k-test-1180', 'gsm8k-test-1181', 'gsm8k-test-1182', 'gsm8k-test-1183', 'gsm8k-test-1184', 'gsm8k-test-1185', 'gsm8k-test-1186', 'gsm8k-test-1187', 'gsm8k-test-1188', 'gsm8k-test-1189', 'gsm8k-test-1190', 'gsm8k-test-1191', 'gsm8k-test-1192', 'gsm8k-test-1193', 'gsm8k-test-1194', 'gsm8k-test-1195', 'gsm8k-test-1196', 'gsm8k-test-1197', 'gsm8k-test-1198', 'gsm8k-test-1199', 'gsm8k-test-1200', 'gsm8k-test-1201', 'gsm8k-test-1202', 'gsm8k-test-1203', 'gsm8k-test-1204', 'gsm8k-test-1205', 'gsm8k-test-1206', 'gsm8k-test-1207', 'gsm8k-test-1208', 'gsm8k-test-1209', 'gsm8k-test-1210', 'gsm8k-test-1211', 'gsm8k-test-1212', 'gsm8k-test-1213', 'gsm8k-test-1214', 'gsm8k-test-1215', 'gsm8k-test-1216', 'gsm8k-test-1217', 'gsm8k-test-1218', 'gsm8k-test-1219', 'gsm8k-test-1220', 'gsm8k-test-1221', 'gsm8k-test-1222', 'gsm8k-test-1223', 'gsm8k-test-1224', 'gsm8k-test-1225', 'gsm8k-test-1226', 'gsm8k-test-1227', 'gsm8k-test-1228', 'gsm8k-test-1229', 'gsm8k-test-1230', 'gsm8k-test-1231', 'gsm8k-test-1232', 'gsm8k-test-1233', 'gsm8k-test-1234', 'gsm8k-test-1235', 'gsm8k-test-1236', 'gsm8k-test-1237', 'gsm8k-test-1238', 'gsm8k-test-1239', 'gsm8k-test-1240', 'gsm8k-test-1241', 'gsm8k-test-1242', 'gsm8k-test-1243', 'gsm8k-test-1244', 'gsm8k-test-1245', 'gsm8k-test-1246', 'gsm8k-test-1247', 'gsm8k-test-1248', 'gsm8k-test-1249', 'gsm8k-test-1250', 'gsm8k-test-1251', 'gsm8k-test-1252', 'gsm8k-test-1253', 'gsm8k-test-1254', 'gsm8k-test-1255', 'gsm8k-test-1256', 'gsm8k-test-1257', 'gsm8k-test-1258', 'gsm8k-test-1259', 'gsm8k-test-1260', 'gsm8k-test-1261', 'gsm8k-test-1262', 'gsm8k-test-1263', 'gsm8k-test-1264', 'gsm8k-test-1265', 'gsm8k-test-1266', 'gsm8k-test-1267', 'gsm8k-test-1268', 'gsm8k-test-1269', 'gsm8k-test-1270', 'gsm8k-test-1271', 'gsm8k-test-1272', 'gsm8k-test-1273', 'gsm8k-test-1274', 'gsm8k-test-1275', 'gsm8k-test-1276', 'gsm8k-test-1277', 'gsm8k-test-1278', 'gsm8k-test-1279', 'gsm8k-test-1280', 'gsm8k-test-1281', 'gsm8k-test-1282', 'gsm8k-test-1283', 'gsm8k-test-1284', 'gsm8k-test-1285', 'gsm8k-test-1286', 'gsm8k-test-1287', 'gsm8k-test-1288', 'gsm8k-test-1289', 'gsm8k-test-1290', 'gsm8k-test-1291', 'gsm8k-test-1292', 'gsm8k-test-1293', 'gsm8k-test-1294', 'gsm8k-test-1295', 'gsm8k-test-1296', 'gsm8k-test-1297', 'gsm8k-test-1298', 'gsm8k-test-1299', 'gsm8k-test-1300', 'gsm8k-test-1301', 'gsm8k-test-1302', 'gsm8k-test-1303', 'gsm8k-test-1304', 'gsm8k-test-1305', 'gsm8k-test-1306', 'gsm8k-test-1307', 'gsm8k-test-1308', 'gsm8k-test-1309', 'gsm8k-test-1310', 'gsm8k-test-1311', 'gsm8k-test-1312', 'gsm8k-test-1313', 'gsm8k-test-1314', 'gsm8k-test-1315', 'gsm8k-test-1316', 'gsm8k-test-1317', 'gsm8k-test-1318'], 'manifest_ref': 'evaluation-items.jsonl', 'manifest_sha256': '2605906367ae053a2354385b46f40b34ddbbeecb37fdec2e495b74c0dffae619'}}}, 'scores': [{'value': 0.7808946171341926, 'arm': 'grpo_gs8', 'metric': 'accuracy', 'seed': 0, 'evaluator_id': '5ce61cac464d6d7739964d336937b9a4de9d9b6a48b9905f59a0d978824a1d70', 'predictions_ref': 'predictions-grpo_gs8.jsonl'}, {'value': 0.7649734647460197, 'arm': 'gs1_kernel', 'metric': 'accuracy', 'seed': 0, 'evaluator_id': '5ce61cac464d6d7739964d336937b9a4de9d9b6a48b9905f59a0d978824a1d70', 'predictions_ref': 'predictions-gs1_kernel.jsonl'}, {'arm': 'base', 'metric': 'accuracy', 'value': 0.7073540561031084, 'evaluator_id': '5ce61cac464d6d7739964d336937b9a4de9d9b6a48b9905f59a0d978824a1d70', 'seed': 0, 'predictions_ref': 'predictions-base.jsonl'}], 'comparisons': [{'metric': 'accuracy', 'arms': ['grpo_gs8', 'gs1_kernel']}]}

evaluator_validation

{'status': 'invalid', 'issues': ['Evaluator 5ce61cac464d6d7739964d336937b9a4de9d9b6a48b9905f59a0d978824a1d70: identity does not match provenance hash.'], 'comparisons': [{'metric': 'accuracy', 'status': 'comparable', 'arms': ['base', 'grpo_gs8', 'gs1_kernel'], 'evaluator_ids': ['5ce61cac464d6d7739964d336937b9a4de9d9b6a48b9905f59a0d978824a1d70'], 'missing_arms': []}]}

feasibility

{'status': 'unknown', 'problems': ['Re-projection required: base_model changed or was not recorded', 'Re-projection required: lora_rank changed or was not recorded', 'Re-projection required: dataset changed or was not recorded', 'Re-projection required: steps changed or was not recorded', 'Re-projection required: batch_size changed or was not recorded', 'Re-projection required: max_seq_len changed or was not recorded', 'Re-projection required: max_tokens changed or was not recorded', 'Re-projection required: arms changed or was not recorded', 'Re-projection required: n_examples changed or was not recorded', 'Re-projection required: n_eval changed or was not recorded'], 'wall_seconds': None, 'gpu_hours': None, 'known_gpu_hours': 0, 'phases': {'setup': {'wall_seconds': 179.169, 'planned_units': 1, 'planned_samples': 0.0, 'gpu_hours': None}, 'training': {'wall_seconds': 2604.825, 'planned_units': 85, 'planned_samples': 5440.0, 'gpu_hours': None}, 'evaluation': {'wall_seconds': 522.1529999999999, 'planned_units': 3957, 'planned_samples': 3957.0, 'gpu_hours': None}, 'recovery': {'wall_seconds': 1200, 'gpu_hours': None}}, 'generation_counts': None, 'generation_applicability': {'check': 'generation_audit', 'status': 'not_applicable', 'enforce': False, 'reasons': ['spec.generation_audit opts out and no feature it covers is true'], 'features': {'generated_training_data': False}}, 'record': {'config': {'pilot': '1 grpo_gs8 step + 2+2 gs1_kernel steps, base eval 1319'}, 'effective_config': {'base_model': 'Qwen/Qwen3.5-2B', 'arms': ['grpo_gs8', 'gs1_kernel'], 'steps': 40, 'batch_size': 64, 'max_tokens': 512, 'max_seq_len': 1024, 'n_eval': 1319, 'gs1_bandwidth': 1.0}, 'primary_endpoint': 'gsm8k_test_exact_match_accuracy_final_checkpoint', 'source': 'activation-pilot-*.json, evaluation-base.json', 'measurements': {'setup': {'seconds': 179.169, 'units': 1, 'planned_units': 1, 'samples': 0, 'gpu_count': None, 'allocation_source': None}, 'training': {'seconds': 153.225, 'units': 5, 'planned_units': 85, 'samples': 320, 'gpu_count': None, 'allocation_source': None}, 'evaluation': {'seconds': 174.051, 'units': 1319, 'planned_units': 3957, 'samples': 1319, 'gpu_count': None, 'allocation_source': None}}, 'generation': None, 'recovery': {'seconds': 1200, 'gpu_count': None, 'allocation_source': None}}, 'recorded_at': 1793805774.0129788}

feasibility_reported_verdict

refuted

feasibility_required

True

generation_audit_validation

{'status': 'not_applicable', 'reasons': ['spec.generation_audit opts out and no feature it covers is true'], 'applicability': {'check': 'generation_audit', 'status': 'not_applicable', 'enforce': False, 'reasons': ['spec.generation_audit opts out and no feature it covers is true'], 'features': {'generated_training_data': False}}}

initialization_provenance

{'version': 1, 'arms': [{'name': 'grpo_gs8', 'restore_count': 1, 'optimizer_checks': [{'after_restore': 0, 'source': 'declared', 'actual': None, 'actual_config': None, 'requested': 'adamw', 'requested_config': {'lr': 4e-05, 'beta1': 0.9, 'beta2': 0.95, 'eps': 1e-08}, 'before_next_update': True, 'reason': 'Shared Spindle accepts adam_params; no concrete active optimizer/config response.'}, {'after_restore': 1, 'source': 'declared', 'actual': None, 'actual_config': None, 'requested': 'adamw', 'requested_config': {'lr': 4e-05, 'beta1': 0.9, 'beta2': 0.95, 'eps': 1e-08}, 'before_next_update': True, 'reason': 'Shared Spindle accepts adam_params; no concrete active optimizer/config response.'}], 'restores': [{'requested': 'tinker://cf264e49a4dd83e5b827d28f49b9c43a/weights/rx-gs1-init-run_5207121d26', 'with_optimizer': False, 'response': {'path': 'tinker://cf264e49a4dd83e5b827d28f49b9c43a/weights/rx-gs1-init-run_5207121d26', 'type': 'load_weights'}, 'checkpoint_step': 0}], 'initialization': {'base_checkpoint': 'Qwen/Qwen3.5-2B', 'checkpoint_uri': 'tinker://cf264e49a4dd83e5b827d28f49b9c43a/weights/rx-gs1-init-run_5207121d26', 'loaded_checkpoint_uri': 'tinker://cf264e49a4dd83e5b827d28f49b9c43a/weights/rx-gs1-init-run_5207121d26', 'load_succeeded': True, 'checkpoint_immutable': True, 'checkpoint_scope': 'full_initialization', 'source': 'server_response', 'evidence_ref': 'full/grpo_gs8/initialization.json', 'before_first_update': True}, 'initial_output': {'available': True, 'batch_sha256': '32e84b95a3a3dbc06415e0d42e7267aff3711da45d7a84cb2f9fb6a92c6b11c2', 'output_sha256': 'd0a5c1e03b499e3b055b40a9b7b75a1fb99b7c3c0e6b2884005b7d1efdd0d1fd', 'before_first_update': True, 'evidence_ref': 'full/grpo_gs8/initial-forward.json'}}, {'name': 'gs1_kernel', 'restore_count': 1, 'optimizer_checks': [{'after_restore': 0, 'source': 'declared', 'actual': None, 'actual_config': None, 'requested': 'adamw', 'requested_config': {'lr': 4e-05, 'beta1': 0.9, 'beta2': 0.95, 'eps': 1e-08}, 'before_next_update': True, 'reason': 'Shared Spindle accepts adam_params; no concrete active optimizer/config response.'}, {'after_restore': 1, 'source': 'declared', 'actual': None, 'actual_config': None, 'requested': 'adamw', 'requested_config': {'lr': 4e-05, 'beta1': 0.9, 'beta2': 0.95, 'eps': 1e-08}, 'before_next_update': True, 'reason': 'Shared Spindle accepts adam_params; no concrete active optimizer/config response.'}], 'restores': [{'requested': 'tinker://cf264e49a4dd83e5b827d28f49b9c43a/weights/rx-gs1-init-run_5207121d26', 'with_optimizer': False, 'response': {'path': 'tinker://cf264e49a4dd83e5b827d28f49b9c43a/weights/rx-gs1-init-run_5207121d26', 'type': 'load_weights'}, 'checkpoint_step': 0}], 'initialization': {'base_checkpoint': 'Qwen/Qwen3.5-2B', 'checkpoint_uri': 'tinker://cf264e49a4dd83e5b827d28f49b9c43a/weights/rx-gs1-init-run_5207121d26', 'loaded_checkpoint_uri': 'tinker://cf264e49a4dd83e5b827d28f49b9c43a/weights/rx-gs1-init-run_5207121d26', 'load_succeeded': True, 'checkpoint_immutable': True, 'checkpoint_scope': 'full_initialization', 'source': 'server_response', 'evidence_ref': 'full/gs1_kernel/initialization.json', 'before_first_update': True}, 'initial_output': {'available': True, 'batch_sha256': '32e84b95a3a3dbc06415e0d42e7267aff3711da45d7a84cb2f9fb6a92c6b11c2', 'output_sha256': 'd0a5c1e03b499e3b055b40a9b7b75a1fb99b7c3c0e6b2884005b7d1efdd0d1fd', 'before_first_update': True, 'evidence_ref': 'full/gs1_kernel/initial-forward.json'}}]}

launch

{'experiment_id': 'exp_32150a771c', 'context_status': 'snapshotted', 'superseded': False, 'mode': 'redo', 'purpose': "Repeat the selected saved experiment version to check reproducibility. This repetition tests that version's hypothesis and methods."}

launch_instructions

{'name': 'launch-instructions-d2809a6d0761.md', 'sha256': 'd2809a6d0761cc56d1d82e3698d3ab83c16c898209ce0d2891bcfa06c8336561', 'url': 'https://rx.modal.dev/api/runs/run_5207121d26/evidence/d2809a6d0761cc56d1d82e3698d3ab83c16c898209ce0d2891bcfa06c8336561'}

reported_verdict

refuted

restoration_validation

{'status': 'unverified', 'required': True, 'checks': [{'check': 'grpo_gs8.initialization', 'status': 'passed', 'reason': 'grpo_gs8: observed initial state identity recorded.'}, {'check': 'grpo_gs8.optimizer.0', 'status': 'unverified', 'reason': 'grpo_gs8: active optimizer after restore 0 is unverified; a client declaration is insufficient.'}, {'check': 'grpo_gs8.optimizer.1', 'status': 'unverified', 'reason': 'grpo_gs8: active optimizer after restore 1 is unverified; a client declaration is insufficient.'}, {'check': 'grpo_gs8.output', 'status': 'passed', 'reason': 'grpo_gs8: fixed-batch full-output fingerprint recorded.'}, {'check': 'gs1_kernel.initialization', 'status': 'passed', 'reason': 'gs1_kernel: observed initial state identity recorded.'}, {'check': 'gs1_kernel.optimizer.0', 'status': 'unverified', 'reason': 'gs1_kernel: active optimizer after restore 0 is unverified; a client declaration is insufficient.'}, {'check': 'gs1_kernel.optimizer.1', 'status': 'unverified', 'reason': 'gs1_kernel: active optimizer after restore 1 is unverified; a client declaration is insufficient.'}, {'check': 'gs1_kernel.output', 'status': 'passed', 'reason': 'gs1_kernel: fixed-batch full-output fingerprint recorded.'}], 'reason': 'grpo_gs8: active optimizer after restore 0 is unverified; a client declaration is insufficient.', 'applicability': 'required'}

slack_started

True

verdict_report

{'effect': -0.01592115238817286, 'margin': -0.03, 'lower': -0.09071051496929317, 'upper': 0.058868210192947426, 'seeds': 1, 'confidence': 'weak', 'reasoning': "GS1 kernel scored 1.6 points below GRPO (0.7650 vs 0.7809) on all 1319 paired GSM8K test items, but the paired 95% Hoeffding lower bound (-0.0907) is below the -0.03 margin, so the success criterion is not met and the hypothesis counts as not supported (the exchange's decision rule itself returned status uncertain/insufficient_evidence; the interval also includes 0). Confidence is weak: one seed, a wide Hoeffding interval at n=1319, GS1 bandwidth 1.0 came from the pre-registered calibration after 0.5 failed the neighbor gate by 0.03, shared Spindle rejects per-model seeds so matched init used one saved checkpoint (identical initial logprob hashes) and the active optimizer is declared not server-verified, and correctness revisions in the spec were placeholders (validator passes on the pinned effective spec). Requester's broader questions (FlashREINFORCE, trained critic, kernel hillclimbing/transfer, offline effective-batch hillclimb) are out of scope of this spec node and were not run; on training time alone GS1 steps were ~13% faster (25.9 s vs 29.7 s mean wall) at equal rollouts."}