MakazhanAlpamys/Soup

soup deploy measure has its own copy of the OK / MINOR / MAJOR ladder (measure_candidate in src/soup_cli/utils/deploy_measure.py, around lines 301-310) and compares the raw float drop with the thresholds, the same defect #1740 fixed in eval quant-check (#1743). An exact five-point drop computed from task counts, 19/20 to 18/20, gives delta = -0.04999999999999993 and is classified MINOR instead of MAJOR. Found while reviewing #1743. Expected The same rule as #1740: round the drop before comparing it with the thresholds (round(-delta, 9) or an equivalent boundary check), at both the minor and the major boundary, with thresholds, scoring and output unchanged. Better still, call the shared classify_delta from eval/quant_check.py instead of keeping a second copy of the ladder, if the thresholds and labels are the same; say in the PR which you chose and why. Acceptance criteria 19/20 -> 18/20 is MAJOR and an exact two-point drop is MINOR through measure_candidate (and through soup deploy m

Two guards in BufferedCanaryOutcomes (src/soup_cli/utils/canary_router.py, from #1712) have no test that fails when they are removed. Both behaviours are documented, so each needs a pin; the two tests below pass on the merged tree and each fails its mutation (checked during the #1712 review on head 4f7b709f). 1. The accept-time rollout check in record What it does. record() checks _is_current(rollout_id) when an outcome arrives and drops a late completion of a superseded rollout on the spot. Without it the sample is buffered and only dropped by the flush-time identity check, which also forces one early flush of the current batch. Mutation that survives. Remove the accept-time _is_current check in record (canary_router.py:385): the 30 tests in tests/test_issue1320_canary_io.py and tests/test_issue815_canary_runtime.py still pass, because the flush-time check keeps the counts right. 2. The pre-record flush that refuses outcomes during an outage What it does. Before buffering a new outcom

What happens today When soup train fails on a Lambda instance, the controller reports the exit code and terminates the instance, and the real error goes with it. A non-zero /home/ubuntu/soup.exit raises remote soup train exited N (cloud/lambda_labs.py:262-263), main() prints Lambda training failed: ... (:316-317), and the finally block terminates the instance (:318-326). The controller never reads remote output: its only ssh commands are the status poll (:217) and the checkpoint scp (:271), and the user_data script (:100-118) keeps soup train output nowhere the controller reads. #1584 changes the config the stub embeds and does not touch the controller. A missing data file was the case in #1430, but the same blind spot hides an out-of-memory error or a failed pip install: the user sees exited 1, and finding out why costs another boot. Modal is not affected: train() does not capture soup train output (cloud/modal.py:179-183), so it is in Modal's own logs. Expected Before terminating, th

What happens today soup train --cloud lambda|modal ships the config and nothing else. commands/train.py:996-1000 passes plan_func the config path, and cloud/lambda_labs.py:351-352 and cloud/modal.py:267-268 re-open it. The remote machine gets soup.yaml and no data: Lambda runs soup train in /home/ubuntu/soup (lambda_labs.py:115-117), Modal in /outputs/<run> (modal.py:174-183). soup init defaults to ./data/train.jsonl (commands/init.py:187) and the templates use local paths too, so the default flow cannot work. #1584 does not change that. It embeds the effective config and adds cloud/_shipped.py, whose refuse_unshipped_inputs exits 2 at plan time, naming each local-file field and suggesting a Hub dataset id. The default flow now stops cleanly before the paid boot, but still cannot use local data. Expected A first change ships local data.train files on both backends, and the refusal in _shipped.py narrows to what is still not shipped. Image/audio/video directories, local model paths, rew

Description #983 refused use_flash_attn / use_liger outside {sft, tts}, and #1587 does the same for reasoning_effort / train_on_eot outside {sft, distill, tts}. Both gates look at the task name only. task: sft is several code paths, and on most of them these four fields are read by nothing. The config loads, the run trains, and nothing says the setting was dropped. This is the within-sft half of #806, and the same split that #1581 makes for freeze_layers. I raised it in my review of #1587 (2026-10-05) and asked for it as its own issue. Who reads each field on main (fe3ca445): reasoning_effort and train_on_eot: build_format_row (data/sft_format.py:65-70), called from SFTTrainerWrapper.setup (trainer/sft.py:899, only on the last branch of the dataset dispatch at :876-899) and from trainer/distill.py:776. train_on_eot has one more reader: preprocess_mask_mode (utils/data_pipeline.py:357-358), which puts +eot into the cache key that soup data preprocess writes (commands/data.py:2563, :263