benchmark: fix calibrate-n option handling#64146
Conversation
|
Review requested:
|
Fix broken parsing for --cv-threshold and --max-increases so the value after the equals sign is used. Use the configured CV threshold when checking overall stability and report the configured thresholds in calibrate-n output. Also handle invalid numeric values for --runs and --cv-threshold consistently with the other numeric options. Signed-off-by: Luan Muniz <luan@luanmuniz.com.br>
|
@RafaelGSS Thanks for the review! I can see the CI is failing due to an issue with the commit message. I noticed the same problem on my other open PR's, and I already fixed them there by amend+force-push-with-lease. I reread the guidelines to make sure, and they say that for this type of issue, someone would help get things sorted here. I will await your instructions on the correct flow to fix the CI here. |
|
@luanmuniz the commit message line should be <= 72 columns. Please commit amend + force push |
ae12dfc to
edb29d4
Compare
|
@RafaelGSS Done |
|
@RafaelGSS Sorry to keep pinging, im waiting the existing PR's to be closed to continue working on the other benchmarks, i don't want to have more PR's open at the same time. Let me know if there is anything i can do to help you move this and the other tickets i have open. Thank you! |
This fixes a few issues in
benchmark/calibrate-n.jsfound while using the tool to calibrate new benchmark values.Before this change,
--cv-thresholdand--max-increaseswere parsed from the wrong substring index. The parsed value still included the=separator, so values such as--cv-threshold=0.035and--max-increases=1were read as=0.035and=1. This made--cv-thresholdbecomeNaNand caused--max-increasesto fall back to the default value.This also updates the stability checks and output so the configured threshold is used consistently. Previously, parts of the tool still compared the overall CV against the default
CV_THRESHOLDand printed hardcoded5%/10%thresholds even when custom values were provided.Changes included:
--cv-threshold--max-increasescvThresholdwhen checking overall stability--runsand--cv-thresholdvalues consistently with the other numeric optionsnvalue and the per-config CV threshold--start-nhelp text to match the actual already-in-place default value