diff options
| author | Stefan Klug <stefan.klug@ideasonboard.com> | 2025-04-01 14:36:11 +0200 |
|---|---|---|
| committer | Stefan Klug <stefan.klug@ideasonboard.com> | 2025-04-03 12:36:07 +0200 |
| commit | 8936e81e3f2f160966f59e1d19103e95098db1da (patch) | |
| tree | 501adec56069c78e18f87038fb998168c8e1d2f5 /src | |
| parent | 781e2f4d0c3bfaf0d53a76b0a0f08aa5c742c7a0 (diff) | |
libipa: histogram: Fix quantile() calculation for fractional results
The calculation of the frac variable is based solely on integers and
therefore results in the fractional part being either 0 or 1.
In the original code from RaspberryPi this is mitigated by casting the
nominator to a double. This works for most cases, but fails when q is
very small because of the quantization introduced by item being an
integer.
Fix both issues by doing the full calculation in double and remove the
should_fail tag.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Diffstat (limited to 'src')
| -rw-r--r-- | src/ipa/libipa/histogram.cpp | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/src/ipa/libipa/histogram.cpp b/src/ipa/libipa/histogram.cpp index 10e44b54..ea042f0a 100644 --- a/src/ipa/libipa/histogram.cpp +++ b/src/ipa/libipa/histogram.cpp @@ -130,7 +130,8 @@ double Histogram::quantile(double q, uint32_t first, uint32_t last) const if (cumulative_[first + 1] == cumulative_[first]) frac = 0; else - frac = (item - cumulative_[first]) / (cumulative_[first + 1] - cumulative_[first]); + frac = (q * total() - cumulative_[first]) + / (cumulative_[first + 1] - cumulative_[first]); return first + frac; } |