libipa: histogram: Fix quantile() calculation for fractional results

The calculation of the frac variable is based solely on integers and
therefore results in the fractional part being either 0 or 1.

In the original code from RaspberryPi this is mitigated by casting the
nominator to a double. This works for most cases, but fails when q is
very small because of the quantization introduced by item being an
integer.

Fix both issues by doing the full calculation in double and remove the
should_fail tag.

Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

0 files changed, 0 insertions, 0 deletions