libcamera: software_isp: Apply black level compensation

Black may not be represented as 0 pixel value for given hardware, it may
be higher.  If this is not compensated then various problems may occur
such as low contrast or suboptimal exposure.

The black pixel value can be either retrieved from a tuning file for the
given hardware, or automatically on the fly.  The former is the right
and correct method, while the latter can be used when a tuning file is
not available for the given hardware.  Since there is currently no
support for tuning files in software ISP, the automatic, hardware
independent way, is always used.  Support for tuning files should be
added in future but it will require more work than this patch.

The patch looks at the image histogram and assumes that black starts
when pixel values start occurring on the left.  A certain amount of the
darkest pixels is ignored; it doesn't matter whether they represent
various kinds of noise or are real, they are better to omit in any case
to make the image looking better.  It also doesn't matter whether the
darkest pixels occur around the supposed black level or are spread
between 0 and the black level, the difference is not important.

An arbitrary threshold of 2% darkest pixels is applied; there is no
magic about that value.

The patch assumes that the black values for different colors are the
same and doesn't attempt any other non-primitive enhancements.  It
cannot completely replace tuning files and simplicity, while providing
visible benefit, is its goal.  Anything more sophisticated is left for
future patches.

A possible cheap enhancement, if needed, could be setting exposure +
gain to minimum values temporarily, before setting the black level.  In
theory, the black level should be fixed but it may not be reached in all
images.  For this reason, the patch updates black level only if the
observed value is lower than the current one; it should be never
increased.

The purpose of the patch is to compensate for hardware properties.
General image contrast enhancements are out of scope of this patch.

Stats are still gathered as an uncorrected histogram, to avoid any
confusion and to represent the raw image data.  Exposure must be
determined after the black level correction -- it has no influence on
the sub-black area and must be correct after applying the black level
correction.  The granularity of the histogram is increased from 16 to 64
to provide a better precision (there is no theory behind either of those
numbers).

Reviewed-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>

4 files changed, 21 insertions, 5 deletions

diff --git a/src/libcamera/software_isp/TODO b/src/libcamera/software_isp/TODO
index fcb02588..4fcee39b 100644
--- a/src/libcamera/software_isp/TODO
+++ b/src/libcamera/software_isp/TODO
@@ -267,3 +267,13 @@ This could be handled better with DelayedControls.
 > 						    V4L2_CID_EXPOSURE }));
 
 You should use the DelayedControls class.
+
+---
+
+13. Improve black level and colour gains application
+
+I think the black level should eventually be moved before debayering, and
+ideally the colour gains as well. I understand the need for optimizations to
+lower the CPU consumption, but at the same time I don't feel comfortable
+building up on top of an implementation that may work a bit more by chance than
+by correctness, as that's not very maintainable.
diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp
index 5b553162..88d6578b 100644
--- a/src/libcamera/software_isp/debayer_cpu.cpp
+++ b/src/libcamera/software_isp/debayer_cpu.cpp
@@ -35,7 +35,7 @@ namespace libcamera {
  * \param[in] stats Pointer to the stats object to use
  */
 DebayerCpu::DebayerCpu(std::unique_ptr<SwStatsCpu> stats)
-	: stats_(std::move(stats)), gammaCorrection_(1.0)
+	: stats_(std::move(stats)), gammaCorrection_(1.0), blackLevel_(0)
 {
 	/*
 	 * Reading from uncached buffers may be very slow.
@@ -699,11 +699,16 @@ void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams
 	}
 
 	/* Apply DebayerParams */
-	if (params.gamma != gammaCorrection_) {
-		for (unsigned int i = 0; i < kGammaLookupSize; i++)
-			gamma_[i] = UINT8_MAX * powf(i / (kGammaLookupSize - 1.0), params.gamma);
+	if (params.gamma != gammaCorrection_ || params.blackLevel != blackLevel_) {
+		const unsigned int blackIndex =
+			params.blackLevel * kGammaLookupSize / 256;
+		std::fill(gamma_.begin(), gamma_.begin() + blackIndex, 0);
+		const float divisor = kGammaLookupSize - blackIndex - 1.0;
+		for (unsigned int i = blackIndex; i < kGammaLookupSize; i++)
+			gamma_[i] = UINT8_MAX * powf((i - blackIndex) / divisor, params.gamma);
 
 		gammaCorrection_ = params.gamma;
+		blackLevel_ = params.blackLevel;
 	}
 
 	if (swapRedBlueGains_)
diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h
index e7a8ba74..689c1075 100644
--- a/src/libcamera/software_isp/debayer_cpu.h
+++ b/src/libcamera/software_isp/debayer_cpu.h
@@ -147,6 +147,7 @@ private:
 	bool enableInputMemcpy_;
 	bool swapRedBlueGains_;
 	float gammaCorrection_;
+	unsigned int blackLevel_;
 	unsigned int measuredFrames_;
 	int64_t frameProcessTime_;
 	/* Skip 30 frames for things to stabilize then measure 30 frames */
diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp
index d746d893..e4e56086 100644
--- a/src/libcamera/software_isp/software_isp.cpp
+++ b/src/libcamera/software_isp/software_isp.cpp
@@ -64,7 +64,7 @@ LOG_DEFINE_CATEGORY(SoftwareIsp)
  */
 SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor)
 	: debayerParams_{ DebayerParams::kGain10, DebayerParams::kGain10,
-			  DebayerParams::kGain10, 0.5f },
+			  DebayerParams::kGain10, 0.5f, 0 },
 	  dmaHeap_(DmaHeap::DmaHeapFlag::Cma | DmaHeap::DmaHeapFlag::System)
 {
 	if (!dmaHeap_.isValid()) {