| Age | Commit message (Collapse) | Author |
|---|
|
Automatic black level setting in software ISP updates the determined
black level value when exposure or gain change. It stores the last
exposure and gain values to detect the change.
BlackLevel::configure() resets the stored black level value but not the
exposure and gain values. This can prevent updating the black value and
cause bad image output, e.g. after suspending and resuming a camera, if
exposure and gain remain unchanged.
Let's store exposure and gain in IPAActiveState. Although the values
are not supposed to be used outside BlackLevel class, storing them in
the context has the advantage of their automatic reset together with the
other context contents and having them in `blc' struct indicates their
relationship to the black value computation.
Bug: https://bugs.libcamera.org/show_bug.cgi?id=259
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Robert Mader <robert.mader@collabora.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
|
|
Report exposure and gain in metadata.
This is more complicated than it could be expected because the exposure
value should be in microseconds but it's handled using V4L2_CID_EXPOSURE
control, which doesn't specify the unit, see
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/control.html.
So the unit conversion is done in the way rkisp1 IPA uses.
This requires getting and passing IPACameraSensorInfo around. To avoid
naming confusion and to improve consistency with rkisp1 IPA,
sensorCtrlInfoMap parameter is renamed to sensorControls.
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
|
|
Provide the requested contrast value, if any, in the metadata to add to
the completed requests.
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
|
|
Provide the determined black level values in the metadata
to add to the completed requests.
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
|
|
Provide the determined colour gains back into the metadata
to add to completed requests.
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
|
|
This patch applies color correction matrix (CCM) in debayering if the
CCM is specified. Not using CCM must still be supported for performance
reasons.
The CCM is applied as follows:
[r1 g1 b1] [r]
[r2 g2 b2] * [g]
[r3 g3 b3] [b]
The CCM matrix (the left side of the multiplication) is constant during
single frame processing, while the input pixel (the right side) changes.
Because each of the color channels is only 8-bit in software ISP, we can
make 9 lookup tables with 256 input values for multiplications of each
of the r_i, g_i, b_i values. This way we don't have to multiply each
pixel, we can use table lookups and additions instead. Gamma (which is
non-linear and thus cannot be a part of the 9 lookup tables values) is
applied on the final values rounded to integers using another lookup
table.
Because the changing part is the pixel value with three color elements,
only three dynamic table lookups are needed. We use three lookup tables
to represent the multiplied matrix values, each of the tables
corresponding to the given matrix column and pixel color.
We use int16_t to store the precomputed multiplications. This seems to
be noticeably (>10%) faster than `float' for the price of slightly less
accuracy and it covers the range of values that sane CCMs produce. The
selection and structure of data is performance critical, for example
using bytes would add significant (>10%) speedup but would be too short
to cover the value range.
The color lookup tables can be represented either as unions,
accommodating tables for both the CCM and non-CCM cases, or as separate
tables for each of the cases, leaving the tables for the other case
unused. The latter is selected as a matter of preference.
The tables are copied (as before), which is not elegant but also not a
big problem. There are patches posted that use shared buffers for
parameters passing in software ISP (see software ISP TODO #5) and they
can be adjusted for the new parameter format.
Color gains from white balance are supposed not to be a part of the
specified CCM. They are applied on it using matrix multiplication,
which is simple and in correspondence with future additions in the form
of matrix multiplication, like saturation adjustment.
With this patch, the reported per-frame slowdown when applying CCM is
about 45% on Debix Model A and about 75% on TI AM69 SK.
Using std::clamp in debayering adds some performance penalty (a few
percent). The clamping is necessary to eliminate out of range values
possibly produced by the CCM. If it could be avoided by adjusting the
precomputed tables some way then performance could be improved a bit.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
Applying color correction matrix (CCM) in software ISP is optional due
to performance reasons. CCM is applied if and only if `Ccm' algorithm
is present in the tuning file.
Software ISP debayering is a performance critical piece of code and we
do not want to use dynamic conditionals there. Therefore we pass
information about CCM application to debayering configuration and let it
select the right versions of debayering functions using templates. This
is a trick similar to the previously used one for adding or not adding
an alpha channel to the output.
Debayering gets this information but it ignores it in this patch.
Actual processing with CCM is added in the followup patch.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
This patch adds color correction matrix (CCM) algorithm to software ISP.
It is based on the corresponding algorithm in rkisp1.
The primary difference against hardware pipelines is that applying the
CCM is optional. Applying CCM causes a significant slowdown, time
needed to process a frame raises by 40-90% on tested platforms. If CCM
is really needed, it can be applied, if not, it's better to stick
without it. This can be configured by presence or omission of Ccm
algorithm in the tuning file.
CCM is changed only if the determined temperature changes by at least
100 K (an arbitrarily selected value), to avoid recomputing the matrices
and lookup tables all the time.
Since the CCM is float, rather than double, to use the same type as in
the rkisp1 pipeline, the type of color gains is changed from double to
float.
The outputs of the algorithm are not used yet, they will be enabled in
followup patches.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
`params' argument of Lut::prepare is actually used, let's remove
maybe_unused from it.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
Image color temperature is a piece of information that should be
reported in metadata, let's put it there.
Metadata is currently not reported in simple pipeline but we should make
at least newly added information ready to be reported.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
Rather than using a custom struct to represent RGB values, let's use the
corresponding type and its facilities.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
The AWB algorithm has data to determine color temperature of the image.
Let's compute the temperature from it and store it into the context.
This piece of information is currently unused but it will be needed in a
followup patch introducing support for color correction matrix.
Let's store the white balance related information under `awb' subsection
of the active state, as the hardware pipelines do.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
Use the proper path to include `libcamera/control_ids.h`.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
This patch introduces support for applying runtime controls to software
ISP. It enables the contrast control as the first control that can be
used.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
Software ISP is currently fully automatic and doesn't allow image
modifications by explicitly set control values. The user has no means
to make the image looking better.
This patch introduces support for contrast control, which can improve
e.g. a flat looking image. Based on the provided contrast value, it
applies a simple S-curve modification to the image.
The contrast algorithm just handles the provided values, while the
S-curve is applied in the gamma algorithm on the computed gamma curve
whenever the contrast value changes. Since the algorithm is applied
only on the lookup table already present, its overhead is negligible.
The contrast value range is 0..2 and corresponds to the whole range from
a completely flat contrast to an infinite contrast, 1.0 being the normal
value. This makes the user visible range intuitive and easy to use in
GUI sliders, while complying with Contrast control definition. There is
no unified range in the hardware pipelines, for example rkisp1 uses
0..1.993 range while rpi uses 0..10 range.
This is a preparation patch without actually providing the control
itself, which is done in the following patch.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
The black level obtained from the tuning file in software ISP is
retrieved in init (because this is the standard algorithm method with
access to tuning data) and stored into context. But the context gets
reset in configure and the black level is lost and never applied.
Let's store the black level from the tuning file into an algorithm
instance variable and put it into the context only later in configure.
This is similar to what rkisp1 IPA does with the values obtained from
the tuning file.
Fixes: 41e3d61c745153779ed5a38634d7266bffd60d61 ("libcamera: software_isp: Clear IPA context on configure and stop")
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Robert Mader <robert.mader@collabora.com>
Reviewed-by: Stanislaw Gruszka <stanislaw.gruszka@linux.intel.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
On my setup, since commit fb8ad13d ("libcamera: software_isp: Move exposure+gain
to an algorithm module"), at start camera output stays very dark for dozen
of seconds, and then later slowly gets to normal. This is because existing
sensor exposure+gain settings are not used at start. We save initial
values in frameContext but in the agc algorithm we use IPA context.
Fix the problem by using in frameContext sensor values, since we already
use those in blc algorithm and change exposure type to int32_t to
unnecessary castings.
Signed-off-by: Stanislaw Gruszka <stanislaw.gruszka@linux.intel.com>
Reviewed-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Robert Mader <robert.mader@collabora.com>
Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
This patch allows obtaining a black level from a tuning file in addition
to the camera sensor helper. If both of them define a black level, the
one from the tuning file takes precedence.
The use cases are:
- A user wants to use a different black level, for whatever reason.
- There is a sensor without known gains but with a known black level.
Because a camera sensor helper cannot be defined without specifying
gains, the only way to specify the black level is using the tuning
file. Software ISP uses its fallback gain handling in such a case.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
The black level in software ISP is unconditionally guessed from the
obtained frames. CameraSensorHelper optionally provides the black level
from camera specifications now. Let's use the value if available.
If the black level is not available from the given CameraSensorHelper
instance, it's still determined on the fly.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Tested-by: Robert Mader <robert.mader@collabora.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
A minor wording improvement suggested on refactoring review.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
The black level is likely to get updated, if ever, only after exposure
or gain changes. Don't compute its possible updates if exposure and
gain are unchanged.
It's probably not worth trying to implement something more
sophisticated. Better to spend the effort on supporting tuning files.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
This is the last step to fully convert software ISP to Algorithm-based
processing.
The newly introduced frameContext.sensor parameters are set, and the
updated code moved, before calling Algorithm::process() to have the
values up-to-date in stats processing.
Resolves software ISP TODO #10.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
It's more natural to represent color gains as floating point numbers
rather than using a particular pixel-related representation.
double is used rather than float because it's a more common floating
point type in libcamera algorithms. Otherwise there is no obvious
reason to select one over the other here.
The constructed color tables still use integer representation for
efficiency.
Black level still uses pixel (integer) values, for consistency with
other libcamera parts.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
After black level handling has been moved to an algorithm module, white
balance and the construction of color tables can be moved to algorithm
modules too.
This time, the moved code is split between stats processing and
parameter construction methods. It is also split to two algorithm
modules:
- White balance computation.
- Gamma table computation and color lookup tables construction. While
this applies the color gains computed by the white balance algorithm,
it is not directly related to white balance. And we may want to
modify the color lookup tables in future according to other parameters
than just gamma and white balance gains.
Gamma table computation and color lookup tables construction could be
split to separate algorithms too. But there is no big need for that now
so they are kept together for simplicity.
This is the only part of the software ISP algorithms that sets the
parameters so emitting setIspParams can be moved to prepare() method.
A more natural representation of the gains (and black level) would be
floating point numbers. This is not done here in order to minimize
changes in code movements. It will be addressed in a followup patch.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Acked-by: Umang Jain <umang.jain@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
The black level determination, already present as a separate class, can
be moved to the prepared Algorithm processing structure. It is the
first of the current software ISP algorithms that is called in the stats
processing sequence, which means it is also the first one that should be
moved to the new structure. Stats processing starts with calling
Algorithm-based processing so the call order of the algorithms is
retained.
Movement of this algorithm is relatively straightforward because all it
does is processing stats.
The comment about getting black level from the tuning files is dropped.
The black level will be taken from CameraSensorHelper if available and
that will be implemented in one of the followup patches.
Black level is now recomputed on each stats processing. In a future
patch, after DelayedControls are used, this will be changed to recompute
the black level only after exposure/gain changes.
The black level depends on the sensor used, the computed value can be
reused in a followup capture sessions with the same sensor. Thus it is
sufficient to (re)set the initial value in BlackLevel::init.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
We are ready to introduce algorithms now. First, let's create
algorithms. The algorithms are not called yet, calls to them will be
added in followup patches.
The maximum number of contexts is set to the same value as in hardware
pipelines.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
|
Software ISP image processing algorithms are currently defined in a
simplified way, different from other libcamera pipelines. This is not
good for several reasons:
- It makes the software ISP code harder to understand due to its
different structuring.
- Adding more algorithms may make the code harder to understand
generally.
- Mass libcamera code changes may not be easily applicable to software
ISP.
- Algorithm sharing with other pipelines is not easily possible.
This patch introduces basic software ISP IPA skeletons structured
similarly to the other pipelines. The newly added files are currently
not used or compiled and the general skeleton structures don't contain
anything particular. It is just a preparation step for a larger
refactoring and the code will be actually used and extended as needed in
followup patches.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
|
