From 9780f46f64070cc40405ab678797334e251458a4 Mon Sep 17 00:00:00 2001 From: Jacopo Mondi Date: Thu, 21 Sep 2023 18:55:38 +0200 Subject: Documentation: Introduce Camera Sensor Model Introduce a documentation page about the 'camera sensor model' implemented by libcamera. The camera sensor model serves to provide to applications a reference description of the processing steps that take place in a camera sensor in order to precisely control the sensor configuration through the forthcoming SensorConfiguration class. Signed-off-by: Jacopo Mondi Reviewed-by: Naushir Patuck Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart Signed-off-by: Laurent Pinchart --- .reuse/dep5 | 6 + Documentation/binning.svg | 5053 +++++++++++++++++++++++++++++++++ Documentation/camera-sensor-model.rst | 174 ++ Documentation/index.rst | 1 + Documentation/meson.build | 1 + Documentation/sensor_model.svg | 4870 +++++++++++++++++++++++++++++++ Documentation/skipping.svg | 1720 +++++++++++ 7 files changed, 11825 insertions(+) create mode 100644 Documentation/binning.svg create mode 100644 Documentation/camera-sensor-model.rst create mode 100644 Documentation/sensor_model.svg create mode 100644 Documentation/skipping.svg diff --git a/.reuse/dep5 b/.reuse/dep5 index d283a0c9..c5ef5e01 100644 --- a/.reuse/dep5 +++ b/.reuse/dep5 @@ -3,6 +3,12 @@ Upstream-Name: libcamera Upstream-Contact: Laurent Pinchart Source: https://git.libcamera.org/libcamera/libcamera.git/ +Files: Documentation/binning.svg + Documentation/camera-sensor-model.rst + Documentation/sensor_model.svg +Copyright: Copyright 2023 Ideas On Board Oy +License: CC-BY-SA-4.0 + Files: Documentation/theme/static/search.png Copyright: 2022 Fonticons, Inc. License: CC-BY-4.0 diff --git a/Documentation/binning.svg b/Documentation/binning.svg new file mode 100644 index 00000000..c6a3b639 --- /dev/null +++ b/Documentation/binning.svg @@ -0,0 +1,5053 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Documentation/camera-sensor-model.rst b/Documentation/camera-sensor-model.rst new file mode 100644 index 00000000..58bf70e1 --- /dev/null +++ b/Documentation/camera-sensor-model.rst @@ -0,0 +1,174 @@ +.. SPDX-License-Identifier: CC-BY-SA-4.0 + +.. _camera-sensor-model: + +.. todo: Move to Doxygen-generated documentation + +The libcamera camera sensor model +================================= + +libcamera defines an abstract camera sensor model in order to provide +a description of each of the processing steps that result in image data being +sent on the media bus and that form the image stream delivered to applications. + +Applications should use the abstract camera sensor model defined here to +precisely control the operations of the camera sensor. + +The libcamera camera sensor model targets image sensors producing frames in +RAW format, delivered through a MIPI CSI-2 compliant bus implementation. + +The abstract sensor model maps libcamera components to the characteristics and +operations of an image sensor, and serves as a reference to model the libcamera +CameraSensor class and SensorConfiguration classes and operations. + +In order to control the configuration of the camera sensor through the +SensorConfiguration class, applications should understand this model and map it +to the combination of image sensor and kernel driver in use. + +The camera sensor model defined here is based on the *MIPI CCS specification*, +particularly on *Section 8.2 - Image readout* of *Chapter 8 - Video Timings*. + + +Glossary +-------- + +.. glossary:: + + Pixel array + The full grid of pixels, active and inactive ones + + Pixel array active area + The portion(s) of the pixel array that contains valid and readable pixels; + corresponds to the libcamera properties::PixelArrayActiveAreas + + Analog crop rectangle + The portion of the *pixel array active area* which is read out and passed + to further processing stages + + Subsampling + Pixel processing techniques that reduce the image size by binning or by + skipping adjacent pixels + + Digital crop + Crop of the sub-sampled image data before scaling + + Frame output + The frame (image) as output on the media bus by the camera sensor + +Camera sensor model +------------------- + +The abstract sensor model is described in the following diagram. + +.. figure:: sensor_model.svg + + +1. The sensor reads pixels from the *pixel array*. The pixels being read out are + selected by the *analog crop rectangle*. + +2. The pixels can be subsampled to reduce the image size without affecting the + field of view. Two subsampling techniques can be used: + + - Binning: combines adjacent pixels of the same colour by averaging or + summing their values, in the analog domain and/or the digital domain. + + .. figure:: binning.svg + + + - Skipping: skips the read out of a number of adjacent pixels. + + .. figure:: skipping.svg + + +3. The output of the optional sub-sampling stage is then cropped after the + conversion of the analogue pixel values in the digital domain. + +4. The resulting output frame is sent on the media bus by the sensor. + +Camera Sensor configuration parameters +-------------------------------------- + +The libcamera camera sensor model defines parameters that allow users to +control: + +1. The image format bit depth + +2. The size and position of the *Analog crop rectangle* + +3. The subsampling factors used to downscale the pixel array readout data to a + smaller frame size without reducing the image *field of view*. Two + configuration parameters are made available to control the downscaling + factor: + + - binning + A vertical and horizontal binning factor can be specified, the image + will be downscaled in its vertical and horizontal sizes by the specified + factor. + + .. code-block:: + :caption: Definition: The horizontal and vertical binning factors + + horizontal_binning = xBin; + vertical_binning = yBin; + + + - skipping + Skipping reduces the image resolution by skipping the read-out of a number + of adjacent pixels. The skipping factor is specified by the 'increment' + number (number of pixels to 'skip') in the vertical and horizontal + directions and for even and odd rows and columns. + + .. code-block:: + :caption: Definition: The horizontal and vertical skipping factors + + horizontal_skipping = (xOddInc + xEvenInc) / 2 + vertical_skipping = (yOddInc + yEvenInc) / 2 + + Different sensors perform the binning and skipping stages in different + orders. For the sake of computing the final output image size the order of + execution is not relevant. The overall down-scaling factor is obtained by + combining the binning and skipping factors. + + .. code-block:: + :caption: Definition: The total scaling factor (binning + sub-sampling) + + total_horizontal_downscale = horizontal_binning + horizontal_skipping + total_vertical_downscale = vertical_binning + vertical_skipping + + +4. The output size is used to specify any additional cropping on the sub-sampled + frame. + +5. The total line length and frame height (*visibile* pixels + *blankings*) as + sent on the MIPI CSI-2 bus. + +6. The pixel transmission rate on the MIPI CSI-2 bus. + +The above parameters are combined to obtain the following high-level +configurations: + +- **frame output size** + + Obtained by applying a crop to the physical pixel array size in the analog + domain, followed by optional binning and sub-sampling (in any order), + followed by an optional crop step in the output digital domain. + +- **frame rate** + + The combination of the *total frame size*, the image format *bit depth* and + the *pixel rate* of the data sent on the MIPI CSI-2 bus allows to compute the + image stream frame rate. The equation is the well known: + + .. code-block:: + + frame_duration = total_frame_size / pixel_rate + frame_rate = 1 / frame_duration + + + where the *pixel_rate* parameter is the result of the sensor's configuration + of the MIPI CSI-2 bus *(the following formula applies to MIPI CSI-2 when + used on MIPI D-PHY physical protocol layer only)* + + .. code-block:: + + pixel_rate = CSI-2_link_freq * 2 * nr_of_lanes / bits_per_sample diff --git a/Documentation/index.rst b/Documentation/index.rst index 43d8b017..63fac72d 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -23,3 +23,4 @@ Sensor driver requirements Lens driver requirements Python Bindings + Camera Sensor Model diff --git a/Documentation/meson.build b/Documentation/meson.build index b2a5bf15..7c150259 100644 --- a/Documentation/meson.build +++ b/Documentation/meson.build @@ -63,6 +63,7 @@ endif if sphinx.found() docs_sources = [ + 'camera-sensor-model.rst', 'coding-style.rst', 'conf.py', 'contributing.rst', diff --git a/Documentation/sensor_model.svg b/Documentation/sensor_model.svg new file mode 100644 index 00000000..02dc55a8 --- /dev/null +++ b/Documentation/sensor_model.svg @@ -0,0 +1,4870 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Pixel array + Analog crop + Subsampling + Digital crop + Media Bus + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1 + + 2 + + 4 + + 3 + + diff --git a/Documentation/skipping.svg b/Documentation/skipping.svg new file mode 100644 index 00000000..7bef37cf --- /dev/null +++ b/Documentation/skipping.svg @@ -0,0 +1,1720 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + x_even_inc = 1 + y_even_inc = 1 + 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 + x_odd_inc = 3 + y_odd_inc = 3 + 0 + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + + + + -- cgit v1.2.1