.. SPDX-License-Identifier: CC-BY-SA-4.0
.. include:: documentation-contents.rst
.. _camera-sensor-model:
.. todo: Move to Doxygen-generated documentation
The libcamera camera sensor model
=================================
libcamera defines an abstract camera sensor model in order to provide
a description of each of the processing steps that result in image data being
sent on the media bus and that form the image stream delivered to applications.
Applications should use the abstract camera sensor model defined here to
precisely control the operations of the camera sensor.
The libcamera camera sensor model targets image sensors producing frames in
RAW format, delivered through a MIPI CSI-2 compliant bus implementation.
The abstract sensor model maps libcamera components to the characteristics and
operations of an image sensor, and serves as a reference to model the libcamera
CameraSensor class and SensorConfiguration classes and operations.
In order to control the configuration of the camera sensor through the
SensorConfiguration class, applications should understand this model and map it
to the combination of image sensor and kernel driver in use.
The camera sensor model defined here is based on the *MIPI CCS specification*,
particularly on *Section 8.2 - Image readout* of *Chapter 8 - Video Timings*.
Glossary
--------
.. glossary::
Pixel array
The full grid of pixels, active and inactive ones
Pixel array active area
The portion(s) of the pixel array that contains valid and readable pixels;
corresponds to the libcamera properties::PixelArrayActiveAreas
Analog crop rectangle
The portion of the *pixel array active area* which is read out and passed
to further processing stages
Subsampling
Pixel processing techniques that reduce the image size by binning or by
skipping adjacent pixels
Digital crop
Crop of the sub-sampled image data before scaling
Frame output
The frame (image) as output on the media bus by the camera sensor
Camera sensor model
-------------------
The abstract sensor model is described in the following diagram.
.. figure:: sensor_model.svg
1. The sensor reads pixels from the *pixel array*. The pixels being read out are
selected by the *analog crop rectangle*.
2. The pixels can be subsampled to reduce the image size without affecting the
field of view. Two subsampling techniques can be used:
- Binning: combines adjacent pixels of the same colour by averaging or
summing their values, in the analog domain and/or the digital domain.
.. figure:: binning.svg
- Skipping: skips the read out of a number of adjacent pixels.
.. figure:: skipping.svg
3. The output of the optional sub-sampling stage is then cropped after the
conversion of the analogue pixel values in the digital domain.
4. The resulting output frame is sent on the media bus by the sensor.
Camera Sensor configuration parameters
--------------------------------------
The libcamera camera sensor model defines parameters that allow users to
control:
1. The image format bit depth
2. The size and position of the *Analog crop rectangle*
3. The subsampling factors used to downscale the pixel array readout data to a
smaller frame size without reducing the image *field of view*. Two
configuration parameters are made available to control the downscaling
factor:
- binning
A vertical and horizontal binning factor can be specified, the image
will be downscaled in its vertical and horizontal sizes by the specified
factor.
.. code-block:: c
:caption: Definition: The horizontal and vertical binning factors
horizontal_binning = xBin;
vertical_binning = yBin;
- skipping
Skipping reduces the image resolution by skipping the read-out of a number
of adjacent pixels. The skipping factor is specified by the 'increment'
number (number of pixels to 'skip') in the vertical and horizontal
directions and for even and odd rows and columns.
.. code-block:: c
:caption: Definition: The horizontal and vertical skipping factors
horizontal_skipping = (xOddInc + xEvenInc) / 2;
vertical_skipping = (yOddInc + yEvenInc) / 2;
Different sensors perform the binning and skipping stages in different
orders. For the sake of computing the final output image size the order of
execution is not relevant. The overall down-scaling factor is obtained by
combining the binning and skipping factors.
.. code-block:: c
:caption: Definition: The total scaling factor (binning + sub-sampling)
total_horizontal_downscale = horizontal_binning + horizontal_skipping;
total_vertical_downscale = vertical_binning + vertical_skipping;
4. The output size is used to specify any additional cropping on the sub-sampled
frame.
5. The total line length and frame height (*visibile* pixels + *blankings*) as
sent on the MIPI CSI-2 bus.
6. The pixel transmission rate on the MIPI CSI-2 bus.
The above parameters are combined to obtain the following high-level
configurations:
- **frame output size**
Obtained by applying a crop to the physical pixel array size in the analog
domain, followed by optional binning and sub-sampling (in any order),
followed by an optional crop step in the output digital domain.
- **frame rate**
The combination of the *total frame size*, the image format *bit depth* and
the *pixel rate* of the data sent on the MIPI CSI-2 bus allows to compute the
image stream frame rate. The equation is the well known:
.. code-block:: c
frame_duration = total_frame_size / pixel_rate;
frame_rate = 1 / frame_duration;
where the *pixel_rate* parameter is the result of the sensor's configuration
of the MIPI CSI-2 bus *(the following formula applies to MIPI CSI-2 when
used on MIPI D-PHY physical protocol layer only)*
.. code-block:: c
pixel_rate = csi_2_link_freq * 2 * nr_of_lanes / bits_per_sample;
>
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* Copyright (C) 2019, Google Inc.
*
* libcamera V4L2 API tests
*
* Validate the function of exporting buffers from a V4L2VideoDevice and
* the ability to import them to another V4L2VideoDevice instance.
* Ensure that the Buffers can successfully be queued and dequeued
* between both devices.
*/
#include <iostream>
#include <libcamera/buffer.h>
#include <libcamera/event_dispatcher.h>
#include <libcamera/timer.h>
#include "libcamera/internal/thread.h"
#include "v4l2_videodevice_test.h"
class BufferSharingTest : public V4L2VideoDeviceTest
{
public:
BufferSharingTest()
: V4L2VideoDeviceTest("vivid", "vivid-000-vid-cap"),
output_(nullptr), framesCaptured_(0), framesOutput_(0) {}
protected:
int init()
{
int ret = V4L2VideoDeviceTest::init();
if (ret)
return ret;
/* media_ already represents VIVID */
MediaEntity *entity = media_->getEntityByName("vivid-000-vid-out");
if (!entity)
return TestSkip;
output_ = new V4L2VideoDevice(entity);
if (!output_) {
std::cout << "Failed to create output device" << std::endl;
return TestFail;
}
ret = output_->open();
if (ret) {
std::cout << "Failed to open output device" << std::endl;
return TestFail;
}
V4L2DeviceFormat format = {};
ret = capture_->getFormat(&format);
if (ret) {
std::cout << "Failed to get capture format" << std::endl;
return TestFail;
}
format.size.width = 320;
format.size.height = 180;
ret = capture_->setFormat(&format);
if (ret) {
std::cout << "Failed to set capture format" << std::endl;
return TestFail;
}
ret = output_->setFormat(&format);
if (ret) {
std::cout << "Failed to set output format" << std::endl;
return TestFail;
}
ret = capture_->allocateBuffers(bufferCount, &buffers_);
if (ret < 0) {
std::cout << "Failed to allocate buffers" << std::endl;
return TestFail;
}
ret = output_->importBuffers(bufferCount);
if (ret < 0) {
std::cout << "Failed to import buffers" << std::endl;
return TestFail;
}
return 0;
}
void captureBufferReady(FrameBuffer *buffer)
{
const FrameMetadata &metadata = buffer->metadata();
std::cout << "Received capture buffer" << std::endl;
if (metadata.status != FrameMetadata::FrameSuccess)
return;
output_->queueBuffer(buffer);
framesCaptured_++;
}
void outputBufferReady(FrameBuffer *buffer)
{
const FrameMetadata &metadata = buffer->metadata();
std::cout << "Received output buffer" << std::endl;
if (metadata.status != FrameMetadata::FrameSuccess)
return;
capture_->queueBuffer(buffer);
framesOutput_++;
}
int run()
{
EventDispatcher *dispatcher = Thread::current()->eventDispatcher();
Timer timeout;
int ret;
capture_->bufferReady.connect(this, &BufferSharingTest::captureBufferReady);
output_->bufferReady.connect(this, &BufferSharingTest::outputBufferReady);
for (const std::unique_ptr<FrameBuffer> &buffer : buffers_) {
if (capture_->queueBuffer(buffer.get())) {
std::cout << "Failed to queue buffer" << std::endl;
return TestFail;
}
}
ret = capture_->streamOn();
if (ret) {
std::cout << "Failed to start streaming on the capture device" << std::endl;
return TestFail;
}
ret = output_->streamOn();
if (ret) {
std::cout << "Failed to start streaming on the output device" << std::endl;
return TestFail;
}
timeout.start(10000);
while (timeout.isRunning()) {
dispatcher->processEvents();
if (framesCaptured_ > 30 && framesOutput_ > 30)
break;
}
if ((framesCaptured_ < 1) || (framesOutput_ < 1)) {
std::cout << "Failed to process any frames within timeout." << std::endl;
return TestFail;
}
if ((framesCaptured_ < 30) || (framesOutput_ < 30)) {
std::cout << "Failed to process 30 frames within timeout." << std::endl;
return TestFail;
}
ret = capture_->streamOff();
if (ret) {
std::cout << "Failed to stop streaming on the capture device" << std::endl;
return TestFail;
}
ret = output_->streamOff();
if (ret) {
std::cout << "Failed to stop streaming on the output device" << std::endl;
return TestFail;
}
return TestPass;
}
void cleanup()
{
std::cout
<< "Captured " << framesCaptured_ << " frames and "
<< "output " << framesOutput_ << " frames"
<< std::endl;
output_->streamOff();
output_->releaseBuffers();
output_->close();
delete output_;
V4L2VideoDeviceTest::cleanup();
}
private:
const unsigned int bufferCount = 4;
V4L2VideoDevice *output_;
unsigned int framesCaptured_;
unsigned int framesOutput_;
};
TEST_REGISTER(BufferSharingTest);
|