diff options
Diffstat (limited to 'Documentation/guides/introduction.rst')
| -rw-r--r-- | Documentation/guides/introduction.rst | 319 |
1 files changed, 319 insertions, 0 deletions
diff --git a/Documentation/guides/introduction.rst b/Documentation/guides/introduction.rst new file mode 100644 index 00000000..f34d2cf7 --- /dev/null +++ b/Documentation/guides/introduction.rst @@ -0,0 +1,319 @@ +.. SPDX-License-Identifier: CC-BY-SA-4.0 + +Developers guide to libcamera +============================= + +The Linux kernel handles multimedia devices through the 'Linux media' subsystem +and provides a set of APIs (application programming interfaces) known +collectively as V4L2 (`Video for Linux 2`_) and the `Media Controller`_ API +which provide an interface to interact and control media devices. + +Included in this subsystem are drivers for camera sensors, CSI2 (Camera +Serial Interface) recievers, and ISPs (Image Signal Processors) + +The usage of these drivers to provide a functioning camera stack is a +responsibility that lies in userspace which is commonly implemented separately +by vendors without a common architecture or API for application developers. + +libcamera provides a complete camera stack for Linux based systems to abstract +functionality desired by camera application developers and process the +configuration of hardware and image control algorithms requried to obtain +desireable results from the camera. + +.. _Video for Linux 2: https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/v4l2.html +.. _Media Controller: https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/mediactl/media-controller.html + + +In this developers guide, we will explore the `Camera Stack`_ and how it is +can be visualised at a high level, and explore the internal `Architecture`_ of +the libcamera library with its components. The current `Platform Support`_ is +detailed, as well as an overview of the `Licensing`_ requirements of the +project. + +This introduction is followed by a walkthrough tutorial to newcomers wishing to +support a new platform with the `Pipeline Handler Writers Guide`_ and for those +looking to make use of the libcamera native API an `Application Writers Guide`_ +provides a tutorial of the key APIs exposed by libcamera. + +.. _Pipeline Handler Writers Guide: pipeline-handler.html +.. _Application Writers Guide: application-developer.html + +.. TODO: Correctly link to the other articles of the guide + +Camera Stack +------------ + +The libcamera library is implemented in userspace, and makes use of underlying +kernel drivers that directly interact with hardware. + +Applications can make use of libcamera through the native `libcamera API`_'s or +through an adaptation layer integrating libcamera into a larger framework. + +.. _libcamera API: https://www.libcamera.org/api-html/index.html + +:: + + Application Layer + / +--------------+ +--------------+ +--------------+ +--------------+ + | | Native | | Framework | | Native | | Android | + | | V4L2 | | Application | | libcamera | | Camera | + | | Application | | (gstreamer) | | Application | | Framework | + \ +--------------+ +--------------+ +--------------+ +--------------+ + + ^ ^ ^ ^ + | | | | + | | | | + v v | v + Adaptation Layer | + / +--------------+ +--------------+ | +--------------+ + | | V4L2 | | gstreamer | | | Android | + | | Compatability| | element | | | Camera | + | | (preload) | |(libcamerasrc)| | | HAL | + \ +--------------+ +--------------+ | +--------------+ + | + ^ ^ | ^ + | | | | + | | | | + v v v v + libcamera Framework + / +--------------------------------------------------------------------+ + | | | + | | libcamera | + | | | + \ +--------------------------------------------------------------------+ + + ^ ^ ^ + Userspace | | | + --------------------- | ---------------- | ---------------- | --------------- + Kernel | | | + v v v + + +-----------+ +-----------+ +-----------+ + | Media | <--> | Video | <--> | V4L2 | + | Device | | Device | | Subdev | + +-----------+ +-----------+ +-----------+ + +The camera stack comprises of four software layers. From bottom to top: + +* The kernel drivers control the camera hardware and expose a low-level + interface to userspace through the Linux kernel V4L2 family of APIs + (Media Controller API, V4L2 Video Device API and V4L2 Subdev API). + +* The libcamera framework is the core part of the stack. It handles all control + of the camera devices in its core component, libcamera, and exposes a native + C++ API to upper layers. + +* The libcamera adaptation layer is an umbrella term designating the components + that interface to libcamera in other frameworks. Notable examples are the V4L2 + compatibility layer, the gstreamer libcamera element, and the Android camera + HAL implementation based on libcamera which are provided as a part of the + libcamera project. + +* The applications and upper level frameworks are based on the libcamera + framework or libcamera adaptation, and are outside of the scope of the + libcamera project, however example native applications (cam, qcam) are + provided for testing. + + +V4L2 Compatibility Layer + V4L2 compatibility is achieved through a shared library that traps all + accesses to camera devices and routes them to libcamera to emulate high-level + V4L2 camera devices. It is injected in a process address space through + `LD_PRELOAD` and is completely transparent for applications. + + The compatibility layer exposes camera device features on a best-effort basis, + and aims for the level of features traditionally available from a UVC camera + designed for video conferencing. + +Android Camera HAL + Camera support for Android is achieved through a generic Android camera HAL + implementation on top of libcamera. The HAL implements features required by + Android and out of scope from libcamera, such as JPEG encoding support. + + This component is used to provide support for ChromeOS platforms + +GStreamer element (gstlibcamerasrc) + A `GStreamer element`_ is provided to allow capture from libcamera supported + devices through GStreamer pipelines, and connect to other elements for further + processing. + + Development of this element is ongoing and is limited to a single stream. + +Native libcamera API + Applications can make use of the libcamera API directly using the C++ + API. An example application and walkthrough using the libcamera API can be + followed in the `Application Writers Guide`_ + +.. _GStreamer element: https://gstreamer.freedesktop.org/documentation/application-development/basics/elements.html + +Architecture +------------ + +While offering a unified API towards upper layers, and presenting itself as a +single library, libcamera isn’t monolithic. It exposes multiple components +through its public API and is built around a set of separate helpers internally. +Hardware abstractractions are handled through the use of device-specific +components where required and dynamically loadable plugins are used to separate +image processing algorithms from the core libcamera codebase. + +:: + + --------------------------< libcamera Public API >--------------------------- + ^ ^ + | | + v v + +-------------+ +---------------------------------------------------+ + | Camera | | Camera Device | + | Manager | | +-----------------------------------------------+ | + +-------------+ | | Device-Agnostic | | + ^ | | | | + | | | +--------------------------+ | + | | | | ~~~~~~~~~~~~~~~~~~~~~~~ | + | | | | { +-----------------+ } | + | | | | } | //// Image //// | { | + | | | | <-> | / Processing // | } | + | | | | } | / Algorithms // | { | + | | | | { +-----------------+ } | + | | | | ~~~~~~~~~~~~~~~~~~~~~~~ | + | | | | ========================== | + | | | | +-----------------+ | + | | | | | // Pipeline /// | | + | | | | <-> | /// Handler /// | | + | | | | | /////////////// | | + | | +--------------------+ +-----------------+ | + | | Device-Specific | + | +---------------------------------------------------+ + | ^ ^ + | | | + v v v + +--------------------------------------------------------------------+ + | Helpers and Support Classes | + | +-------------+ +-------------+ +-------------+ +-------------+ | + | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | | + | | Support | | Allocator | | IPC | | Manager | | + | +-------------+ +-------------+ +-------------+ +-------------+ | + | +-------------+ +-------------+ | + | | Pipeline | | ... | | + | | Runner | | | | + | +-------------+ +-------------+ | + +--------------------------------------------------------------------+ + + /// Device-Specific Components + ~~~ Sandboxing + + +Camera Manager + The Camera Manager enumerates cameras and instantiates Pipeline Handlers to + manage each Camera that libcamera supports. The Camera Manager supports + hotplug detection and notification events when supported by the underlying + kernel devices. + + There is only ever one instance of the Camera Manager running per application. + Each application's instance of the Camera Manager ensures that only a single + application can take control of a camera device at once. + + Read the `Camera Manager API`_ documentation for more details. + +.. _Camera Manager API: http://libcamera.org/api-html/classlibcamera_1_1CameraManager.html + +Camera Device + The Camera class represents a single item of camera hardware that is capable + of producing one or more image streams, and provides the API to interact with + the underlying device. + + If a system has multiple instances of the same hardware attached, each has it's + own instance of the camera class. + + The API exposes full control of the device to upper layers of libcamera through + the public API, making it the highest level object libcamera exposes, and the + object that all other API operations interact with from configuration to + capture. + + Read the `Camera API`_ documentation for more details. + +.. _Camera API: http://libcamera.org/api-html/classlibcamera_1_1Camera.html + +Pipeline Handler + The Pipeline Handler manages the complex pipelines exposed by the kernel + drivers through the Media Controller and V4L2 APIs. It abstracts pipeline + handling to hide device-specific details from the rest of the library, and + implements both pipeline configuration based on stream configuration, and + pipeline runtime execution and scheduling when needed by the device. + + The Pipeline Handler lives in the same process as the rest of the library, and + has access to all helpers and kernel camera-related devices. + + Hardware abstraction is handled by device specific Pipeline Handlers which are + derived from the Pipeline Handler base class allowing commonality to be shared + among the implementations. + + Derived pipeline handlers create Camera device instances based on the devices + they detect and support on the running system, and are responsible for + managing the interactions with a camera device. + + More details can be found in the `PipelineHandler API`_ documentation, and the + `Pipeline Handler Writers Guide`_. + +.. _PipelineHandler API: http://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html + +Image Processing Algorithms + An image processing algorithm (IPA) component is a loadable plugin that + implements 3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) and other + algorithms. + + The algorithms run on the CPU and interact with the camera devices through the + Pipeline Handler to control hardware image processing based on the parameters + supplied by upper layers, maintaining state and closing the control loop + of the ISP. + + The component is sandboxed and can only interact with libcamera through the + API provided by the Pipeline Handler and an IPA has no direct access to kernel + camera devices. + + Open source IPA modules built with libcamera can be run in the same process + space as libcamera, however external IPA modules are run in a separate process + from the main libcamera process. IPA modules have a restricted view of the + system, including no access to networking APIs and limited access to file + systems. + + IPA modules are only required for platforms and devices with an ISP controlled + by the host CPU. Camera sensors which have an integrated ISP are not + controlled through the IPA module. + +Platform Support +---------------- + +The library currently supports the following hardware platforms specifically +with dedicated pipeline handlers: + + - Intel IPU3 (ipu3) + - Rockchip RK3399 (rkisp1) + - RaspberryPi 3 and 4 (raspberrypi) + +Furthermore, generic platform support is provided for the following: + + - USB video device class cameras (uvcvideo) + - iMX7, Allwinner Sun6i (simple) + - Virtual media controller driver for test use cases (vimc) + +Licensing +--------- + +The libcamera core, is covered by the `LGPL-2.1-or-later`_ license. Pipeline +Handlers are a part of the libcamera code base and need to be contributed +upstream by device vendors. IPA modules included in libcamera are covered by a +free software license, however third-parties may develop IPA modules outside of +libcamera and distribute them under a closed-source license, provided they do +not include source code from the libcamera project. + +The libcamera project itself contains multiple libraries, applications and +utilities. Licenses are expressed through SPDX tags in text-based files that +support comments, and through the .reuse/dep5 file otherwise. A copy of all +licenses are stored in the LICENSES directory, and a full summary of the +licensing used throughout the project can be found in the COPYING.rst document. + +Applications which link dynamically against libcamera and use only the public +API are an independent work of the authors and have no license restrictions +imposed upon them from libcamera. + +.. _LGPL-2.1-or-later: https://spdx.org/licenses/LGPL-2.1-or-later.html
\ No newline at end of file |