diff options
Diffstat (limited to 'Documentation/docs.rst')
| -rw-r--r-- | Documentation/docs.rst | 400 |
1 files changed, 0 insertions, 400 deletions
diff --git a/Documentation/docs.rst b/Documentation/docs.rst deleted file mode 100644 index a6e8a59a..00000000 --- a/Documentation/docs.rst +++ /dev/null @@ -1,400 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-SA-4.0 - -.. contents:: - :local: - -************* -Documentation -************* - -.. toctree:: - :hidden: - - API <api-html/index> - -API -=== - -The libcamera API is extensively documented using Doxygen. The :ref:`API -nightly build <api>` contains the most up-to-date API documentation, built from -the latest master branch. - -Feature Requirements -==================== - -Device enumeration ------------------- - -The library shall support enumerating all camera devices available in the -system, including both fixed cameras and hotpluggable cameras. It shall -support cameras plugged and unplugged after the initialization of the -library, and shall offer a mechanism to notify applications of camera plug -and unplug. - -The following types of cameras shall be supported: - -* Internal cameras designed for point-and-shoot still image and video - capture usage, either controlled directly by the CPU, or exposed through - an internal USB bus as a UVC device. - -* External UVC cameras designed for video conferencing usage. - -Other types of camera, including analog cameras, depth cameras, thermal -cameras, external digital picture or movie cameras, are out of scope for -this project. - -A hardware device that includes independent camera sensors, such as front -and back sensors in a phone, shall be considered as multiple camera devices -for the purpose of this library. - -Independent Camera Devices --------------------------- - -When multiple cameras are present in the system and are able to operate -independently from each other, the library shall expose them as multiple -camera devices and support parallel operation without any additional usage -restriction apart from the limitations inherent to the hardware (such as -memory bandwidth, CPU usage or number of CSI-2 receivers for instance). - -Independent processes shall be able to use independent cameras devices -without interfering with each other. A single camera device shall be -usable by a single process at a time. - -Multiple streams support ------------------------- - -The library shall support multiple video streams running in parallel -for each camera device, within the limits imposed by the system. - -Per frame controls ------------------- - -The library shall support controlling capture parameters for each stream -on a per-frame basis, on a best effort basis based on the capabilities of the -hardware and underlying software stack (including kernel drivers and -firmware). It shall apply capture parameters to the frame they target, and -report the value of the parameters that have effectively been used for each -captured frame. - -When a camera device supports multiple streams, the library shall allow both -control of each stream independently, and control of multiple streams -together. Streams that are controlled together shall be synchronized. No -synchronization is required for streams controlled independently. - -Capability Enumeration ----------------------- - -The library shall expose capabilities of each camera device in a way that -allows applications to discover those capabilities dynamically. Applications -shall be allowed to cache capabilities for as long as they are using the -library. If capabilities can change at runtime, the library shall offer a -mechanism to notify applications of such changes. Applications shall not -cache capabilities in long term storage between runs. - -Capabilities shall be discovered dynamically at runtime from the device when -possible, and may come, in part or in full, from platform configuration -data. - -Device Profiles ---------------- - -The library may define different camera device profiles, each with a minimum -set of required capabilities. Applications may use those profiles to quickly -determine the level of features exposed by a device without parsing the full -list of capabilities. Camera devices may implement additional capabilities -on top of the minimum required set for the profile they expose. - -3A and Image Enhancement Algorithms ------------------------------------ - -The camera devices shall implement auto exposure, auto gain and auto white -balance. Camera devices that include a focus lens shall implement auto -focus. Additional image enhancement algorithms, such as noise reduction or -video stabilization, may be implemented. - -All algorithms may be implemented in hardware or firmware outside of the -library, or in software in the library. They shall all be controllable by -applications. - -The library shall be architectured to isolate the 3A and image enhancement -algorithms in a component with a documented API, respectively called the 3A -component and the 3A API. The 3A API shall be stable, and shall allow both -open-source and closed-source implementations of the 3A component. - -The library may include statically-linked open-source 3A components, and -shall support dynamically-linked open-source and closed-source 3A -components. - -Closed-source 3A Component Sandboxing -------------------------------------- - -For security purposes, it may be desired to run closed-source 3A components -in a separate process. The 3A API would in such a case be transported over -IPC. The 3A API shall make it possible to use any IPC mechanism that -supports passing file descriptors. - -The library may implement an IPC mechanism, and shall support third-party -platform-specific IPC mechanisms through the implementation of a -platform-specific 3A API wrapper. No modification to the library shall be -needed to use such third-party IPC mechanisms. - -The 3A component shall not directly access any device node on the system. -Such accesses shall instead be performed through the 3A API. The library -shall validate all accesses and restrict them to what is absolutely required -by 3A components. - -V4L2 Compatibility Layer ------------------------- - -The project shall support traditional V4L2 application through an additional -libcamera wrapper library. The wrapper library shall trap all accesses to -camera devices through `LD_PRELOAD`, and route them through libcamera to -emulate a high-level V4L2 camera device. It shall expose camera device -features on a best-effort basis, and aim for the level of features -traditionally available from a UVC camera designed for video conferencing. - -Android Camera HAL v3 Compatibility ------------------------------------ - -The library API shall expose all the features required to implement an -Android Camera HAL v3 on top of libcamera. Some features of the HAL may be -omitted as long as they can be implemented separately in the HAL, such as -JPEG encoding, or YUV reprocessing. - - -Camera Stack -============ - -:: - - a c / +-------------+ +-------------+ +-------------+ +-------------+ - p a | | Native | | Framework | | Native | | Android | - p t | | V4L2 | | Application | | libcamera | | Camera | - l i | | Application | | (gstreamer) | | Application | | Framework | - i o \ +-------------+ +-------------+ +-------------+ +-------------+ - n ^ ^ ^ ^ - | | | | - l a | | | | - i d v v | v - b a / +-------------+ +-------------+ | +-------------+ - c p | | V4L2 | | Camera | | | Android | - a t | | Compat. | | Framework | | | Camera | - m a | | | | (gstreamer) | | | HAL | - e t \ +-------------+ +-------------+ | +-------------+ - r i ^ ^ | ^ - a o | | | | - n | | | | - / | ,................................................ - | | ! : Language : ! - l f | | ! : Bindings : ! - i r | | ! : (optional) : ! - b a | | \...............................................' - c m | | | | | - a e | | | | | - m w | v v v v - e o | +----------------------------------------------------------------+ - r r | | | - a k | | libcamera | - | | | - \ +----------------------------------------------------------------+ - ^ ^ ^ - Userspace | | | - ------------------------ | ---------------- | ---------------- | --------------- - Kernel | | | - v v v - +-----------+ +-----------+ +-----------+ - | Media | <--> | Video | <--> | V4L2 | - | Device | | Device | | Subdev | - +-----------+ +-----------+ +-----------+ - -The camera stack comprises four software layers. From bottom to top: - -* The kernel drivers control the camera hardware and expose a - low-level interface to userspace through the Linux kernel V4L2 - family of APIs (Media Controller API, V4L2 Video Device API and - V4L2 Subdev API). - -* The libcamera framework is the core part of the stack. It - handles all control of the camera devices in its core component, - libcamera, and exposes a native C++ API to upper layers. Optional - language bindings allow interfacing to libcamera from other - programming languages. - - Those components live in the same source code repository and - all together constitute the libcamera framework. - -* The libcamera adaptation is an umbrella term designating the - components that interface to libcamera in other frameworks. - Notable examples are a V4L2 compatibility layer, a gstreamer - libcamera element, and an Android camera HAL implementation based - on libcamera. - - Those components can live in the libcamera project source code - in separate repositories, or move to their respective project's - repository (for instance the gstreamer libcamera element). - -* The applications and upper level frameworks are based on the - libcamera framework or libcamera adaptation, and are outside of - the scope of the libcamera project. - - -libcamera Architecture -====================== - -:: - - ---------------------------< libcamera Public API >--------------------------- - ^ ^ - | | - v v - +-------------+ +-------------------------------------------------+ - | Camera | | Camera Device | - | Devices | | +---------------------------------------------+ | - | Manager | | | Device-Agnostic | | - +-------------+ | | | | - ^ | | +------------------------+ | - | | | | ~~~~~~~~~~~~~~~~~~~~~ | - | | | | { +---------------+ } | - | | | | } | ////Image//// | { | - | | | | <-> | /Processing// | } | - | | | | } | /Algorithms// | { | - | | | | { +---------------+ } | - | | | | ~~~~~~~~~~~~~~~~~~~~~ | - | | | | ======================== | - | | | | +---------------+ | - | | | | | //Pipeline/// | | - | | | | <-> | ///Handler/// | | - | | | | | ///////////// | | - | | +--------------------+ +---------------+ | - | | Device-Specific | - | +-------------------------------------------------+ - | ^ ^ - | | | - v v v - +--------------------------------------------------------------------+ - | Helpers and Support Classes | - | +-------------+ +-------------+ +-------------+ +-------------+ | - | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | | - | | Support | | Allocator | | IPC | | Manager | | - | +-------------+ +-------------+ +-------------+ +-------------+ | - | +-------------+ +-------------+ | - | | Pipeline | | ... | | - | | Runner | | | | - | +-------------+ +-------------+ | - +--------------------------------------------------------------------+ - - /// Device-Specific Components - ~~~ Sandboxing - -While offering a unified API towards upper layers, and presenting -itself as a single library, libcamera isn't monolithic. It exposes -multiple components through its public API, is built around a set of -separate helpers internally, uses device-specific components and can -load dynamic plugins. - -Camera Devices Manager - The Camera Devices Manager provides a view of available cameras - in the system. It performs cold enumeration and runtime camera - management, and supports a hotplug notification mechanism in its - public API. - - To avoid the cost associated with cold enumeration of all devices - at application start, and to arbitrate concurrent access to camera - devices, the Camera Devices Manager could later be split to a - separate service, possibly with integration in platform-specific - device management. - -Camera Device - The Camera Device represents a camera device to upper layers. It - exposes full control of the device through the public API, and is - thus the highest level object exposed by libcamera. - - Camera Device instances are created by the Camera Devices - Manager. An optional function to create new instances could be exposed - through the public API to speed up initialization when the upper - layer knows how to directly address camera devices present in the - system. - -Pipeline Handler - The Pipeline Handler manages complex pipelines exposed by the kernel drivers - through the Media Controller and V4L2 APIs. It abstracts pipeline handling to - hide device-specific details to the rest of the library, and implements both - pipeline configuration based on stream configuration, and pipeline runtime - execution and scheduling when needed by the device. - - This component is device-specific and is part of the libcamera code base. As - such it is covered by the same free software license as the rest of libcamera - and needs to be contributed upstream by device vendors. The Pipeline Handler - lives in the same process as the rest of the library, and has access to all - helpers and kernel camera-related devices. - -Image Processing Algorithms - Together with the hardware image processing and hardware statistics - collection, the Image Processing Algorithms implement 3A (Auto-Exposure, - Auto-White Balance and Auto-Focus) and other algorithms. They run on the CPU - and interact with the kernel camera devices to control hardware image - processing based on the parameters supplied by upper layers, closing the - control loop of the ISP. - - This component is device-specific and is loaded as an external plugin. It can - be part of the libcamera code base, in which case it is covered by the same - license, or provided externally as an open-source or closed-source component. - - The component is sandboxed and can only interact with libcamera through - internal APIs specifically marked as such. In particular it will have no - direct access to kernel camera devices, and all its accesses to image and - metadata will be mediated by dmabuf instances explicitly passed to the - component. The component must be prepared to run in a process separate from - the main libcamera process, and to have a very restricted view of the system, - including no access to networking APIs and limited access to file systems. - - The sandboxing mechanism isn't defined by libcamera. One example - implementation will be provided as part of the project, and platforms vendors - will be able to provide their own sandboxing mechanism as a plugin. - - libcamera should provide a basic implementation of Image Processing - Algorithms, to serve as a reference for the internal API. Device vendors are - expected to provide a full-fledged implementation compatible with their - Pipeline Handler. One goal of the libcamera project is to create an - environment in which the community will be able to compete with the - closed-source vendor binaries and develop a high quality open source - implementation. - -Helpers and Support Classes - While Pipeline Handlers are device-specific, implementations are expected to - share code due to usage of identical APIs towards the kernel camera drivers - and the Image Processing Algorithms. This includes without limitation handling - of the MC and V4L2 APIs, buffer management through dmabuf, and pipeline - discovery, configuration and scheduling. Such code will be factored out to - helpers when applicable. - - Other parts of libcamera will also benefit from factoring code out to - self-contained support classes, even if such code is present only once in the - code base, in order to keep the source code clean and easy to read. This - should be the case for instance for plugin management. - - -V4L2 Compatibility Layer ------------------------- - -V4L2 compatibility is achieved through a shared library that traps all -accesses to camera devices and routes them to libcamera to emulate high-level -V4L2 camera devices. It is injected in a process address space through -`LD_PRELOAD` and is completely transparent for applications. - -The compatibility layer exposes camera device features on a best-effort basis, -and aims for the level of features traditionally available from a UVC camera -designed for video conferencing. - - -Android Camera HAL ------------------- - -Camera support for Android is achieved through a generic Android -camera HAL implementation on top of libcamera. The HAL will implement internally -features required by Android and missing from libcamera, such as JPEG encoding -support. - -The Android camera HAL implementation will initially target the -LIMITED hardware level, with support for the FULL level then being gradually -implemented. |