Documentation: Add architecture documentation

The documentation is copied mostly verbatim from the website, with small
modifications to the ascii art diagrams to make them compile.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Acked-by: Jacopo Mondi <jacopo@jmondi.org>
Acked-by: Kieran Bingham <kieran.bingham@ideasonboard.com>

1 files changed, 383 insertions, 0 deletions

diff --git a/Documentation/docs.rst b/Documentation/docs.rst
new file mode 100644
index 00000000..4bbcd9aa
--- /dev/null
+++ b/Documentation/docs.rst
@@ -0,0 +1,383 @@
+*************
+Documentation
+*************
+
+Feature Requirements
+====================
+
+Device enumeration
+------------------
+
+The library shall support enumerating all camera devices available in the
+system, including both fixed cameras and hotpluggable cameras. It shall
+support cameras plugged and unplugged after the initialization of the
+library, and shall offer a mechanism to notify applications of camera plug
+and unplug.
+
+The following types of cameras shall be supported:
+
+* Internal cameras designed for point-and-shoot still image and video
+  capture usage, either controlled directly by the CPU, or exposed through
+  an internal USB bus as a UVC device.
+
+* External UVC cameras designed for video conferencing usage.
+
+Other types of camera, including analog cameras, depth cameras, thermal
+cameras, external digital picture or movie cameras, are out of scope for
+this project.
+
+A hardware device that includes independent camera sensors, such as front
+and back sensors in a phone, shall be considered as multiple camera devices
+for the purpose of this library.
+
+Independent Camera Devices
+--------------------------
+
+When multiple cameras are present in the system and are able to operate
+independently from each other, the library shall expose them as multiple
+camera devices and support parallel operation without any additional usage
+restriction apart from the limitations inherent to the hardware (such as
+memory bandwidth, CPU usage or number of CSI-2 receivers for instance).
+
+Independent processes shall be able to use independent cameras devices
+without interfering with each other. A single camera device shall be
+usable by a single process at a time.
+
+Multiple streams support
+------------------------
+
+The library shall support multiple video streams running in parallel
+for each camera device, within the limits imposed by the system.
+
+Per frame controls
+------------------
+
+The library shall support controlling capture parameters for each stream
+on a per-frame basis, on a best effort basis based on the capabilities of the
+hardware and underlying software stack (including kernel drivers and
+firmware). It shall apply capture parameters to the frame they target, and
+report the value of the parameters that have effectively been used for each
+captured frame.
+
+When a camera device supports multiple streams, the library shall allow both
+control of each stream independently, and control of multiple streams
+together. Streams that are controlled together shall be synchronized. No
+synchronization is required for streams controlled independently.
+
+Capability Enumeration
+----------------------
+
+The library shall expose capabilities of each camera device in a way that
+allows applications to discover those capabilities dynamically. Applications
+shall be allowed to cache capabilities for as long as they are using the
+library. If capabilities can change at runtime, the library shall offer a
+mechanism to notify applications of such changes. Applications shall not
+cache capabilities in long term storage between runs.
+
+Capabilities shall be discovered dynamically at runtime from the device when
+possible, and may come, in part or in full, from platform configuration
+data.
+
+Device Profiles
+---------------
+
+The library may define different camera device profiles, each with a minimum
+set of required capabilities. Applications may use those profiles to quickly
+determine the level of features exposed by a device without parsing the full
+list of capabilities. Camera devices may implement additional capabilities
+on top of the minimum required set for the profile they expose.
+
+3A and Image Enhancement Algorithms
+-----------------------------------
+
+The camera devices shall implement auto exposure, auto gain and auto white
+balance. Camera devices that include a focus lens shall implement auto
+focus. Additional image enhancement algorithms, such as noise reduction or
+video stabilization, may be implemented.
+
+All algorithms may be implemented in hardware or firmware outside of the
+library, or in software in the library. They shall all be controllable by
+applications.
+
+The library shall be architectured to isolate the 3A and image enhancement
+algorithms in a component with a documented API, respectively called the 3A
+component and the 3A API. The 3A API shall be stable, and shall allow both
+open-source and closed-source implementations of the 3A component.
+
+The library may include statically-linked open-source 3A components, and
+shall support dynamically-linked open-source and closed-source 3A
+components.
+
+Closed-source 3A Component Sandboxing
+-------------------------------------
+
+For security purposes, it may be desired to run closed-source 3A components
+in a separate process. The 3A API would in such a case be transported over
+IPC. The 3A API shall make it possible to use any IPC mechanism that
+supports passing file descriptors.
+
+The library may implement an IPC mechanism, and shall support third-party
+platform-specific IPC mechanisms through the implementation of a
+platform-specific 3A API wrapper. No modification to the library shall be
+needed to use such third-party IPC mechanisms.
+
+The 3A component shall not directly access any device node on the system.
+Such accesses shall instead be performed through the 3A API. The library
+shall validate all accesses and restrict them to what is absolutely required
+by 3A components.
+
+V4L2 Compatibility Layer
+------------------------
+
+The project shall support traditional V4L2 application through an additional
+libcamera wrapper library. The wrapper library shall trap all accesses to
+camera devices through LD_PRELOAD, and route them through libcamera to
+emulate a high-level V4L2 camera device. It shall expose camera device
+features on a best-effort basis, and aim for the level of features
+traditionally available from a UVC camera designed for video conferencing.
+
+Android Camera HAL v3 Compatibility
+-----------------------------------
+
+The library API shall expose all the features required to implement an
+Android Camera HAL v3 on top of libcamera. Some features of the HAL may be
+omitted as long as they can be implemented separately in the HAL, such as
+JPEG encoding, or YUV reprocessing.
+
+
+Camera Stack
+============
+
+::
+
+ |  a c /    +-------------+  +-------------+  +-------------+  +-------------+
+ |  p a |    |   Native    |  |  Framework  |  |   Native    |  |   Android   |
+ |  p t |    |    V4L2     |  | Application |  |  libcamera  |  |   Camera    |
+ |  l i |    | Application |  | (gstreamer) |  | Application |  |  Framework  |
+ |  i o \    +-------------+  +-------------+  +-------------+  +-------------+
+ |    n             ^                ^                ^                ^
+ |                  |                |                |                |
+ |  l a             |                |                |                |
+ |  i d             v                v                |                v
+ |  b a /    +-------------+  +-------------+         |         +-------------+
+ |  c p |    |    V4L2     |  |   Camera    |         |         |   Android   |
+ |  a t |    |   Compat.   |  |  Framework  |         |         |   Camera    |
+ |  m a |    |             |  | (gstreamer) |         |         |     HAL     |
+ |  e t \    +-------------+  +-------------+         |         +-------------+
+ |  r i             ^                ^                |                ^
+ |  a o             |                |                |                |
+ |    n             |                |                |                |
+ |      /           |         ,................................................
+ |      |           |         !      :            Language             :      !
+ |  l f |           |         !      :            Bindings             :      !
+ |  i r |           |         !      :           (optional)            :      !
+ |  b a |           |         \...............................................'
+ |  c m |           |                |                |                |
+ |  a e |           |                |                |                |
+ |  m w |           v                v                v                v
+ |  e o |    +----------------------------------------------------------------+
+ |  r r |    |                                                                |
+ |  a k |    |                           libcamera                            |
+ |      |    |                                                                |
+ |      \    +----------------------------------------------------------------+
+ |                          ^                  ^                  ^
+ |  Userspace               |                  |                  |
+ | ------------------------ | ---------------- | ---------------- | ---------------
+ |  Kernel                  |                  |                  |
+ |                          v                  v                  v
+ |                    +-----------+      +-----------+      +-----------+
+ |                    |   Media   | <--> |   Video   | <--> |   V4L2    |
+ |                    |  Device   |      |  Device   |      |  Subdev   |
+ |                    +-----------+      +-----------+      +-----------+
+
+The camera stack comprises four software layers. From bottom to top:
+
+* The kernel drivers control the camera hardware and expose a
+  low-level interface to userspace through the Linux kernel V4L2
+  family of APIs (Media Controller API, V4L2 Video Device API and
+  V4L2 Subdev API).
+
+* The libcamera framework is the core part of the stack. It
+  handles all control of the camera devices in its core component,
+  libcamera, and exposes a native C++ API to upper layers. Optional
+  language bindings allow interfacing to libcamera from other
+  programming languages.
+
+  Those components live in the same source code repository and
+  all together constitute the libcamera framework.
+
+* The libcamera adaptation is an umbrella term designating the
+  components that interface to libcamera in other frameworks.
+  Notable examples are a V4L2 compatibility layer, a gstreamer
+  libcamera element, and an Android camera HAL implementation based
+  on libcamera.
+
+  Those components can live in the libcamera project source code
+  in separate repositories, or move to their respective project's
+  repository (for instance the gstreamer libcamera element).</p>
+
+* The applications and upper level frameworks are based on the
+  libcamera framework or libcamera adaptation, and are outside of
+  the scope of the libcamera project.
+
+
+libcamera Architecture
+======================
+
+::
+
+ | ---------------------------< libcamera Public API >---------------------------
+ |                  ^                                      ^
+ |                  |                                      |
+ |                  v                                      v
+ |           +-------------+  +-------------------------------------------------+
+ |           |   Camera    |  |  Camera Device                                  |
+ |           |   Devices   |  | +---------------------------------------------+ |
+ |           |   Manager   |  | | Device-Agnostic                             | |
+ |           +-------------+  | |                                             | |
+ |                  ^         | |                    +------------------------+ |
+ |                  |         | |                    |   ~~~~~~~~~~~~~~~~~~~~~  |
+ |                  |         | |                    |  {  +---------------+  } |
+ |                  |         | |                    |  }  | ////Image//// |  { |
+ |                  |         | |                    | <-> | /Processing// |  } |
+ |                  |         | |                    |  }  | /Algorithms// |  { |
+ |                  |         | |                    |  {  +---------------+  } |
+ |                  |         | |                    |   ~~~~~~~~~~~~~~~~~~~~~  |
+ |                  |         | |                    | ======================== |
+ |                  |         | |                    |     +---------------+    |
+ |                  |         | |                    |     | //Pipeline/// |    |
+ |                  |         | |                    | <-> | ///Handler/// |    |
+ |                  |         | |                    |     | ///////////// |    |
+ |                  |         | +--------------------+     +---------------+    |
+ |                  |         |                                 Device-Specific |
+ |                  |         +-------------------------------------------------+
+ |                  |                     ^                        ^
+ |                  |                     |                        |
+ |                  v                     v                        v
+ |         +--------------------------------------------------------------------+
+ |         | Helpers and Support Classes                                        |
+ |         | +-------------+  +-------------+  +-------------+  +-------------+ |
+ |         | |  MC & V4L2  |  |   Buffers   |  | Sandboxing  |  |   Plugins   | |
+ |         | |   Support   |  |  Allocator  |  |     IPC     |  |   Manager   | |
+ |         | +-------------+  +-------------+  +-------------+  +-------------+ |
+ |         | +-------------+  +-------------+                                   |
+ |         | |  Pipeline   |  |     ...     |                                   |
+ |         | |   Runner    |  |             |                                   |
+ |         | +-------------+  +-------------+                                   |
+ |         +--------------------------------------------------------------------+
+ |
+ |           /// Device-Specific Components
+ |           ~~~ Sandboxing
+
+While offering a unified API towards upper layers, and presenting
+itself as a single library, libcamera isn't monolithic. It exposes
+multiple components through its public API, is built around a set of
+separate helpers internally, uses device-specific components and can
+load dynamic plugins.
+
+Camera Devices Manager
+  The Camera Devices Manager provides a view of available cameras
+  in the system. It performs cold enumeration and runtime camera
+  management, and supports a hotplug notification mechanism in its
+  public API.
+
+  To avoid the cost associated with cold enumeration of all devices
+  at application start, and to arbitrate concurrent access to camera
+  devices, the Camera Devices Manager could later be split to a
+  separate service, possibly with integration in platform-specific
+  device management.
+
+Camera Device
+  The Camera Device represents a camera device to upper layers. It
+  exposes full control of the device through the public API, and is
+  thus the highest level object exposed by libcamera.
+
+  Camera Device instances are created by the Camera Devices
+  Manager. An optional method to create new instances could be exposed
+  through the public API to speed up initialization when the upper
+  layer knows how to directly address camera devices present in the
+  system.
+
+Pipeline Handler
+  The Pipeline Handler manages complex pipelines exposed by the kernel drivers
+  through the Media Controller and V4L2 APIs. It abstracts pipeline handling to
+  hide device-specific details to the rest of the library, and implements both
+  pipeline configuration based on stream configuration, and pipeline runtime
+  execution and scheduling when needed by the device.
+
+  This component is device-specific and is part of the libcamera code base. As
+  such it is covered by the same free software license as the rest of libcamera
+  and needs to be contributed upstream by device vendors. The Pipeline Handler
+  lives in the same process as the rest of the library, and has access to all
+  helpers and kernel camera-related devices.</p>
+
+Image Processing Algorithms
+  Together with the hardware image processing and hardware statistics
+  collection, the Image Processing Algorithms implement 3A (Auto-Exposure,
+  Auto-White Balance and Auto-Focus) and other algorithms. They run on the CPU
+  and interact with the kernel camera devices to control hardware image
+  processing based on the parameters supplied by upper layers, closing the
+  control loop of the ISP.
+
+  This component is device-specific and is loaded as an external plugin. It can
+  be part of the libcamera code base, in which case it is covered by the same
+  license, or provided externally as an open-source or closed-source component.
+
+  The component is sandboxed and can only interact with libcamera through
+  internal APIs specifically marked as such. In particular it will have no
+  direct access to kernel camera devices, and all its accesses to image and
+  metadata will be mediated by dmabuf instances explicitly passed to the
+  component. The component must be prepared to run in a process separate from
+  the main libcamera process, and to have a very restricted view of the system,
+  including no access to networking APIs and limited access to file systems.
+
+  The sandboxing mechanism isn't defined by libcamera. One example
+  implementation will be provided as part of the project, and platforms vendors
+  will be able to provide their own sandboxing mechanism as a plugin.
+
+  libcamera should provide a basic implementation of Image Processing
+  Algorithms, to serve as a reference for the internal API. Device vendors are
+  expected to provide a full-fledged implementation compatible with their
+  Pipeline Handler. One goal of the libcamera project is to create an
+  environment in which the community will be able to compete with the
+  closed-source vendor binaries and develop a high quality open source
+  implementation.
+
+Helpers and Support Classes
+  While Pipeline Handlers are device-specific, implementations are expected to
+  share code due to usage of identical APIs towards the kernel camera drivers
+  and the Image Processing Algorithms. This includes without limitation handling
+  of the MC and V4L2 APIs, buffer management through dmabuf, and pipeline
+  discovery, configuration and scheduling. Such code will be factored out to
+  helpers when applicable.
+
+  Other parts of libcamera will also benefit from factoring code out to
+  self-contained support classes, even if such code is present only once in the
+  code base, in order to keep the source code clean and easy to read. This
+  should be the case for instance for plugin management.
+
+
+V4L2 Compatibility Layer
+------------------------
+
+V4L2 compatibility is achieved through a shared library that traps all
+accesses to camera devices and routes them to libcamera to emulate high-level
+V4L2 camera devices. It is injected in a process address space through
+LD_PRELOAD and is completely transparent for applications.
+
+The compatibility layer exposes camera device features on a best-effort basis,
+and aims for the level of features traditionally available from a UVC camera
+designed for video conferencing.
+
+
+Android Camera HAL
+------------------
+
+Camera support for Android is achieved through a generic Android
+camera HAL implementation on top of libcamera. The HAL will implement internally
+features required by Android and missing from libcamera, such as JPEG encoding
+support.
+
+The Android camera HAL implementation will initially target the
+LIMITED harware level, with support for the FULL level then being gradually
+implemented.