Documentation: Add IPA writers guide

Add a guide about writing IPAs.

Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Sebastian Fricke <sebastian.fricke@posteo.net>

1 files changed, 511 insertions, 0 deletions

diff --git a/Documentation/guides/ipa.rst b/Documentation/guides/ipa.rst
new file mode 100644
index 00000000..e53486d1
--- /dev/null
+++ b/Documentation/guides/ipa.rst
@@ -0,0 +1,511 @@
+.. SPDX-License-Identifier: CC-BY-SA-4.0
+
+IPA Writer's Guide
+==================
+
+IPA modules are Image Processing Algorithm modules. They provide functionality
+that the pipeline handler can use for image processing.
+
+This guide covers the definition of the IPA interface, and how to plumb the
+connection between the pipeline handler and the IPA.
+
+The IPA interface and protocol
+------------------------------
+
+The IPA interface defines the interface between the pipeline handler and the
+IPA. Specifically, it defines the functions that the IPA exposes that the
+pipeline handler can call, and the signals that the pipeline handler can
+connect to, in order to receive data from the IPA asynchronously. In addition,
+it contains any custom data structures that the pipeline handler and IPA may
+pass to each other.
+
+The IPA protocol refers to the agreement between the pipeline handler and the
+IPA regarding the expected response(s) from the IPA for given calls to the IPA.
+This protocol doesn't need to be declared anywhere in code, but it shall be
+documented, as there may be multiple IPA implementations for one pipeline
+handler.
+
+As part of the design of libcamera, IPAs may be isolated in a separate process,
+or run in the same process but a different thread from libcamera. The pipeline
+handler and IPA shall not have to change their operation based on whether the
+IPA is isolated or not, but the possibility of isolation needs to be kept in
+mind. Therefore all data that is passed between them must be serializable, so
+they must be defined separately in the `mojo Interface Definition Language`_
+(IDL), and a code generator will generate headers and serializers corresponding
+to the definitions. Every interface is defined in a mojom file and includes:
+
+- the functions that the pipeline handler can call from the IPA
+- signals in the pipeline handler that the IPA can emit
+- any data structures that are to be passed between the pipeline handler and the IPA
+
+All IPA modules of a given pipeline handler use the same IPA interface. The IPA
+interface definition is thus written by the pipeline handler author, based on
+how they design the interactions between the pipeline handler and the IPA.
+
+The entire IPA interface, including the functions, signals, and any custom
+structs shall be defined in a file named {pipeline_name}.mojom under
+include/libcamera/ipa/.
+
+.. _mojo Interface Definition Language: https://chromium.googlesource.com/chromium/src.git/+/master/mojo/public/tools/bindings/README.md
+
+Namespacing
+-----------
+
+To avoid name collisions between data types defined by different IPA interfaces
+and data types defined by libcamera, each IPA interface must be defined in its
+own namespace.
+
+The namespace is specific with mojo's module directive. It must be the first
+non-comment line in the mojo data definition file. For example, the Raspberry
+Pi IPA interface uses:
+
+.. code-block:: none
+
+        module ipa.rpi;
+
+This will become the ipa::rpi namespace in C++ code.
+
+Data containers
+---------------
+
+Since the data passed between the pipeline handler and the IPA must support
+serialization, any custom data containers must be defined with the mojo IDL.
+
+The following list of libcamera objects are supported in the interface
+definition, and may be used as function parameter types or struct field types:
+
+- libcamera.CameraSensorInfo
+- libcamera.ControlInfoMap
+- libcamera.ControlList
+- libcamera.FileDescriptor
+- libcamera.IPABuffer
+- libcamera.IPASettings
+- libcamera.IPAStream
+- libcamera.Point
+- libcamera.Rectangle
+- libcamera.Size
+- libcamera.SizeRange
+
+To use them, core.mojom must be included in the mojo data definition file:
+
+.. code-block:: none
+
+        import "include/libcamera/ipa/core.mojom";
+
+Other custom structs may be defined and used as well. There is no requirement
+that they must be defined before usage. enums and structs are supported.
+
+The following is an example of a definition of an enum, for the purpose of
+being used as flags:
+
+.. code-block:: none
+
+        enum ConfigParameters {
+                ConfigLsTable = 0x01,
+                ConfigStaggeredWrite = 0x02,
+                ConfigSensor = 0x04,
+                ConfigDropFrames = 0x08,
+        };
+
+The following is an example of a definition of a struct:
+
+.. code-block:: none
+
+        struct ConfigInput {
+                uint32 op;
+                uint32 transform;
+                libcamera.FileDescriptor lsTableHandle;
+                int32 lsTableHandleStatic = -1;
+                map<uint32, libcamera.IPAStream> streamConfig;
+                array<libcamera.IPABuffer> buffers;
+        };
+
+This example has some special things about it. First of all, it uses the
+FileDescriptor data type. This type must be used to ensure that the file
+descriptor that it contains is translated properly across the IPC boundary
+(when the IPA is in an isolated process).
+
+This does mean that if the file descriptor should be sent without being
+translated (for example, for the IPA to tell the pipeline handler which
+fd *that the pipeline handler holds* to act on), then it must be in a
+regular int32 type.
+
+This example also illustrates that struct fields may have default values, as
+is assigned to lsTableHandleStatic. This is the value that the field will
+take when the struct is constructed with the default constructor.
+
+Arrays and maps are supported as well. They are translated to C++ vectors and
+maps, respectively. The members of the arrays and maps are embedded, and cannot
+be const.
+
+Note that nullable fields, static-length arrays, handles, and unions, which
+are supported by mojo, are not supported by our code generator.
+
+The Main IPA interface
+----------------------
+
+The IPA interface is split in two parts, the Main IPA interface, which
+describes the functions that the pipeline handler can call from the IPA,
+and the Event IPA interface, which describes the signals received by the
+pipeline handler that the IPA can emit. Both must be defined. This section
+focuses on the Main IPA interface.
+
+The main interface must be named as IPA{pipeline_name}Interface.
+
+The functions that the pipeline handler can call from the IPA may be
+synchronous or asynchronous. Synchronous functions do not return until the IPA
+returns from the function, while asynchronous functions return immediately
+without waiting for the IPA to return.
+
+At a minimum, the following three functions must be present (and implemented):
+
+- init();
+- start();
+- stop();
+
+All three of these functions are synchronous. The parameters for start() and
+init() may be customized.
+
+A configure() method is recommended. Any ControlInfoMap instances that will be
+used by the IPA must be sent to the IPA from the pipeline handler, at configure
+time, for example.
+
+All input parameters will become const references, except for arithmetic types,
+which will be passed by value. Output parameters will become pointers, unless
+the first output parameter is an int32, or there is only one primitive output
+parameter, in which case it will become a regular return value.
+
+const is not allowed inside of arrays and maps. mojo arrays will become C++
+std::vector<>.
+
+By default, all methods defined in the main interface are synchronous. This
+means that in the case of IPC (i.e. isolated IPA), the function call will not
+return until the return value or output parameters are ready. To specify an
+asynchronous function, the [async] attribute can be used. Asynchronous
+methods must not have any return value or output parameters, since in the
+case of IPC the call needs to return immediately.
+
+It is also possible that the IPA will not be run in isolation. In this case,
+the IPA thread will not exist until start() is called. This means that in the
+case of no isolation, asynchronous calls cannot be made before start(). Since
+the IPA interface must be the same regardless of isolation, the same
+restriction applies to the case of isolation, and any function that will be
+called before start() must be synchronous.
+
+In addition, any call made after start() and before stop() must be
+asynchronous. The motivation for this is to avoid damaging real-time
+performance of the pipeline handler. If the pipeline handler wants some data
+from the IPA, the IPA should return the data asynchronously via an event
+(see "The Event IPA interface").
+
+The following is an example of a main interface definition:
+
+.. code-block:: none
+
+        interface IPARPiInterface {
+                init(libcamera.IPASettings settings, string sensorName)
+                        => (int32 ret, bool metadataSupport);
+                start() => (int32 ret);
+                stop();
+
+                configure(libcamera.CameraSensorInfo sensorInfo,
+                          map<uint32, libcamera.IPAStream> streamConfig,
+                          map<uint32, libcamera.ControlInfoMap> entityControls,
+                          ConfigInput ipaConfig)
+                        => (int32 ret, ConfigOutput results);
+
+                mapBuffers(array<IPABuffer> buffers);
+                unmapBuffers(array<uint32> ids);
+
+                [async] signalStatReady(uint32 bufferId);
+                [async] signalQueueRequest(libcamera.ControlList controls);
+                [async] signalIspPrepare(ISPConfig data);
+        };
+
+
+The first three functions are the required functions. Functions do not need to
+have return values, like stop(), mapBuffers(), and unmapBuffers(). In the case
+of asynchronous functions, as explained before, they *must not* have return
+values.
+
+The Event IPA interface
+-----------------------
+
+The event IPA interface describes the signals received by the pipeline handler
+that the IPA can emit. It must be defined. If there are no event functions,
+then it may be empty. These emissions are meant to notify the pipeline handler
+of some event, such as request data is ready, and *must not* be used to drive
+the camera pipeline from the IPA.
+
+The event interface must be named as IPA{pipeline_name}EventInterface.
+
+Methods defined in the event interface are implicitly asynchronous.
+Thus they cannot return any value. Specifying the [async] tag is not
+necessary.
+
+Methods defined in the event interface will become signals in the IPA
+interface. The IPA can emit signals, while the pipeline handler can connect
+slots to them.
+
+The following is an example of an event interface definition:
+
+.. code-block:: none
+
+        interface IPARPiEventInterface {
+                statsMetadataComplete(uint32 bufferId,
+                                      libcamera.ControlList controls);
+                runIsp(uint32 bufferId);
+                embeddedComplete(uint32 bufferId);
+                setIsp(libcamera.ControlList controls);
+                setStaggered(libcamera.ControlList controls);
+        };
+
+Compiling the IPA interface
+---------------------------
+
+After the IPA interface is defined in include/libcamera/ipa/{pipeline_name}.mojom,
+an entry for it must be added in meson so that it can be compiled. The filename
+must be added to the ipa_mojom_files object in include/libcamera/ipa/meson.build.
+
+For example, adding the raspberrypi.mojom file to meson:
+
+.. code-block:: none
+
+        ipa_mojom_files = [
+            'raspberrypi.mojom',
+        ]
+
+This will cause the mojo data definition file to be compiled. Specifically, it
+generates five files:
+
+- a header describing the custom data structures, and the complete IPA
+  interface (at {$build_dir}/include/libcamera/ipa/{pipeline}_ipa_interface.h)
+
+- a serializer implementing de/serialization for the custom data structures (at
+  {$build_dir}/include/libcamera/ipa/{pipeline}_ipa_serializer.h)
+
+- a proxy header describing a specialized IPA proxy (at
+  {$build_dir}/include/libcamera/ipa/{pipeline}_ipa_proxy.h)
+
+- a proxy source implementing the IPA proxy (at
+  {$build_dir}/src/libcamera/proxy/{pipeline}_ipa_proxy.cpp)
+
+- a proxy worker source implementing the other end of the IPA proxy (at
+  {$build_dir}/src/libcamera/proxy/worker/{pipeline}_ipa_proxy_worker.cpp)
+
+The IPA proxy serves as the layer between the pipeline handler and the IPA, and
+handles threading vs isolation transparently. The pipeline handler and the IPA
+only require the interface header and the proxy header. The serializer is only
+used internally by the proxy.
+
+Using the custom data structures
+--------------------------------
+
+To use the custom data structures that are defined in the mojo data definition
+file, the following header must be included:
+
+.. code-block:: C++
+
+   #include <libcamera/ipa/{pipeline_name}_ipa_interface.h>
+
+The POD types of the structs simply become their C++ counterparts, eg. uint32
+in mojo will become uint32_t in C++. mojo map becomes C++ std::map, and mojo
+array becomes C++ std::vector. All members of maps and vectors are embedded,
+and are not pointers. The members cannot be const.
+
+The names of all the fields of structs can be used in C++ in exactly the same
+way as they are defined in the data definition file. For example, the following
+struct as defined in the mojo file:
+
+.. code-block:: none
+
+   struct SensorConfig {
+        uint32 gainDelay = 1;
+        uint32 exposureDelay;
+        uint32 sensorMetadata;
+   };
+
+Will become this in C++:
+
+.. code-block:: C++
+
+   struct SensorConfig {
+        uint32_t gainDelay;
+        uint32_t exposureDelay;
+        uint32_t sensorMetadata;
+   };
+
+The generated structs will also have two constructors, a constructor that
+fills all fields with the default values, and a second constructor that takes
+a value for every field. The default value constructor will fill in the fields
+with the specified default value if it exists. In the above example, `gainDelay_`
+will be initialized to 1. If no default value is specified, then it will be
+filled in as zero (or -1 for a FileDescriptor type).
+
+All fields and constructors/destructors in these generated structs are public.
+
+Using the IPA interface (pipeline handler)
+------------------------------------------
+
+The following headers are necessary to use an IPA in the pipeline handler
+(with raspberrypi as an example):
+
+.. code-block:: C++
+
+   #include <libcamera/ipa/raspberrypi_ipa_interface.h>
+   #include <libcamera/ipa/raspberrypi_ipa_proxy.h>
+
+The first header includes definitions of the custom data structures, and
+the definition of the complete IPA interface (including both the Main and
+the Event IPA interfaces). The name of the header file comes from the name
+of the mojom file, which in this case was raspberrypi.mojom.
+
+The second header includes the definition of the specialized IPA proxy. It
+exposes the complete IPA interface. We will see how to use it in this section.
+
+In the pipeline handler, we first need to construct a specialized IPA proxy.
+From the point of view of the pipeline hander, this is the object that is the
+IPA.
+
+To do so, we invoke the IPAManager:
+
+.. code-block:: C++
+
+        std::unique_ptr<ipa::rpi::IPAProxyRPi> ipa_ =
+                IPAManager::createIPA<ipa::rpi::IPAProxyRPi>(pipe_, 1, 1);
+
+The ipa::rpi namespace comes from the namespace that we defined in the mojo
+data definition file, in the "Namespacing" section. The name of the proxy,
+IPAProxyRPi, comes from the name given to the main IPA interface,
+IPARPiInterface, in the "The Main IPA interface" section.
+
+The return value of IPAManager::createIPA shall be error-checked, to confirm
+that the returned pointer is not a nullptr.
+
+After this, before initializing the IPA, slots should be connected to all of
+the IPA's signals, as defined in the Event IPA interface:
+
+.. code-block:: C++
+
+	ipa_->statsMetadataComplete.connect(this, &RPiCameraData::statsMetadataComplete);
+	ipa_->runIsp.connect(this, &RPiCameraData::runIsp);
+	ipa_->embeddedComplete.connect(this, &RPiCameraData::embeddedComplete);
+	ipa_->setIsp.connect(this, &RPiCameraData::setIsp);
+	ipa_->setStaggered.connect(this, &RPiCameraData::setStaggered);
+
+The slot functions have a function signature based on the function definition
+in the Event IPA interface. All plain old data (POD) types are as-is (with
+their C++ versions, eg. uint32 -> uint32_t), and all structs are const references.
+
+For example, for the following entry in the Event IPA interface:
+
+.. code-block:: none
+
+   statsMetadataComplete(uint32 bufferId, ControlList controls);
+
+A function with the following function signature shall be connected to the
+signal:
+
+.. code-block:: C++
+
+   void statsMetadataComplete(uint32_t bufferId, const ControlList &controls);
+
+After connecting the slots to the signals, the IPA should be initialized
+(using the main interface definition example from earlier):
+
+.. code-block:: C++
+
+   IPASettings settings{};
+   bool metadataSupport;
+   int ret = ipa_->init(settings, "sensor name", &metadataSupport);
+
+At this point, any IPA functions that were defined in the Main IPA interface
+can be called as if they were regular member functions, for example (based on
+the main interface definition example from earlier):
+
+.. code-block:: C++
+
+   ipa_->start();
+   int ret = ipa_->configure(sensorInfo_, streamConfig, entityControls, ipaConfig, &result);
+   ipa_->signalStatReady(RPi::BufferMask::STATS | static_cast<unsigned int>(index));
+
+Remember that any functions designated as asynchronous *must not* be called
+before start().
+
+Notice that for both init() and configure(), the first output parameter is a
+direct return, since it is an int32, while the other output parameter is a
+pointer-based output parameter.
+
+Using the IPA interface (IPA Module)
+-----------------------------
+
+The following header is necessary to implement an IPA Module (with raspberrypi
+as an example):
+
+.. code-block:: C++
+
+   #include <libcamera/ipa/raspberrypi_ipa_interface.h>
+
+This header includes definitions of the custom data structures, and
+the definition of the complete IPA interface (including both the Main and
+the Event IPA interfaces). The name of the header file comes from the name
+of the mojom file, which in this case was raspberrypi.mojom.
+
+The IPA module must implement the IPA interface class that is defined in the
+header. In the case of our example, that is ipa::rpi::IPARPiInterface. The
+ipa::rpi namespace comes from the namespace that we defined in the mojo data
+definition file, in the "Namespacing" section. The name of the interface is the
+same as the name given to the Main IPA interface.
+
+The function signature rules are the same as for the slots in the pipeline
+handler side; PODs are passed by value, and structs are passed by const
+reference. For the Main IPA interface, output values are also allowed (only
+for synchronous calls), so there may be output parameters as well. If the
+first output parameter is a POD it will be returned by value, otherwise
+it will be returned by an output parameter pointer. The second and any other
+output parameters will also be returned by output parameter pointers.
+
+For example, for the following function specification in the Main IPA interface
+definition:
+
+.. code-block:: none
+
+   configure(libcamera.CameraSensorInfo sensorInfo,
+             uint32 exampleNumber,
+             map<uint32, libcamera.IPAStream> streamConfig,
+             map<uint32, libcamera.ControlInfoMap> entityControls,
+             ConfigInput ipaConfig)
+   => (int32 ret, ConfigOutput results);
+
+We will need to implement a function with the following function signature:
+
+.. code-block:: C++
+
+        int configure(const CameraSensorInfo &sensorInfo,
+                      uint32_t exampleNumber,
+                      const std::map<unsigned int, IPAStream> &streamConfig,
+                      const std::map<unsigned int, ControlInfoMap> &entityControls,
+                      const ipa::rpi::ConfigInput &data,
+                      ipa::rpi::ConfigOutput *response);
+
+The return value is int, because the first output parameter is int32.  The rest
+of the output parameters (in this case, only response) become output parameter
+pointers. The non-POD input parameters become const references, and the POD
+input parameter is passed by value.
+
+At any time after start() and before stop() (though usually only in response to
+an IPA call), the IPA may send data to the pipeline handler by emitting
+signals. These signals are defined in the C++ IPA interface class (which is in
+the generated and included header).
+
+For example, for the following function defined in the Event IPA interface:
+
+.. code-block:: none
+
+   statsMetadataComplete(uint32 bufferId, libcamera.ControlList controls);
+
+We can emit a signal like so:
+
+.. code-block:: C++
+
+   statsMetadataComplete.emit(bufferId & RPi::BufferMask::ID, libcameraMetadata_);