From 0906ddb2681a53aff9ecccb7e933e9f04974022f Mon Sep 17 00:00:00 2001 From: Paul Elder Date: Sat, 5 Dec 2020 19:31:05 +0900 Subject: Documentation: Add IPA writers guide Add a guide about writing IPAs. Signed-off-by: Paul Elder Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart Reviewed-by: Sebastian Fricke --- Documentation/guides/ipa.rst | 511 +++++++++++++++++++++++++++++++++++++++++++ Documentation/index.rst | 1 + Documentation/meson.build | 1 + 3 files changed, 513 insertions(+) create mode 100644 Documentation/guides/ipa.rst (limited to 'Documentation') diff --git a/Documentation/guides/ipa.rst b/Documentation/guides/ipa.rst new file mode 100644 index 00000000..e53486d1 --- /dev/null +++ b/Documentation/guides/ipa.rst @@ -0,0 +1,511 @@ +.. SPDX-License-Identifier: CC-BY-SA-4.0 + +IPA Writer's Guide +================== + +IPA modules are Image Processing Algorithm modules. They provide functionality +that the pipeline handler can use for image processing. + +This guide covers the definition of the IPA interface, and how to plumb the +connection between the pipeline handler and the IPA. + +The IPA interface and protocol +------------------------------ + +The IPA interface defines the interface between the pipeline handler and the +IPA. Specifically, it defines the functions that the IPA exposes that the +pipeline handler can call, and the signals that the pipeline handler can +connect to, in order to receive data from the IPA asynchronously. In addition, +it contains any custom data structures that the pipeline handler and IPA may +pass to each other. + +The IPA protocol refers to the agreement between the pipeline handler and the +IPA regarding the expected response(s) from the IPA for given calls to the IPA. +This protocol doesn't need to be declared anywhere in code, but it shall be +documented, as there may be multiple IPA implementations for one pipeline +handler. + +As part of the design of libcamera, IPAs may be isolated in a separate process, +or run in the same process but a different thread from libcamera. The pipeline +handler and IPA shall not have to change their operation based on whether the +IPA is isolated or not, but the possibility of isolation needs to be kept in +mind. Therefore all data that is passed between them must be serializable, so +they must be defined separately in the `mojo Interface Definition Language`_ +(IDL), and a code generator will generate headers and serializers corresponding +to the definitions. Every interface is defined in a mojom file and includes: + +- the functions that the pipeline handler can call from the IPA +- signals in the pipeline handler that the IPA can emit +- any data structures that are to be passed between the pipeline handler and the IPA + +All IPA modules of a given pipeline handler use the same IPA interface. The IPA +interface definition is thus written by the pipeline handler author, based on +how they design the interactions between the pipeline handler and the IPA. + +The entire IPA interface, including the functions, signals, and any custom +structs shall be defined in a file named {pipeline_name}.mojom under +include/libcamera/ipa/. + +.. _mojo Interface Definition Language: https://chromium.googlesource.com/chromium/src.git/+/master/mojo/public/tools/bindings/README.md + +Namespacing +----------- + +To avoid name collisions between data types defined by different IPA interfaces +and data types defined by libcamera, each IPA interface must be defined in its +own namespace. + +The namespace is specific with mojo's module directive. It must be the first +non-comment line in the mojo data definition file. For example, the Raspberry +Pi IPA interface uses: + +.. code-block:: none + + module ipa.rpi; + +This will become the ipa::rpi namespace in C++ code. + +Data containers +--------------- + +Since the data passed between the pipeline handler and the IPA must support +serialization, any custom data containers must be defined with the mojo IDL. + +The following list of libcamera objects are supported in the interface +definition, and may be used as function parameter types or struct field types: + +- libcamera.CameraSensorInfo +- libcamera.ControlInfoMap +- libcamera.ControlList +- libcamera.FileDescriptor +- libcamera.IPABuffer +- libcamera.IPASettings +- libcamera.IPAStream +- libcamera.Point +- libcamera.Rectangle +- libcamera.Size +- libcamera.SizeRange + +To use them, core.mojom must be included in the mojo data definition file: + +.. code-block:: none + + import "include/libcamera/ipa/core.mojom"; + +Other custom structs may be defined and used as well. There is no requirement +that they must be defined before usage. enums and structs are supported. + +The following is an example of a definition of an enum, for the purpose of +being used as flags: + +.. code-block:: none + + enum ConfigParameters { + ConfigLsTable = 0x01, + ConfigStaggeredWrite = 0x02, + ConfigSensor = 0x04, + ConfigDropFrames = 0x08, + }; + +The following is an example of a definition of a struct: + +.. code-block:: none + + struct ConfigInput { + uint32 op; + uint32 transform; + libcamera.FileDescriptor lsTableHandle; + int32 lsTableHandleStatic = -1; + map streamConfig; + array buffers; + }; + +This example has some special things about it. First of all, it uses the +FileDescriptor data type. This type must be used to ensure that the file +descriptor that it contains is translated properly across the IPC boundary +(when the IPA is in an isolated process). + +This does mean that if the file descriptor should be sent without being +translated (for example, for the IPA to tell the pipeline handler which +fd *that the pipeline handler holds* to act on), then it must be in a +regular int32 type. + +This example also illustrates that struct fields may have default values, as +is assigned to lsTableHandleStatic. This is the value that the field will +take when the struct is constructed with the default constructor. + +Arrays and maps are supported as well. They are translated to C++ vectors and +maps, respectively. The members of the arrays and maps are embedded, and cannot +be const. + +Note that nullable fields, static-length arrays, handles, and unions, which +are supported by mojo, are not supported by our code generator. + +The Main IPA interface +---------------------- + +The IPA interface is split in two parts, the Main IPA interface, which +describes the functions that the pipeline handler can call from the IPA, +and the Event IPA interface, which describes the signals received by the +pipeline handler that the IPA can emit. Both must be defined. This section +focuses on the Main IPA interface. + +The main interface must be named as IPA{pipeline_name}Interface. + +The functions that the pipeline handler can call from the IPA may be +synchronous or asynchronous. Synchronous functions do not return until the IPA +returns from the function, while asynchronous functions return immediately +without waiting for the IPA to return. + +At a minimum, the following three functions must be present (and implemented): + +- init(); +- start(); +- stop(); + +All three of these functions are synchronous. The parameters for start() and +init() may be customized. + +A configure() method is recommended. Any ControlInfoMap instances that will be +used by the IPA must be sent to the IPA from the pipeline handler, at configure +time, for example. + +All input parameters will become const references, except for arithmetic types, +which will be passed by value. Output parameters will become pointers, unless +the first output parameter is an int32, or there is only one primitive output +parameter, in which case it will become a regular return value. + +const is not allowed inside of arrays and maps. mojo arrays will become C++ +std::vector<>. + +By default, all methods defined in the main interface are synchronous. This +means that in the case of IPC (i.e. isolated IPA), the function call will not +return until the return value or output parameters are ready. To specify an +asynchronous function, the [async] attribute can be used. Asynchronous +methods must not have any return value or output parameters, since in the +case of IPC the call needs to return immediately. + +It is also possible that the IPA will not be run in isolation. In this case, +the IPA thread will not exist until start() is called. This means that in the +case of no isolation, asynchronous calls cannot be made before start(). Since +the IPA interface must be the same regardless of isolation, the same +restriction applies to the case of isolation, and any function that will be +called before start() must be synchronous. + +In addition, any call made after start() and before stop() must be +asynchronous. The motivation for this is to avoid damaging real-time +performance of the pipeline handler. If the pipeline handler wants some data +from the IPA, the IPA should return the data asynchronously via an event +(see "The Event IPA interface"). + +The following is an example of a main interface definition: + +.. code-block:: none + + interface IPARPiInterface { + init(libcamera.IPASettings settings, string sensorName) + => (int32 ret, bool metadataSupport); + start() => (int32 ret); + stop(); + + configure(libcamera.CameraSensorInfo sensorInfo, + map streamConfig, + map entityControls, + ConfigInput ipaConfig) + => (int32 ret, ConfigOutput results); + + mapBuffers(array buffers); + unmapBuffers(array ids); + + [async] signalStatReady(uint32 bufferId); + [async] signalQueueRequest(libcamera.ControlList controls); + [async] signalIspPrepare(ISPConfig data); + }; + + +The first three functions are the required functions. Functions do not need to +have return values, like stop(), mapBuffers(), and unmapBuffers(). In the case +of asynchronous functions, as explained before, they *must not* have return +values. + +The Event IPA interface +----------------------- + +The event IPA interface describes the signals received by the pipeline handler +that the IPA can emit. It must be defined. If there are no event functions, +then it may be empty. These emissions are meant to notify the pipeline handler +of some event, such as request data is ready, and *must not* be used to drive +the camera pipeline from the IPA. + +The event interface must be named as IPA{pipeline_name}EventInterface. + +Methods defined in the event interface are implicitly asynchronous. +Thus they cannot return any value. Specifying the [async] tag is not +necessary. + +Methods defined in the event interface will become signals in the IPA +interface. The IPA can emit signals, while the pipeline handler can connect +slots to them. + +The following is an example of an event interface definition: + +.. code-block:: none + + interface IPARPiEventInterface { + statsMetadataComplete(uint32 bufferId, + libcamera.ControlList controls); + runIsp(uint32 bufferId); + embeddedComplete(uint32 bufferId); + setIsp(libcamera.ControlList controls); + setStaggered(libcamera.ControlList controls); + }; + +Compiling the IPA interface +--------------------------- + +After the IPA interface is defined in include/libcamera/ipa/{pipeline_name}.mojom, +an entry for it must be added in meson so that it can be compiled. The filename +must be added to the ipa_mojom_files object in include/libcamera/ipa/meson.build. + +For example, adding the raspberrypi.mojom file to meson: + +.. code-block:: none + + ipa_mojom_files = [ + 'raspberrypi.mojom', + ] + +This will cause the mojo data definition file to be compiled. Specifically, it +generates five files: + +- a header describing the custom data structures, and the complete IPA + interface (at {$build_dir}/include/libcamera/ipa/{pipeline}_ipa_interface.h) + +- a serializer implementing de/serialization for the custom data structures (at + {$build_dir}/include/libcamera/ipa/{pipeline}_ipa_serializer.h) + +- a proxy header describing a specialized IPA proxy (at + {$build_dir}/include/libcamera/ipa/{pipeline}_ipa_proxy.h) + +- a proxy source implementing the IPA proxy (at + {$build_dir}/src/libcamera/proxy/{pipeline}_ipa_proxy.cpp) + +- a proxy worker source implementing the other end of the IPA proxy (at + {$build_dir}/src/libcamera/proxy/worker/{pipeline}_ipa_proxy_worker.cpp) + +The IPA proxy serves as the layer between the pipeline handler and the IPA, and +handles threading vs isolation transparently. The pipeline handler and the IPA +only require the interface header and the proxy header. The serializer is only +used internally by the proxy. + +Using the custom data structures +-------------------------------- + +To use the custom data structures that are defined in the mojo data definition +file, the following header must be included: + +.. code-block:: C++ + + #include + +The POD types of the structs simply become their C++ counterparts, eg. uint32 +in mojo will become uint32_t in C++. mojo map becomes C++ std::map, and mojo +array becomes C++ std::vector. All members of maps and vectors are embedded, +and are not pointers. The members cannot be const. + +The names of all the fields of structs can be used in C++ in exactly the same +way as they are defined in the data definition file. For example, the following +struct as defined in the mojo file: + +.. code-block:: none + + struct SensorConfig { + uint32 gainDelay = 1; + uint32 exposureDelay; + uint32 sensorMetadata; + }; + +Will become this in C++: + +.. code-block:: C++ + + struct SensorConfig { + uint32_t gainDelay; + uint32_t exposureDelay; + uint32_t sensorMetadata; + }; + +The generated structs will also have two constructors, a constructor that +fills all fields with the default values, and a second constructor that takes +a value for every field. The default value constructor will fill in the fields +with the specified default value if it exists. In the above example, `gainDelay_` +will be initialized to 1. If no default value is specified, then it will be +filled in as zero (or -1 for a FileDescriptor type). + +All fields and constructors/destructors in these generated structs are public. + +Using the IPA interface (pipeline handler) +------------------------------------------ + +The following headers are necessary to use an IPA in the pipeline handler +(with raspberrypi as an example): + +.. code-block:: C++ + + #include + #include + +The first header includes definitions of the custom data structures, and +the definition of the complete IPA interface (including both the Main and +the Event IPA interfaces). The name of the header file comes from the name +of the mojom file, which in this case was raspberrypi.mojom. + +The second header includes the definition of the specialized IPA proxy. It +exposes the complete IPA interface. We will see how to use it in this section. + +In the pipeline handler, we first need to construct a specialized IPA proxy. +From the point of view of the pipeline hander, this is the object that is the +IPA. + +To do so, we invoke the IPAManager: + +.. code-block:: C++ + + std::unique_ptr ipa_ = + IPAManager::createIPA(pipe_, 1, 1); + +The ipa::rpi namespace comes from the namespace that we defined in the mojo +data definition file, in the "Namespacing" section. The name of the proxy, +IPAProxyRPi, comes from the name given to the main IPA interface, +IPARPiInterface, in the "The Main IPA interface" section. + +The return value of IPAManager::createIPA shall be error-checked, to confirm +that the returned pointer is not a nullptr. + +After this, before initializing the IPA, slots should be connected to all of +the IPA's signals, as defined in the Event IPA interface: + +.. code-block:: C++ + + ipa_->statsMetadataComplete.connect(this, &RPiCameraData::statsMetadataComplete); + ipa_->runIsp.connect(this, &RPiCameraData::runIsp); + ipa_->embeddedComplete.connect(this, &RPiCameraData::embeddedComplete); + ipa_->setIsp.connect(this, &RPiCameraData::setIsp); + ipa_->setStaggered.connect(this, &RPiCameraData::setStaggered); + +The slot functions have a function signature based on the function definition +in the Event IPA interface. All plain old data (POD) types are as-is (with +their C++ versions, eg. uint32 -> uint32_t), and all structs are const references. + +For example, for the following entry in the Event IPA interface: + +.. code-block:: none + + statsMetadataComplete(uint32 bufferId, ControlList controls); + +A function with the following function signature shall be connected to the +signal: + +.. code-block:: C++ + + void statsMetadataComplete(uint32_t bufferId, const ControlList &controls); + +After connecting the slots to the signals, the IPA should be initialized +(using the main interface definition example from earlier): + +.. code-block:: C++ + + IPASettings settings{}; + bool metadataSupport; + int ret = ipa_->init(settings, "sensor name", &metadataSupport); + +At this point, any IPA functions that were defined in the Main IPA interface +can be called as if they were regular member functions, for example (based on +the main interface definition example from earlier): + +.. code-block:: C++ + + ipa_->start(); + int ret = ipa_->configure(sensorInfo_, streamConfig, entityControls, ipaConfig, &result); + ipa_->signalStatReady(RPi::BufferMask::STATS | static_cast(index)); + +Remember that any functions designated as asynchronous *must not* be called +before start(). + +Notice that for both init() and configure(), the first output parameter is a +direct return, since it is an int32, while the other output parameter is a +pointer-based output parameter. + +Using the IPA interface (IPA Module) +----------------------------- + +The following header is necessary to implement an IPA Module (with raspberrypi +as an example): + +.. code-block:: C++ + + #include + +This header includes definitions of the custom data structures, and +the definition of the complete IPA interface (including both the Main and +the Event IPA interfaces). The name of the header file comes from the name +of the mojom file, which in this case was raspberrypi.mojom. + +The IPA module must implement the IPA interface class that is defined in the +header. In the case of our example, that is ipa::rpi::IPARPiInterface. The +ipa::rpi namespace comes from the namespace that we defined in the mojo data +definition file, in the "Namespacing" section. The name of the interface is the +same as the name given to the Main IPA interface. + +The function signature rules are the same as for the slots in the pipeline +handler side; PODs are passed by value, and structs are passed by const +reference. For the Main IPA interface, output values are also allowed (only +for synchronous calls), so there may be output parameters as well. If the +first output parameter is a POD it will be returned by value, otherwise +it will be returned by an output parameter pointer. The second and any other +output parameters will also be returned by output parameter pointers. + +For example, for the following function specification in the Main IPA interface +definition: + +.. code-block:: none + + configure(libcamera.CameraSensorInfo sensorInfo, + uint32 exampleNumber, + map streamConfig, + map entityControls, + ConfigInput ipaConfig) + => (int32 ret, ConfigOutput results); + +We will need to implement a function with the following function signature: + +.. code-block:: C++ + + int configure(const CameraSensorInfo &sensorInfo, + uint32_t exampleNumber, + const std::map &streamConfig, + const std::map &entityControls, + const ipa::rpi::ConfigInput &data, + ipa::rpi::ConfigOutput *response); + +The return value is int, because the first output parameter is int32. The rest +of the output parameters (in this case, only response) become output parameter +pointers. The non-POD input parameters become const references, and the POD +input parameter is passed by value. + +At any time after start() and before stop() (though usually only in response to +an IPA call), the IPA may send data to the pipeline handler by emitting +signals. These signals are defined in the C++ IPA interface class (which is in +the generated and included header). + +For example, for the following function defined in the Event IPA interface: + +.. code-block:: none + + statsMetadataComplete(uint32 bufferId, libcamera.ControlList controls); + +We can emit a signal like so: + +.. code-block:: C++ + + statsMetadataComplete.emit(bufferId & RPi::BufferMask::ID, libcameraMetadata_); diff --git a/Documentation/index.rst b/Documentation/index.rst index 7e3fe2f6..1f4fc485 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -17,6 +17,7 @@ Developer Guide Application Writer's Guide Pipeline Handler Writer's Guide + IPA Writer's guide Tracing guide Environment variables Sensor driver requirements diff --git a/Documentation/meson.build b/Documentation/meson.build index a82ee7c8..c8521574 100644 --- a/Documentation/meson.build +++ b/Documentation/meson.build @@ -53,6 +53,7 @@ if sphinx.found() 'environment_variables.rst', 'guides/application-developer.rst', 'guides/introduction.rst', + 'guides/ipa.rst', 'guides/pipeline-handler.rst', 'guides/tracing.rst', 'index.rst', -- cgit v1.2.1