summaryrefslogtreecommitdiff
path: root/src/ipa/ipu3
diff options
context:
space:
mode:
authorKieran Bingham <kieran.bingham@ideasonboard.com>2021-09-12 22:30:17 +0100
committerJean-Michel Hautbois <jeanmichel.hautbois@ideasonboard.com>2021-10-26 14:08:17 +0200
commit372698ece3492771bececb6fd5f7dd9f74469518 (patch)
treecd326f9b69b48dad85de1f2ce71b646f40c4468c /src/ipa/ipu3
parenta1f0df865409fa2d65e42ab7451fdfc98089c760 (diff)
Documentation: IPU3 IPA Design guide
The IPU3 IPA implements the basic 3A using the ImgU ISP. Provide an overview document to describe its operations, and provide a block diagram to help visualise how the components are put together to assist any new developers exploring the code. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Signed-off-by: Jean-Michel Hautbois <jeanmichel.hautbois@ideasonboard.com> Reviewed-by: Umang Jain <umang.jain@ideasonboard.com> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Jean-Michel Hautbois <jeanmichel.hautbois@ideasonboard.com>
Diffstat (limited to 'src/ipa/ipu3')
-rw-r--r--src/ipa/ipu3/ipu3-ipa-design-guide.rst155
1 files changed, 155 insertions, 0 deletions
diff --git a/src/ipa/ipu3/ipu3-ipa-design-guide.rst b/src/ipa/ipu3/ipu3-ipa-design-guide.rst
new file mode 100644
index 00000000..89c71108
--- /dev/null
+++ b/src/ipa/ipu3/ipu3-ipa-design-guide.rst
@@ -0,0 +1,155 @@
+IPU3 IPA Architecture Design and Overview
+=========================================
+
+The IPU3 IPA is built as a modular and extensible framework with an
+upper layer to manage the interactions with the pipeline handler, and
+the image processing algorithms split to compartmentalise the processing
+required for each processing block, making use of the fixed-function
+accelerators provided by the ImgU ISP.
+
+The core IPU3 class is responsible for initialisation and construction
+of the algorithm components, processing controls set by the requests
+from applications, and managing events from the pipeline handler.
+
+::
+
+ ┌───────────────────────────────────────────┐
+ │ IPU3 Pipeline Handler │
+ │ ┌────────┐ ┌────────┐ ┌────────┐ │
+ │ │ │ │ │ │ │ │
+ │ │ Sensor ├───►│ CIO2 ├───►│ ImgU ├──►
+ │ │ │ │ │ │ │ │
+ │ └────────┘ └────────┘ └─▲────┬─┘ │ P: Parameter Buffer
+ │ │P │ │ S: Statistics Buffer
+ │ │ │S │
+ └─┬───┬───┬──────┬────┬────┬────┬─┴────▼─┬──┘ 1: init()
+ │ │ │ │ ▲ │ ▲ │ ▲ │ ▲ │ 2: configure()
+ │1 │2 │3 │4│ │4│ │4│ │4│ │5 3: mapBuffers(), start()
+ ▼ ▼ ▼ ▼ │ ▼ │ ▼ │ ▼ │ ▼ 4: processEvent()
+ ┌──────────────────┴────┴────┴────┴─────────┐ 5: stop(), unmapBuffers()
+ │ IPU3 IPA │
+ │ ┌───────────────────────┐ │
+ │ ┌───────────┐ │ Algorithms │ │
+ │ │IPAContext │ │ ┌─────────┐ │ │
+ │ │ ┌───────┐ │ │ │ ... │ │ │
+ │ │ │ │ │ │ ┌─┴───────┐ │ │ │
+ │ │ │ SC │ │ │ │ Tonemap ├─┘ │ │
+ │ │ │ │ ◄───► ┌─┴───────┐ │ │ │
+ │ │ ├───────┤ │ │ │ AWB ├─┘ │ │
+ │ │ │ │ │ │ ┌─┴───────┐ │ │ │
+ │ │ │ FC │ │ │ │ AGC ├─┘ │ │
+ │ │ │ │ │ │ │ │ │ │
+ │ │ └───────┘ │ │ └─────────┘ │ │
+ │ └───────────┘ └───────────────────────┘ │
+ └───────────────────────────────────────────┘
+ SC: IPASessionConfiguration
+ FC: IPAFrameContext(s)
+
+The IPA instance is constructed and initialised at the point a Camera is
+created by the IPU3 pipeline handler. The initialisation call provides
+details about which camera sensor is being used, and the controls that
+it has available, along with their default values and ranges.
+
+Buffers
+~~~~~~~
+
+The IPA will have Parameter and Statistics buffers shared with it from
+the IPU3 Pipeline handler. These buffers will be passed to the IPA using
+the ``mapBuffers()`` call before the ``start()`` operation occurs.
+
+The IPA will map the buffers into CPU-accessible memory, associated with
+a buffer ID, and further events for sending or receiving parameter and
+statistics buffers will reference the ID to avoid expensive memory
+mapping operations, or the passing of file handles during streaming.
+
+After the ``stop()`` operation occurs, these buffers will be unmapped
+when requested by the pipeline handler using the ``unmapBuffers()`` call
+and no further access to the buffers is permitted.
+
+Context
+~~~~~~~
+
+Algorithm calls will always have the ``IPAContext`` available to them.
+This context comprises of two parts:
+
+- IPA Session Configuration
+- IPA Frame Context
+
+The session configuration structure ``IPASessionConfiguration``
+represents constant parameters determined before streaming commenced
+during ``configure()``.
+
+The IPA Frame Context provides the storage for algorithms for a single
+frame operation.
+
+The ``IPAFrameContext`` structure may be extended to an array, list, or
+queue to store historical state for each frame, allowing algorithms to
+obtain and reference results of calculations which are deeply pipelined.
+This may only be done if an algorithm needs to know the context that was
+applied at the frame the statistics were produced for, rather than the
+previous or current frame.
+
+Presently there is a single ``IPAFrameContext`` without historical data,
+and the context is maintained and updated through successive processing
+operations.
+
+Operating
+~~~~~~~~~
+
+There are three main interactions with the algorithms for the IPU3 IPA
+to operate when running:
+
+- configure()
+- processEvent(``EventFillParams``)
+- processEvent(``EventStatReady``)
+
+The configuration phase allows the pipeline-handler to inform the IPA of
+the current stream configurations, which is then passed into each
+algorithm to provide an opportunity to identify and track state of the
+hardware, such as image size or ImgU pipeline configurations.
+
+Pre-frame preparation
+~~~~~~~~~~~~~~~~~~~~~
+
+When configured, the IPA is notified by the pipeline handler of the
+Camera ``start()`` event, after which incoming requests will be queued
+for processing, requiring a parameter buffer (``ipu3_uapi_params``) to
+be populated for the ImgU. This is given to the IPA through the
+``EventFillParams`` event, and then passed directly to each algorithm
+through the ``prepare()`` call allowing the ISP configuration to be
+updated for the needs of each component that the algorithm is
+responsible for.
+
+The algorithm should set the use flag (``ipu3_uapi_flags``) for any
+structure that it modifies, and it should take care to ensure that any
+structure set by a use flag is fully initialised to suitable values.
+
+The parameter buffer is returned to the pipeline handler through the
+``ActionParamFilled`` event, and from there queued to the ImgU along
+with a raw frame captured with the CIO2.
+
+Post-frame completion
+~~~~~~~~~~~~~~~~~~~~~
+
+When the capture of an image is completed, and successfully processed
+through the ImgU, the generated statistics buffer
+(``ipu3_uapi_stats_3a``) is given to the IPA through the
+``EventStatReady`` event. This provides the IPA with an opportunity to
+examine the results of the ISP and run the calculations required by each
+algorithm on the new data. The algorithms may require context from the
+operations of other algorithms, for example, the AWB might choose to use
+a scene brightness determined by the AGC. It is important that the
+algorithms are ordered to ensure that required results are determined
+before they are needed.
+
+The ordering of the algorithm processing is determined by their
+placement in the ``IPU3::algorithms_`` ordered list.
+
+Sensor Controls
+~~~~~~~~~~~~~~~
+
+The AutoExposure and AutoGain (AGC) algorithm differs slightly from the
+others as it requires operating directly on the sensor, as opposed to
+through the ImgU ISP. To support this, there is a dedicated action
+`ActionSetSensorControls` to allow the IPA to request controls to be set
+on the camera sensor through the pipeline handler.