libcamera: Avoid variable-length arrays

Unlike in C where they have been standardized since C99, variable-length
arrays in C++ are an extension supported by gcc and clang. Clang started
warning about this with -Wall in version 18:

src/libcamera/ipc_unixsocket.cpp:250:11: error: variable length arrays in C++ are a Clang extension [-Werror,-Wvla-cxx-extension]
  250 |         char buf[CMSG_SPACE(num * sizeof(uint32_t))];
      |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

One simple option is to disable the warning. However, usage of VLAs in
C++ is discouraged by some, usually due to security reasons, based on
the rationale that developers are often unaware of unintentional use of
VLAs and how they may affect the security of the code when the array
size is not properly validated.

This rationale may sound dubious, as the most commonly proposed fix is
to replace VLAs with vectors (or just arrays dynamically allocated with
new() wrapped in unique pointers), without adding any size validation.
This will not produce much better results. However, keeping the VLA
warning and converting the code to dynamic allocation may still be
slightly better, as it can prompt developers to notice VLAs and check if
size validation is required.

For these reasons, convert all VLAs to std::vector. Most of the VLAs
don't need extra size validation, as the size is bound through different
constraints (e.g. image width for line buffers). An arguable exception
may be the buffers in IPCUnixSocket::sendData() and
IPCUnixSocket::recvData() as the number of fds is not bound-checked
locally, but we will run out of file descriptors before we could
overflow the buffer size calculation.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Milan Zamazal <mzamazal@redhat.com>
Acked-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>

1 files changed, 3 insertions, 3 deletions

diff --git a/src/android/jpeg/encoder_libjpeg.cpp b/src/android/jpeg/encoder_libjpeg.cpp
index 7fc6287e..cb242b5e 100644
--- a/src/android/jpeg/encoder_libjpeg.cpp
+++ b/src/android/jpeg/encoder_libjpeg.cpp
@@ -125,7 +125,7 @@ void EncoderLibJpeg::compressRGB(const std::vector<Span<uint8_t>> &planes)
  */
 void EncoderLibJpeg::compressNV(const std::vector<Span<uint8_t>> &planes)
 {
-	uint8_t tmprowbuf[compress_.image_width * 3];
+	std::vector<uint8_t> tmprowbuf(compress_.image_width * 3);
 
 	/*
 	 * \todo Use the raw api, and only unpack the cb/cr samples to new line
@@ -149,10 +149,10 @@ void EncoderLibJpeg::compressNV(const std::vector<Span<uint8_t>> &planes)
 	const unsigned char *src_c = planes[1].data();
 
 	JSAMPROW row_pointer[1];
-	row_pointer[0] = &tmprowbuf[0];
+	row_pointer[0] = tmprowbuf.data();
 
 	for (unsigned int y = 0; y < compress_.image_height; y++) {
-		unsigned char *dst = &tmprowbuf[0];
+		unsigned char *dst = tmprowbuf.data();
 
 		const unsigned char *src_y = src + y * y_stride;
 		const unsigned char *src_cb = src_c + (y / vertSubSample) * c_stride + cb_pos;