lc-compliance: Add a libcamera compliance tool

Add a compliance tool to ease testing of cameras. In contrast to the
unit-tests under test/ that aims to test the internal components of
libcamera the compliance tool aims to test application use-cases and to
some extent the public API.

This change adds the boilerplate code of a simple framework for the
creation of tests. The tests aim both to demonstrate the tool and to
catch real problems. The tests added are:

 - Test that if one queues exactly N requests to a camera exactly N
   requests are eventually completed.

 - Test that a configured camera can be started and stopped multiple
   times in an attempt to exercise cleanup code paths otherwise not
   often tested with 'cam' for example.

Signed-off-by: Niklas Söderlund <niklas.soderlund@ragnatech.se>
Acked-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Tested-by: Jean-Michel Hautbois <jeanmichel.hautbois@ideasonboard.com>

1 files changed, 75 insertions, 0 deletions

diff --git a/src/lc-compliance/results.cpp b/src/lc-compliance/results.cpp
new file mode 100644
index 00000000..f149f785
--- /dev/null
+++ b/src/lc-compliance/results.cpp
@@ -0,0 +1,75 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (C) 2020, Google Inc.
+ *
+ * results.cpp - Test result aggregator
+ */
+
+#include "results.h"
+
+#include <iostream>
+
+void Results::add(const Result &result)
+{
+	if (result.first == Pass)
+		passed_++;
+	else if (result.first == Fail)
+		failed_++;
+	else if (result.first == Skip)
+		skipped_++;
+
+	printResult(result);
+}
+
+void Results::add(Status status, const std::string &message)
+{
+	add({ status, message });
+}
+
+void Results::fail(const std::string &message)
+{
+	add(Fail, message);
+}
+
+void Results::pass(const std::string &message)
+{
+	add(Pass, message);
+}
+
+void Results::skip(const std::string &message)
+{
+	add(Skip, message);
+}
+
+int Results::summary() const
+{
+	if (failed_ + passed_ + skipped_ != planned_) {
+		std::cout << "Planned and executed number of tests differ "
+			  << failed_ + passed_ + skipped_ << " executed "
+			  << planned_ << " planned" << std::endl;
+
+		return -EINVAL;
+	}
+
+	std::cout << planned_ << " tests executed, "
+		  << passed_ << " tests passed, "
+		  << skipped_ << " tests skipped and "
+		  << failed_ << " tests failed " << std::endl;
+
+	return 0;
+}
+
+void Results::printResult(const Result &result)
+{
+	std::string prefix;
+
+	/* \todo Make parsable as TAP. */
+	if (result.first == Pass)
+		prefix = "PASS";
+	else if (result.first == Fail)
+		prefix = "FAIL";
+	else if (result.first == Skip)
+		prefix = "SKIP";
+
+	std::cout << "- " << prefix << ": " << result.second << std::endl;
+}