summaryrefslogtreecommitdiff
path: root/utils/raspberrypi/ctt
AgeCommit message (Collapse)Author
2024-05-08libcamera: Drop file name from header comment blocksLaurent Pinchart
Source files in libcamera start by a comment block header, which includes the file name and a one-line description of the file contents. While the latter is useful to get a quick overview of the file contents at a glance, the former is mostly a source of inconvenience. The name in the comments can easily get out of sync with the file name when files are renamed, and copy & paste during development have often lead to incorrect names being used to start with. Readers of the source code are expected to know which file they're looking it. Drop the file name from the header comment block. The change was generated with the following script: ---------------------------------------- dirs="include/libcamera src test utils" declare -rA patterns=( ['c']=' \* ' ['cpp']=' \* ' ['h']=' \* ' ['py']='# ' ['sh']='# ' ) for ext in ${!patterns[@]} ; do files=$(for dir in $dirs ; do find $dir -name "*.${ext}" ; done) pattern=${patterns[${ext}]} for file in $files ; do name=$(basename ${file}) sed -i "s/^\(${pattern}\)${name} - /\1/" "$file" done done ---------------------------------------- This misses several files that are out of sync with the comment block header. Those will be addressed separately and manually. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
2024-04-20libcamera: Fix output spelling errorUmang Jain
There are two places where "output" is spelled as "ouput". Fix it. Signed-off-by: Umang Jain <umang.jain@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
2024-01-09utils: raspberrypi: ctt: Improve the Macbeth Chart search reliabilityDavid Plowman
Previously the code would brighten up images in case the Macbeth Chart is slightly dark, and also zoom in on sections of it to look for charts occupying less of the field of view. But it would not do both together. This change makes the search for smaller charts also repeat that search for the brightened up images that it made earlier, thereby increasing the chances of success for non-optimal tuning images. There are also a couple of very small drive-by typo fixes. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> Reviewed-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
2023-07-28utils: raspberrypi: ctt: Code tidyingBen Benson
Altered the way that some lines are laid out, made functions more attractive to look at, and tidied up messy areas. Signed-off-by: Ben Benson <ben.benson@raspberrypi.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Reviewed-by: Naushir Patuck <naush@raspberrypi.com> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
2023-07-28utils: raspberrypi: ctt: Improved color matrix fittingBen Benson
Added code which optimises the color matrices based off delta E values for the calibration images. Working in LAB color space. Signed-off-by: Ben Benson <ben.benson@raspberrypi.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Reviewed-by: Naushir Patuck <naush@raspberrypi.com> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
2022-10-20utils: raspberrypi: ctt: Fix alsc green averagingPaul Elder
The alsc component of ctt meant to average the two green channels into one, but used incorrect indexing resulting in only the first green channel being used. Fix this. Signed-off-by: Paul Elder <paul.elder@ideasonboard.com> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2022-08-03utils: raspberrypi: ctt: dng_load_image: Work with DNG files from Picamera2William Vinnicombe
The DNG specification is based on the TIFF file format and recommends storing the raw image data in a SubIFD and the Exif tags in an Exif IFD. Other options are allowed, even if not recommended, such as storing both the raw image data and the Exif data in IFD0, as done by the TIFF/EP specification. libcamera-apps use pyexiv2 to produce DNG files, following the DNG recommendation, while applications based on picamera2 use PiDNG, which adopts the TIFF/EP structure. Why it does so is not currently clear (see https://github.com/schoolpost/PiDNG/issues/65 for discussions on this topic), but as files based on the DNG and TIFF/EP variants exist in the wild, both need to be supported by ctt. Add code to identify which tags are being used, and then load the metadata from the correct tags. Signed-off-by: William Vinnicombe <william.vinnicombe@raspberrypi.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Reviewed-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
2022-08-02utils: raspberrypi: ctt: Add alsc_only methodWilliam Vinnicombe
The ctt would not work if only passed alsc images. Add alsc_only.py to run alsc calibration only, and modify check_imgs to allow for no macbeth chart images. Example usage would be ./alsc_only.py -i tuning-images/ -o sensor.json with the same optional arguments as the original ctt. Signed-off-by: William Vinnicombe <william.vinnicombe@raspberrypi.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Reviewed-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
2022-07-28utils: raspberrypi: Add tuning file conversion scriptNaushir Patuck
Add a script to convert the Raspberry Pi camera tuning file format from version 1.0 to 2.0. This script also adds a root level version key set to 2.0 to the config file, allowing the controller to distinguish between the two formats. Signed-off-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Tested-by: Naushir Patuck <naush@raspberrypi.com>
2022-07-28utils: raspberrypi: ctt: Output version 2.0 format tuning filesNaushir Patuck
Update the ctt_pretty_print_json.py script to generate the new version 2.0 format camera tuning file. This script can be called through the command line to prettify an existing JSON file, or programatically by the CTT to format a new JSON config dictionary. Update the CTT to produce a version 2.0 format json structure and use ctt_pretty_print_json.pretty_print to prettify the output. Signed-off-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Tested-by: Naushir Patuck <naush@raspberrypi.com>
2022-07-27raspberrypi: Update Copyright statement in all Raspberry Pi source filesNaushir Patuck
s/Raspberry Pi (Trading) Limited/Raspberry Pi Ltd/ to reflect the new Raspberry Pi entity name. Signed-off-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
2022-07-06utils: raspberrypi: ctt: load_image: Ignore JPEG files with no raw dataWilliam Vinnicombe
The load_image function would throw errors with JPEG or JPG files containing no raw data. Prevent throwing these errors by returning 0 if an error has occurred. Signed-off-by: William Vinnicombe <william.vinnicombe@raspberrypi.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
2021-08-03utils: raspberrypi: ctt: Fix namespace for sklearn NearestCentroid functionDavid Plowman
Starting in version 0.22, the NearestCentroid function is only available in the sklearn.neighbors namespace, when it was previously available in both the sklearn.neighbors.nearest_centroid and sklearn.neighbors namespaces. Use sklearn.neighbors as it works on all versions of sklearn. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> Reviewed-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
2021-08-02utils: raspberrypi: ctt: Fix usage of findHomography functionDavid Plowman
The OpenCV findHomography function now raises an unhandled error if it receives fewer than 4 points whereas previously the limit was 3. This makes no material difference to the behaviour of the tuning tool as it will continue to search for the Macbeth chart at different scales. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> Reviewed-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
2020-11-20src: ipa: raspberrypi: Change 'sport' exposure mode name to 'short'David Plowman
The names have to match for the setting to work. Use the libcamera terminology for consistency (even though it touches more files). Signed-off-by: David Plowman <david.plowman@raspberrypi.com> Reviewed-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
2020-07-03utils: raspberrypi: ctt: json_pretty_print: Add newline at end of outputLaurent Pinchart
Make sure the output ends with a newline. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Tested-by: David Plowman <david.plowman@raspberrypi.com>
2020-07-03utils: raspberrypi: ctt: json_pretty_print: Avoid spaces at end of linesLaurent Pinchart
Avoid outputting spaces at end of lines by recording the need for a space and outputting it before the next character only if not a newline. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Tested-by: David Plowman <david.plowman@raspberrypi.com>
2020-07-03utils: raspberrypi: ctt: json_pretty_print: Collapse newlinesLaurent Pinchart
Simplify the newline skipping logic by simply collapsing newlines. If a newline has been output, all subsequent newlines will be skipped until the next non-newline character is output. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Tested-by: David Plowman <david.plowman@raspberrypi.com>
2020-07-03utils: raspberrypi: ctt: json_pretty_print: Fix indentation handlingLaurent Pinchart
Indentation is handled by outputting spaces right after outputting a newline character. That works in most cases, but would result in the input '{}' being printed as { } instead of { } Fix it by outputting the indentation before outputting the next character after a newline. The indentation value will be updated by then. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Tested-by: David Plowman <david.plowman@raspberrypi.com>
2020-07-03utils: raspberrypi: ctt: json_pretty_print: Add character write methodLaurent Pinchart
Add a write method to the JSONPrettyPrinter class to output a character. This will be used to handle state updates when outputting individual characters. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Tested-by: David Plowman <david.plowman@raspberrypi.com>
2020-07-03utils: raspberrypi: ctt: json_pretty_print: Skip all spacesLaurent Pinchart
Skip all white space characters, not just ' '. This makes a difference if the input JSON data is already formatted. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Tested-by: David Plowman <david.plowman@raspberrypi.com>
2020-07-03utils: raspberrypi: ctt: json_pretty_print: Make test output to stdoutLaurent Pinchart
The standalone test mode output to a file name "pretty.json". To make the test mode more versatile, output to stdout instead. The user can then decide how to use the output. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Tested-by: David Plowman <david.plowman@raspberrypi.com>
2020-07-03utils: raspberrypi: ctt: json_pretty_print: Make output file a class memberLaurent Pinchart
Instead of passing the output file to every method of the printer class, make it a class member. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Tested-by: David Plowman <david.plowman@raspberrypi.com>
2020-07-03utils: raspberrypi: ctt: json_pretty_print: Turn printer into a classLaurent Pinchart
Instead of passing a state dictionary to every method, turn the printer into a class and store the state internally. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Tested-by: David Plowman <david.plowman@raspberrypi.com>
2020-07-03utils: raspberrypi: ctt: json_pretty_print: Fix printer testLaurent Pinchart
The ctt_pretty_print_json.py file supports being run standalone to test the code. It however suffers from multiple issues: - The same input file name is hardcoded, and doesn't exist in the repository - The input file name is used instead of JSON data Fix both issues and make the input file selectable on the command line. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com> Tested-by: David Plowman <david.plowman@raspberrypi.com>
2020-06-29utils: raspberrypi: ctt: Fix pycodestyle W605Laurent Pinchart
W605 invalid escape sequence '\.' Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Niklas Söderlund <niklas.soderlund@ragnatech.se> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E302Laurent Pinchart
E302 expected 2 blank lines, found 0 Note that issues are still flagged, due to the use of docstrings as multi-lines comments. This will be addressed separately. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E305Laurent Pinchart
E305 expected 2 blank lines after class or function definition Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E741Laurent Pinchart
E741 ambiguous variable name 'l' Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle W504Laurent Pinchart
W504 line break after binary operator Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E722Laurent Pinchart
E722 do not use bare 'except' Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E721Laurent Pinchart
E721 do not compare types, use 'isinstance()' Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E713Laurent Pinchart
E713 test for membership should be 'not in' Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E116 and E117Laurent Pinchart
E116 unexpected indentation (comment) E117 over-indented (comment) Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E123 and E126Laurent Pinchart
E123 closing bracket does not match indentation of opening bracket's line E126 continuation line over-indented for hanging indent Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E711 and E712Laurent Pinchart
E711 comparison to None should be 'if cond is None:' E711 comparison to None should be 'if cond is not None:' E712 comparison to False should be 'if cond is False:' or 'if not cond:' E712 comparison to True should be 'if cond is True:' or 'if cond:' Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E222Laurent Pinchart
E222 multiple spaces after operator Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E261 and E262Laurent Pinchart
E261 at least two spaces before inline comment E262 inline comment should start with '# ' Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E303Laurent Pinchart
E303 too many blank lines Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E701Laurent Pinchart
E701 multiple statements on one line (colon) Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E228Laurent Pinchart
E228 missing whitespace around modulo operator Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E225Laurent Pinchart
E225 missing whitespace around operator Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E128Laurent Pinchart
E128 continuation line under-indented for visual indent Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E251Laurent Pinchart
E251 unexpected spaces around keyword / parameter equals Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E211Laurent Pinchart
E211 whitespace before '[' Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E241Laurent Pinchart
E241 multiple spaces after ':' E241 multiple spaces after ',' Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E203Laurent Pinchart
E203 whitespace before ':' Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E201 and E202Laurent Pinchart
E201 whitespace after '(' E201 whitespace after '{' E201 whitespace after '[' E202 whitespace before '}' E202 whitespace before ']' Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-13utils: raspberrypi: ctt: Fix pycodestyle E231Laurent Pinchart
E231 missing whitespace after ',' E231 missing whitespace after ':' Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
2020-05-11libcamera: utils: Raspberry Pi Camera Tuning ToolNaushir Patuck
Initial implementation of the Raspberry Pi (BCM2835) Camera Tuning Tool. All code is licensed under the BSD-2-Clause terms. Copyright (c) 2019-2020 Raspberry Pi Trading Ltd. Signed-off-by: Naushir Patuck <naush@raspberrypi.com> Acked-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
n>)) self.__to_line = int(match.group(3)) self.__from = DiffHunkSide(self.__from_line) self.__to = DiffHunkSide(self.__to_line) self.lines = [] def __repr__(self): s = '%s@@ -%u,%u +%u,%u @@\n' % \ (Colours.fg(Colours.Cyan), self.__from.start, len(self.__from), self.__to.start, len(self.__to)) for line in self.lines: if line[0] == '-': s += Colours.fg(Colours.Red) elif line[0] == '+': s += Colours.fg(Colours.Green) if line[0] == '-': spaces = 0 for i in range(len(line)): if line[-i-1].isspace(): spaces += 1 else: break spaces = len(line) - spaces line = line[0:spaces] + Colours.bg(Colours.Red) + line[spaces:] s += line s += Colours.reset() s += '\n' return s[:-1] def append(self, line): if line[0] == ' ': self.__from.untouched.append(self.__from_line) self.__from_line += 1 self.__to.untouched.append(self.__to_line) self.__to_line += 1 elif line[0] == '-': self.__from.touched.append(self.__from_line) self.__from_line += 1 elif line[0] == '+': self.__to.touched.append(self.__to_line) self.__to_line += 1 self.lines.append(line.rstrip('\n')) def intersects(self, lines): for line in lines: if line in self.__from.touched: return True return False def side(self, side): if side == 'from': return self.__from else: return self.__to def parse_diff(diff): hunks = [] hunk = None for line in diff: if line.startswith('@@'): if hunk: hunks.append(hunk) hunk = DiffHunk(line) elif hunk is not None: hunk.append(line) if hunk: hunks.append(hunk) return hunks # ------------------------------------------------------------------------------ # Commit, Staged Changes & Amendments # class CommitFile: def __init__(self, name): info = name.split() self.__status = info[0][0] # For renamed files, store the new name if self.__status == 'R': self.__filename = info[2] else: self.__filename = info[1] @property def filename(self): return self.__filename @property def status(self): return self.__status class Commit: def __init__(self, commit): self.commit = commit self._parse() def _parse(self): # Get the commit title and list of files. ret = subprocess.run(['git', 'show', '--pretty=oneline', '--name-status', self.commit], stdout=subprocess.PIPE).stdout.decode('utf-8') files = ret.splitlines() self._files = [CommitFile(f) for f in files[1:]] self._title = files[0] def files(self, filter='AM'): return [f.filename for f in self._files if f.status in filter] @property def title(self): return self._title def get_diff(self, top_level, filename): diff = subprocess.run(['git', 'diff', '%s~..%s' % (self.commit, self.commit), '--', '%s/%s' % (top_level, filename)], stdout=subprocess.PIPE).stdout.decode('utf-8') return parse_diff(diff.splitlines(True)) def get_file(self, filename): return subprocess.run(['git', 'show', '%s:%s' % (self.commit, filename)], stdout=subprocess.PIPE).stdout.decode('utf-8') class StagedChanges(Commit): def __init__(self): Commit.__init__(self, '') def _parse(self): ret = subprocess.run(['git', 'diff', '--staged', '--name-status'], stdout=subprocess.PIPE).stdout.decode('utf-8') self._title = "Staged changes" self._files = [CommitFile(f) for f in ret.splitlines()] def get_diff(self, top_level, filename): diff = subprocess.run(['git', 'diff', '--staged', '--', '%s/%s' % (top_level, filename)], stdout=subprocess.PIPE).stdout.decode('utf-8') return parse_diff(diff.splitlines(True)) class Amendment(StagedChanges): def __init__(self): StagedChanges.__init__(self) def _parse(self): # Create a title using HEAD commit ret = subprocess.run(['git', 'show', '--pretty=oneline', '--no-patch'], stdout=subprocess.PIPE).stdout.decode('utf-8') self._title = 'Amendment of ' + ret.strip() # Extract the list of modified files ret = subprocess.run(['git', 'diff', '--staged', '--name-status', 'HEAD~'], stdout=subprocess.PIPE).stdout.decode('utf-8') self._files = [CommitFile(f) for f in ret.splitlines()] def get_diff(self, top_level, filename): diff = subprocess.run(['git', 'diff', '--staged', 'HEAD~', '--', '%s/%s' % (top_level, filename)], stdout=subprocess.PIPE).stdout.decode('utf-8') return parse_diff(diff.splitlines(True)) # ------------------------------------------------------------------------------ # Helpers # class ClassRegistry(type): def __new__(cls, clsname, bases, attrs): newclass = super().__new__(cls, clsname, bases, attrs) if bases: bases[0].subclasses.append(newclass) return newclass # ------------------------------------------------------------------------------ # Commit Checkers # class CommitChecker(metaclass=ClassRegistry): subclasses = [] def __init__(self): pass # # Class methods # @classmethod def checkers(cls): for checker in cls.subclasses: yield checker class CommitIssue(object): def __init__(self, msg): self.msg = msg class HeaderAddChecker(CommitChecker): @classmethod def check(cls, commit, top_level): issues = [] meson_files = [f for f in commit.files('M') if os.path.basename(f) == 'meson.build'] for filename in commit.files('A'): if not filename.startswith('include/libcamera/') or \ not filename.endswith('.h'): continue meson = os.path.dirname(filename) + '/meson.build' header = os.path.basename(filename) issue = CommitIssue('Header %s added without corresponding update to %s' % (filename, meson)) if meson not in meson_files: issues.append(issue) continue diff = commit.get_diff(top_level, meson) found = False for hunk in diff: for line in hunk.lines: if line[0] != '+': continue if line.find("'%s'" % header) != -1: found = True break if found: break if not found: issues.append(issue) return issues # ------------------------------------------------------------------------------ # Style Checkers # class StyleChecker(metaclass=ClassRegistry): subclasses = [] def __init__(self): pass # # Class methods # @classmethod def checkers(cls, filename): for checker in cls.subclasses: if checker.supports(filename): yield checker @classmethod def supports(cls, filename): for pattern in cls.patterns: if fnmatch.fnmatch(os.path.basename(filename), pattern): return True return False @classmethod def all_patterns(cls): patterns = set() for checker in cls.subclasses: patterns.update(checker.patterns) return patterns class StyleIssue(object): def __init__(self, line_number, line, msg): self.line_number = line_number self.line = line self.msg = msg class IncludeChecker(StyleChecker): patterns = ('*.cpp', '*.h') headers = ('assert', 'ctype', 'errno', 'fenv', 'float', 'inttypes', 'limits', 'locale', 'setjmp', 'signal', 'stdarg', 'stddef', 'stdint', 'stdio', 'stdlib', 'string', 'time', 'uchar', 'wchar', 'wctype') include_regex = re.compile('^#include <c([a-z]*)>') def __init__(self, content): super().__init__() self.__content = content def check(self, line_numbers): issues = [] for line_number in line_numbers: line = self.__content[line_number - 1] match = IncludeChecker.include_regex.match(line) if not match: continue header = match.group(1) if header not in IncludeChecker.headers: continue issues.append(StyleIssue(line_number, line, 'C compatibility header <%s.h> is preferred' % header)) return issues class LogCategoryChecker(StyleChecker): log_regex = re.compile('\\bLOG\((Debug|Info|Warning|Error|Fatal)\)') patterns = ('*.cpp',) def __init__(self, content): super().__init__() self.__content = content def check(self, line_numbers): issues = [] for line_number in line_numbers: line = self.__content[line_number-1] if not LogCategoryChecker.log_regex.search(line): continue issues.append(StyleIssue(line_number, line, 'LOG() should use categories')) return issues class MesonChecker(StyleChecker): patterns = ('meson.build',) def __init__(self, content): super().__init__() self.__content = content def check(self, line_numbers): issues = [] for line_number in line_numbers: line = self.__content[line_number-1] if line.find('\t') != -1: issues.append(StyleIssue(line_number, line, 'meson.build should use spaces for indentation')) return issues class Pep8Checker(StyleChecker): patterns = ('*.py',) results_regex = re.compile('stdin:([0-9]+):([0-9]+)(.*)') def __init__(self, content): super().__init__() self.__content = content def check(self, line_numbers): issues = [] data = ''.join(self.__content).encode('utf-8') try: ret = subprocess.run(['pycodestyle', '--ignore=E501', '-'], input=data, stdout=subprocess.PIPE) except FileNotFoundError: issues.append(StyleIssue(0, None, "Please install pycodestyle to validate python additions")) return issues results = ret.stdout.decode('utf-8').splitlines() for item in results: search = re.search(Pep8Checker.results_regex, item) line_number = int(search.group(1)) position = int(search.group(2)) msg = search.group(3) if line_number in line_numbers: line = self.__content[line_number - 1] issues.append(StyleIssue(line_number, line, msg)) return issues class ShellChecker(StyleChecker): patterns = ('*.sh',) results_line_regex = re.compile('In - line ([0-9]+):') def __init__(self, content): super().__init__() self.__content = content def check(self, line_numbers): issues = [] data = ''.join(self.__content).encode('utf-8') try: ret = subprocess.run(['shellcheck', '-Cnever', '-'], input=data, stdout=subprocess.PIPE) except FileNotFoundError: issues.append(StyleIssue(0, None, "Please install shellcheck to validate shell script additions")) return issues results = ret.stdout.decode('utf-8').splitlines() for nr, item in enumerate(results): search = re.search(ShellChecker.results_line_regex, item) if search is None: continue line_number = int(search.group(1)) line = results[nr + 1] msg = results[nr + 2] # Determined, but not yet used position = msg.find('^') + 1 if line_number in line_numbers: issues.append(StyleIssue(line_number, line, msg)) return issues # ------------------------------------------------------------------------------ # Formatters # class Formatter(metaclass=ClassRegistry): subclasses = [] def __init__(self): pass # # Class methods # @classmethod def formatters(cls, filename): for formatter in cls.subclasses: if formatter.supports(filename): yield formatter @classmethod def supports(cls, filename): for pattern in cls.patterns: if fnmatch.fnmatch(os.path.basename(filename), pattern): return True return False @classmethod def all_patterns(cls): patterns = set() for formatter in cls.subclasses: patterns.update(formatter.patterns) return patterns class CLangFormatter(Formatter): patterns = ('*.c', '*.cpp', '*.h') @classmethod def format(cls, filename, data): ret = subprocess.run(['clang-format', '-style=file', '-assume-filename=' + filename], input=data.encode('utf-8'), stdout=subprocess.PIPE) return ret.stdout.decode('utf-8') class DoxygenFormatter(Formatter): patterns = ('*.c', '*.cpp') return_regex = re.compile(' +\\* +\\\\return +[a-z]') @classmethod def format(cls, filename, data): lines = [] in_doxygen = False for line in data.split('\n'): if line.find('/**') != -1: in_doxygen = True if not in_doxygen: lines.append(line) continue line = cls.return_regex.sub(lambda m: m.group(0)[:-1] + m.group(0)[-1].upper(), line) if line.find('*/') != -1: in_doxygen = False lines.append(line) return '\n'.join(lines) class DPointerFormatter(Formatter): # Ensure consistent naming of variables related to the d-pointer design # pattern. patterns = ('*.cpp', '*.h') # The clang formatter runs first, we can thus rely on appropriate coding # style. declare_regex = re.compile(r'^(\t*)(const )?([a-zA-Z0-9_]+) \*( ?const )?([a-zA-Z0-9_]+) = (LIBCAMERA_[DO]_PTR)\(([a-zA-Z0-9_]+)\);$') @classmethod def format(cls, filename, data): lines = [] for line in data.split('\n'): match = cls.declare_regex.match(line) if match: indent = match.group(1) or '' const = match.group(2) or '' macro = match.group(6) klass = match.group(7) if macro == 'LIBCAMERA_D_PTR': var = 'Private *const d' else: var = f'{klass} *const o' line = f'{indent}{const}{var} = {macro}({klass});' lines.append(line) return '\n'.join(lines) class IncludeOrderFormatter(Formatter): patterns = ('*.cpp', '*.h') include_regex = re.compile('^#include ["<]([^">]*)[">]') @classmethod def format(cls, filename, data): lines = [] includes = [] # Parse blocks of #include statements, and output them as a sorted list # when we reach a non #include statement. for line in data.split('\n'): match = IncludeOrderFormatter.include_regex.match(line) if match: # If the current line is an #include statement, add it to the # includes group and continue to the next line. includes.append((line, match.group(1))) continue # The current line is not an #include statement, output the sorted # stashed includes first, and then the current line. if len(includes): includes.sort(key=lambda i: i[1]) for include in includes: lines.append(include[0]) includes = [] lines.append(line) # In the unlikely case the file ends with an #include statement, make # sure we output the stashed includes. if len(includes): includes.sort(key=lambda i: i[1]) for include in includes: lines.append(include[0]) includes = [] return '\n'.join(lines) class StripTrailingSpaceFormatter(Formatter): patterns = ('*.c', '*.cpp', '*.h', '*.py', 'meson.build') @classmethod def format(cls, filename, data): lines = data.split('\n') for i in range(len(lines)): lines[i] = lines[i].rstrip() + '\n' return ''.join(lines) # ------------------------------------------------------------------------------ # Style checking # def check_file(top_level, commit, filename): # Extract the line numbers touched by the commit. commit_diff = commit.get_diff(top_level, filename) lines = [] for hunk in commit_diff: lines.extend(hunk.side('to').touched) # Skip commits that don't add any line. if len(lines) == 0: return 0 # Format the file after the commit with all formatters and compute the diff # between the unformatted and formatted contents. after = commit.get_file(filename) formatted = after for formatter in Formatter.formatters(filename): formatted = formatter.format(filename, formatted) after = after.splitlines(True) formatted = formatted.splitlines(True) diff = difflib.unified_diff(after, formatted) # Split the diff in hunks, recording line number ranges for each hunk, and # filter out hunks that are not touched by the commit. formatted_diff = parse_diff(diff) formatted_diff = [hunk for hunk in formatted_diff if hunk.intersects(lines)] # Check for code issues not related to formatting. issues = [] for checker in StyleChecker.checkers(filename): checker = checker(after) for hunk in commit_diff: issues += checker.check(hunk.side('to').touched) # Print the detected issues. if len(issues) == 0 and len(formatted_diff) == 0: return 0 print('%s---' % Colours.fg(Colours.Red), filename) print('%s+++' % Colours.fg(Colours.Green), filename) if len(formatted_diff): for hunk in formatted_diff: print(hunk) if len(issues): issues = sorted(issues, key=lambda i: i.line_number) for issue in issues: print('%s#%u: %s' % (Colours.fg(Colours.Yellow), issue.line_number, issue.msg)) if issue.line is not None: print('+%s%s' % (issue.line.rstrip(), Colours.reset())) return len(formatted_diff) + len(issues) def check_style(top_level, commit): separator = '-' * len(commit.title) print(separator) print(commit.title) print(separator) issues = 0 # Apply the commit checkers first. for checker in CommitChecker.checkers(): for issue in checker.check(commit, top_level): print('%s%s%s' % (Colours.fg(Colours.Yellow), issue.msg, Colours.reset())) issues += 1 # Filter out files we have no checker for. patterns = set() patterns.update(StyleChecker.all_patterns()) patterns.update(Formatter.all_patterns()) files = [f for f in commit.files() if len([p for p in patterns if fnmatch.fnmatch(os.path.basename(f), p)])] for f in files: issues += check_file(top_level, commit, f) if issues == 0: print("No issue detected") else: print('---') print("%u potential %s detected, please review" % (issues, 'issue' if issues == 1 else 'issues')) return issues def extract_commits(revs): """Extract a list of commits on which to operate from a revision or revision range. """ ret = subprocess.run(['git', 'rev-parse', revs], stdout=subprocess.PIPE, stderr=subprocess.PIPE) if ret.returncode != 0: print(ret.stderr.decode('utf-8').splitlines()[0]) return [] revlist = ret.stdout.decode('utf-8').splitlines() # If the revlist contains more than one item, pass it to git rev-list to list # each commit individually. if len(revlist) > 1: ret = subprocess.run(['git', 'rev-list', *revlist], stdout=subprocess.PIPE) revlist = ret.stdout.decode('utf-8').splitlines() revlist.reverse() return [Commit(x) for x in revlist] def git_top_level(): """Get the absolute path of the git top-level directory.""" ret = subprocess.run(['git', 'rev-parse', '--show-toplevel'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) if ret.returncode != 0: print(ret.stderr.decode('utf-8').splitlines()[0]) return None return ret.stdout.decode('utf-8').strip() def main(argv): # Parse command line arguments parser = argparse.ArgumentParser() parser.add_argument('--staged', '-s', action='store_true', help='Include the changes in the index. Defaults to False') parser.add_argument('--amend', '-a', action='store_true', help='Include changes in the index and the previous patch combined. Defaults to False') parser.add_argument('revision_range', type=str, default=None, nargs='?', help='Revision range (as defined by git rev-parse). Defaults to HEAD if not specified.') args = parser.parse_args(argv[1:]) # Check for required dependencies. for command, mandatory in dependencies.items(): found = shutil.which(command) if mandatory and not found: print("Executable %s not found" % command) return 1 dependencies[command] = found # Get the top level directory to pass absolute file names to git diff # commands, in order to support execution from subdirectories of the git # tree. top_level = git_top_level() if top_level is None: return 1 commits = [] if args.staged: commits.append(StagedChanges()) if args.amend: commits.append(Amendment()) # If none of --staged or --amend was passed if len(commits) == 0: # And no revisions were passed, then default to HEAD if not args.revision_range: args.revision_range = 'HEAD' if args.revision_range: commits += extract_commits(args.revision_range) issues = 0 for commit in commits: issues += check_style(top_level, commit) print('') if issues: return 1 else: return 0 if __name__ == '__main__': sys.exit(main(sys.argv))