guibot.finder module

class guibot.finder.CVParameter(value, min_val=None, max_val=None, delta=10.0, tolerance=1.0, fixed=True, enumerated=False)[source]

Bases: object

A class for a single parameter used for CV backend configuration.

__init__(value, min_val=None, max_val=None, delta=10.0, tolerance=1.0, fixed=True, enumerated=False)[source]

Build a computer vision parameter.

Parameters:
  • value (bool or int or float or str or None) – value of the parameter
  • min_val (int or float or None) – lower boundary for the parameter range
  • max_val (int or float or None) – upper boundary for the parameter range
  • delta (float) – delta for the calibration and random value (no calibration if delta < tolerance)
  • tolerance (float) – tolerance of calibration
  • fixed (bool) – whether the parameter is prevented from calibration
  • enumerated (bool) – whether the parameter value belongs to an enumeration or to a range (distance matters)

As a rule of thumb a good choice for the parameter delta is one fourth of the range since the delta will be used as standard deviation when generating a random value for the parameter from a normal distribution. The delta to tolerance ratio is basically the number of failing trials before the parameter converges and is usually set to ten.

__repr__()[source]

Provide a representation of the parameter for storing and reporting.

Returns:special syntax representation of the parameter
Return type:str
static from_string(raw)[source]

Parse a CV parameter from string.

Parameters:raw (str) – string representation for the parameter
Returns:parameter parsed from the representation
Return type:CVParameter
Raises:ValueError if unsupported type is encountered
random_value(mu=None, sigma=None)[source]

Return a random value of the CV parameter given its range and type.

Parameters:
  • mu (bool or int or float or str or None) – mean for a normal distribution, uniform distribution if None
  • sigma (bool or int or float or str or None) – standard deviation for a normal distribution, quarter range if None (maximal range is equivalent to maximal data type values)
Returns:

a random value comforming to the CV parameter range and type

Return type:

bool or int or float or str or None

Note

Only uniform distribution is used for boolean values.

class guibot.finder.Finder(configure=True, synchronize=True)[source]

Bases: guibot.config.LocalConfig

Base for all image matching functionality and backends.

The image finding methods include finding one or all matches above the similarity defined in the configuration of each backend.

There are many parameters that could contribute for a good match. They can all be manually adjusted or automatically calibrated.

static from_match_file(filename)[source]

Read the configuration from a match file with the given filename.

Parameters:filename (str) – match filename for the configuration
Returns:target finder with the parsed (and generated) settings
Return type:finder.Finder
Raises:IOError if the respective match file couldn’t be read

The influence of the read configuration is that of an overwrite, i.e. all parameters will be generated (if not already present) and then the ones read from the configuration file will be overwritten.

static to_match_file(finder, filename)[source]

Write the configuration to a match file with the given filename.

Parameters:
  • finder (finder.Finder) – match configuration to save
  • filename (str) – match filename for the configuration
__init__(configure=True, synchronize=True)[source]

Build a finder and its CV backend settings.

configure_backend(backend=None, category='find', reset=False)[source]

Custom implementation of the base method.

See base method for details.

synchronize_backend(backend=None, category='find', reset=False)[source]

Custom implementation of the base method.

See base method for details.

can_calibrate(category, mark)[source]

Fix the parameters for a given category backend algorithm, i.e. disallow the calibrator to change them.

Parameters:
  • mark (bool) – whether to mark for calibration
  • category (str) – backend category whose parameters are marked
Raises:

UnsupportedBackendError if category is not among the supported backend categories

copy()[source]

Deep copy the current finder and its configuration.

Returns:a copy of the current finder with identical configuration
Return type:Finder
find(needle, haystack)[source]

Find all needle targets in a haystack image.

Parameters:
  • needle (target.Target or [target.Target]) – image, text, pattern, or a list or chain of such to look for
  • haystack (target.Image) – image to look in
Returns:

all found matches (one in most use cases)

Return type:

[match.Match]

Raises:

NotImplementedError if the base class method is called

log(lvl)[source]

Log images with an arbitrary logging level.

Parameters:lvl (int) – logging level for the message
class guibot.finder.AutoPyFinder(configure=True, synchronize=True)[source]

Bases: guibot.finder.Finder

Simple matching backend provided by AutoPy.

__init__(configure=True, synchronize=True)[source]

Build a CV backend using AutoPy.

configure_backend(backend=None, category='autopy', reset=False)[source]

Custom implementation of the base method.

See base method for details.

find(needle, haystack)[source]

Custom implementation of the base method.

Parameters:needle (Image) – target iamge to search for

See base method for details.

Warning

AutoPy has a bug when finding multiple matches so it will currently only return a single match.

class guibot.finder.ContourFinder(configure=True, synchronize=True)[source]

Bases: guibot.finder.Finder

Contour matching backend provided by OpenCV.

Essentially, we will find all countours in a binary image, preprocessed with Gaussian blur and adaptive threshold and return the ones with area (size) similar to the searched image.

__init__(configure=True, synchronize=True)[source]

Build a CV backend using OpenCV’s contour matching.

configure_backend(backend=None, category='contour', reset=False)[source]

Custom implementation of the base method.

See base method for details.

configure(threshold_filter=None, reset=True)[source]

Custom implementation of the base method.

Parameters:threshold_filter (str or None) – name of a preselected backend
find(needle, haystack)[source]

Custom implementation of the base method.

Parameters:needle (Image) – target iamge to search for

See base method for details.

First extract all contours from a binary (boolean, threshold) version of the needle and haystack and then match the needle contours with one or more sets of contours in the haystack image. The number of needle matches depends on the set similarity and can be improved by requiring minimal area for the contours to be considered.

log(lvl)[source]

Custom implementation of the base method.

See base method for details.

class guibot.finder.TemplateFinder(configure=True, synchronize=True)[source]

Bases: guibot.finder.Finder

Template matching backend provided by OpenCV.

__init__(configure=True, synchronize=True)[source]

Build a CV backend using OpenCV’s template matching.

configure_backend(backend=None, category='template', reset=False)[source]

Custom implementation of the base method.

See base method for details.

find(needle, haystack)[source]

Custom implementation of the base method.

Parameters:needle (Image) – target iamge to search for
Raises:UnsupportedBackendError if the choice of template matches is not among the supported ones

See base method for details.

log(lvl)[source]

Custom implementation of the base method.

See base method for details.

class guibot.finder.FeatureFinder(configure=True, synchronize=True)[source]

Bases: guibot.finder.Finder

Feature matching backend provided by OpenCV.

Note

SURF and SIFT are proprietary algorithms and are not available by default in newer OpenCV versions (>3.0).

__init__(configure=True, synchronize=True)[source]

Build a CV backend using OpenCV’s feature matching.

configure_backend(backend=None, category='feature', reset=False)[source]

Custom implementation of the base method.

Some relevant parameters are:
  • detect filter - works for certain detectors and
    determines how many initial features are detected in an image (e.g. hessian threshold for SURF detector)
  • match filter - determines what part of all matches
    returned by feature matcher remain good matches
  • project filter - determines what part of the good
    matches are considered inliers
  • ratio test - boolean for whether to perform a ratio test
  • symmetry test - boolean for whether to perform a symmetry test

See base method for details.

configure(feature_detect=None, feature_extract=None, feature_match=None, reset=True)[source]

Custom implementation of the base method.

Parameters:
  • feature_detect (str or None) – name of a preselected backend
  • feature_extract (str or None) – name of a preselected backend
  • feature_match (str or None) – name of a preselected backend
synchronize_backend(backend=None, category='feature', reset=False)[source]

Custom implementation of the base method.

See base method for details.

synchronize(feature_detect=None, feature_extract=None, feature_match=None, reset=True)[source]

Custom implementation of the base method.

Parameters:
  • feature_detect (str or None) – name of a preselected backend
  • feature_extract (str or None) – name of a preselected backend
  • feature_match (str or None) – name of a preselected backend
find(needle, haystack)[source]

Custom implementation of the base method.

Parameters:needle (Image) – target iamge to search for

See base method for details.

Warning

Finding multiple matches is currently not supported and this will currently only return a single match.

Available methods are: a combination of feature detector, extractor, and matcher.

log(lvl)[source]

Custom implementation of the base method.

See base method for details.

class guibot.finder.CascadeFinder(classifier_datapath='.', configure=True, synchronize=True)[source]

Bases: guibot.finder.Finder

Cascade matching backend provided by OpenCV.

This matcher uses Haar cascade for object detection. It is the most advanced method for object detection excluding convolutional neural networks. However, it requires the generation of a Haar cascade (if such is not already provided) of the needle to be found.

TODO: Currently no similarity requirement can be applied due to the cascade classifier API.

__init__(classifier_datapath='.', configure=True, synchronize=True)[source]

Build a CV backend using OpenCV’s cascade matching options.

configure_backend(backend=None, category='cascade', reset=False)[source]

Custom implementation of the base method.

See base method for details.

find(needle, haystack)[source]

Custom implementation of the base method.

Parameters:needle (Pattern) – target pattern (cascade) to search for

See base method for details.

class guibot.finder.TextFinder(configure=True, synchronize=True)[source]

Bases: guibot.finder.ContourFinder

Text matching backend provided by OpenCV.

This matcher will find a text (string) needle in the haystack, eventually relying on Tesseract or simpler kNN-based OCR, using extremal regions or contours before recognition, and returning a match if the string is among the recognized strings using string metric similar to Hamming distance.

Extremal Region Filter algorithm described in: Neumann L., Matas J.: Real-Time Scene Text Localization and Recognition, CVPR 2012

__init__(configure=True, synchronize=True)[source]

Build a CV backend using OpenCV’s text matching options.

configure_backend(backend=None, category='text', reset=False)[source]

Custom implementation of the base method.

See base method for details.

configure(text_detector=None, text_recognizer=None, threshold_filter=None, threshold_filter2=None, threshold_filter3=None, reset=True)[source]

Custom implementation of the base method.

Parameters:
  • text_detector (str or None) – name of a preselected backend
  • text_recognizer (str or None) – name of a preselected backend
  • threshold_filter (str or None) – threshold filter for the text detection stage
  • threshold_filter2 (str or None) – additional threshold filter for the OCR stage
  • threshold_filter3 (str or None) – additional threshold filter for distance transformation
synchronize_backend(backend=None, category='text', reset=False)[source]

Custom implementation of the base method.

See base method for details.

synchronize(text_detector=None, text_recognizer=None, threshold_filter=None, threshold_filter2=None, threshold_filter3=None, reset=True)[source]

Custom implementation of the base method.

Parameters:
  • text_detector (str or None) – name of a preselected backend
  • text_recognizer (str or None) – name of a preselected backend
  • threshold_filter (str or None) – threshold filter for the text detection stage
  • threshold_filter2 (str or None) – additional threshold filter for the OCR stage
  • threshold_filter3 (str or None) – additional threshold filter for distance transformation
find(needle, haystack)[source]

Custom implementation of the base method.

Parameters:needle (Text) – target text to search for

See base method for details.

log(lvl)[source]

Custom implementation of the base method.

See base method for details.

class guibot.finder.TemplateFeatureFinder(configure=True, synchronize=True)[source]

Bases: guibot.finder.TemplateFinder, guibot.finder.FeatureFinder

Hybrid matcher using both OpenCV’s template and feature matching.

Feature matching is robust at small regions not too abundant of features where template matching is too picky. Template matching is good at large feature abundant regions and can be used as a heuristic for the feature matching. The current matcher will perform template matching first and then feature matching on the survived template matches to select among them one more time.

A separate (usually lower) front similarity is used for the first stage template matching in order to remove a lot of noise that would otherwise be distracting for the second stage feature matching.

__init__(configure=True, synchronize=True)[source]

Build a CV backend using OpenCV’s template and feature matching.

configure_backend(backend=None, category='tempfeat', reset=False)[source]

Custom implementation of the base method.

See base method for details.

configure(template_match=None, feature_detect=None, feature_extract=None, feature_match=None, reset=True)[source]

Custom implementation of the base methods.

See base methods for details.

synchronize(feature_detect=None, feature_extract=None, feature_match=None, reset=True)[source]

Custom implementation of the base method.

See base method for details.

find(needle, haystack)[source]

Custom implementation of the base method.

See base method for details.

Use template matching to deal with feature dense regions and guide a final feature matching stage.

log(lvl)[source]

Custom implementation of the base method.

See base method for details.

class guibot.finder.DeepFinder(classifier_datapath='.', configure=True, synchronize=True)[source]

Bases: guibot.finder.Finder

Deep learning matching backend provided by PyTorch.

The current implementation contains a basic convolutional neural network which can be trained to produce needle locations from a haystack image.

__init__(classifier_datapath='.', configure=True, synchronize=True)[source]

Build a CV backend using OpenCV’s text matching options.

configure_backend(backend=None, category='deep', reset=False)[source]

Custom implementation of the base method.

See base method for details.

synchronize_backend(backend=None, category='deep', reset=False)[source]

Custom implementation of the base method.

See base method for details.

find(needle, haystack)[source]

Custom implementation of the base method.

Parameters:needle (Pattern) – target pattern (cascade) to search for

See base method for details.

train(epochs, train_samples, train_targets, data_filename=None)[source]

Train the convolutional neural network.

Parameters:
  • epochs (int) – number of training epochs (train on all samples for each)
  • train_samples (str) – filename for the samples dataset
  • train_targets (str) – filename for the targets dataset
  • data_filename – file name for storing the trained model (won’t store if None)
  • data_filename – str or None
test(train_samples, train_targets)[source]

Test the convolutional neural network.

Parameters:
  • train_samples (str) – filename for the samples dataset
  • train_targets (str) – filename for the targets dataset
log(lvl)[source]

Custom implementation of the base method.

See base method for details.

class guibot.finder.CustomFinder(configure=True, synchronize=True)[source]

Bases: guibot.finder.Finder

Custom matching backend with in-house CV algorithms.

Warning

This matcher is currently not supported by our configuration.

Todo

“in-house-raw” performs regular knn matching, but “in-house-region” performs a special filtering and replacement of matches based on positional information (it does not have ratio and symmetry tests and assumes that the needle is transformed preserving the relative positions of each pair of matches, i.e. no rotation is allowed, but scaling for example is supported)

__init__(configure=True, synchronize=True)[source]

Build a CV backend using custom matching.

configure_backend(backend=None, category='custom', reset=False)[source]

Custom implementation of the base method.

See base method for details.

find(needle, haystack)[source]

Custom implementation of the base method.

See base method for details.

Todo

This custom feature matching backend needs more serious reworking before it even makes sense to get properly documented.

detect_features(needle, haystack)[source]

In-house feature detection algorithm.

Parameters:
  • needle (target.Image) – image to look for
  • haystack (target.Image) – image to look in

Warning

This method is currently not fully implemented. The current MSER might not be used in the actual implementation.

regionMatch(desc1, desc2, kp1, kp2, refinements=50, recalc_interval=10, variants_k=100, variants_ratio=0.33)[source]

Use location information to better decide on matched features.

Parameters:
  • desc1 – descriptors of the first image
  • desc2 – descriptors of the second image
  • kp1 – key points of the first image
  • kp2 – key points of the second image
  • refinements (int) – number of points to relocate
  • recalc_interval (int) – recalculation on a number of refinements
  • variants_k (int) – kNN parameter for to limit the alternative variants of a badly positioned feature
  • variants_ratio (float) – internal ratio test for knnMatch autostop (see below)
Returns:

obtained matches

The knn distance is now only a heuristic for the search of best matched set as is information on relative location with regard to the other matches.

Todo

handle a subset of matches (ignoring some matches if not all features are detected)

Todo

disable kernel mapping (multiple needle feature mapped to a single haystack feature)

knnMatch(desc1, desc2, k=1, desc4kp=1, autostop=0.0)[source]

Performs k-Nearest Neighbor matching.

Parameters:
  • desc1 – descriptors of the first image
  • desc2 – descriptors of the second image
  • k (int) – categorization up to k-th nearest neighbor
  • desc4kp (int) – legacy parameter for the old SURF() feature detector where desc4kp = len(desc2) / len(kp2) or analogically len(desc1) / len(kp1) i.e. needle row 5 is a descriptor vector for needle keypoint 5
  • autostop (float) – stop automatically if the ratio (dist to k)/(dist to k+1) is close to 0, i.e. the k+1-th neighbor is too far.
Returns:

obtained matches

class guibot.finder.HybridFinder(configure=True, synchronize=True)[source]

Bases: guibot.finder.Finder

Match a target through a sequence of differently configured attempts.

This matcher can work with any other matcher in the background and with unique or repeating matchers for each step. If a step fails, the matcher tries the next available along the fallback chain or fails if the end of the chain is reached.

__init__(configure=True, synchronize=True)[source]

Build a hybrid matcher.

configure_backend(backend=None, category='hybrid', reset=False)[source]

Custom implementation of the base method.

See base method for details.

synchronize_backend(backend=None, category='hybrid', reset=False)[source]

Custom implementation of the base method.

See base method for details.

find(needle, haystack)[source]

Custom implementation of the base method.

See base method for details.