guibot.finder module

SUMMARY

Computer vision finders (CV backends) to perform find targets on screen.

INTERFACE

class guibot.finder.CVParameter(value, min_val=None, max_val=None, delta=10.0, tolerance=1.0, fixed=True, enumerated=False)

Bases: object

A class for a single parameter used for CV backend configuration.

__init__(value, min_val=None, max_val=None, delta=10.0, tolerance=1.0, fixed=True, enumerated=False)

Build a computer vision parameter.

Parameters:

• value (bool or int or float or str or None) – value of the parameter
• min_val (int or float or None) – lower boundary for the parameter range
• max_val (int or float or None) – upper boundary for the parameter range
• delta (float) – delta for the calibration and random value (no calibration if delta < tolerance)
• tolerance (float) – tolerance of calibration
• fixed (bool) – whether the parameter is prevented from calibration
• enumerated (bool) – whether the parameter value belongs to an enumeration or to a range (distance matters)

As a rule of thumb a good choice for the parameter delta is one fourth of the range since the delta will be used as standard deviation when generating a random value for the parameter from a normal distribution. The delta to tolerance ratio is basically the number of failing trials before the parameter converges and is usually set to ten.

__repr__()

Provide a representation of the parameter for storing and reporting.

Returns:special syntax representation of the parameter Return type:str

__eq__(other)

Custom implementation for equality check.

Returns:whether this instance is equal to another Return type:bool

static from_string(raw)

Parse a CV parameter from string.

Parameters:raw (str) – string representation for the parameter Returns:parameter parsed from the representation Return type:CVParameter Raises:ValueError if unsupported type is encountered

random_value(mu=None, sigma=None)

Return a random value of the CV parameter given its range and type.

Parameters:

• mu (bool or int or float or str or None) – mean for a normal distribution, uniform distribution if None
• sigma (bool or int or float or str or None) – standard deviation for a normal distribution, quarter range if None (maximal range is equivalent to maximal data type values)

Returns:

a random value comforming to the CV parameter range and type

Return type:

bool or int or float or str or None

Note

Only uniform distribution is used for boolean values.

__hash__ = None

class guibot.finder.Finder(configure=True, synchronize=True)

Bases: guibot.config.LocalConfig

Base for all image matching functionality and backends.

The image finding methods include finding one or all matches above the similarity defined in the configuration of each backend.

There are many parameters that could contribute for a good match. They can all be manually adjusted or automatically calibrated.

static from_match_file(filename)

Read the configuration from a match file with the given filename.

Parameters:filename (str) – match filename for the configuration Returns:target finder with the parsed (and generated) settings Return type:finder.Finder Raises:IOError if the respective match file couldn’t be read

The influence of the read configuration is that of an overwrite, i.e. all parameters will be generated (if not already present) and then the ones read from the configuration file will be overwritten.

static to_match_file(finder, filename)

Write the configuration to a match file with the given filename.

Parameters:

• finder (finder.Finder) – match configuration to save
• filename (str) – match filename for the configuration

__init__(configure=True, synchronize=True)

Build a finder and its CV backend settings.

configure_backend(backend=None, category='find', reset=False)

Custom implementation of the base method.

See base method for details.

synchronize_backend(backend=None, category='find', reset=False)

Custom implementation of the base method.

See base method for details.

can_calibrate(category, mark)

Fix the parameters for a given category backend algorithm, i.e. disallow the calibrator to change them.

Parameters:

• mark (bool) – whether to mark for calibration
• category (str) – backend category whose parameters are marked

Raises:

UnsupportedBackendError if category is not among the supported backend categories

copy()

Deep copy the current finder and its configuration.

Returns:a copy of the current finder with identical configuration Return type:Finder

find(needle, haystack)

Find all needle targets in a haystack image.

Parameters:

• needle (target.Target or [target.Target]) – image, text, pattern, or a list or chain of such to look for
• haystack (target.Image) – image to look in

Returns:

all found matches (one in most use cases)

Return type:

[match.Match]

Raises:

NotImplementedError if the base class method is called

log(lvl)

Log images with an arbitrary logging level.

Parameters:lvl (int) – logging level for the message

class guibot.finder.AutoPyFinder(configure=True, synchronize=True)

Bases: guibot.finder.Finder

Simple matching backend provided by AutoPy.

__init__(configure=True, synchronize=True)

Build a CV backend using AutoPy.

configure_backend(backend=None, category='autopy', reset=False)

Custom implementation of the base method.

See base method for details.

find(needle, haystack)

Custom implementation of the base method.

Parameters:needle (Image) – target iamge to search for

See base method for details.

Warning

AutoPy has a bug when finding multiple matches so it will currently only return a single match.

class guibot.finder.ContourFinder(configure=True, synchronize=True)

Bases: guibot.finder.Finder

Contour matching backend provided by OpenCV.

Essentially, we will find all countours in a binary image, preprocessed with Gaussian blur and adaptive threshold and return the ones with area (size) similar to the searched image.

__init__(configure=True, synchronize=True)

Build a CV backend using OpenCV’s contour matching.

configure_backend(backend=None, category='contour', reset=False)

Custom implementation of the base method.

See base method for details.

configure(threshold_filter=None, reset=True, **kwargs)

Custom implementation of the base method.

Parameters:threshold_filter (str or None) – name of a preselected backend

find(needle, haystack)

Custom implementation of the base method.

Parameters:needle (Image) – target iamge to search for

See base method for details.

First extract all contours from a binary (boolean, threshold) version of the needle and haystack and then match the needle contours with one or more sets of contours in the haystack image. The number of needle matches depends on the set similarity and can be improved by requiring minimal area for the contours to be considered.

log(lvl)

Custom implementation of the base method.

See base method for details.

class guibot.finder.TemplateFinder(configure=True, synchronize=True)

Bases: guibot.finder.Finder

Template matching backend provided by OpenCV.

__init__(configure=True, synchronize=True)

Build a CV backend using OpenCV’s template matching.

configure_backend(backend=None, category='template', reset=False)

Custom implementation of the base method.

See base method for details.

find(needle, haystack)

Custom implementation of the base method.

Parameters:needle (Image) – target iamge to search for Raises:UnsupportedBackendError if the choice of template matches is not among the supported ones

See base method for details.

log(lvl)

Custom implementation of the base method.

See base method for details.

class guibot.finder.FeatureFinder(configure=True, synchronize=True)

Bases: guibot.finder.Finder

Feature matching backend provided by OpenCV.

Note

SURF and SIFT are proprietary algorithms and are not available by default in newer OpenCV versions (>3.0).

__init__(configure=True, synchronize=True)

Build a CV backend using OpenCV’s feature matching.

configure_backend(backend=None, category='feature', reset=False)

Custom implementation of the base method.

Some relevant parameters are:

• detect filter - works for certain detectors and

  determines how many initial features are detected in an image (e.g. hessian threshold for SURF detector)

• match filter - determines what part of all matches

  returned by feature matcher remain good matches

• project filter - determines what part of the good

  matches are considered inliers

• ratio test - boolean for whether to perform a ratio test
• symmetry test - boolean for whether to perform a symmetry test

See base method for details.

configure(feature_detect=None, feature_extract=None, feature_match=None, reset=True, **kwargs)

Custom implementation of the base method.

Parameters:

• feature_detect (str or None) – name of a preselected backend
• feature_extract (str or None) – name of a preselected backend
• feature_match (str or None) – name of a preselected backend

synchronize_backend(backend=None, category='feature', reset=False)

Custom implementation of the base method.

See base method for details.

synchronize(feature_detect=None, feature_extract=None, feature_match=None, reset=True)

Custom implementation of the base method.

Parameters:

• feature_detect (str or None) – name of a preselected backend
• feature_extract (str or None) – name of a preselected backend
• feature_match (str or None) – name of a preselected backend

find(needle, haystack)

Custom implementation of the base method.

Parameters:needle (Image) – target iamge to search for

See base method for details.

Warning

Finding multiple matches is currently not supported and this will currently only return a single match.

Available methods are: a combination of feature detector, extractor, and matcher.

log(lvl)

Custom implementation of the base method.

See base method for details.

class guibot.finder.CascadeFinder(classifier_datapath='.', configure=True, synchronize=True)

Bases: guibot.finder.Finder

Cascade matching backend provided by OpenCV.

This matcher uses Haar cascade for object detection. It is the most advanced method for object detection excluding convolutional neural networks. However, it requires the generation of a Haar cascade (if such is not already provided) of the needle to be found.

TODO: Currently no similarity requirement can be applied due to the cascade classifier API.

__init__(classifier_datapath='.', configure=True, synchronize=True)

Build a CV backend using OpenCV’s cascade matching options.

configure_backend(backend=None, category='cascade', reset=False)

Custom implementation of the base method.

See base method for details.

find(needle, haystack)

Custom implementation of the base method.

Parameters:needle (Pattern) – target pattern (cascade) to search for

See base method for details.

class guibot.finder.TextFinder(configure=True, synchronize=True)

Bases: guibot.finder.ContourFinder

Text matching backend provided by OpenCV.

This matcher will find a text (string) needle in the haystack, eventually relying on Tesseract or simpler kNN-based OCR, using extremal regions or contours before recognition, and returning a match if the string is among the recognized strings using string metric similar to Hamming distance.

Extremal Region Filter algorithm described in: Neumann L., Matas J.: Real-Time Scene Text Localization and Recognition, CVPR 2012

__init__(configure=True, synchronize=True)

Build a CV backend using OpenCV’s text matching options.

configure_backend(backend=None, category='text', reset=False)

Custom implementation of the base method.

See base method for details.

configure(text_detector=None, text_recognizer=None, threshold_filter=None, threshold_filter2=None, threshold_filter3=None, reset=True, **kwargs)

Custom implementation of the base method.

Parameters:

• text_detector (str or None) – name of a preselected backend
• text_recognizer (str or None) – name of a preselected backend
• threshold_filter (str or None) – threshold filter for the text detection stage
• threshold_filter2 (str or None) – additional threshold filter for the OCR stage
• threshold_filter3 (str or None) – additional threshold filter for distance transformation

synchronize_backend(backend=None, category='text', reset=False)

Custom implementation of the base method.

See base method for details.

synchronize(text_detector=None, text_recognizer=None, threshold_filter=None, threshold_filter2=None, threshold_filter3=None, reset=True)

Custom implementation of the base method.

Parameters:

• text_detector (str or None) – name of a preselected backend
• text_recognizer (str or None) – name of a preselected backend
• threshold_filter (str or None) – threshold filter for the text detection stage
• threshold_filter2 (str or None) – additional threshold filter for the OCR stage
• threshold_filter3 (str or None) – additional threshold filter for distance transformation

find(needle, haystack)

Custom implementation of the base method.

Parameters:needle (Text) – target text to search for

See base method for details.

log(lvl)

Custom implementation of the base method.

See base method for details.

class guibot.finder.TemplateFeatureFinder(configure=True, synchronize=True)

Bases: guibot.finder.TemplateFinder, guibot.finder.FeatureFinder

Hybrid matcher using both OpenCV’s template and feature matching.

Feature matching is robust at small regions not too abundant of features where template matching is too picky. Template matching is good at large feature abundant regions and can be used as a heuristic for the feature matching. The current matcher will perform template matching first and then feature matching on the survived template matches to select among them one more time.

A separate (usually lower) front similarity is used for the first stage template matching in order to remove a lot of noise that would otherwise be distracting for the second stage feature matching.

__init__(configure=True, synchronize=True)

Build a CV backend using OpenCV’s template and feature matching.

configure_backend(backend=None, category='tempfeat', reset=False)

Custom implementation of the base method.

See base method for details.

configure(template_match=None, feature_detect=None, feature_extract=None, feature_match=None, reset=True, **kwargs)

Custom implementation of the base methods.

See base methods for details.

synchronize(feature_detect=None, feature_extract=None, feature_match=None, reset=True)

Custom implementation of the base method.

See base method for details.

find(needle, haystack)

Custom implementation of the base method.

See base method for details.

Use template matching to deal with feature dense regions and guide a final feature matching stage.

log(lvl)

Custom implementation of the base method.

See base method for details.

class guibot.finder.DeepFinder(classifier_datapath='.', configure=True, synchronize=True)

Bases: guibot.finder.Finder

Deep learning matching backend provided by PyTorch.

The current implementation contains a basic convolutional neural network which can be trained to produce needle locations from a haystack image.

__init__(classifier_datapath='.', configure=True, synchronize=True)

Build a CV backend using OpenCV’s text matching options.

configure_backend(backend=None, category='deep', reset=False)

Custom implementation of the base method.

See base method for details.

synchronize_backend(backend=None, category='deep', reset=False)

Custom implementation of the base method.

See base method for details.

find(needle, haystack)

Custom implementation of the base method.

Parameters:needle (Pattern) – target pattern (cascade) to search for

See base method for details.

log(lvl)

Custom implementation of the base method.

See base method for details.

class guibot.finder.HybridFinder(configure=True, synchronize=True)

Bases: guibot.finder.Finder

Match a target through a sequence of differently configured attempts.

This matcher can work with any other matcher in the background and with unique or repeating matchers for each step. If a step fails, the matcher tries the next available along the fallback chain or fails if the end of the chain is reached.

__init__(configure=True, synchronize=True)

Build a hybrid matcher.

configure_backend(backend=None, category='hybrid', reset=False)

Custom implementation of the base method.

See base method for details.

synchronize_backend(backend=None, category='hybrid', reset=False)

Custom implementation of the base method.

See base method for details.

find(needle, haystack)

Custom implementation of the base method.

See base method for details.