guibot.guibot_proxy module

Frontend with serialization compatible API allowing the use of Pyro4 modified guibot.GuiBot object (creating and running the same object remotely and manipulating it locally). All the methods delegate their calls to this object with some additional postprocessing to make the execution remote so for information about the API please refer to it and region.Region.

guibot.guibot_proxy.serialize_custom_error(class_obj)

Serialization method for the errors.UnsupportedBackendError which was chosen just as a sample.

Parameters:class_obj (classobj) – class object for the serialized error class Returns:serialization dictionary with the class name, arguments, and attributes Return type:{str, str or getset_descriptor or dictproxy}

guibot.guibot_proxy.register_exception_serialization()

We put here any exceptions that are too complicated for the default serialization and define their serialization methods.

Note

This would not be needed if we were using the Pickle serializer but its security problems at the moment made us prefer the serpent serializer paying for it with some extra setup steps and functions below.

class guibot.guibot_proxy.GuiBotProxy(dc=None, cv=None)

Bases: guibot.guibot.GuiBot

The proxy guibot object is just a wrapper around the actual guibot object that takes care of returning easily serializable Pyro4 proxy objects instead of the real ones or their serialized copies.

It allows you to move the mouse, type text and do any other GuiBot action from code which is executed on another machine somewhere on the network.

__init__(dc=None, cv=None)

Build a guibot object.

Parameters:

• dc (desktopcontrol.DesktopControl or None) – DC backend used for any desktop control
• cv (finder.Finder or None) – CV backend used for any target finding

We will initialize with default region of full screen and default desktop control and computer vision backends if none are provided.

nearby(rrange=50)

Obtain a region containing the previous one but enlarged by a number of pixels on each side.

Parameters:rrange (int) – number of pixels to add Returns:new region enlarged by rrange on all sides Return type:Region

above(rrange=0)

Obtain a region containing the previous one but enlarged by a number of pixels on the upper side.

Parameters:rrange (int) – number of pixels to add Returns:new region enlarged by rrange on upper side Return type:Region

below(rrange=0)

Obtain a region containing the previous one but enlarged by a number of pixels on the lower side.

Parameters:rrange (int) – number of pixels to add Returns:new region enlarged by rrange on lower side Return type:Region

left(rrange=0)

Obtain a region containing the previous one but enlarged by a number of pixels on the left side.

Parameters:rrange (int) – number of pixels to add Returns:new region enlarged by rrange on left side Return type:Region

right(rrange=0)

Obtain a region containing the previous one but enlarged by a number of pixels on the right side.

Parameters:rrange (int) – number of pixels to add Returns:new region enlarged by rrange on right side Return type:Region

find(target, timeout=10)

Find a target (image, text, etc.) on the screen.

Parameters:

• target (str or target.Target) – target to look for
• timeout (int) – timeout before giving up

Returns:

match obtained from finding the target within the region

Return type:

match.Match

Raises:

errors.FindError if no match is found

This method is the main entrance to all our target finding capabilities and is the milestone for all target expect methods.

find_all(target, timeout=10, allow_zero=False)

Find multiples of a target on the screen.

Parameters:

• target (str or target.Target) – target to look for
• timeout (int) – timeout before giving up
• allow_zero (bool) – whether to allow zero matches or raise error

Returns:

matches obtained from finding the target within the region

Return type:

[match.Match]

Raises:

errors.FindError if no matches are found and zero matches are not allowed

This method is similar the one above but allows for more than one match.

sample(target)

Sample the similarity between a target and the screen, i.e. an empirical probability that the target is on the screen.

Parameters:target (str or target.Target) – target to look for Returns:similarity with best match on the screen Return type:float

Note

Not all matchers support a ‘similarity’ value. The ones that don’t will return zero similarity (similarly to the target logging case).

exists(target, timeout=0)

Check if a target exists on the screen using the matching success as a threshold for the existence.

Parameters:

• target (str or target.Target) – target to look for
• timeout (int) – timeout before giving up

Returns:

match obtained from finding the target within the region or nothing if no match is found

Return type:

match.Match or None

wait(target, timeout=30)

Wait for a target to appear (be matched) with a given timeout as failing tolerance.

Parameters:

• target (str or target.Target) – target to look for
• timeout (int) – timeout before giving up

Returns:

match obtained from finding the target within the region

Return type:

match.Match

Raises:

errors.FindError if no match is found

wait_vanish(target, timeout=30)

Wait for a target to disappear (be unmatched, i.e. matched without success) with a given timeout as failing tolerance.

Parameters:

• target (str or target.Target) – target to look for
• timeout (int) – timeout before giving up

Returns:

whether the target disappeared from the region

Return type:

bool

Raises:

errors.NotFindError if match is still found

hover(target_or_location)

Hover the mouse over a target or location.

Parameters:target_or_location (match.Match or location.Location or str or target.Target) – target or location to hover to Returns:match from finding the target or nothing if hovering over a known location Return type:match.Match or None

click(target_or_location, modifiers=None)

Click on a target or location using the left mouse button and optionally holding special keys.

Parameters:

• target_or_location (match.Match or location.Location or str or target.Target) – target or location to click on
• modifiers ([str]) – special keys to hold during clicking (see inputmap.KeyModifier for extensive list)

Returns:

match from finding the target or nothing if clicking on a known location

Return type:

match.Match or None

The special keys refer to a list of key modifiers, e.g.:

self.click('my_target', [KeyModifier.MOD_CTRL, 'x']).

right_click(target_or_location, modifiers=None)

Click on a target or location using the right mouse button and optionally holding special keys.

Arguments and return values are analogical to Region.click().

double_click(target_or_location, modifiers=None)

Double click on a target or location using the left mouse button and optionally holding special keys.

Arguments and return values are analogical to Region.click().

multi_click(target_or_location, count=3, modifiers=None)

Click N times on a target or location using the left mouse button and optionally holding special keys.

Arguments and return values are analogical to Region.click().

click_expect(click_image_or_location, expect_image_or_location=None, modifiers=None, timeout=60)

Click on an image or location and wait for another one to appear.

Parameters:

• click_image_or_location (Image or Location) – image or location to click on
• expect_image_or_location (Image or Location or None) – image or location to wait for
• modifiers ([Key] or None) – key modifiers when clicking
• timout (int) – time in seconds to wait for

Returns:

match obtained from finding the second target within the region

Return type:

match.Match

click_vanish(click_image_or_location, expect_image_or_location=None, modifiers=None, timeout=60)

Click on an image or location and wait for another one to disappear.

Parameters:

• click_image_or_location (Image or Location) – image or location to click on
• expect_image_or_location (Image or Location or None) – image or location to wait for
• modifiers ([Key] or None) – key modifiers when clicking
• timout (int) – time in seconds to wait for

Returns:

whether the second target disappeared from the region

Return type:

bool

click_at_index(anchor, index=0, find_number=3, timeout=10)

Find all instances of an anchor image and click on the one with the desired index given that they are horizontally then vertically sorted.

Parameters:

• anchor (str or target.Target) – image to find all matches of
• index (int) – index of the match to click on (assuming >=1 matches), sorted according to their (x,y) coordinates
• find_number (int) – expected number of matches which is necessary for fast failure in case some elements are not visualized and/or proper matching result
• timeout (int) – timeout before which the number of matches should be found

Returns:

match from finding the target of the desired index

Return type:

match.Match

Note

This method is a good replacement of a number of coincident limitations regarding the Windows version of autopy and Pyro and therefore the (Windows) virtual user:

• autopy has an old BUG regarding capturing the screen at a region with boundaries, different than the entire screen -> subregioning which is the main way to deal with any kind of highly repeating and homogeneous interface, is totally unavailable here.
• Pyro4 cannot serialize generators, so this is an implementation of a “generator step” involving clicking on consecutive matches.
• The serialized virtual user now returns a list of proxified matches when calling find_all, but they are all essentially useless as they don’t proxify their returned objects and cannot be sent back as arguments. The special proxy interface of the virtual user was implemented only to handle the most basic case - serialize the objects returned by the main shared class by proxifying them (turning them into remote objects as well, which already have a well-defined serialization method) and nothing more.

mouse_down(target_or_location, button=None)

Hold down an arbitrary mouse button on a target or location.

Parameters:

• target_or_location (match.Match or location.Location or str or target.Target) – target or location to toggle on
• button (int or None) – button index depending on backend (default is left button) (see inputmap.MouseButton for extensive list)

Returns:

match from finding the target or nothing if toggling on a known location

Return type:

match.Match or None

mouse_up(target_or_location, button=None)

Release an arbitrary mouse button on a target or location.

Parameters:

• target_or_location (match.Match or location.Location or str or target.Target) – target or location to toggle on
• button (int or None) – button index depending on backend (default is left button) (see inputmap.MouseButton for extensive list)

Returns:

match from finding the target or nothing if toggling on a known location

Return type:

match.Match or None

drag_drop(src_target_or_location, dst_target_or_location, modifiers=None)

Drag from and drop at a target or location optionally holding special keys.

Parameters:

• src_target_or_location (match.Match or location.Location or str or target.Target) – target or location to drag from
• dst_target_or_location (match.Match or location.Location or str or target.Target) – target or location to drop at
• modifiers ([str]) – special keys to hold during dragging and dropping (see inputmap.KeyModifier for extensive list)

Returns:

match from finding the target or nothing if dropping at a known location

Return type:

match.Match or None

drag_from(target_or_location, modifiers=None)

Drag from a target or location optionally holding special keys.

Arguments and return values are analogical to Region.drag_drop() but with target_or_location as src_target_or_location.

drop_at(target_or_location, modifiers=None)

Drop at a target or location optionally holding special keys.

Arguments and return values are analogical to Region.drag_drop() but with target_or_location as dst_target_or_location.

press_keys(keys)

Press a single key or a list of keys simultaneously.

Parameters:keys ([str] or str (possibly special keys in both cases)) – characters or special keys depending on the backend (see inputmap.Key for extensive list) Returns:self Return type:Region

Thus, the line self.press_keys([Key.ENTER]) is equivalent to the line self.press_keys(Key.ENTER). Other examples are:

self.press_keys([Key.CTRL, 'X'])
self.press_keys(['a', 'b', 3])

press_at(target_or_location=None, keys=None)

Press a single key or a list of keys simultaneously at a specified target or location.

This method is similar to Region.press_keys() but with an extra argument like Region.click().

type_text(text, modifiers=None)

Type a list of consecutive character keys (without special keys).

Parameters:

• text ([str] or str (no special keys in both cases)) – characters or strings (independent of the backend)
• modifiers ([str]) – special keys to hold during typing (see inputmap.KeyModifier for extensive list)

Returns:

self

Return type:

Region

Thus, the line self.type_text(['hello']) is equivalent to the line self.type_text('hello'). Other examples are:

self.type_text('ab3') # compare with press_keys()
self.type_text(['Hello', ' ', 'user3614']) # in cases with appending

Special keys are only allowed as modifiers here - simply call Region.press_keys() multiple times for consecutively typing special keys.

type_at(target_or_location=None, text='', modifiers=None)

Type a list of consecutive character keys (without special keys) at a specified target or location.

This method is similar to Region.type_text() but with an extra argument like Region.click().

fill_at(anchor, text, dx, dy, del_flag=True, esc_flag=True, mark_clicks=1)

Fills a new text at a text box with variable content using an anchor image and a displacement from that image.

Parameters:

• anchor (Match or Location or Target or str) – target of reference for the input field
• text (str) – text to fill in
• dx (int) – displacement from the anchor in the x direction
• dy (int) – displacement from the anchor in the y direction
• del_flag (bool) – whether to delete the highlighted text
• esc_flag (bool) – whether to escape any possible fill suggestions
• mark_clicks (int) – 0, 1, 2, … clicks to highlight previous text

Returns:

self

Return type:

Region

Raises:

exceptions.ValueError if mark_click is not acceptable value

If the delete flag is set the previous content will be deleted or otherwise the new text will be added in the end of the current text. If the escape flag is set an escape will be pressed after typing in order to avoid any entry suggestions from a dropdown list that could cover important image matching areas.

Since different interfaces behave differently, one might need a single, double or triple click to mark the already present text that has to be replaced.

select_at(anchor, image_or_index, dx, dy, dw=0, dh=0, ret_flag=True, mark_clicks=1)

Select an option at a dropdown list using either an integer index or an option image if the order cannot be easily inferred.

Parameters:

• anchor (Match or Location or Target or str) – target of reference for the input dropdown menu
• image_or_index (str or int) – item image or item index
• dx (int) – displacement from the anchor in the x direction
• dy (int) – displacement from the anchor in the y direction
• dw (int) – width to add to the displacement for an image search area
• dh (int) – height to add to the displacement for an image search area
• ret_flag (bool) – whether to press Enter after selecting
• mark_clicks (int) – 0, 1, 2, … clicks to highlight previous text

Returns:

self

Return type:

Region

It uses an anchor image which is rather constant and a displacement to locate the dropdown location. It moves down to the option if index is used where index 0 represents the current selection.

To avoid the limitations of the index method, an image of the option can be provided and will be matched in the area with and under the dropdown list. This also handles cases where the option coincides with the previously selected option. For more details see the really cool note in the end of this method.