This is UltimateRPA Documentation
Finding an element

Search methods – an introduction

This chapter demonstrates, with several examples, how to find GUI elements in an application. This chapter assumes that you can already use Inspector and you know what condition factory is.

The following example will show the concept using the RpaFormsTest application. Import the urpa module. Create a condition_factory object "cf". Define the main function. To run the script in debug mode use the method set_debug_mode and pass the argument True. Running the robotic script in debug mode highlights the GUI element with a green frame to indicate the element being worked on. The application path, in the argument of the function exec_app, may be absolute or relative to the script.

import urpa
def main():
app = urpa.exec_app("TutorialApps/RpaFormsTest.exe")

The path to the app "TutorialApps/RpaFormsTest.exe" is relative to the script file directory and assumes script placed in the main directory of UltimateRPA tools. If your script is saved elsewhere, remember to change the path.

Launch RpaFormsTest.exe application manually. Use the Inspector to identify the GUI elements on the app.

Figure 1 RpaFormsTest application

Finding a single element

First, find the label ID. Left-click to activate crosshair in Inspector, hold the mouse button down and drag the cursor over the text ID in RpaFormsTest. The GUI element is framed in green. If the desired element is selected, release the mouse button. In the right half of the Inspector, mark the Name checkbox and select find_first in the Search function. A search command is generated in the Expression section. It should look like this: app.find_first("ID")).

Figure 2 RpaFormsTest application in Inspector

To find one GUI element, use the method find_first. You may already have come across the use of this method in the chapter First robotization. This method returns the first GUI element matching the condition defined in the condition parameter. You can read about how to create conditions in the chapter condition factory. This method returns an object of the class AppElement that can be used for user interactions or to query its properties. This is covered in more detail in the next chapter. To run a search, simply copy an expression from Inspector into the code.

Figure 3 Expression in Inspector
import urpa
def main():
app = urpa.exec_app("TutorialApps/RpaFormsTest.exe")

Run the robot using the keyboard shortcut (ALT + F9). Alternatively, select Run and then External Run in the PyScripter upper menu. The debugging mode marks the element ID found with a green frame. The find_first method always finds the first GUI element in the object tree that matches the condition parameter.

Finding multiple elements

In certain cases, however, you need to find all of the GUI elements for the defined condition at once. In this case, use the method find_all. This method always returns a Python list of all found GUI elements matching the condition parameter. If the method does not find any GUI element, it returns an empty list. As with find_first, the method's first argument is a condition object defining the properties according to which you want to find GUI elements. In addition, this method has a second optional element argument, zero by default, which defines the minimum number of GUI elements the robot has to find. In our test case, we will try to find the text Attribute twice in RpaFormsTest. Use Inspector and move crosshair to any "Attribute" text. Check the Name checkbox. In the left part of Inspector, under the search box, you can see that there are two elements in the application that have a Name equal to Attribute. In the Search function, select find_all and the command you have generated should look like this: app.find_all("Attribute")). Copy this and replace the previous search with it in your script. As the application contains this element twice, set the function's second argument to the integer 2 so that the robot always searches for all the Attributes that should be in the application.

Figure 4 Two elements "Attribute" in Inspector
import urpa
def main():
app = urpa.exec_app("TutorialApps/RpaFormsTest.exe")
app.find_all("Attribute"), 2)
Figure 5 Highlighted elements in debug mode

Run the application and you can see that both of the elements with the name "Attribute" have been marked with a green frame.

Running a search from another GUI element

Sometimes you find yourself in a situation where you are unable to find an element in the application clearly. In these cases, you can draw on the assistance of another reference element that is located near to the element you wish to find. In the example provided here, we want to find an edit box besides the text ID in RpaFormsTest. Use the method find_first_right_to. Select the text ID as the reference object. We will be searching for an edit box to the right of that object. Although Inspector does not generate an entire expression for this method, you can still help yourself by preparing a condition in it. The first condition is for the reference object. With the crosshair, find the text ID in Inspector and release the mouse button. Select the Name checkbox and choose (none) as the Search function. The following piece of code is generated:"ID"). Copy this and, in your script, save it in the ref_elem_condition variable. Next, create a condition for the object you want to find. Move the crosshair to the edit box to the right of the text ID. In the menu, check the Class Name checkbox and leave (none) in the Search function. The following piece of code is generated: cf.class_name("Edit"). Save this in the right_elem_condition variable. Then, for the app object, invoke the method app find_first_right_to with ref_elem_condition as the first parameter and right_elem_condition as the second.

import urpa
def main():
app = urpa.exec_app("TutorialApps/RpaFormsTest.exe")
ref_elem_condition ="ID")
right_elem_conditon = cf.class_name("Edit")
app.find_first_right_to(ref_elem_condition, right_elem_conditon)
Figure 6 Element found at the right side of reference element

Run the script. Because the debug mode is active, you can see a green frame around the edit box.

In the same way, use the methods find_first_down_to, find_first_left_to and find_first_up_to.

Running a search from a point

In certain cases, not even the solution above is enough, so you need to use the method find_from_point. This also uses a reference GUI element to run searches. From its top left corner, search for the required GUI element using x and y coordinates. Using this method, you can find the edit box for the second Attribute. The first "Attribute" GUI element is used as the reference GUI element. Use Inspector to create a condition for the reference GUI element that looks like this:"Attribute"). Copy this into the script, and save it in the ref_elem_condition object. For the GUI element, you want to find, create the condition again using Inspector. Drag the crosshair to the edit box for the second "Attribute" text and check the Class Name checkbox. This generates the code cf.class_name("Edit"), which is represented by the elem_condition variable object in your script. Finally invoke the app object method find_from_point with ref_elem_condition as the first parameter and elem_condition as the second. The third and fourth parameters comprise the x, y coordinates in pixels from the upper left corner of the reference GUI element. In this particular case, for example, the values 70 and 30 can be used.

import urpa
def main():
app = urpa.exec_app("TutorialApps/RpaFormsTest.exe")
ref_elem_condition ="Attribute")
elem_condition = cf.class_name("Edit")
app.find_from_point(ref_elem_condition, elem_condition, 70, 30)

Run the robot, and the edit box for the second "Attribute" text is framed in green.

Figure 7 Element found by position from the reference element

Timeout parameter

All GUI element search methods have an optional timeout parameter. This indicates how long a robot can search for an object in ms. The robot's default setting is five seconds (5,000 ms). However, this is not enough in certain cases. When using RpaFormsTest, you may have noticed that if you run this application it takes a while to load the last field of the form called Quantity. Now we will try to find this element right after the application starts. The code below is used for this purpose.

import urpa
def main():
app = urpa.exec_app("TutorialApps/RpaFormsTest.exe")

In this case, the application runs and starts to search for an object that is not yet loaded. This takes 5 seconds, and then the robot terminates with the error ElementNotFoundError.

Figure 8 Error raised when an element is not found

You can easily correct this problem by increasing the timeout, for example, to 10 seconds (10,000 ms). Modify the last line of the previous code to app.find_first("Quantity"), 10000).

import urpa
def main():
app = urpa.exec_app("TutorialApps/RpaFormsTest.exe")
app.find_first("Quantity"), 10000)

Now when you run the application and wait a while, the Quantity field appears and it is immediately framed in green. In the next section, we show you a simple example of what you can do if you expect a situation where you may not find an element and the robot would terminate with an error.

Figure 9 Element "Quantity" found

If a search fails

If the robot does not find the GUI element matching the properties or condition entered, it throws an error ElementNotFoundError. The error might be captured in the try and except block, and processed without ending the script.

print("Element not found!")

Running a search in a parent GUI element

All of the scripts above demonstrate a case where you run a search for a GUI element or GUI elements throughout the application. However, UltimateRPA allows for a GUI element to be retrieved only in part of an application, i.e. within a parent GUI element. You can see how interdependent the GUI elements are in the left part of Inspector – in the "tree".

Figure 10 Inspector tree showing RpaTableTest GUI element dependencies

For this example, we use the application RpaTableTest.

Figure 11 RpaTableTest application window

Now launch the application manually and view it in Inspector. Move the Inspector's crosshair to any point of the application. This lets you access the GUI element tree on the left. What you see in the tree may vary depending on which GUI element you have selected in the application. However, you should always see the RpaTableTest[Window] element and, under that, the GUI List element, representing a table in the application. If you unpack this GUI element, you find elements 1[ListItem] to 22 [ListItem], representing the rows of the table. Select row number 3 and create an expression for it with the method find_first, which will look like this: app.find_first(cf.list_item().name("3")). Copy the expression into the code and save it to the parent_element variable. This is how to find the parent GUI element. Now, we will try to find the text GoodFactory in that element. Move the Inspector’s crosshair to any GoodFactory text. On the right, check the name. Inspector generates this expression: app.find_first(cf.text().name("GoodFactory")). Copy this and paste it into your code under the last line, and then paste it into the code once more. We will explain this in a moment, but first we replace the app with the parent_element object in the first line, so that the line will look like this: parent_element.find_first(cf.text().name("GoodFactory")). In this line of code, you search for the text GoodFactory only in the third row of the table; in the expression below it, you search for the first GoodFactory text in the entire application.

import urpa
def main():
app = urpa.exec_app("TutorialApps/RpaTableTest.exe")
parent_element = app.find_first(cf.list_item().name("3"))

When you run the robot, first the third row of the table is framed in green, followed by the GoodFactory text on the third line and, finally, the GoodFactory in the first line.

Figure 12 Searching element by specifying a parent element

All of the methods introduced in this tutorial can also be used to run searches in a parent GUI element.

Recommended search procedures

Unique identification

Make sure you identify the GUI elements uniquely to allow the robot to perform the actions on the desired element. By using the search methods and defining the Conditions properly through the Inspector we can make sure the GUI elements are uniquely identified.

Multiple applications

Finding and accessing a GUI element happens only in an application's GUI tree. A GUI element can be found after the application is identified of which the element is part of. Sometimes an application runs another process (application) from within and the GUI element is part of that different application. In this case the application needs to be identified first and then the GUI element can be successfully searched for within that application context. You can read about how to work with multiple applications in the chapter.