Example: Batch get webpage search results
Scenario
Create a visual flow to enter search keywords into Baidu. The flow saves the titles and corresponding page numbers from the first five pages of search results to a local Excel file.
For a detailed guide, watch the following video tutorial:
Example description
This example focuses on browser and Excel operations. To interact with a webpage, you must capture controls to retrieve elements such as input boxes and buttons. Then, you can perform the required operations on these elements. Because you need to retrieve title data from multiple pages, you will use a loop. Pay attention to the notes as you follow the procedure.
A control is a specific element that the Robotic Process Automation (RPA) bot can interact with, such as the Baidu input box, the "Baidu Search" button, the "Next Page" button, and search result titles. For more information, see Capture controls.
This example is a quick start guide for visual widgets. It is not a best practice.
Step 1: Access Baidu and search for "Alibaba Cloud"
1. On the visual project canvas, click the plus (+) icon that appears after the start node. Select and add the Open New Webpage widget.
2. In the properties area of the Open New Webpage widget, enter `www.baidu.com` as the URL to open.
The output shown in the figure is the result produced after the widget completes its task. The widget assigns a default name to the output. You can use this name to reference the output in subsequent steps. For example, in this step, the default name `v_web_obj_1` represents the successfully opened webpage. To interact with elements on this webpage later, you must reference it by this name. After you configure the widget properties, you can click the execute (triangle) button to test whether the webpage opens automatically.
3. Add a Fill Input Box widget after the Open New Webpage widget.
The search box supports fuzzy search. You can enter Pinyin or Chinese characters.
4. In the properties area of the Fill Input Box widget, select `v_web_obj_1` (the output from the previous step) for Opened Browser Object. For Input Content, enter the search term `Alibaba Cloud`. In the Input Box Control field, click the button on the right and select Capture New Control.
All webpage operations require three elements: a webpage object, a control object, and the operation content.
5. Use the capture control feature to capture the Baidu input box. Name the control 1_BaiduInputBox.
For more information about capturing the controls mentioned below, see Capture controls.
6. In the Input Box Control field, select the control that you just captured, `1_BaiduInputBox`. Click the run (triangle) button to verify that the steps execute correctly. Note that you must open the webpage before you can perform operations on its objects.
7. Capture the Baidu Search button control and name it 2_BaiduSearchButton.
8. After the capture is complete, add a Click Control (Webpage) widget after the Fill Input Box (Webpage) widget. In the properties area of the Click Control widget, set Opened Browser Object to `v_web_obj_1` (the browser output from the Open New Webpage widget) and Target Control to `2_BaiduSearchButton`. This completes the Baidu search simulation.
Step 2: Use similar controls to capture multiple titles
1. After the search is complete, capture all titles related to Alibaba Cloud using the Capture Similar Controls feature. Name the captured control 3_MultipleTitles.
In the following figure, first use the Capture Control button to capture the element in red box 1. Then, use the Capture Similar Controls button to capture the element in red box 2.
2. After you capture the control, return to the visual canvas. Drag the Loop Through Similar Controls widget to the canvas. In its properties area, set Opened Window or Browser Object to v_web_obj_1 (the browser object output from the initial Open New Webpage widget). Set Target Control to `3_MultipleTitles`, the control that you just captured.
In the Loop Through Similar Controls widget, the Current Loop Item (`v_element_item_1`) in the Outputs section of the properties area refers to the single control object that is operated on in each iteration of the loop. For example, if there are 10 similar controls, referencing `v_element_item_1` inside the loop body means that the operation is performed on each control sequentially. If the flow stops at the Loop Through Similar Controls widget and returns an `UnboundLocalError: local variable 'v_element_index_1' referenced before assignment` error, set a pre-execution delay of 0.5 seconds in the advanced options of the Loop Through Similar Controls widget.
3. Inside the Loop Through Similar Controls widget, add a Get Text (Webpage) widget. In its properties area, set Input to `v_web_obj_1` (the browser object output from the initial Open New Webpage widget). Set Target Control to `v_element_item_1` (the Current Loop Item output from the previous step). Set Control Index to `v_element_index_1` (the Current Loop Index output from the previous step).
Step 3: Write the collected titles to Excel
1. To record the data, you need to use Excel. From the widget panel, drag the Launch Excel widget to a position before the Loop Through Similar Controls widget. In the properties area of the Launch Excel widget, select the New Excel radio button.
2. After you select New Excel, click the Browse button. Select a location to save the Excel file. In the File name field, enter a custom name for the file and include the `.xlsx` extension. Then, click Save.
3. Add a Get Active Sheet widget after the Launch Excel widget. In its properties area, set Opened Excel Object to `v_excel_obj_1`, the output from the previous step.
Launching Excel opens an Excel workbook. A workbook can contain multiple worksheets (sheets). Data operations must be performed on a specific worksheet. This step retrieves the active sheet, which is the default sheet that is displayed when the workbook opens.
4. Inside the loop body, add a Write a Row to End of Excel widget. In its properties area, set Opened Sheet Object to `v_sheet_obj_1`, the sheet object output from the previous step. For Content to Write, you must provide an array in the format `[XXX, XXX, ...]`. Follow the text prompt. Click the button on the right and select Edit Expression from the options.
An array, such as `[XXX, XXX, ...]`, is a data structure in programming. It represents a collection of multiple data items that are separated by commas. A row of data can have multiple columns. Therefore, you must use this data structure when you write a row. For example, writing `["Title1", "Title2", "Title3"]` to a row corresponds to the following data in the table:
5. In the Edit Expression window, enter `[v_web_text_1, "RPA bot auto-fill test"]`. Then, click OK. This completes the flow for scraping search result titles from a single page.
In this expression, `v_web_text_1` is the output variable from the preceding Get Text (Webpage) widget. To enter text instead of a variable name, you must enclose the text in double quotation marks. This is similar to how text is handled in Excel formulas.
6. To scrape data from multiple pages, you must capture the Next Page button on the Baidu search results page. The capture method is the same as described previously.
7. On the canvas, add a Loop by Times widget. In its properties area, set Loop End Value to `5`.
In this example, the loop start value is 1, the end value is 5, and the step size is 1. The output Current Loop Item (`v_range_item_1`) iterates from 1 to 5 (1, 2, 3, 4, 5). To demonstrate this, the following steps record this value in the Excel file.
8. Drag the Loop Through Similar Controls widget into the Loop by Times widget.
Nesting the Loop Through Similar Controls widget inside the Loop by Times widget means that for each page, after all titles are written to the Excel file, the flow clicks the Next Page button and repeats the operation. This process continues until the loop is complete.
9. In the properties area of the Write a Row to End of Excel widget, click the gray box in the Content to Write field to open the Edit Expression interface. Add `v_range_item_1` to the end of the original expression. The final content to write is `[v_web_text_1,"RPA bot auto-fill test",v_range_item_1]`.
The `v_range_item_1` variable is the value from the Current Loop Item field in the Outputs section of the Loop by Times widget from Step 7.
10. After the Loop Through Similar Controls widget, add a Click Control (Webpage) widget. Set Opened Browser Object to `v_web_obj_1` and Target Control to `Next Page Button`. Then, click Advanced Options and enter `2` in the Delay After Execution box.
Place the Click Control widget inside the Loop by Times widget but after the Loop Through Similar Controls widget. The logic is to retrieve all titles on the current page, click to the next page, and then repeat these actions until the loop completes its preset count of 5. The delay is set to allow the webpage enough time to load. When you click the Next Page button, the page content does not refresh instantly due to network latency. The bot works best in a stable page environment. Waiting 2 seconds after clicking the Next Page button ensures that the page content is stable before the next operation begins.
11. After the loop ends, add a Save Excel widget. Set Input to `v_excel_obj_1`, the output from the Launch Excel widget. You have now built a complete flow. Click Start Debug to run the flow.