|
|
# Writing Tests
|
|
|
|
|
|
https://invent.kde.org/sdk/selenium-webdriver-at-spi/-/blob/master/examples/calculatortest.py
|
|
|
# GUI Testing with selenium-webdriver-at-spi
|
|
|
|
|
|
## Glossary
|
|
|
|
|
|
- Selenium: A testing system used mostly for weby stuff
|
|
|
- Appium: An extension to selenium making it more useful for app testing
|
|
|
- Driver: The server component of selenium
|
|
|
| Term | Explanation |
|
|
|
|--------------|-----------------|
|
|
|
| [Selenium](https://www.selenium.dev/) | A browser automation framework
|
|
|
| Driver | The server component of Selenium
|
|
|
| [Appium](https://appium.io/) | An extension to Selenium making it more useful for app testing
|
|
|
| [AT-SPI2](https://www.freedesktop.org/wiki/Accessibility/AT-SPI2/) | A protocol over DBus, which GUI toolkit widgets use to provide their content to screen readers
|
|
|
|
|
|
|
|
|
## Requirements
|
|
|
## Installing Dependencies
|
|
|
|
|
|
Basic understanding of how to use `meson`, `cmake`, `autotools`, and `make` to build and install software is expected.
|
|
|
|
|
|
- A supported python3 version (at the time of writing >=3.7)
|
|
|
- https://gitlab.gnome.org/GNOME/at-spi2-core (build & install -- can also use distro packages instead)
|
|
|
- https://gitlab.gnome.org/GNOME/pyatspi2 (build & install -- can also use distro packages instead)
|
|
|
- https://gitlab.gnome.org/GNOME/accerciser (build & install -- can also use distro packages instead)
|
|
|
- https://gitlab.gnome.org/GNOME/gobject-introspection (build & install -- can and should use distro packages instead)
|
|
|
- https://invent.kde.org/sdk/selenium-webdriver-at-spi (build & install)
|
|
|
- Install selenium-webdriver-at-spi dependencies with `pip3 install -r requirements.txt`
|
|
|
- Make sure ~/.local/bin is in your $PATH
|
|
|
### Distribution-specific instructions
|
|
|
|
|
|
## Running Tests
|
|
|
If your distribution is listed below, just install the listed packages.
|
|
|
|
|
|
| Distribution | Install command |
|
|
|
|--------------|-----------------|
|
|
|
| openSUSE Tumbleweed | `sudo zypper install accerciser at-spi2-core cmake-full extra-cmake-modules gcc gcc-c++ git gobject-introspection-devel kcoreaddons-devel kpipewire-devel kwayland-devel kwindowsystem-devel libQt5Core-devel libQt5DBus-devel libqt5-qtwayland-devel libqt5*private*devel plasma-wayland-protocols python3*pycairo*` |
|
|
|
|
|
|
Running tests is best done through cmake respectively ctest when working on an existing code base. For boiler plate logic see for example https://invent.kde.org/plasma/kdeplasma-addons/-/blob/master/appiumtests/CMakeLists.txt
|
|
|
### General-purpose instructions
|
|
|
|
|
|
You can also run tests manually. Like so: `selenium-webdriver-at-spi-run ./examples/calculatortest.py`. The run wrapper makes sure the server side components are correctly started and shut down as necessary; it must be used for things to work correctly!
|
|
|
Otherwise, manually install the following dependencies:
|
|
|
|
|
|
## Test
|
|
|
- A supported `python3` version (at the time of writing >=3.7)
|
|
|
- https://gitlab.gnome.org/GNOME/at-spi2-core
|
|
|
- https://gitlab.gnome.org/GNOME/pyatspi2
|
|
|
- https://gitlab.gnome.org/GNOME/accerciser
|
|
|
- https://gitlab.gnome.org/GNOME/gobject-introspection
|
|
|
|
|
|
You can write tests in any language you want. For the purposes of this guide we are going to use python in the hopes that most readers are familiar enough with that language. The complete source code is available [here](https://invent.kde.org/sdk/selenium-webdriver-at-spi/-/blob/master/examples/calculatortest.py).
|
|
|
## Installing selenium-webdriver-at-spi
|
|
|
|
|
|
```
|
|
|
git clone https://invent.kde.org/sdk/selenium-webdriver-at-spi.git
|
|
|
cd selenium-webdriver-at-spi
|
|
|
pip3 install -r requirements.txt
|
|
|
echo export PATH="~/.local/bin:$PATH" >> ${HOME}/.bashrc
|
|
|
mkdir build
|
|
|
cd build
|
|
|
cmake ..
|
|
|
make
|
|
|
sudo make install
|
|
|
cd ..
|
|
|
```
|
|
|
|
|
|
## Writing Tests
|
|
|
|
|
|
You can write tests in any language you want. For the purposes of this guide we are going to use Python in the hopes that most readers are familiar enough with that language. The complete source code is available [here](https://invent.kde.org/sdk/selenium-webdriver-at-spi/-/blob/master/examples/calculatortest.py).
|
|
|
|
|
|
|
|
|
|
|
|
We start off by creating our test class:
|
|
|
|
|
|
```python
|
|
|
import unittest
|
... | ... | @@ -46,7 +70,7 @@ if __name__ == '__main__': |
|
|
unittest.TextTestRunner(verbosity=2).run(suite)
|
|
|
```
|
|
|
|
|
|
We start off by creating our test class.
|
|
|
Next, we'll define some boilerplate setup logic:
|
|
|
|
|
|
```python
|
|
|
@classmethod
|
... | ... | @@ -70,25 +94,31 @@ We start off by creating our test class. |
|
|
self.driver.quit()
|
|
|
```
|
|
|
|
|
|
We'll also define some boilerplate setup logic. This will start the app org.kde.kcalc.desktop through its desktop file, and expect that it correctly set its desktop file id on the window as well - when not dealing with a proper GUI app you may want to use another startup method: You can also pass command lines to instead fork a process manually. For example you might start a Plasma applet with `"plasmawindowed org.kde.plasma.calculator"`. Valid app startup options are:
|
|
|
This will start the app org.kde.kcalc.desktop through its desktop file, and expect that it correctly set its desktop file id on the window as well - when not dealing with a proper GUI app you may want to use another startup method: You can also pass command lines to instead fork a process manually. For example you might start a Plasma applet with `"plasmawindowed org.kde.plasma.calculator"`. Valid app startup options are:
|
|
|
|
|
|
|
|
|
- desktop file id: `desired_caps["app"] = "org.kde.kcalc.desktop"` the app will be started by its desktop file name similar to how plasma would start it
|
|
|
- command line: `desired_caps["app"] = "plasmawindowed org.kde.plasma.calculator"` the app will be fork()ed off without any expectation of having desktop file ids available
|
|
|
- pid: `desired_caps["app"] = "12356"` you are in charge of starting the app and pass the pid in. please note that you are in charge of ensuring that this pid actually terminates properly once your test is done!
|
|
|
| Option | Example | Explanation |
|
|
|
|----------------- |-------------------------------------------------------------------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
|
| desktop file id | `desired_caps["app"] = "org.kde.kcalc.desktop"` | The app will be started by its desktop file name similar to how plasma would start it |
|
|
|
| command line | `desired_caps["app"] = "plasmawindowed org.kde.plasma.calculator"` | The app will be fork()ed off without any expectation of having desktop file ids available |
|
|
|
| pid | `desired_caps["app"] = "12356"` | You are in charge of starting the app and pass the pid in. please note that you are in charge of ensuring that this pid actually terminates properly once your test is done! |
|
|
|
|
|
|
Let's write our first test. A simple addition should do. To write selenium tests we need to tell the driver to find specific UI elements and interact with them (e.g. click them). There are a number of options for finding elements based on at-spi properties:
|
|
|
|
|
|
- name: `self.driver.find_element(by=AppiumBy.NAME, value="AC")`
|
|
|
- description: `self.driver.find_element(by='description', value="Result Display")`
|
|
|
- accessibility id: `self.driver.find_element(by=AppiumBy.ACCESSIBILITY_ID, value="QGuiApplication.QQuickWindow_QML_28.developerPage")` the ID is constructed from objectNames and the object tree. The id is matched from the end (e.g. in the example value="developerPage" would also match). On the QML side you can also set an objectName when you need to find an Item by its id rather than name or description; Mind that this requires a Qt 5 Patch Collection build to work correctly.
|
|
|
- class name: `self.driver.find_element(by=AppiumBy.CLASS_NAME, value="[push button | AC]")` the class name is comprised of the `type` and `name`, you can easily find this identifier in accerciser's API Browser tab (combobox might need changing away from the "Accessible").
|
|
|
- xpath: `//dialog[@name="Duplicate?"]//push_button[@name="Yes"]` based on an XML representation of the object tree. The xml may be accessed via `http://127.0.0.1:4723/session/$$SESSION-UUID$$/sourceRaw`. http://xpather.com/ is a useful tool to test xpath queries.
|
|
|
Let's write our first test case. A simple addition should do. To write Selenium tests we need to tell the driver to find specific UI elements and interact with them (e.g. click them). There are a number of options for finding elements based on `at-spi` properties:
|
|
|
|
|
|
To figure out what to actually look for we can look at at-spi directly. To do this we'll use the tool "accerciser". On the left hand side you can navigate the various accessible elements. On the right hand side you can inspect the element. The most pertinent tab here is 'Interface Viewer', it let's us find most of the locator types as well as inspect interaction options we have in the "Action" group as well as state assertion options in the "States" list view.
|
|
|
| Option | Example | Explanation |
|
|
|
|------------------ |--------------------------------------------------------------------------------------------------------------------- |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
|
| name | `self.driver.find_element(by=AppiumBy.NAME, value="AC")` | |
|
|
|
| description | `self.driver.find_element(by='description', value="Result Display")` | |
|
|
|
| accessibility id | `self.driver.find_element(by=AppiumBy.ACCESSIBILITY_ID, value="QGuiApplication.QQuickWindow_QML_28.developerPage")` | The ID is constructed from objectNames and the object tree. The id is matched from the end (e.g. in the example value="developerPage" would also match). On the QML side you can also set an objectName when you need to find an Item by its id rather than name or description; Mind that this requires a Qt 5 Patch Collection build to work correctly. |
|
|
|
| class name | `self.driver.find_element(by=AppiumBy.CLASS_NAME, value="[push button \| AC]")` | The class name is comprised of the `type` and `name` , you can easily find this identifier in Accerciser's API Browser tab (combobox might need changing away from the "Accessible"). |
|
|
|
| xpath | `//dialog[@name="Duplicate?"]//push_button[@name="Yes"]` | Based on an XML representation of the object tree. The xml may be accessed via `http://127.0.0.1:4723/session/$$SESSION-UUID$$/sourceRaw` . <br/> http://xpather.com/ is a useful tool to test xpath queries. |
|
|
|
|
|
|
To figure out what to actually look for we can look at `at-spi` directly. To do this we'll use the tool "Accerciser". On the left hand side you can navigate the various accessible elements of currently opened applications. On the right hand side you can inspect the element. The most pertinent tab here is 'Interface Viewer', it let's us find most of the locator types as well as inspect interaction options we have in the "Action" group and state assertion options in the "States" list view.
|
|
|
|
|
|
![Screenshot_20221125_133136-1](uploads/8d0bdc1f09cf0f173bd28e067fc9a9a5/Screenshot_20221125_133136-1.png)
|
|
|
|
|
|
Let's sketch out a simple addition test.
|
|
|
Let's sketch out a simple addition test:
|
|
|
|
|
|
```python
|
|
|
def test_addition(self):
|
... | ... | @@ -100,11 +130,34 @@ Let's sketch out a simple addition test. |
|
|
|
|
|
1+7=8. Easy enough. We'll have to replicate the interaction with the UI. We need to click the numbers and actions in order. For simplicity we'll find the elements by name, but be mindful that finding by name can easily be ambiguous (e.g. in kcalc's specific case the result display may have the same name as a button). When ambiguity is a possibility it's generally a better idea to use one of the other locator strategies.
|
|
|
|
|
|
|
|
|
Lastly we'll find the result display element, obtain its text property, and assert it being 8:
|
|
|
|
|
|
```python
|
|
|
displaytext = self.driver.find_element(by='description', value="Result Display").text
|
|
|
self.assertEqual(displaytext, "8")
|
|
|
```
|
|
|
|
|
|
Lastly we'll find the result display element, obtain its text property, and assert it being 8. We now have our first test completed. To run this test we'll simply execute it through the test runner `selenium-webdriver-at-spi-run ./calculatortest.py`
|
|
|
## Running Tests
|
|
|
|
|
|
We now have our first test completed. To run this test we'll simply execute it through the test runner like this:
|
|
|
|
|
|
`selenium-webdriver-at-spi-run examples/calculatortest.py`
|
|
|
|
|
|
The run wrapper makes sure the server side components are correctly started and shut down as necessary; it must be used for things to work correctly!
|
|
|
|
|
|
When working on an existing code base, running tests is best done through `cmake` respectively `ctest`. For boiler plate logic see [this](https://invent.kde.org/plasma/kdeplasma-addons/-/blob/master/appiumtests/CMakeLists.txt) example.
|
|
|
|
|
|
A Wayland compositor window opens and the test cases are executed sequentially:
|
|
|
|
|
|
![Screenshot_20230308_012038](uploads/66096bf0c26c86d725672bfc2f0238e8/Screenshot_20230308_012038.png)
|
|
|
|
|
|
Afterwards, the console output should show that the tests succeeded:
|
|
|
|
|
|
```
|
|
|
----------------------------------------------------------------------
|
|
|
Ran 6 tests in 38.490s
|
|
|
|
|
|
The complete test can be found at https://invent.kde.org/sdk/selenium-webdriver-at-spi/-/blob/master/examples/calculatortest.py |
|
|
OK
|
|
|
kwin_wayland_backend: Destroyed Wayland display
|
|
|
``` |
|
|
\ No newline at end of file |