Revision control
Copy as Markdown
QMLBenchrunner
==============
QMLBenchrunner is a simple script and python application to programmatically
clone a Qt Git repository, run associated QML Benchmark tests, and post
results to the qt testresults timeseries database.
Prerequisites
-------------
-Qt library
-initialized and configured Qt modules
-Qmlbench repository
- Python3
- 'requests' module
Windows-specific Prerequisites:
* (Qt 5.x) Flex & Bison for windows - Included with qmlbenchrunner, or downloadable from
If Win Flex-Bison is downloaded, you must rename the executables to "flex.exe" and "bison.exe"
and specify the location of your flex-bison executables using the powershell parameter specified
in the parameters section below.
* VS Build Tools 2019 (VS 2017 is supported for Qt 5.x) - https://visualstudio.microsoft.com/downloads/
* JOM - A version is included with qmlbenchrunner. If a new version is required, it can be found at
Usage
-----
QMLBenchrunner will build the required Qt Git repositories as well as qmlbench into the local machine.
Always set environment variables TIMESCALEDBUSER and TIMESCALEDBPASSWORD in the console before calling the build_and_test
script. Results will still be saved to disk in "results.json" even if unable to write to the database.
Optional environment variables:
TIMESCALEDBUSER=username #Set if writing to a database.
TIMESCALEDBPASSWORD=password #Set if writing to a database.
BADTESTS=/path/to/bad/test #Specify a space-separated list of known bad tests or directories
that are too difficult for the client to complete and would cause a crash.
EXTRA_CONFIGURE_ARGS="string of additional Configure arguments"
### Linux: ###
Some machines may require XAUTHORITY to be specified. If problems are encountered, set XAUTHORITY
and DISPLAY in the console before running this script.
**build_and_test.sh arguments are strictly positional. Do not skip arguments that are required.**
Args:
QtVersion (required) | MachineName (required) | BuildCores (required) |
Example:
export TIMESCALEDBUSER=dbuser1
export TIMESCALEDBPASSWORD=dbuser1password
export XAUTHORITY=/home/user1/.Xauthority
export DISPLAY=:0
qmlbenchrunner/build_and_test.sh 5.15 $NODE_NAME 8
### Windows: ###
Qmlbenchrunner should be executed from Powershell for best compatibility. Because QMLBench is a
graphical application, if qmlbenchrunner is being executed via a jenkins slave, the slave must use
the java web start method. Running jenkins as a Windows service will not display QMLBench on the
real user desktop.
Args:
QtVersion (Required)
MachineName (Required)
BuildCores (Optional)
Annotate (Optional)
QtDeclarativeVersion (Optional)
FlexBisonDir = (Optional, defaults to "[current working directory]\flex_bison\" if omitted)
Example:
$env:TIMESCALEDBUSER=dbuser1
$env:TIMESCALEDBPASSWORD=dbuser1password
$env:BADTESTS=""
.\qmlbenchrunner\build_and_test_windows.ps1 -QtVersion 5.15 -BuildCores 7 -MachineName
$ENV:NODE_NAME -FlexBisonDir C:\flex_bison -Annotate
### macOS: ###
See instructions for Linux. Some trouble has been observed with Jenkins macOS clients not setting
PATH correctly. If you experience issues when submitting results to a database, make sure that
Python3 is set in the PATH environment variable used by Jenkins scripts.
Example:
export PATH+=:/Library/Frameworks/Python.framework/Versions/3.6/bin/
qmlbenchrunner/build_and_test.sh dev $NODE_NAME 6 annotate
### Embedded ###
The build_and_test_embedded.sh script is intended to be used with a Jenkins host, but can be used
on it' own. The script file will need to be modified based on your specific environment as detailed
below.
Terminology for this section:
* Host - The machine this script will run on.
* Client - The target embedded device that will execute QMLBench.
Assumptions:
1) The host is configured for cross-compilation to a given target device.
2) An official Boot2Qt SDK is installed on the host.
3) The target embedded device (client) is accessible via SSH over a local network by IP and the
host has connected to it at least once.
4) The host is accessible via ssh over a local network by IP and the host's ssh configuration is configured to accept connections from the client.
Required script alterations:
1) Update the sysrootDir with your Boot2Qt version number.
2) Update the user and IP of the host in order to pass the results.json file back from the
client.
Qt5 Required host configuration (customize for your target device):
DEVMKSPEC=linux-imx6-g++
DEVNAME=apalis-imx6
CROSSCOMPILE=x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi/arm-poky-linux-gnueabi-
ARMSYSROOT=cortexa9hf-neon-poky-linux-gnueabi
DEVIP=IP of the target device to execute the tests on.
Qt6 Required host configuration:
CMAKE_GENERATOR="Unix Makefiles"
LB_TOOLCHAIN: Full path of the Boot2Qt sdk toolchain to use when cross-compiling.
TOOLCHAIN_FILE: Full path of the cmake toolchain file to use when cross-compiling.
DEVIP: IP of the target device to execute the tests on.
Running the script:
**build_and_test_embedded_[qt5|qt6].sh arguments are strictly positional. Do not skip arguments that are
required.**
Args:
QtVersion (required) | MachineName (required) | BuildCores (required for qt5, omit in qt6) |
Annotate (required, set to "False" if not desired) |
QtDeclarativeVersion (optional, leave missing if same as main QtVersion)
qmlbenchrunner/build_and_test_embedded.sh 5.15 $NODE_NAME 8 annotate
Required client configuration:
1) Verify that at least 1GB of free space exists on the client.
NB! This script will wipe out /opt/qt on the device. Verify that this directory does not
contain anything you want to keep.