Post by Holger FreytherI understand that in the beginning infrastructure will be solely operated
by Nokia, would it make sense to setup a Jenkins outside of Nokia?
I am not sure. The only immediate benefits I can see from doing that is
(1) removes one blocker for non-Nokia entities to add test machines with
their own configurations, (2) the full build logs could immediately be
made public. The disadvantage would be disruption on the existing
codereview/CI setup for all developers.
The build logs issue is tracked by
https://bugreports.qt.nokia.com/browse/QTQAINFRA-139 and should be
solvable with a less drastic solution than an entire switchover of the CI
tool.
Post by Holger FreytherIs there
a list of what Pulse/Scripts do that Jenkins is not capable of doing right
now?
No, not right now. It would make sense to create this.
In our setup, the responsibilities of Pulse/Jenkins/whatever else at
that layer consists of "merely":
- schedule arbitrary test scripts over a pool of machines
- gather and store build logs, test results
- reading the contents of some git repositories
- run arbitrary pre/post build commands for a few things (e.g.
automatic reboot of virtual machines after each test run)
I expect these requirements are simple enough to be satisfied by many
tools. The actual test logic and test machine setups are not
implemented in Pulse at all, they are almost all public and could be
immediately used outside of Nokia's current Pulse setup.
- http://qt.gitorious.org/qt/qtqa <- the scripts used for testing Qt5
in CI
- http://qt.gitorious.org/qtqa/testconfig <- glue between CI tool
(e.g. Pulse) and the above scripts
- http://qt.gitorious.org/qtqa/sysadmin <- metadata for configuration
of test machines
There is some documentation at http://wiki.qt-project.org/index.php/Public_Autotest_Infrastructure
which should be expanded.
Post by Holger FreytherI assume the Qt Project starts off while still using your internal
and non-free pulse system while an alternative is built?
Yes, as far as the CI system is concerned, the Qt project going live meant
nothing other than the firewall rules blocking access to
codereview.qt-project.org from outside of Nokia being dropped.
Post by Holger FreytherPost by Rohan McGovernIf they are generic enough to be usable for e.g. "any" DirectFB backend,
then it may be appropriate to check them into the source tree, if they
are fairly stable. If they're likely to be unstable, or they are very
large, we don't have any good solution at the moment :(
The question is a general one. How do we have custom results? In this
specific case the DirectFB software image decoders generate different
results to Qt, but if I would build a SoC platform and I decide to run
Qt tests and my hardware is giving a slightly different result i would
want to override some results? Any idea?
In case of image results it would make sense to look for 'foo.png'
in images/foo.png, images/QPA-DirectFB/foo.png,
images/QPA-DirectFB/HW/foo.png,
That would be fine, we've used that approach for testdata sometimes in
the past. However it's hard to decide how much detail needs to go into
the identification of the testdata.
For example, maybe changing the version of system libraries such as
freetype could change the font rendering slightly, so would you then
include the freetype version number in the string as well?
Post by Holger Freythermaybe even have qrc:/ look into an external path?
Not sure about that.