Discussion:
3rdpart platform plugins, tests and CI
Holger Freyther
2011-10-17 08:05:25 UTC
Permalink
Hi again,

this is from the non-mainstream plugin (directFB, EGLFS) and non
Nokia point of view. I wonder if inside the Qt Project there is a
plan to setup Buildbot/Jenkins (and slowly add the features your
current CI system provides?). Is there some kind of 'infrastructure'
team for the Qt project and what would be the place to discuss things.

The other topic is platform specific results.E.g. on DirectFB I
might use the DirectFB image decoders, or for a specific product
use HW decoders that decode some pixels differently. How would I
provide some platform specific image results?

holger
Rohan McGovern
2011-10-20 00:08:07 UTC
Permalink
Post by Holger Freyther
Hi again,
this is from the non-mainstream plugin (directFB, EGLFS) and non
Nokia point of view. I wonder if inside the Qt Project there is a
plan to setup Buildbot/Jenkins (and slowly add the features your
current CI system provides?).
Hi Holger,

I would say there is intent but nothing that could seriously be called
a "plan" right now.

With respect to specific tools, Jenkins has a bit of traction, because
it is similar to the non-free tool we are currently using (Pulse)
and it is also coincidentally being used in a few places within Nokia
already.
Post by Holger Freyther
Is there some kind of 'infrastructure'
team for the Qt project and what would be the place to discuss things.
There is a loosely defined team taking care of the existing CI
and code review tool within Nokia. I didn't see any discussions yet
about establishing a new team under the "Qt Project" umbrella, so I
imagine the Nokia team will also act as the Qt Project team at least
for a while. However there is a definite priority problem here, which
could be avoided by a team with a bit more distance to Nokia.

This list seems a fine place for discussions for now, all the relevant
people are subscribed.
Post by Holger Freyther
The other topic is platform specific results.E.g. on DirectFB I
might use the DirectFB image decoders, or for a specific product
use HW decoders that decode some pixels differently. How would I
provide some platform specific image results?
Am I understanding correctly that you want to provide these image
results for use as baselines, for regression checking?

In that case, if they are really for a specific product, then I think
they should not be provided with Qt, but rather with whatever
scripts/packaging metadata/etc you are using to deploy Qt for that
product;

If they are generic enough to be usable for e.g. "any" DirectFB backend,
then it may be appropriate to check them into the source tree, if they
are fairly stable. If they're likely to be unstable, or they are very
large, we don't have any good solution at the moment :(
Holger Freyther
2011-10-21 08:36:13 UTC
Permalink
Post by Rohan McGovern
Hi Holger,
Hi Rohan,
Post by Rohan McGovern
With respect to specific tools, Jenkins has a bit of traction, because
it is similar to the non-free tool we are currently using (Pulse)
and it is also coincidentally being used in a few places within Nokia
already.
I understand that in the beginning infrastructure will be solely operated
by Nokia, would it make sense to setup a Jenkins outside of Nokia? Is there
a list of what Pulse/Scripts do that Jenkins is not capable of doing right
now? I assume the Qt Project starts off while still using your internal
and non-free pulse system while an alternative is built?
Post by Rohan McGovern
If they are generic enough to be usable for e.g. "any" DirectFB backend,
then it may be appropriate to check them into the source tree, if they
are fairly stable. If they're likely to be unstable, or they are very
large, we don't have any good solution at the moment :(
The question is a general one. How do we have custom results? In this
specific case the DirectFB software image decoders generate different
results to Qt, but if I would build a SoC platform and I decide to run
Qt tests and my hardware is giving a slightly different result i would
want to override some results? Any idea?

In case of image results it would make sense to look for 'foo.png'
in images/foo.png, images/QPA-DirectFB/foo.png,
images/QPA-DirectFB/HW/foo.png, maybe even have qrc:/ look into an
external path?
Rohan McGovern
2011-10-24 02:01:27 UTC
Permalink
Post by Holger Freyther
I understand that in the beginning infrastructure will be solely operated
by Nokia, would it make sense to setup a Jenkins outside of Nokia?
I am not sure. The only immediate benefits I can see from doing that is
(1) removes one blocker for non-Nokia entities to add test machines with
their own configurations, (2) the full build logs could immediately be
made public. The disadvantage would be disruption on the existing
codereview/CI setup for all developers.

The build logs issue is tracked by
https://bugreports.qt.nokia.com/browse/QTQAINFRA-139 and should be
solvable with a less drastic solution than an entire switchover of the CI
tool.
Post by Holger Freyther
Is there
a list of what Pulse/Scripts do that Jenkins is not capable of doing right
now?
No, not right now. It would make sense to create this.

In our setup, the responsibilities of Pulse/Jenkins/whatever else at
that layer consists of "merely":

- schedule arbitrary test scripts over a pool of machines
- gather and store build logs, test results
- reading the contents of some git repositories
- run arbitrary pre/post build commands for a few things (e.g.
automatic reboot of virtual machines after each test run)

I expect these requirements are simple enough to be satisfied by many
tools. The actual test logic and test machine setups are not
implemented in Pulse at all, they are almost all public and could be
immediately used outside of Nokia's current Pulse setup.

- http://qt.gitorious.org/qt/qtqa <- the scripts used for testing Qt5
in CI
- http://qt.gitorious.org/qtqa/testconfig <- glue between CI tool
(e.g. Pulse) and the above scripts
- http://qt.gitorious.org/qtqa/sysadmin <- metadata for configuration
of test machines

There is some documentation at http://wiki.qt-project.org/index.php/Public_Autotest_Infrastructure
which should be expanded.
Post by Holger Freyther
I assume the Qt Project starts off while still using your internal
and non-free pulse system while an alternative is built?
Yes, as far as the CI system is concerned, the Qt project going live meant
nothing other than the firewall rules blocking access to
codereview.qt-project.org from outside of Nokia being dropped.
Post by Holger Freyther
Post by Rohan McGovern
If they are generic enough to be usable for e.g. "any" DirectFB backend,
then it may be appropriate to check them into the source tree, if they
are fairly stable. If they're likely to be unstable, or they are very
large, we don't have any good solution at the moment :(
The question is a general one. How do we have custom results? In this
specific case the DirectFB software image decoders generate different
results to Qt, but if I would build a SoC platform and I decide to run
Qt tests and my hardware is giving a slightly different result i would
want to override some results? Any idea?
In case of image results it would make sense to look for 'foo.png'
in images/foo.png, images/QPA-DirectFB/foo.png,
images/QPA-DirectFB/HW/foo.png,
That would be fine, we've used that approach for testdata sometimes in
the past. However it's hard to decide how much detail needs to go into
the identification of the testdata.
For example, maybe changing the version of system libraries such as
freetype could change the font rendering slightly, so would you then
include the freetype version number in the string as well?
Post by Holger Freyther
maybe even have qrc:/ look into an external path?
Not sure about that.
Holger Hans Peter Freyther
2011-10-24 14:21:38 UTC
Permalink
On 10/24/2011 04:01 AM, Rohan McGovern wrote:

Hi Rohan,
Post by Rohan McGovern
The build logs issue is tracked by
https://bugreports.qt.nokia.com/browse/QTQAINFRA-139 and should be
solvable with a less drastic solution than an entire switchover of the CI
tool.
I was not asking for a drastic move. I asked for building up a
Jenkins/whatever system in parallel and once it can do everything the current
CI system can do, it would be nice to switch over. The only blocker from my
side is that the CI/Pulse system is a bit mystic (If there is documentation I
did not read it).

Is this something where Nokia could provide a FreeBSD Jail/Linux Container
(still insecure, so probably OpenVZ)/Virtual Machine? Or should this be setup
entirely outside of the Qt Project and migrated once it is working?
Post by Rohan McGovern
There is some documentation at http://wiki.qt-project.org/index.php/Public_Autotest_Infrastructure
which should be expanded.
ah great, will try to read it.
Post by Rohan McGovern
That would be fine, we've used that approach for testdata sometimes in
the past. However it's hard to decide how much detail needs to go into
the identification of the testdata.
For example, maybe changing the version of system libraries such as
freetype could change the font rendering slightly, so would you then
include the freetype version number in the string as well?
In general yes. If the FreeType version in OSX Snow Lion is providing
different results then I would propose to have different results, in contrast
if some 'embedded' guy is using an old version I will ask him to upgrade. :)



cheers
holger

Loading...