Generating headers with moc

[Peter Kuemmel]

The discussion about public signals has shown that
with current C++ and moc techniques it is not possible
to elegantly solve the problem of protected signals
with statically checked connection calls.

Because of this I propose that in Qt5 the moc mechanism
should be expanded by the generation of public headers.

For instance qobject.h:

#ifndef QOBJECT_H
#define QOBJECT_H

#ifndef QT_NO_QOBJECT

#include <QtCore/qobjectdefs.h>
#include <QtCore/qstring.h>
#include <QtCore/qbytearray.h>

// generated by moc into the build directory
#include <qobject_moc.h>


This would give much more flexibility with generated code.
This extension would add new ways of programming Qt classes
and would allow new features.

Additionally, relying on generated code is always better
than on hand written boilerplate code.

I know, many people think this a step in the wrong direction,
but moccing is a reality in Qt, and Qt never had a C++ purist
ideology, on the contrary Qt always was very pragmatically.


It's time to open another door in C++/Qt development.

Peter

[Peter Kuemmel]

Post by Peter Kuemmel
The discussion about public signals has shown that
with current C++ and moc techniques it is not possible
to elegantly solve the problem of protected signals
with statically checked connection calls.
Because of this I propose that in Qt5 the moc mechanism
should be expanded by the generation of public headers.
#ifndef QOBJECT_H
#define QOBJECT_H
#ifndef QT_NO_QOBJECT
#include <QtCore/qobjectdefs.h>
#include <QtCore/qstring.h>
#include <QtCore/qbytearray.h>
// generated by moc into the build directory
#include <qobject_moc.h>
This would give much more flexibility with generated code.
This extension would add new ways of programming Qt classes
and would allow new features.
Additionally, relying on generated code is always better
than on hand written boilerplate code.
I know, many people think this a step in the wrong direction,
but moccing is a reality in Qt, and Qt never had a C++ purist
ideology, on the contrary Qt always was very pragmatically.
It's time to open another door in C++/Qt development.

moc-generated headers would also in the support of templeted
QObjects:

template<class T>
class TObjectGen : public QObject
{
public:
void setT(T* t) { m_t = t; }
T* t() { return m_t; }

private:
T* m_t;
};

template<class T>
class TObject;

Q_QOBJECT_TEMPLATE(TObject, int)

// for all used types moc must be triggered to generate the meta code
// e.g Q_QOBJECT_TEMPLATE(TObject, int) will generate a header
// with this code:
//
// template<>
// class TObject<int> : public TObjectGen
// {
// Q_OBJECT
//
// ***
//


This is only a demonstration, and I haven't tested it with a
compiler.

Peter

[Peter Kuemmel]

Another idea for generated headers is to introduce
a API which allows the generation of customized code,
similar to

https://developer.mozilla.org/En/Dehydra/Using_Dehydra

(there the AI is used to statically analyze code, but
it could also be used to generate new code or code which
breaks compiling when an error is found.)

Peter

[Peter Kuemmel]

With such a new moc, language bindings also would
become much simpler. Especially the QML binding.

ATM external tools are needed generate the binding
code. But if moc is able to generate such code many
projects could use the some code base.

Concrete binding rules could be loaded as scripts or
plugins when moc is running.

Peter

[Konstantin Tokarev]

Post by Peter Kuemmel
With such a new moc, language bindings also would
become much simpler. Especially the QML binding.
ATM external tools are needed generate the binding
code. But if moc is able to generate such code many
projects could use the some code base.
Concrete binding rules could be loaded as scripts or
plugins when moc is running.

I don't think it should be in the scope of moc.

[Thiago Macieira]

Post by Peter Kuemmel
#ifndef QOBJECT_H
#define QOBJECT_H
#ifndef QT_NO_QOBJECT
#include <QtCore/qobjectdefs.h>
#include <QtCore/qstring.h>
#include <QtCore/qbytearray.h>
// generated by moc into the build directory
#include <qobject_moc.h>

This means that qobject_moc.h depends on qobject.h and qobject.h depends on
qobject_moc.h.

make will scream out.

[Peter Kuemmel]

-------- Original-Nachricht --------

Datum: Sat, 24 Sep 2011 17:01:15 +0200
Betreff: Re: [Qt5-feedback] Generating headers with moc

Post by Peter Kuemmel
#ifndef QOBJECT_H
#define QOBJECT_H
#ifndef QT_NO_QOBJECT
#include <QtCore/qobjectdefs.h>
#include <QtCore/qstring.h>
#include <QtCore/qbytearray.h>
// generated by moc into the build directory
#include <qobject_moc.h>

This means that qobject_moc.h depends on qobject.h and qobject.h depends on
qobject_moc.h.
make will scream out.

Indeed. Would this work? Content of file 'QObject':

#include ".../qobject.h"
#include "qobject_moc.h"

Peter

[Thiago Macieira]

Post by Peter Kuemmel

Post by Thiago Macieira
make will scream out.

#include ".../qobject.h"
#include "qobject_moc.h"

Yes, that would be better, but it would be another generated header. The
QObject forwarding header is just something Qt does to be nice. For other
people who don't have syncqt, that's a new file they need to deal with and
install.

What's more, having public generated headers means we need to preprocess all
headers before we can start compiling.

I also would like to know how moc deals with its input including headers that
it generated by itself.

[Peter Kuemmel]

-------- Original-Nachricht --------

Datum: Sat, 24 Sep 2011 18:05:07 +0200
Betreff: Re: [Qt5-feedback] Generating headers with moc

Post by Peter Kuemmel

Post by Thiago Macieira
make will scream out.

#include ".../qobject.h"
#include "qobject_moc.h"

Yes, that would be better, but it would be another generated header. The
QObject forwarding header is just something Qt does to be nice. For other
people who don't have syncqt, that's a new file they need to
deal with and install.

So sometimes the file 'QObject' is simply a copy of qobject.h?
Then this would not be possible any more.

What's more, having public generated headers means we need to preprocess
all headers before we can start compiling.

Yes compilation would slow down.

I also would like to know how moc deals with its input including headers
that it generated by itself.

The bootsrap code can't rely on the moc's generated headers. This would
limit the usage of Qt features for the boostrap code, or the boostrap
code must use some pre-generated headers. Isn't it the same problem like
the self-compiling of compilers?

Peter

[Thiago Macieira]

Post by Peter Kuemmel

Post by Thiago Macieira
Yes, that would be better, but it would be another generated header. The
QObject forwarding header is just something Qt does to be nice. For other
people who don't have syncqt, that's a new file they need to
deal with and install.

So sometimes the file 'QObject' is simply a copy of qobject.h?
Then this would not be possible any more.

No, today it is always a forwarding header.

My point is that syncqt is a tool we wrote for Qt. It's not used by anyone
else. Your plan was suggesting that every header processed by moc would
produce *two* headers to be installed along with the library: one containing
the output from moc and another with the forwarding.

Two aren't necessary. We only need one: the one from moc.

Post by Peter Kuemmel

Post by Thiago Macieira
What's more, having public generated headers means we need to preprocess
all headers before we can start compiling.

Yes compilation would slow down.

Post by Thiago Macieira
I also would like to know how moc deals with its input including headers
that it generated by itself.

The bootsrap code can't rely on the moc's generated headers. This would
limit the usage of Qt features for the boostrap code, or the boostrap
code must use some pre-generated headers. Isn't it the same problem like
the self-compiling of compilers?

That's not what I was talking about. Take the example that QIODevice derives
from QObject. In order to derive from QObject, the qiodevice.h header needs to
include the full "qobject" headers -- that includes what moc generated too. So
in order for moc to run on qiodevice.h, it needs to be run first on qobject.h.
And moc needs to understand whatever it produced too.

I don't want to shoot you down without listening to your ideas. I'd like to
give you a chance to present them and prove them. So if you feel this strongly
about the feature, go ahead and create a proof of concept.

I've only said I don't like the idea of installed generated headers,
especially if they are for almost every single class. But if the advantages
that you are proposing are worth it, we should use it.

In other words, if you want to see this, you need to create the proof of
concept, prove the 8 points I outlined in the other email and prove to us that
this is worth the hassle.

[Peter Kuemmel]

Post by Thiago Macieira
That's not what I was talking about. Take the example that QIODevice derives
from QObject. In order to derive from QObject, the qiodevice.h header needs to
include the full "qobject" headers -- that includes what moc generated too. So
in order for moc to run on qiodevice.h, it needs to be run first on qobject.h.
And moc needs to understand whatever it produced too.

Indeed, this is really a big problem, moc would have to check
dependencies.

Post by Thiago Macieira
I don't want to shoot you down without listening to your ideas. I'd like to
give you a chance to present them and prove them. So if you feel this strongly
about the feature, go ahead and create a proof of concept.
I've only said I don't like the idea of installed generated headers,
especially if they are for almost every single class. But if the advantages
that you are proposing are worth it, we should use it.
In other words, if you want to see this, you need to create the proof of
concept, prove the 8 points I outlined in the other email and prove to us that
this is worth the hassle.
--
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
Software Architect - Intel Open Source Technology Center
E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358

[Oswald Buddenhagen]

Post by Thiago Macieira

Post by Peter Kuemmel
#ifndef QOBJECT_H
#define QOBJECT_H
#ifndef QT_NO_QOBJECT
#include <QtCore/qobjectdefs.h>
#include <QtCore/qstring.h>
#include <QtCore/qbytearray.h>
// generated by moc into the build directory
#include <qobject_moc.h>

This means that qobject_moc.h depends on qobject.h and qobject.h depends on
qobject_moc.h.

this is actually wrong. there is no dependency of qobject.h on
qobject_moc.h. the *objects* which include qobject.h would also depend
on qobject_moc.h.

[Thiago Macieira]

Post by Oswald Buddenhagen

Post by Thiago Macieira

Post by Peter Kuemmel
#ifndef QOBJECT_H
#define QOBJECT_H
#ifndef QT_NO_QOBJECT
#include <QtCore/qobjectdefs.h>
#include <QtCore/qstring.h>
#include <QtCore/qbytearray.h>
// generated by moc into the build directory
#include <qobject_moc.h>

This means that qobject_moc.h depends on qobject.h and qobject.h depends on
qobject_moc.h.

this is actually wrong. there is no dependency of qobject.h on
qobject_moc.h. the *objects* which include qobject.h would also depend
on qobject_moc.h.

Not if you put the #include like it was in the above example.

To break the dependency, you need to either have a third header that includes
the original and the generated, or the user needs to remember to include the
generated one.

[Oswald Buddenhagen]

Post by Thiago Macieira

Post by Oswald Buddenhagen

Post by Thiago Macieira

Post by Peter Kuemmel
#ifndef QOBJECT_H
#define QOBJECT_H
#ifndef QT_NO_QOBJECT
#include <QtCore/qobjectdefs.h>
#include <QtCore/qstring.h>
#include <QtCore/qbytearray.h>
// generated by moc into the build directory
#include <qobject_moc.h>

This means that qobject_moc.h depends on qobject.h and qobject.h depends on
qobject_moc.h.

this is actually wrong. there is no dependency of qobject.h on
qobject_moc.h. the *objects* which include qobject.h would also depend
on qobject_moc.h.

Not if you put the #include like it was in the above example.

no. including a file does *not* add a dependency of the "includer" on the
"includee". dependencies are always from generated artifacts to their
sources. the _moc.h is generated from the .h, so it has a dependency on
it (and the files it includes; excluding the _moc.h's would be qmake's
responsibility - it can do that, because it knows that the inclusion of
the generated files themselves does not change what will go into the
generated files). the .o is generated from the .cpp and the .h and the
_moc.h, so it has dependencies on them.

[Joerg Bornemann]

Post by Oswald Buddenhagen
no. including a file does *not* add a dependency of the "includer" on the
"includee". dependencies are always from generated artifacts to their
sources. the _moc.h is generated from the .h, so it has a dependency on
it (and the files it includes; excluding the _moc.h's would be qmake's
responsibility - it can do that, because it knows that the inclusion of
the generated files themselves does not change what will go into the
generated files). the .o is generated from the .cpp and the .h and the
_moc.h, so it has dependencies on them.

And we already have such a case with our current moc:

--myobject.cpp--
class MyObject :public QObject {
Q_OBJECT
// ...
};

#include "myobject.moc"
----------------

If includes would add dependencies then we would have a circle:

myobject.cpp -> myobject.moc -> myobject.cpp


Jörg

[Thiago Macieira]

Post by Joerg Bornemann

Post by Oswald Buddenhagen
no. including a file does *not* add a dependency of the "includer" on the
"includee". dependencies are always from generated artifacts to their
sources. the _moc.h is generated from the .h, so it has a dependency on
it (and the files it includes; excluding the _moc.h's would be qmake's
responsibility - it can do that, because it knows that the inclusion of
the generated files themselves does not change what will go into the
generated files). the .o is generated from the .cpp and the .h and the
_moc.h, so it has dependencies on them.

--myobject.cpp--
class MyObject :public QObject {
Q_OBJECT
// ...
};
#include "myobject.moc"
----------------
myobject.cpp -> myobject.moc -> myobject.cpp

I spent half an hour yesterday trying to wrap my brain around what Oswald had
said. I was writing a reply disproving what he had said and I ended up proving
he was right instead. Since it was so convoluted with my thought process, I
discarded it and didn't send.

My problem was thinking of the source file as a target in make. That isn't the
case: neither myobject.cpp nor myobject.h are targets. They simply exist as
files.

myobject.o is a target. That's how your loop above breaks. The make rules are:

myobject.o: myobject.cpp myobject.moc
myobject.moc: myobject.cpp

I got confused because in the case of a header, there is no myobject.o.
Instead, the dependency of myobject.h on myobject.moc.h becomes an indirect
dependency for the next target using the header. That is, if myotherobject.cpp
includes myobject.h, it becomes:

myotherobject.o: myobject.h myobject.moc.h \
myotherobject.h myotherobject.cpp myotherobject.moc

[Oswald Buddenhagen]

Post by Peter Kuemmel
Because of this I propose that in Qt5 the moc mechanism
should be expanded by the generation of public headers.

if one takes it further and allows pre-processing the actual source
code, one could solve the fragile base class problem within qt scope.
while d pointers solve it for data, virtual functions and later added
reimplementations pose a very real problem for qt's binary
compatibility (especially non-qobject classes).

fwiw, in java this problem is solved by the compiler (gjc has a mode to
produce such code), but it is not generically solvable for c++ due to
constraints of the language. it would be solvable for qt due to language
limitations we can impose on classes for which we have strong binary
compatibility requirements. of course, one could do it in the compiler
by adding an appropriate language extension, too.

just thinking aloud ...

[Thiago Macieira]

Post by Oswald Buddenhagen
if one takes it further and allows pre-processing the actual source
code, one could solve the fragile base class problem within qt scope.
while d pointers solve it for data, virtual functions and later added
reimplementations pose a very real problem for qt's binary
compatibility (especially non-qobject classes).
fwiw, in java this problem is solved by the compiler (gjc has a mode to
produce such code), but it is not generically solvable for c++ due to
constraints of the language. it would be solvable for qt due to language
limitations we can impose on classes for which we have strong binary
compatibility requirements. of course, one could do it in the compiler
by adding an appropriate language extension, too.

What are you proposing?

[Oswald Buddenhagen]

Post by Thiago Macieira

Post by Oswald Buddenhagen
one could solve the fragile base class problem within qt scope.

What are you proposing?

as i said, i'm so far only thinking aloud.

the relevant paper which underlies the gjc implementation is here:
http://www.usenix.org/event/jvm02/yu/yu_html/index.html

here is an entirely different approach:
http://gcc.gnu.org/ml/gcc/2000-12/msg00581.html
i've been pondering this myself, and i think it would be doable. the
problem is that it would produce executables which are *stuffed*
with relocations, so it would be kind of the antithesis of generating
PIC code. it would mandate prelinking, and it would conflict with the
IA64 ABI (for which i still don't know why it requires relocation-free
shared objects).

[Konstantin Tokarev]

Post by Oswald Buddenhagen

 Because of this I propose that in Qt5 the moc mechanism
 should be expanded by the generation of public headers.

if one takes it further and allows pre-processing the actual source
code, one could solve the fragile base class problem within qt scope.
while d pointers solve it for data, virtual functions and later added
reimplementations pose a very real problem for qt's binary
compatibility (especially non-qobject classes).
fwiw, in java this problem is solved by the compiler (gjc has a mode to
produce such code), but it is not generically solvable for c++ due to
constraints of the language. it would be solvable for qt due to language
limitations we can impose on classes for which we have strong binary
compatibility requirements. of course, one could do it in the compiler
by adding an appropriate language extension, too.

This could be implemented within Clang plugin (or fork), but depending on it
inside Qt will cause lock-in to particular compiler which may not be appropriate
for many usages...

[Oswald Buddenhagen]

Post by Konstantin Tokarev

Post by Oswald Buddenhagen
of course, one could do it in the compiler
by adding an appropriate language extension, too.

This could be implemented within Clang plugin (or fork),

same with gcc, i think.

Post by Konstantin Tokarev
but depending on it inside Qt will cause lock-in to particular
compiler which may not be appropriate for many usages...

i don't think this is a real problem. using such a special compiler
would add the feature of extended binary compatibility. not using one
would, well, do nothing to the status quo. any missing keywords could be
#defined out, just like "inline" for older C compilers, various dll
import/export-related keywords, gcc's ish __attribute__()s in general,
etc.
an interesting question would be in how far mixing of modified and
unmodified object code would work. in java, it apparently works.