VFS in qt5

[Иван Комиссаров]

As long as QFileEngines are marked as deprecated, Qt will have no possibility to implement virtual filesystem.

So i would like to discuss features of new VFS.

I have 2 questions. First - should it work using QFile and QDir classes as front-end or it should provide its own API like QVFSFile? Second - if we provide new API, should it be synchronous (like QFile) or asynchronous (like Q*Socket)?

Ivan.

[Thiago Macieira]

Post by Иван Комиссаров
As long as QFileEngines are marked as deprecated, Qt will have no
possibility to implement virtual filesystem.
So i would like to discuss features of new VFS.
I have 2 questions. First - should it work using QFile and QDir classes as
front-end or it should provide its own API like QVFSFile? Second - if we
provide new API, should it be synchronous (like QFile) or asynchronous
(like Q*Socket)?

I would prefer a new API, more suited for the task. Whether it is synchronous
or asynchronous, it's up to the implementation.

Experience in KDE shows asynchronous is better when a non-negligible delay is
expected.

[Georg Rudoy]

Post by Thiago Macieira

Post by Иван Комиссаров
As long as QFileEngines are marked as deprecated, Qt will have no
possibility to implement virtual filesystem.
So i would like to discuss features of new VFS.
I have 2 questions. First - should it work using QFile and QDir classes as
front-end or it should provide its own API like QVFSFile? Second - if we
provide new API, should it be synchronous (like QFile) or asynchronous
(like Q*Socket)?

I would prefer a new API, more suited for the task. Whether it is synchronous
or asynchronous, it's up to the implementation.

Personally, I think QIODevice-derived thing would be just enough,
maybe with additional protocol-specific methods/properties for things
like FTP mode changes, or such.

Post by Thiago Macieira
Experience in KDE shows asynchronous is better when a non-negligible delay is
expected.

Agreed. Synchronous QFile-style approach just doesn't scale well to
operations with remote sources (thus with much greater latencies and
more faulty ones) and even with local sources that require a lot of
work to be done (think of accessing a file inside a .xz archive).

After all, it's much easier to make a sync API out of async one given
all the waitFor*-functions than the reverse.

[Иван Комиссаров]

Ok, asynchronous data reading is not a problem a well as QIODevice has asynchronous API.

But what about other QFile methods? I mean copy, rename, link and setPermissions. Should they be asynchronous too?

Next question - what about QVFSDir? Should directory parsing be asynchronous?

Post by Georg Rudoy

Post by Thiago Macieira

Post by Иван Комиссаров
As long as QFileEngines are marked as deprecated, Qt will have no
possibility to implement virtual filesystem.
So i would like to discuss features of new VFS.
I have 2 questions. First - should it work using QFile and QDir classes as
front-end or it should provide its own API like QVFSFile? Second - if we
provide new API, should it be synchronous (like QFile) or asynchronous
(like Q*Socket)?

I would prefer a new API, more suited for the task. Whether it is synchronous
or asynchronous, it's up to the implementation.

Personally, I think QIODevice-derived thing would be just enough,
maybe with additional protocol-specific methods/properties for things
like FTP mode changes, or such.

Post by Thiago Macieira
Experience in KDE shows asynchronous is better when a non-negligible delay is
expected.

Agreed. Synchronous QFile-style approach just doesn't scale well to
operations with remote sources (thus with much greater latencies and
more faulty ones) and even with local sources that require a lot of
work to be done (think of accessing a file inside a .xz archive).
After all, it's much easier to make a sync API out of async one given
all the waitFor*-functions than the reverse.
--
Georg Rudoy
LeechCraft — http://leechcraft.org

_______________________________________________
Qt5-feedback mailing list
Qt5-***@qt.nokia.com
http://lists.qt.nokia.com/mailman/listinfo/qt5-feedback

[s***@accenture.com]

If the file is on a remote file system, then those functions can also potentially have a high latency.

For example, if you just lost connectivity, then the synchronous function would not complete until it fails with a supervisory timeout at the network layer (e.g. TCP reset in the case of TCP/IP connectivity)

-----Original Message-----
On Behalf Of ???? ??????????
Sent: Sunday, September 25, 2011 19:03
Subject: Re: [Qt5-feedback] VFS in qt5
Ok, asynchronous data reading is not a problem a well as QIODevice has asynchronous API.
But what about other QFile methods? I mean copy, rename, link and
setPermissions. Should they be asynchronous too?
Next question - what about QVFSDir? Should directory parsing be asynchronous?

Post by Georg Rudoy

Post by Thiago Macieira

Post by Иван Комиссаров
As long as QFileEngines are marked as deprecated, Qt will have no
possibility to implement virtual filesystem.
So i would like to discuss features of new VFS.
I have 2 questions. First - should it work using QFile and QDir

classes as

Post by Georg Rudoy

Post by Thiago Macieira

Post by Иван Комиссаров
front-end or it should provide its own API like QVFSFile? Second -

if we

Post by Georg Rudoy

Post by Thiago Macieira

Post by Иван Комиссаров
provide new API, should it be synchronous (like QFile) or

asynchronous

Post by Georg Rudoy

Post by Thiago Macieira

Post by Иван Комиссаров
(like Q*Socket)?

I would prefer a new API, more suited for the task. Whether it is

synchronous

Post by Georg Rudoy

Post by Thiago Macieira
or asynchronous, it's up to the implementation.

Personally, I think QIODevice-derived thing would be just enough,
maybe with additional protocol-specific methods/properties for things
like FTP mode changes, or such.

Post by Thiago Macieira
Experience in KDE shows asynchronous is better when a non-negligible

delay is

Post by Georg Rudoy

Post by Thiago Macieira
expected.

Agreed. Synchronous QFile-style approach just doesn't scale well to
operations with remote sources (thus with much greater latencies and
more faulty ones) and even with local sources that require a lot of
work to be done (think of accessing a file inside a .xz archive).
After all, it's much easier to make a sync API out of async one given
all the waitFor*-functions than the reverse.
--
Georg Rudoy
LeechCraft - http://leechcraft.org

_______________________________________________
Qt5-feedback mailing list
http://lists.qt.nokia.com/mailman/listinfo/qt5-feedback
_______________________________________________
Qt5-feedback mailing list
http://lists.qt.nokia.com/mailman/listinfo/qt5-feedback

________________________________
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.

[Thiago Macieira]

Post by s***@accenture.com
If the file is on a remote file system, then those functions can also
potentially have a high latency.
For example, if you just lost connectivity, then the synchronous function
would not complete until it fails with a supervisory timeout at the network
layer (e.g. TCP reset in the case of TCP/IP connectivity)

Or a timeout in the VFS implementation, depending on the implementation.

I know an asynchronous API is much harder to deal with and leads to more
complex code. Synchronous would make it easier, but you'd have to use a lot
more threading to ensure the UI or other functions aren't blocked. Threading
also has the problem of timeouts.

One of the worst problems with NFS (at least, on Linux with kernel mounts) is
that if the server goes away, the applications hang indefinitely. There's no
soft reset. Usually, the only way to get them back is by making the server
available again or by rebooting.

I'm not saying our VFS layer would have anywhere near those problems. But it
goes to show the kind of failure modes the API needs to handle.

Just like the case of installing generated headers, I have my preconceived
notion that asynchronous is better. But I'd like to keep an open mind -- and
ask everyone else to do to -- and see the benefits of either approach. Whoever
is designing this new addon has the right to make the choices and present
their case for why they chose a particular way.

BTW, if anyone has suggestions on how to write cancellable synchronous code
that is also non-blocking (co-threads?), I'd love to read some more. Vala (the
language) has some concepts in that area, but I don't know how they're
implemented.

[Olivier Goffart]

Post by Thiago Macieira
I know an asynchronous API is much harder to deal with and leads to more
complex code.

Not necessarily...

connect (manager->get(QNetworkRequest(url)), &QNetworkReply::finished, [=](){
QByteArray content = qobject_cast<QNetworkReply*>(sender())->readAll();
parse(content);
});



That is, using C++11 lambda and the proposed conneciton syntax:
http://developer.qt.nokia.com/wiki/New_Signal_Slot_Syntax#45f5369b0d8445adbaf54cd74a36b823

[Thiago Macieira]

Post by Olivier Goffart

Post by Thiago Macieira
I know an asynchronous API is much harder to deal with and leads to more
complex code.

Not necessarily...
connect (manager->get(QNetworkRequest(url)), &QNetworkReply::finished,
[=](){ QByteArray content =
qobject_cast<QNetworkReply*>(sender())->readAll(); parse(content);
});
http://developer.qt.nokia.com/wiki/New_Signal_Slot_Syntax#45f5369b0d8445adba
f54cd74a36b823

It helps, but it's still more complex than direct, unbroken synchronous code.

[Adriano Rezende]

Post by Olivier Goffart

Post by Thiago Macieira
I know an asynchronous API is much harder to deal with and leads to more
complex code.

Not necessarily...
connect (manager->get(QNetworkRequest(url)), &QNetworkReply::finished, [=](){
               QByteArray content = qobject_cast<QNetworkReply*>(sender())->readAll();
               parse(content);
       });
http://developer.qt.nokia.com/wiki/New_Signal_Slot_Syntax#45f5369b0d8445adbaf54cd74a36b823

I think this new C++11 lambda feature should be used wisely.
Personally, I wouldn't like to see many of these blocks in a source code.

Br,
Adriano

[Georg Rudoy]

Post by Thiago Macieira
Just like the case of installing generated headers, I have my preconceived
notion that asynchronous is better.

Either do I.

Post by Thiago Macieira
But I'd like to keep an open mind -- and
ask everyone else to do to -- and see the benefits of either approach. Whoever
is designing this new addon has the right to make the choices and present
their case for why they chose a particular way.

I'd still like to emphasize that it's much easier to make sync API of
async one rather then do the reverse. IMHO that single argument should
be enough to decide in favour of async API.

--
Georg Rudoy
LeechCraft — http://leechcraft.org

[Matthias Hörmann]

Post by Georg Rudoy
I'd still like to emphasize that it's much easier to make sync API of
async one rather then do the reverse. IMHO that single argument should
be enough to decide in favour of async API.

And how would you do such a thing easily? Assuming we are talking about
Qt signal/slot asynchronous APIs those are a real pain to turn into a
synchronous
API without ending up in a horrible mess of nested QEventLoops that sometimes
abort for no obvious reason (they do when the inner event loop has no reason to
quit but one further out does).

There are ways of implementating asynchronous APIs that do not have this problem
of course but Qt's usual way of implementing them is not among those as far as I
can tell as a regular Qt user.
--
Mit freundlichen Grüßen,

Matthias Hörmann

fon: +49 (0) 521 - 329647-29
fax: +49 (0) 521 - 329647-40
email: ***@saltation.de


---------------
saltation GmbH & Co. KG | Niederwall 43 | 33602 Bielefeld
Sitz Bielefeld | Amtsgericht Bielefeld HRA 15344
Persönlich haftende Gesellschafterin:
saltation Beteiligungs-GmbH | Niederwall 43 | 33602 Bielefeld
Sitz Bielefeld | Amtsgericht Bielefeld HRB 39339
Geschäftsführer: Daniel Brün
---------------

[Georg Rudoy]

Just as with current API for sockets, for example, using the waitFor* family
of functions.

--
Georg Rudoy
http://leechcraft.org

Post by Georg Rudoy
I'd still like to emphasize that it's much easier to make sync API of
async one rather then do the reverse. IMHO that single argument should
be enough to decide in favour of async API.

And how would you do such a thing easily? Assuming we are talking about
Qt signal/slot asynchronous APIs those are a real pain to turn into a
synchronous
API without ending up in a horrible mess of nested QEventLoops that sometimes
abort for no obvious reason (they do when the inner event loop has no reason to
quit but one further out does).
There are ways of implementating asynchronous APIs that do not have this problem
of course but Qt's usual way of implementing them is not among those as far as I
can tell as a regular Qt user.
--
Mit freundlichen GrÌßen,
Matthias Hörmann
fon: +49 (0) 521 - 329647-29
fax: +49 (0) 521 - 329647-40
---------------
saltation GmbH & Co. KG | Niederwall 43 | 33602 Bielefeld
Sitz Bielefeld | Amtsgericht Bielefeld HRA 15344
saltation Beteiligungs-GmbH | Niederwall 43 | 33602 Bielefeld
Sitz Bielefeld | Amtsgericht Bielefeld HRB 39339
GeschÀftsfÌhrer: Daniel BrÌn
---------------

[Thiago Macieira]

Post by Georg Rudoy
Just as with current API for sockets, for example, using the waitFor* family
of functions.

Those are very hard to create in a higher level of the stack. Take, for
example, QNetworkReply: the function is simply a "return false" since it
cannot block everything else. In order for this request to complete, other
requests may have to complete too.

The easiest way to make it blocking is to move the asynchronous code into a
separate thread and let it run there. But that's also the solution to making
synchronous code asynchronous.

[s***@accenture.com]

Just as with current API for sockets, for example, using the waitFor* family of functions.

This isn't a good example - sockets can only do that because they can do something special with the socket descriptor internally. (a blocking select() call)
In general, it's better to know whether the request is asynchronous or synchronous before starting it.
That gives more flexibility to the implementation. A waitFor function in the general case requires a nested event loop.

I'd suggest a parallel API for synchronous usage.
e.g.
write(data) - emits bytesWritten() when done
writeSynchronous(data, timeout) - blocks until the write completed or timeout occurs.

Or to do it at the class level:
QSyncVFSDirIterator - sync implementation, functions take timeout parameters and block until done
QAsyncVFSDirIterator - async implementation, functions emit signals when done

The synchronous implementation can be done by running the asynchronous version in a worker thread.
Or where the underlying API has synchronous and asynchronous versions, it can use the synchronous version.

________________________________
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.

[m***@nokia.com]

Post by Thiago Macieira
BTW, if anyone has suggestions on how to write cancellable synchronous code
that is also non-blocking (co-threads?), I'd love to read some more. Vala (the
language) has some concepts in that area, but I don't know how they're
implemented.

I've been toying around with a new API for QtConcurrent that supports the "threaded synchronous" style with cancellation. If using futures it could look like this:

future f = run([]{
contents = QFile::readAll(fileName);

process(contents);
});

f.cancel() // cancels the task at any point.

QFile:readAll() would register itself with the future and be implemented on top of a suitable async API that supports cancellation. How does it return from the readAll() call? By throwing a QCancelledException.

Morten

[Thiago Macieira]

Post by m***@nokia.com

Post by Thiago Macieira
BTW, if anyone has suggestions on how to write cancellable synchronous code
that is also non-blocking (co-threads?), I'd love to read some more. Vala
(the language) has some concepts in that area, but I don't know how
they're implemented.

I've been toying around with a new API for QtConcurrent that supports the
"threaded synchronous" style with cancellation. If using futures it could
future f = run([]{
contents = QFile::readAll(fileName);
process(contents);
});
f.cancel() // cancels the task at any point.
QFile:readAll() would register itself with the future and be implemented on
top of a suitable async API that supports cancellation. How does it return
from the readAll() call? By throwing a QCancelledException.

When I said cancellable, I think I used the wrong term. I didn't mean
cancelling it programmatically, though it of course makes sense (the user may
press the Cancel button).

I was thinking of operations that might fail with bad results. For example,
when you read a file, you expect the operation to succeed. Most file APIs do not
deal with errors like EIO and cannot recover from it since they are quite
rare. On a network scenario, such calls can and will fail often.

I was thinking of something like the async client example in
http://live.gnome.org/Vala/GIONetworkingSample. You simply write a bunch of
code that would be otherwise blocking, but at suitable points where it might
block, the execution yields to something else. It helps that Vala is a C code
generator, so it can simply split up the function into chunks and set the next
chunk as callback to the previous one.

That's why I thought of co-threads. Whenever we run into a blocking call
(EWOULDBLOCK), the execution suspends and something else takes over. until
we're asynchronously notified that the blocking call can now succeed. At that
point, it resumes from where it left off. Co-threads could be implemented using
setjmp/longjmp, for example.

I imagine though that this would make all hell break loose. You'd write:

1 QNetworkReply reply = manager.get(...);
2 QByteArray line = synchronous reply.readLine();
3 process(line);

Between lines 2 and 3, the execution might be suspended and other code would
run, which means the rest of the program's state could completely change.

[Girish Ramakrishnan]

Hi,

Post by Иван Комиссаров
As long as QFileEngines are marked as deprecated, Qt will have no possibility to implement virtual filesystem.
So i would like to discuss features of new VFS.
I have 2 questions. First - should it work using QFile and QDir classes as front-end or it should provide its own API like QVFSFile? Second - if we provide new API, should it be synchronous (like QFile) or asynchronous (like Q*Socket)?

What exactly is the use case for having VFS at the QFile and QDir
level? I would prefer QFile and QDir to always operate on the actual
file system and have nothing to do with virtual file systems. The
current implementation of file engines is quite flawed - it has a
malloc cost for every QFile/QDir (i think this is fixed in 4.8 by
using the QFileSystemEngine) and more importantly it injects special
handling for certain files throughout the application. This most often
breaks application code. As an example, if you implemented a tar file
engine which makes .tar files appear as directories, it changes the
behavior of your file dialogs (you can step into tar files now). It
also breaks all the file iteration code because .tar files don't
appear as files anymore.

I haven't thought this through but I would like to see a QNAM style
API for virtual file systems (with iteratation support etc). And just
like one has to add explicit network support (and not expect some
magic QFile/QDir handling), one has to add explicit vfs support in the
code and expose API like QWebPage::setNetworkAccessManager.

Girish

[l***@nokia.com]

Post by Girish Ramakrishnan
Hi,

Post by Иван Комиссаров
As long as QFileEngines are marked as deprecated, Qt will have no
possibility to implement virtual filesystem.
So i would like to discuss features of new VFS.
I have 2 questions. First - should it work using QFile and QDir classes
as front-end or it should provide its own API like QVFSFile? Second - if
we provide new API, should it be synchronous (like QFile) or
asynchronous (like Q*Socket)?

What exactly is the use case for having VFS at the QFile and QDir
level? I would prefer QFile and QDir to always operate on the actual
file system and have nothing to do with virtual file systems. The
current implementation of file engines is quite flawed - it has a
malloc cost for every QFile/QDir (i think this is fixed in 4.8 by
using the QFileSystemEngine) and more importantly it injects special
handling for certain files throughout the application. This most often
breaks application code. As an example, if you implemented a tar file
engine which makes .tar files appear as directories, it changes the
behavior of your file dialogs (you can step into tar files now). It
also breaks all the file iteration code because .tar files don't
appear as files anymore.
I haven't thought this through but I would like to see a QNAM style
API for virtual file systems (with iteratation support etc). And just
like one has to add explicit network support (and not expect some
magic QFile/QDir handling), one has to add explicit vfs support in the
code and expose API like QWebPage::setNetworkAccessManager.

Agree with Girish. The QFileEngine API is flawed. I actually want to
remove it from Qt 5. At the minimum make it private for Qt 5.0 and remove
at a later stage once the code is refactored to not require it anymore.

VFS support would be good to have, but let's make it explicit. We should
also keep it outside of QtCore to be able to support network based
protocols as ftp and http.

Cheers,
Lars

[Иван Комиссаров]

Post by l***@nokia.com
Agree with Girish. The QFileEngine API is flawed. I actually want to
remove it from Qt 5. At the minimum make it private for Qt 5.0 and remove
at a later stage once the code is refactored to not require it anymore.
VFS support would be good to have, but let's make it explicit. We should
also keep it outside of QtCore to be able to support network based
protocols as ftp and http.

What's the difference where to place VFS? I think we will use some kind of plugins for it - so the plugin will depend on network, not VFS itself.

[Иван Комиссаров]

I think the problem is then that using libQtCore pulls in VFS plugins
which pull in libQtNetwork. This is bad if all you want is libQtCore
(e.g. embedded systems). I'm also all for including VFS support
explicitly. After all, this approach has worked well for KDE since
years.

If you need only QtCore, you don't use plugins that depend on QtNetwork - what's the problem?

[David Faure]

Post by Иван Комиссаров
As long as QFileEngines are marked as deprecated, Qt will have no
possibility to implement virtual filesystem.
So i would like to discuss features of new VFS.
I have 2 questions. First - should it work using QFile and QDir classes as
front-end or it should provide its own API like QVFSFile? Second - if we
provide new API, should it be synchronous (like QFile) or asynchronous
(like Q*Socket)?

With Qt 5 I think we should start thinking much more in terms of "a collection
of libraries on top of a few core libraries". For something to be available,
it doesn't have to be "inside Qt" (i.e. Core, Gui, Network).

What I mean by that, is that we'll soon have a set of independent libraries
that can be used with Qt, including libkarchive (ZIP/TAR/XZ/gzip/bzip2...),
libkauth, libkplotting, solid (hardware detection), itemmodels (common proxies
etc.)... and soon KIO, i.e. an existing, proven stable, implementation of a
VFS.

Why reinvent the wheel in Qt when you can use an existing framework?
(I'm not saying to use the KDE 4 kio, well you can but it comes with more
dependencies than one might want. However the goal is for KDE Frameworks 5 to
provide these pieces exactly in a way that make it easy for any Qt application
developer to use them.).

To give you an idea, KIO supports at least the following protocols:

about cgi fish gzip knewsservice magnet network pop3 settings thumbnail zip
akonadi data floppy help man nfs pop3s sftp trash applications
desktop fonts http konvirc mbox nntp programs sieve videodvd ar
feed ftp https konvircs metainfo nntps remote smb webcal bookmarks
filenamesearch ghelp imap4 ldap mms perldoc rtsp smtp webdav bzip2 file
groupwise imaps ldaps mmst plasma rtspt smtps webdavs bzip finger
groupwises info lzma mmsu pnm rtspu tar xz

(some of these are internal, but you can still recognize many useful virtual
filesystems)

[Иван Комиссаров]

Post by David Faure
With Qt 5 I think we should start thinking much more in terms of "a collection
of libraries on top of a few core libraries". For something to be available,
it doesn't have to be "inside Qt" (i.e. Core, Gui, Network).
What I mean by that, is that we'll soon have a set of independent libraries
that can be used with Qt, including libkarchive (ZIP/TAR/XZ/gzip/bzip2...),
libkauth, libkplotting, solid (hardware detection), itemmodels (common proxies
etc.)... and soon KIO, i.e. an existing, proven stable, implementation of a
VFS.
Why reinvent the wheel in Qt when you can use an existing framework?
(I'm not saying to use the KDE 4 kio, well you can but it comes with more
dependencies than one might want. However the goal is for KDE Frameworks 5 to
provide these pieces exactly in a way that make it easy for any Qt application
developer to use them.).
about cgi fish gzip knewsservice magnet network pop3 settings thumbnail zip
akonadi data floppy help man nfs pop3s sftp trash applications
desktop fonts http konvirc mbox nntp programs sieve videodvd ar
feed ftp https konvircs metainfo nntps remote smb webcal bookmarks
filenamesearch ghelp imap4 ldap mms perldoc rtsp smtp webdav bzip2 file
groupwise imaps ldaps mmst plasma rtspt smtps webdavs bzip finger
groupwises info lzma mmsu pnm rtspu tar xz
(some of these are internal, but you can still recognize many useful virtual
filesystems)

As you said, KIO provides lot of dependencies. I don't think this dependencies will be removed in future.

Also, KIO doesn't seems to be 'vfs' in my opinion - it just provides asynchronous methods for common operations. I'm not sure, but i don't think people will be glad to see al this "job" stuff.

[Thiago Macieira]

Post by Иван Комиссаров
Also, KIO doesn't seems to be 'vfs' in my opinion - it just provides
asynchronous methods for common operations. I'm not sure, but i don't think
people will be glad to see al this "job" stuff.

Just rename the classes.

QNetworkReply is a "job". You post the job using QNetworkAccessManager.

We're not suggesting using KIO. The API was designed back in 2000 for KDE 2
and, while it works, I'd rather have a smaller, leaner API for Qt. Some of
that learning I applied in QNetworkReply, for example, which works as a hybrid
of KIO::TransferJob and KIO::StoredTransferJob.

[Иван Комиссаров]

Post by Thiago Macieira

Post by Иван Комиссаров
Also, KIO doesn't seems to be 'vfs' in my opinion - it just provides
asynchronous methods for common operations. I'm not sure, but i don't think
people will be glad to see al this "job" stuff.

Just rename the classes.
QNetworkReply is a "job". You post the job using QNetworkAccessManager.
We're not suggesting using KIO. The API was designed back in 2000 for KDE 2
and, while it works, I'd rather have a smaller, leaner API for Qt. Some of
that learning I applied in QNetworkReply, for example, which works as a hybrid
of KIO::TransferJob and KIO::StoredTransferJob.

Well, i meant people won't be happy to see Jobs as they are. API is ok in general, but it uses inheritance from KJob which is not necessary for VFS and Qt. So it should be renamed to class that looks like job, behaves like a job, but not a "job" in API. We can't do that in KIO, so we have to reinvent wheel)

[Alexander Neundorf]

Post by Иван Комиссаров

Post by Thiago Macieira

Post by Иван Комиссаров
Also, KIO doesn't seems to be 'vfs' in my opinion - it just provides
asynchronous methods for common operations. I'm not sure, but i don't
think people will be glad to see al this "job" stuff.

Just rename the classes.
QNetworkReply is a "job". You post the job using QNetworkAccessManager.
We're not suggesting using KIO. The API was designed back in 2000 for KDE
2 and, while it works, I'd rather have a smaller, leaner API for Qt.
Some of that learning I applied in QNetworkReply, for example, which
works as a hybrid of KIO::TransferJob and KIO::StoredTransferJob.

Well, i meant people won't be happy to see Jobs as they are. API is ok in
general, but it uses inheritance from KJob which is not necessary for VFS
and Qt. So it should be renamed to class that looks like job, behaves like
a job, but not a "job" in API. We can't do that in KIO, so we have to
reinvent wheel)

No, it can be done.
The work on KDE frameworks is to a verry big part motivated by making the bits
and pieces more easily available to Qt-only applications.
Source-compatibility also should be kept as much as possible, but this can
also be achieved via KDE4-libs wrappers (if it's only name changes).

So, it's not an "impossible".

Alex

[Иван Комиссаров]

So, should i start implementing new VFS or someone will port KIO to use pure qt and include it in qt5?)

Post by Alexander Neundorf

Post by Иван Комиссаров

Post by Thiago Macieira

Post by Иван Комиссаров
Also, KIO doesn't seems to be 'vfs' in my opinion - it just provides
asynchronous methods for common operations. I'm not sure, but i don't
think people will be glad to see al this "job" stuff.

Just rename the classes.
QNetworkReply is a "job". You post the job using QNetworkAccessManager.
We're not suggesting using KIO. The API was designed back in 2000 for KDE
2 and, while it works, I'd rather have a smaller, leaner API for Qt.
Some of that learning I applied in QNetworkReply, for example, which
works as a hybrid of KIO::TransferJob and KIO::StoredTransferJob.

Well, i meant people won't be happy to see Jobs as they are. API is ok in
general, but it uses inheritance from KJob which is not necessary for VFS
and Qt. So it should be renamed to class that looks like job, behaves like
a job, but not a "job" in API. We can't do that in KIO, so we have to
reinvent wheel)

No, it can be done.
The work on KDE frameworks is to a verry big part motivated by making the bits
and pieces more easily available to Qt-only applications.
Source-compatibility also should be kept as much as possible, but this can
also be achieved via KDE4-libs wrappers (if it's only name changes).
So, it's not an "impossible".
Alex

[Alexander Neundorf]

Post by Иван Комиссаров
So, should i start implementing new VFS or someone will port KIO to use
pure qt and include it in qt5?)

Join the KDE frameworks effort and make KIO usable without dragging in all of
kdelibs (as it is now) :-)

David Faure and Stephen Kelly (both also on this list) can tell you more.

Alex

[Robin Burchell]

Hi,

Post by Иван Комиссаров
Second - if we provide new API, should it be synchronous (like QFile) or asynchronous (like Q*Socket)?

My personal thoughts would be to drop the synchronous like a brick,
because if it exists, people who don't know any better *will* (ab)use
it because it's "easier" or simply because examples/other people do
and they don't know what that really means, and then you end up with
poorly-performing applications. At least, that's the experience I've
had with watching people write applications using the Qt Mobility
Contacts API - it has an asynchronous API, but the synchronous one
gets used a lot without thought, until it turns out to be the cause of
a performance problem after some analysis.

Plus a great deal of the things you'd probably want VFS backends for
are going to be asynchronous, anyway.

BR,

Robin

[Oswald Buddenhagen]

Post by Robin Burchell

Post by Иван Комиссаров
Second - if we provide new API, should it be synchronous (like QFile) or asynchronous (like Q*Socket)?

My personal thoughts would be to drop the synchronous like a brick,
because if it exists, people who don't know any better *will* (ab)use
it because it's "easier" or simply because examples/other people do
and they don't know what that really means, and then you end up with
poorly-performing applications.

an incorrectly used async api (nested event loops, in particular) is a
source of much harder to debug problems, so it's not as clear-cut at it
may seem.

i wonder whether it would be possible to make a qthread subclass which
would be a generic sync wrapper for async apis, so one could dispose of
the waitFor*() in the i/o devices without requiring the users to write
lots of boilerplate code.

[Thiago Macieira]

Post by Oswald Buddenhagen
i wonder whether it would be possible to make a qthread subclass which
would be a generic sync wrapper for async apis, so one could dispose of
the waitFor*() in the i/o devices without requiring the users to write
lots of boilerplate code.

Making that class is easy.

Making the I/O classes work properly with being moved across threads (moving
their timers and socket notifiers) and deal with concurrent access to the I/O
buffers, that's not trivial at all.

[Oswald Buddenhagen]

Post by Thiago Macieira

Post by Oswald Buddenhagen
i wonder whether it would be possible to make a qthread subclass which
would be a generic sync wrapper for async apis, so one could dispose of
the waitFor*() in the i/o devices without requiring the users to write
lots of boilerplate code.

Making that class is easy.
Making the I/O classes work properly with being moved across threads (moving
their timers and socket notifiers) and deal with concurrent access to the I/O
buffers, that's not trivial at all.

that would be easy if we require that prior to wrapping only setters can
be used and setters must have no internal side effects. of course, that
means that opening the device in a class-specific way needs to be done
in the thread already. doing *that* generically may turn out, uhm,
challenging. it may be more appropriate to have a QFooWaiter for every
relevant QFoo.

[Thiago Macieira]

Post by Oswald Buddenhagen

Post by Thiago Macieira

Post by Oswald Buddenhagen
i wonder whether it would be possible to make a qthread subclass which
would be a generic sync wrapper for async apis, so one could dispose of
the waitFor*() in the i/o devices without requiring the users to write
lots of boilerplate code.

Making that class is easy.
Making the I/O classes work properly with being moved across threads
(moving their timers and socket notifiers) and deal with concurrent
access to the I/O buffers, that's not trivial at all.

that would be easy if we require that prior to wrapping only setters can
be used and setters must have no internal side effects. of course, that
means that opening the device in a class-specific way needs to be done
in the thread already. doing *that* generically may turn out, uhm,
challenging. it may be more appropriate to have a QFooWaiter for every
relevant QFoo.

You can't do that if there are shared resources. For example, QNetworkReply is
just a small window into the QNetworkAccessManager's resources. You cannot
move just the QNR. You need to move the entire QNAM along with all currently
active QNRs.

That severely limits the usefulness of moving to a thread. Even if you manage
to do that, the waitFor function might still trigger other signals emitting,
from other objects.

[Alexander Neundorf]

Post by Иван Комиссаров
As long as QFileEngines are marked as deprecated, Qt will have no
possibility to implement virtual filesystem.
So i would like to discuss features of new VFS.
I have 2 questions. First - should it work using QFile and QDir classes as
front-end or it should provide its own API like QVFSFile? Second - if we
provide new API, should it be synchronous (like QFile) or asynchronous
(like Q*Socket)?

Of course asynchronous would be better, because of delays, etc.

OTOH, we are happily using a synchronous QFileEngine over the network and have
no problems at all with it (ok, we know that usually the other side is on the
same local network and not really far away...).

Alex