Announcing libasync, a cross-platform D event loop

[Etienne via Digitalmars-d]

It's finally here: https://github.com/etcimon/libasync

We all know how event loops are the foundation of more popular libraries
Qt and Nodejs.. we now have a natively compiling async library entirely
written in D.

This event library was tested on Win32, Linux x64, Mac OS x64, with DMD
2.066, offers the more low-level async objects: timers, file i/o, dns
resolver, tcp, udp, listeners, signals (cross-thread), notifications
(same thread), and more recently (and with great efforts for
implementing with OS X / BSD) a directory watcher.

e.g. You can run a timer with:

import std.datetime; import std.stdio; import libasync.all;
EventLoop evl = new EventLoop;
auto timer = new AsyncTimer(evl);
timer.duration(2.seconds).periodic().run({ writeln("Another 2 seconds
have passed"); });
while(evl.loop()) continue;

The tests may be most revealing:
https://github.com/etcimon/libasync/blob/master/source/libasync/test.d

A (lightly tested) vibe.d driver using all those async objects is also
available and currently ongoing a pull request:

https://github.com/etcimon/vibe.d/tree/native-events

The incentive was to make vibe.d compile in completely native D, I'm now
moving onto a botan C++ => D wrapper for it, I plan on moving objects to
D over the years until the TLS library can be completely native. I thank
Walter for the efforts on extern(C++)

Finally, I release this on the basis of an MIT license, looking forward
to seeing our community flourishing with yet more native libraries. Code on

[Andrei Alexandrescu via Digitalmars-d]

Post by Etienne via Digitalmars-d
It's finally here: https://github.com/etcimon/libasync

This is fantastic! We really need something like this in a Facebook project.

Would be appropriate for you to consider making a bid for adding it to
Phobos? In that case one issue to address is integrating std.concurrency
with the event loop, i.e. registering an event handler should be
possible for a thread message as well. The Boost license would need to
be added. What do you think?


Andrei

[Etienne via Digitalmars-d]

Post by Andrei Alexandrescu via Digitalmars-d
This is fantastic! We really need something like this in a Facebook project.

That's pleasing to hear, although I'm pretty sure Facebook is far from
being the only organization who will benefit from this ;)

Post by Andrei Alexandrescu via Digitalmars-d
Would be appropriate for you to consider making a bid for adding it

to Phobos?

Thanks for the invitation, I'll start working on a phobos fork with this
added into std.async and re-licensed to Boost. Would that be an
appropriate namespace?

Post by Andrei Alexandrescu via Digitalmars-d
In that case one issue to address is integrating std.concurrency
with the event loop, i.e. registering an event handler should be
possible for a thread message as well.

Of course, the libasync.threads module already has a good foothold on
the strategy, so a newer/better std.concurrency could deprecate parts of
this module by offering an adapted message queue that allows threads
with an event loop to communicate with threads that have one or not, in
both directions.

Post by Andrei Alexandrescu via Digitalmars-d
Do I understand correctly that it is similar library to boost.asio

(EventLoop being equivalent of asio::io_service)?

That's right, but instead of io_service::run() you call
eventloop.loop(). In both cases, it's a wrapper around epoll_wait
(linux), kqueue (osx/bsd) or MsgWaitForMultipleObjects (windows) with a
timeout parameter. It waits for the objects supplied by the library. The
"strands" feature is pretty much covered by the vibe.d tasks if you're
using the driver I wrote for it.

[Andrei Alexandrescu via Digitalmars-d]

Post by Etienne via Digitalmars-d

Post by Andrei Alexandrescu via Digitalmars-d
This is fantastic! We really need something like this in a Facebook project.

That's pleasing to hear, although I'm pretty sure Facebook is far from
being the only organization who will benefit from this ;)

Post by Andrei Alexandrescu via Digitalmars-d
Would be appropriate for you to consider making a bid for adding it

to Phobos?
Thanks for the invitation, I'll start working on a phobos fork with this
added into std.async and re-licensed to Boost. Would that be an
appropriate namespace?

Yes. Thanks!

Post by Etienne via Digitalmars-d

Post by Andrei Alexandrescu via Digitalmars-d
In that case one issue to address is integrating std.concurrency
with the event loop, i.e. registering an event handler should be
possible for a thread message as well.

Of course, the libasync.threads module already has a good foothold on
the strategy, so a newer/better std.concurrency could deprecate parts of
this module by offering an adapted message queue that allows threads
with an event loop to communicate with threads that have one or not, in
both directions.

Yah, I think that's where the most design effort around porting would go.

Andrei

[Martin Nowak via Digitalmars-d]

Post by Etienne via Digitalmars-d

Post by Andrei Alexandrescu via Digitalmars-d
This is fantastic! We really need something like this in a Facebook project.

That's pleasing to hear, although I'm pretty sure Facebook is far from
being the only organization who will benefit from this ;)

Who doesn't need something like this?

Post by Etienne via Digitalmars-d

Post by Andrei Alexandrescu via Digitalmars-d
Would be appropriate for you to consider making a bid for adding it

to Phobos?
Thanks for the invitation, I'll start working on a phobos fork with this
added into std.async and re-licensed to Boost. Would that be an
appropriate namespace?

I still have to try out the library, but I'd like to see that.

One thing that always bothers me with async libraries is that now every
IO class has it async cousin, so there is Socket and AsyncSocket,
resolveDNS and asyncResolveDNS.
With Fibers and a Scheduler it's actually possible to present the same
API to asynchronous and synchronous code. I'd really like to see this at
some point, but a cross-platform event loop in phobos is a great first step.

[Etienne Cimon via Digitalmars-d]

Post by Martin Nowak via Digitalmars-d
One thing that always bothers me with async libraries is that now every
IO class has it async cousin, so there is Socket and AsyncSocket,
resolveDNS and asyncResolveDNS.
With Fibers and a Scheduler it's actually possible to present the same
API to asynchronous and synchronous code. I'd really like to see this at
some point, but a cross-platform event loop in phobos is a great first step.

Exactly, though I'm pretty sure this could be solved by moving vibe core
to std.vibe, and then adding it as a dependency for std.socket and
std.concurrency. It doesn't feel right having the future/promise aside
of fibers with a scheduler, it needs to be done correctly from the start.

[Szymon Gatner via Digitalmars-d]

Post by Etienne via Digitalmars-d
It's finally here: https://github.com/etcimon/libasync
We all know how event loops are the foundation of more popular
libraries Qt and Nodejs.. we now have a natively compiling
async library entirely written in D.
This event library was tested on Win32, Linux x64, Mac OS x64,
timers, file i/o, dns resolver, tcp, udp, listeners, signals
(cross-thread), notifications (same thread), and more recently
(and with great efforts for implementing with OS X / BSD) a
directory watcher.
import std.datetime; import std.stdio; import libasync.all;
EventLoop evl = new EventLoop;
auto timer = new AsyncTimer(evl);
timer.duration(2.seconds).periodic().run({ writeln("Another 2
seconds have passed"); });
while(evl.loop()) continue;
https://github.com/etcimon/libasync/blob/master/source/libasync/test.d
A (lightly tested) vibe.d driver using all those async objects
https://github.com/etcimon/vibe.d/tree/native-events
The incentive was to make vibe.d compile in completely native
D, I'm now moving onto a botan C++ => D wrapper for it, I plan
on moving objects to D over the years until the TLS library can
be completely native. I thank Walter for the efforts on
extern(C++)
Finally, I release this on the basis of an MIT license, looking
forward to seeing our community flourishing with yet more
native libraries. Code on

Do I understand correctly that it is similar library to
boost.asio (EventLoop being equivalent of asio::io_service)?

[via Digitalmars-d]

Post by Etienne via Digitalmars-d
It's finally here: https://github.com/etcimon/libasync
We all know how event loops are the foundation of more popular
libraries Qt and Nodejs.. we now have a natively compiling
async library entirely written in D.

I am very excited about this. Thank you for this awesome
contribution, you made my day :)

[Andrei Alexandrescu via Digitalmars-d]

Post by Etienne via Digitalmars-d
It's finally here: https://github.com/etcimon/libasync

http://www.reddit.com/r/programming/comments/2hcm9n/announcing_libasync_a_crossplatform_d_event_loop/

Andrei

[Jacob Carlborg via Digitalmars-d]

Post by Etienne via Digitalmars-d
It's finally here: https://github.com/etcimon/libasync
We all know how event loops are the foundation of more popular libraries
Qt and Nodejs.. we now have a natively compiling async library entirely
written in D.
This event library was tested on Win32, Linux x64, Mac OS x64, with DMD
2.066, offers the more low-level async objects: timers, file i/o, dns
resolver, tcp, udp, listeners, signals (cross-thread), notifications
(same thread), and more recently (and with great efforts for
implementing with OS X / BSD) a directory watcher.

Shouldn't the directory watcher use FSEvents on OS X?

[Etienne via Digitalmars-d]

Post by Jacob Carlborg via Digitalmars-d
Shouldn't the directory watcher use FSEvents on OS X?

I've thought about it but it isn't compatible with other BSD platforms
and has no docs about using it with kqueue, it ended up looking more
complicated and unstable because I could read complaints everywhere I
looked. It ended up putting me in a direction where I needed separate
threads with their own event loops and communication through signals,
which didn't fit in with what I was used to doing from what the other
platforms offered.

I found the solution with kqueue's vnode, which acts like inotify. It
doesn't have recursion nor file modification events when watching
folders so individual files had to be watched and the directories
checked against cache data every time an event came in. It seems a lot
more flexible on the long run, and making feature additions /
compatibility adjustments didn't sound like a nightmare on the long run.

[Jacob Carlborg via Digitalmars-d]

Post by Etienne via Digitalmars-d
I've thought about it but it isn't compatible with other BSD platforms
and has no docs about using it with kqueue, it ended up looking more
complicated and unstable because I could read complaints everywhere I
looked. It ended up putting me in a direction where I needed separate
threads with their own event loops and communication through signals,
which didn't fit in with what I was used to doing from what the other
platforms offered.

The only complains I've seen with FSEvents is that it doesn't support
notifying about file changes, only directory changes. But that has been
fixed in OS X 10.7 so that complain is moot.

Post by Etienne via Digitalmars-d
I found the solution with kqueue's vnode, which acts like inotify. It
doesn't have recursion nor file modification events when watching
folders so individual files had to be watched and the directories
checked against cache data every time an event came in.

That sounds like a quite big limitation.

[Etienne via Digitalmars-d]

Post by Jacob Carlborg via Digitalmars-d
The only complains I've seen with FSEvents is that it doesn't support
notifying about file changes, only directory changes. But that has been
fixed in OS X 10.7 so that complain is moot.

Good, and according to the stats I've seen, 95% of users are on the 2
most recent OS X versions!

Post by Jacob Carlborg via Digitalmars-d

Post by Etienne via Digitalmars-d
I found the solution with kqueue's vnode, which acts like inotify. It
doesn't have recursion nor file modification events when watching
folders so individual files had to be watched and the directories
checked against cache data every time an event came in.

That sounds like a quite big limitation.

It is, but I managed to get around it. Although I haven't yet
implemented the renaming/move operation detection. I might as well plan
on writing the FSEvent implementation for Mac OS X and state some
limitations with FreeBSD in that case =) Thanks for the hint!

[ketmar via Digitalmars-d]

On Wed, 24 Sep 2014 09:13:31 -0400

Post by Etienne via Digitalmars-d
It's finally here: https://github.com/etcimon/libasync
We all know how event loops are the foundation of more popular
libraries Qt and Nodejs.. we now have a natively compiling async
library entirely written in D.

it's great! and it will be even greater ;-) to have this in phobos.

thank you for your work.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20140925/aadf58d7/attachment.sig>

[Zhao Puming via Digitalmars-d]

Great work Etienne!

will libasync make it into phobos?

Post by Etienne via Digitalmars-d
It's finally here: https://github.com/etcimon/libasync
We all know how event loops are the foundation of more popular
libraries Qt and Nodejs.. we now have a natively compiling
async library entirely written in D.
This event library was tested on Win32, Linux x64, Mac OS x64,
timers, file i/o, dns resolver, tcp, udp, listeners, signals
(cross-thread), notifications (same thread), and more recently
(and with great efforts for implementing with OS X / BSD) a
directory watcher.
import std.datetime; import std.stdio; import libasync.all;
EventLoop evl = new EventLoop;
auto timer = new AsyncTimer(evl);
timer.duration(2.seconds).periodic().run({ writeln("Another 2
seconds have passed"); });
while(evl.loop()) continue;
https://github.com/etcimon/libasync/blob/master/source/libasync/test.d
A (lightly tested) vibe.d driver using all those async objects
https://github.com/etcimon/vibe.d/tree/native-events
The incentive was to make vibe.d compile in completely native
D, I'm now moving onto a botan C++ => D wrapper for it, I plan
on moving objects to D over the years until the TLS library can
be completely native. I thank Walter for the efforts on
extern(C++)
Finally, I release this on the basis of an MIT license, looking
forward to seeing our community flourishing with yet more
native libraries. Code on

[Sean Kelly via Digitalmars-d]

Post by Zhao Puming via Digitalmars-d
Great work Etienne!
will libasync make it into phobos?

I certainly hope so. We need async functionality if we're to
ever have a decent socket package in Phobos.

[H. S. Teoh via Digitalmars-d]

Post by Zhao Puming via Digitalmars-d
Great work Etienne!
will libasync make it into phobos?

I certainly hope so. We need async functionality if we're to ever
have a decent socket package in Phobos.

Adam has also written an event loop (arsd.eventloop), which also has
timers and a signals-and-slots subsystem:

https://github.com/adamdruppe/arsd/blob/master/eventloop.d

I'm not sure if it supports multithreading, but its event registration
API is very cool. I think we should pick the best of both projects for
inclusion in Phobos.


T

[Andrej Mitrovic via Digitalmars-d]

Post by Etienne via Digitalmars-d
It's finally here: https://github.com/etcimon/libasync

Small nitpick with the spelling: asynchroneous => asynchronous. I
personally don't care about spelling but many people tend to
(unfortunately) look the other way when they find typos.

Btw, have you had a look at http://wiki.dlang.org/Event_system ? I'm
just wondering how much of that idea page / proposal is covered.

Anyway, this is a solid initiative. I look forward to seeing it in Phobos!

[Etienne via Digitalmars-d]

Post by Andrej Mitrovic via Digitalmars-d
Small nitpick with the spelling: asynchroneous => asynchronous. I
personally don't care about spelling but many people tend to
(unfortunately) look the other way when they find typos.

Hm, my french messed with my mind on that one :-P we say asynchrone or
asynchroné

Post by Andrej Mitrovic via Digitalmars-d
Btw, have you had a look at http://wiki.dlang.org/Event_system ? I'm
just wondering how much of that idea page / proposal is covered.
Anyway, this is a solid initiative. I look forward to seeing it in Phobos!

I originally developed it with vibe.d's usage in mind which implicitely
forced me to cover most of this list, but currently it's lacking
AsyncObject which would allow to monitor file descriptors and cover a
good chunk of this list. I'm sure it will be a very useful resource for
upcoming decisions.

[Adam Wilson via Digitalmars-d]

Post by Etienne via Digitalmars-d
It's finally here: https://github.com/etcimon/libasync
We all know how event loops are the foundation of more popular libraries
Qt and Nodejs.. we now have a natively compiling async library entirely
written in D.
This event library was tested on Win32, Linux x64, Mac OS x64, with DMD
2.066, offers the more low-level async objects: timers, file i/o, dns
resolver, tcp, udp, listeners, signals (cross-thread), notifications
(same thread), and more recently (and with great efforts for
implementing with OS X / BSD) a directory watcher.
import std.datetime; import std.stdio; import libasync.all;
EventLoop evl = new EventLoop;
auto timer = new AsyncTimer(evl);
timer.duration(2.seconds).periodic().run({ writeln("Another 2 seconds
have passed"); });
while(evl.loop()) continue;
https://github.com/etcimon/libasync/blob/master/source/libasync/test.d
A (lightly tested) vibe.d driver using all those async objects is also
https://github.com/etcimon/vibe.d/tree/native-events
The incentive was to make vibe.d compile in completely native D, I'm now
moving onto a botan C++ => D wrapper for it, I plan on moving objects to
D over the years until the TLS library can be completely native. I thank
Walter for the efforts on extern(C++)
Finally, I release this on the basis of an MIT license, looking forward
to seeing our community flourishing with yet more native libraries. Code on

You mentioned Botan. I already have a C++ => D Wrapper project going over
here: https://github.com/ellipticbit/titanium

I am working out a bug where the memory corrupts itself when passing data
back to D but it works and most of the leg-work is done. And I am
definitely open to pull-requests.

[Etienne via Digitalmars-d]

Post by Adam Wilson via Digitalmars-d
You mentioned Botan. I already have a C++ => D Wrapper project going
over here: https://github.com/ellipticbit/titanium
I am working out a bug where the memory corrupts itself when passing
data back to D but it works and most of the leg-work is done. And I am
definitely open to pull-requests.

Your wrapper will probably have to wrap a new Botan. I decided to
translate everything to D because I need the complete interface.

https://github.com/etcimon/botan

[Joakim via Digitalmars-d]

Post by Etienne via Digitalmars-d

Post by Adam Wilson via Digitalmars-d
You mentioned Botan. I already have a C++ => D Wrapper project going
over here: https://github.com/ellipticbit/titanium
I am working out a bug where the memory corrupts itself when
passing
data back to D but it works and most of the leg-work is done.
And I am
definitely open to pull-requests.

Your wrapper will probably have to wrap a new Botan. I decided
to translate everything to D because I need the complete
interface.
https://github.com/etcimon/botan

How long do you think that's going to take? What do you plan to
do about ongoing C++ patches added to the original C++ botan
version? Maybe developing something like Daniel Murphy's DDMD
magicport for botan would save you some time from doing it all
manually.

[Etienne via Digitalmars-d]

How long do you think that's going to take? What do you plan to do
about ongoing C++ patches added to the original C++ botan version?
Maybe developing something like Daniel Murphy's DDMD magicport for botan
would save you some time from doing it all manually.

I see it done in a week before the testing, search and replace does the
trick for most of it, the longest part is merging the .h and .cpp files.
The patches don't seem very frequent (last update was in april, mostly
minor), but I can also apply those manually if necessary. However, I
intend on branching off completely and maintaining it myself with new
algorithms, a certificate factory and a better BER/DER serialization
engine (I have an ASN1 library in the works as well).

[Etienne via Digitalmars-d]

Post by Etienne via Digitalmars-d
engine (I have an ASN1 library in the works as well).

It's nearly finished, it will allow BER/DER serialization to take place
from UDAs and native types at compile-time:

https://github.com/globecsys/asn1.d

[Dmitry Olshansky via Digitalmars-d]

Post by Etienne via Digitalmars-d
It's finally here: https://github.com/etcimon/libasync
We all know how event loops are the foundation of more popular libraries
Qt and Nodejs.. we now have a natively compiling async library entirely
written in D.
This event library was tested on Win32, Linux x64, Mac OS x64, with DMD
2.066, offers the more low-level async objects: timers, file i/o, dns
resolver, tcp, udp, listeners, signals (cross-thread), notifications
(same thread), and more recently (and with great efforts for
implementing with OS X / BSD) a directory watcher.

Awesome! Bookmarked for future ;)