Discussion:
Is D right for me?
Gour D.
2010-10-05 07:18:59 UTC
Permalink
Hello!

I'd like some help to resolve the subject's question...

Let me start with my background...Finished Computer Engineering
studies and during that period was introduced to several programming
languages (including some assembly stuff) starting with Fortran & C, a
little Prolog and did my thesis by programming simulator for coloured
Petri nets using Zortech C++ - that was in 1990 :-)

I also playing with Smalltalk a bit...I'm saying as small emphasis
that D would not be my 1st programming language.

Later, my life went away from programming into totally different area
and today programming is not my occupation, but should serve my hobby
projects.

After my 'programming come-back', I explored Ruby a bit, checked Ocaml
shortly (did not like syntax) and wanting something 'new', I ended up
tinkering with Haskell.

For my web purposes I wanted to use Django, but realized that not
being web developer, there is no sense to write so many code, when I
can achieve the same by learning some PHP (which I do these days) in
order to tweak and/or write some missing module for my preferred CMS
(Made Simple).

Similarly, to create invoices for our startup 'company', I abandoned
GNUCash which requires tinkering with Scheme (Guile) and decided to
use SimpleInvoices (PHP & MySQL web app).

The keyword here is 'pragmatism', iow. understanding that one needs to
make some compromises in order to "get job done".

Now, we're back at D...Saw a Reddit thread yesterday which inspired
me to think (once again) about Haskell vs D...

So, we want a general programming language to work on our open-source
(we plan GPL) hobby project which is desktop GUI application and
besides the need to develop several libs for it, it needs to use C-lib
(Swiss Ephemeris).

Are there other alternatives?

Well, I do not like Java, VMs (Scala included)., want something
'modern' to avoid manual memory management, pointers etc.,
higher-level...which eliminates C(++).

Scripting languages (Perl, Python) are too slow and I'm aware of some
projects from the same domain which switched from Python to C++.

I'm not interested in LISP-family ala Clojure, neither inspired by
C#,Go...

We want to develop on Linux (running x86_64 I7 cpu) and have app
working on Mac and possibly Windoze.

For a long time I was thinking about gtk2hs bindings, but since Mac
platform became important for us (supervisor of the project recently
switched to it), I abandoned GTK. I was even advised by one dev
working on GTK Mac port that wx(haskell) is better solution if we
target Mac.

However, we would like to write kind of 'desktop-lite' app here idea
to use Qt & Meego was born, since there is no wxQT port.

I was sorry to discover yesterday that QtD project is suspended. :-(

So, let's recap in regards to Haskell vs D.

a) I like Haskell syntax, its type-system, purity and the concept of
separating pure code from non-pure (e.g. IO), HOF. Community is very
friendly and growing (1st time when i visited #haskell it was <100
users, today probably >600), there are lot of packages available on
Hackage, GHC is keeping strong, Cabal build system is nice,
QuickCheck...

Otoh, many libs/packages are not adequately documented, there is joke
that one needs PhD to use the language, lot of papers but with strong
influence from academia and one has to encounter lot of terminology
from category theory etc. although maybe wanting to 'just get the job
done' - iow, Haskell could become more pragmatic. Moreover, to get
better performance, laziness with its non-determinism might be a
problem and/or profiling to discover leaks is not straightforward
and/or code becomes more ugly. :-)

D, from the other side, is younger language, community is not so
big, language is, afaict, evolving very rapidly and it's not easy to
tell which compiler to use, which libs etc.

Moreover, I'm a bit worried on the state of GUI libs available for D,
especially about QtD.

Moreover, 64-bit is not ready (yet), although I'm told it should come
soon. What about ARM if we want to target MeeGo in the future?

I also did not research what is the state of database support...Now
we're thinking to use sqlite3 as back-end.

Our project will be developed in free time and we want language which
is easy to maintain because the project (with all desired features)
might evolve into a big one during the period of several years.

I also have experience that some potential developers did not join our
team since Haskell was to hard to grok for them (coming from C++), so
D might be an easier path with less steep learning curve, but I also
wonder about myself whether I could pick D quickly enough (I'll buy
book, of course) after long exposure to Haskell and FP.

I read The Case for D article and saw Andrei's Google talk - it was
funny to see Google people being like little children when questioned
by him :-)

So, can you offer some advice, what could be better choice between
Haskell & D for our planned project with the following features:

a) maintainable code

b) decent performance

c) higher-level programming and suitable for general programming tasks

d) good library support (database stuff, data structures, Qt GUI...)

e) vibrant community and active development so that there is some
guarantee that the language won't fall in oblivion if some devs leave
the project, iow. 'bus-factor > 2' ?

(It would be nice if someone familiar with both languages can share...)


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101005/70cb1301/attachment-0001.pgp>
Simen kjaeraas
2010-10-05 10:52:22 UTC
Permalink
Post by Gour D.
D, from the other side, is younger language, community is not so
big, language is, afaict, evolving very rapidly and it's not easy to
tell which compiler to use, which libs etc.
This is a big problem for D at this point. The language is no longer
evolving (much), and we're at a point in time where libraries and
toolchain parts need to be written.
Post by Gour D.
Moreover, I'm a bit worried on the state of GUI libs available for D,
especially about QtD.
Moreover, 64-bit is not ready (yet), although I'm told it should come
soon.
It will. Latest news (2 days ago) say it's now getting as far as
main(), which is good.
Post by Gour D.
What about ARM if we want to target MeeGo in the future?
I believe GDC supports ARM.
Post by Gour D.
I also did not research what is the state of database support...Now
we're thinking to use sqlite3 as back-end.
There's a list here:
http://www.wikiservice.at/d/wiki.cgi?DatabaseBindings

However, most of those are for D1, and a large percentage seem to be
abandoned.

SQLite seems to be well supported, with 7 projects claiming support.
Post by Gour D.
I also have experience that some potential developers did not join our
team since Haskell was to hard to grok for them (coming from C++), so
D might be an easier path with less steep learning curve, but I also
wonder about myself whether I could pick D quickly enough (I'll buy
book, of course) after long exposure to Haskell and FP.
I'm sure you can. D also supports programming styles closer to those
of FP, making such a transition easier (I hope :p)
Post by Gour D.
So, can you offer some advice, what could be better choice between
a) maintainable code
This is likely a bit subjective, and much more dependent upon the
programmers themselves than the language used.

That said, D supports a variety of features that boost
maintainability:

- Contract programming in the form of pre and post contracts for
functions[1].
- Class invariants[2].
- Built in unit testing[3].
- Documentation comments[4].

Of course, other features of D may increase maintainability, but
those are the ones most directly associated with it.
Post by Gour D.
b) decent performance
D is generally as fast as C, though some abstractions of course cost
more than others.
Post by Gour D.
c) higher-level programming and suitable for general programming tasks
My impression (not having used Haskell), D wins hands down on the
latter, and is a bit weaker on the former.
Post by Gour D.
d) good library support (database stuff, data structures, Qt GUI...)
Likely Haskell is better here (as noted above, D has some problems in
this regard).
Post by Gour D.
e) vibrant community and active development so that there is some
guarantee that the language won't fall in oblivion if some devs leave
the project, iow. 'bus-factor > 2' ?
The bus-factor of D is sadly close to 1. If Walter should choose to
leave, we have a problem. On the other hand, I don't think a mere bus
would keep him from continuing the project.
Post by Gour D.
(It would be nice if someone familiar with both languages can share...)
Here I can't help. I don't know Haskell.

In closing,

[1]: http://digitalmars.com/d/2.0/dbc.html
[2]: http://digitalmars.com/d/2.0/class.html#invariants
[3]: http://digitalmars.com/d/2.0/unittest.html
[4]: http://digitalmars.com/d/2.0/ddoc.html
--
Simen
Gour D.
2010-10-05 12:50:44 UTC
Permalink
On Tue, 05 Oct 2010 12:52:22 +0200
Simen> This is a big problem for D at this point. The language is no
Simen> longer evolving (much), and we're at a point in time where
Simen> libraries and toolchain parts need to be written.

That's nice to hear and it's solvable.

Simen> It will. Latest news (2 days ago) say it's now getting as far as
Simen> main(), which is good.

Great!

Simen> I believe GDC supports ARM.

Hmm, baes on http://dgcc.sourceforge.net/ it looks it is not overly
active?

Simen> There's a list here:
Simen> http://www.wikiservice.at/d/wiki.cgi?DatabaseBindings
Simen>
Simen> However, most of those are for D1, and a large percentage seem
Simen> to be abandoned.

:-(

Simen> SQLite seems to be well supported, with 7 projects claiming
Simen> support.

Why so many?

Similar to Haskell where one can find bunch of libs doing practically
the same thing, but most of them half-baked.

Simen> I'm sure you can. D also supports programming styles closer to
Simen> those of FP, making such a transition easier (I hope :p)

This is certainly bonus.

Simen> > a) maintainable code
Simen>
Simen> This is likely a bit subjective, and much more dependent upon the
Simen> programmers themselves than the language used.

I agree. Otoh, afaict, D use modules/packages, so code can be nicely
organized, as well as in Haskell.


Simen> - Contract programming in the form of pre and post contracts for
Simen> functions[1].
Simen> - Class invariants[2].
Simen> - Built in unit testing[3].
Simen> - Documentation comments[4].
Simen>
Simen> Of course, other features of D may increase maintainability, but
Simen> those are the ones most directly associated with it.

Not bad.;)

Simen> > b) decent performance
Simen>
Simen> D is generally as fast as C, though some abstractions of course
Simen> cost more than others.

This is, probably, more than we'd need, but definitely no fear as with
e.g. Python & co.

Simen> > c) higher-level programming and suitable for general
Simen> > programming tasks
Simen>
Simen> My impression (not having used Haskell), D wins hands down on the
Simen> latter, and is a bit weaker on the former.

Still, I believe, D provides much more comfortable higher-order
experience than C++.

Simen> > d) good library support (database stuff, data structures, Qt
Simen> > GUI...)
Simen>
Simen> Likely Haskell is better here (as noted above, D has some
Simen> problems in this regard).

Lack of GUI libs for D2 is serious concern atm.

Simen> The bus-factor of D is sadly close to 1. If Walter should choose
Simen> to leave, we have a problem. On the other hand, I don't think a
Simen> mere bus would keep him from continuing the project.

Uhh...this is almost like a showstopper or, at least, very strong
anti-adoption pattern. :-(

It is even worse than Haskell where GHC has bus-factor >=2 and there
are other compilers like uhc, lhc, jhc...there is even Haskell
committee working on Haskell' (prime) standard.

Simen> Here I can't help. I don't know Haskell.

Thanks a lot. It is helpful, although with a little discouraging
end. :-(


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101005/be4c60ff/attachment.pgp>
Daniel Gibson
2010-10-05 13:39:30 UTC
Permalink
Post by Gour D.
On Tue, 05 Oct 2010 12:52:22 +0200
Simen> I believe GDC supports ARM.
Hmm, baes on http://dgcc.sourceforge.net/ it looks it is not overly
active?
try http://bitbucket.org/goshawk/gdc/wiki/Home :-)
Post by Gour D.
Simen> The bus-factor of D is sadly close to 1. If Walter should choose
Simen> to leave, we have a problem. On the other hand, I don't think a
Simen> mere bus would keep him from continuing the project.
Uhh...this is almost like a showstopper or, at least, very strong
anti-adoption pattern. :-(
I don't think it's as serious, because afaik Walter is not the only one
developing the dmd compiler (and thus familiar with it) and, more
importantly, there are alternative D compilers (gdc and ldc, with at
least gdc being actively developed).

So even if Walter, for whatever reason, stops developing D, there is -
IMHO - a good chance that others will continue his efforts and keep D alive.

Cheers,
- Daniel
Amber
2010-10-05 13:52:12 UTC
Permalink
Post by Daniel Gibson
Post by Gour D.
On Tue, 05 Oct 2010 12:52:22 +0200
Simen> I believe GDC supports ARM.
Hmm, baes on http://dgcc.sourceforge.net/ it looks it is not overly
active?
try http://bitbucket.org/goshawk/gdc/wiki/Home :-)
Post by Gour D.
Simen> The bus-factor of D is sadly close to 1. If Walter should choose
Simen> to leave, we have a problem. On the other hand, I don't think a
Simen> mere bus would keep him from continuing the project.
Uhh...this is almost like a showstopper or, at least, very strong
anti-adoption pattern. :-(
I don't think it's as serious, because afaik Walter is not the only one
developing the dmd compiler (and thus familiar with it) and, more
importantly, there are alternative D compilers (gdc and ldc, with at
least gdc being actively developed).
So even if Walter, for whatever reason, stops developing D, there is -
IMHO - a good chance that others will continue his efforts and keep D alive.
Cheers,
- Daniel
I've also heard there is an unannounced compiler in the works.
Gour D.
2010-10-05 13:52:29 UTC
Permalink
On Tue, 05 Oct 2010 15:39:30 +0200
Daniel> try http://bitbucket.org/goshawk/gdc/wiki/Home :-)

Ahh, this looks much better. Thanks. ;)

Daniel> I don't think it's as serious, because afaik Walter is not the
Daniel> only one developing the dmd compiler (and thus familiar with
Daniel> it) and, more importantly, there are alternative D compilers
Daniel> (gdc and ldc, with at least gdc being actively developed).

So, both gdc & ldc are open-source?

What about standard libs?

Daniel> So even if Walter, for whatever reason, stops developing D,
Daniel> there is - IMHO - a good chance that others will continue his
Daniel> efforts and keep D alive.

This is not so disheartening. :-)

btw, I've asked similar/same question on SO
http://stackoverflow.com/questions/3863111/haskell-or-d-for-gui-desktop-application

if you want to contribute (I'm not much advised for D, so far). ;)


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101005/89da4062/attachment.pgp>
Daniel Gibson
2010-10-05 14:00:48 UTC
Permalink
Post by Gour D.
On Tue, 05 Oct 2010 15:39:30 +0200
Daniel> try http://bitbucket.org/goshawk/gdc/wiki/Home :-)
Ahh, this looks much better. Thanks. ;)
Daniel> I don't think it's as serious, because afaik Walter is not the
Daniel> only one developing the dmd compiler (and thus familiar with
Daniel> it) and, more importantly, there are alternative D compilers
Daniel> (gdc and ldc, with at least gdc being actively developed).
So, both gdc & ldc are open-source?
Yes.
Post by Gour D.
What about standard libs?
They're open source, too (boost license for phobos, is said to be even
more liberal than BSD license, there is an alternative standard lib for
D1 - tango[1] - that uses a BSD license and the "Academic Free License
v3.0"[2]).
Also, there are already several people maintaining/developing Phobos
(not just Walter and Andrei Alexandrescu).

Cheers,
- Daniel

[1] http://www.dsource.org/projects/tango/
[2] http://www.dsource.org/projects/tango/wiki/LibraryLicense
Gour D.
2010-10-05 14:34:42 UTC
Permalink
On Tue, 05 Oct 2010 16:00:48 +0200
Daniel> > So, both gdc & ldc are open-source?
Daniel>
Daniel> Yes.

Nice. It means that, in the future, we could target ARM as well (for
MeeGo).


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101005/bd6c79ad/attachment.pgp>
Don
2010-10-05 14:01:41 UTC
Permalink
Post by Daniel Gibson
Post by Gour D.
On Tue, 05 Oct 2010 12:52:22 +0200
Simen> I believe GDC supports ARM.
Hmm, baes on http://dgcc.sourceforge.net/ it looks it is not overly
active?
try http://bitbucket.org/goshawk/gdc/wiki/Home :-)
Post by Gour D.
Simen> The bus-factor of D is sadly close to 1. If Walter should choose
Simen> to leave, we have a problem. On the other hand, I don't think a
Simen> mere bus would keep him from continuing the project.
Uhh...this is almost like a showstopper or, at least, very strong
anti-adoption pattern. :-(
I don't think it's as serious, because afaik Walter is not the only one
developing the dmd compiler (and thus familiar with it) and, more
importantly, there are alternative D compilers (gdc and ldc, with at
least gdc being actively developed).
So even if Walter, for whatever reason, stops developing D, there is -
IMHO - a good chance that others will continue his efforts and keep D alive.
Look at the changelog for the last three releases. During that time,
Walter has worked almost exclusively on the backend for the 64-bit
compiler. If there were no community involvement, there would have been
almost no progress on the 32-bit compiler. Yet the rate of compiler bug
fixing has not fallen.

I would estimate the truck factor as between 2.0 and 2.5. Two years ago,
the truck factor was 1.0, but not any more.
Gour D.
2010-10-05 14:37:09 UTC
Permalink
On Tue, 05 Oct 2010 16:01:41 +0200
Don> I would estimate the truck factor as between 2.0 and 2.5. Two
Don> years ago, the truck factor was 1.0, but not any more.

Nice, nice...Still SO people say: "Neither Haskell nor D is popular
enough for it to be at all likely that you will ever attract a single
other developer to your project..." :-)

If just QtD hadn't been suspended...


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101005/337f7c11/attachment.pgp>
Andrei Alexandrescu
2010-10-05 16:29:03 UTC
Permalink
Post by Gour D.
On Tue, 05 Oct 2010 16:01:41 +0200
Don> I would estimate the truck factor as between 2.0 and 2.5. Two
Don> years ago, the truck factor was 1.0, but not any more.
Nice, nice...Still SO people say: "Neither Haskell nor D is popular
enough for it to be at all likely that you will ever attract a single
other developer to your project..." :-)
If developer attraction is a concern, you're likely better off with D.
Programmers who have used at least one Algol-like language (C, C++,
Java, C#) will have no problem feeling comfortable in D. With Haskell
you'd need to stick with "the choir".
Post by Gour D.
If just QtD hadn't been suspended...
I agree that's a bummer. I suggest you write the developers and ask what
would revive their interest. The perspective of a solid client is bound
to be noticeable.


Andrei
Gour D.
2010-10-05 18:52:27 UTC
Permalink
On Tue, 05 Oct 2010 11:29:03 -0500
Andrei> If developer attraction is a concern, you're likely better off
Andrei> with D. Programmers who have used at least one Algol-like
Andrei> language (C, C++, Java, C#) will have no problem feeling
Andrei> comfortable in D. With Haskell you'd need to stick with "the
Andrei> choir".

Yeah, I'm aware of it...

btw, let me personally congatulate for a great talk at Google!

Andrei> I agree that's a bummer. I suggest you write the developers and
Andrei> ask what would revive their interest. The perspective of a
Andrei> solid client is bound to be noticeable.

You think that D beginner with a open-source project is "solid client"?

Otoh, there is nothing to lose...


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101005/cab9e7fd/attachment.pgp>
Walter Bright
2010-10-05 23:06:00 UTC
Permalink
Post by Gour D.
Andrei> I agree that's a bummer. I suggest you write the developers and
Andrei> ask what would revive their interest. The perspective of a
Andrei> solid client is bound to be noticeable.
You think that D beginner with a open-source project is "solid client"?
Otoh, there is nothing to lose...
Few things work better than customers letting a company know they are interested
in such-and-such a product.
Gour D.
2010-10-06 06:23:19 UTC
Permalink
On Tue, 05 Oct 2010 16:06:00 -0700
Walter> Few things work better than customers letting a company know
Walter> they are interested in such-and-such a product.

Even a non-paying customer in the open-source world?

Well, I'm going to send email to two people considering them important
for QtD, based on what I could deduce...


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101006/63ee561f/attachment-0001.pgp>
Walter Bright
2010-10-09 19:44:37 UTC
Permalink
Post by Gour D.
Walter> Few things work better than customers letting a company know
Walter> they are interested in such-and-such a product.
Even a non-paying customer in the open-source world?
At least it shows interest. No emails tells the open source developer "nobody
cares, so I'll just abandon it".
Post by Gour D.
Well, I'm going to send email to two people considering them important
for QtD, based on what I could deduce...
Good!
Jonathan M Davis
2010-10-10 07:46:47 UTC
Permalink
Post by Walter Bright
Post by Gour D.
Walter> Few things work better than customers letting a company know
Walter> they are interested in such-and-such a product.
Even a non-paying customer in the open-source world?
At least it shows interest. No emails tells the open source developer
"nobody cares, so I'll just abandon it".
Yes, a lack of positive feedback can be frustrating even if you have the best
code ever. And as much as the developers of QtD likely want to use it for their
own stuff, it's likely not worth doing it just for themselves. It's just too much
work.

Of course, projects like QtD suffer from the same sort of problem as a compiler
does in that it's not necessarily very useful until it's complete. Lots of
people may be interested in using QtD, but if it's not at least close to done,
it's not going to be useable enough to use in any major project, so people won't
use, they won't report bugs on it, and the won't give any kind of feedback on
the project. So, the poor QtD people then have to get a _lot_ of code done
before they see any kind of positive feedback from the community, and when they
_do_ start getting feedback, much of it is likely to be negative because feature
X hasn't been implemented yet or feature Y is buggy. A lot of people have given
up on D for similar reasons. Hopefully enough of the problems that they were
having with dmd get fixed soon enough that they're able to actually continue
working on the project without getting too frustrated over it. QtD is a huge
service to the D community.

- Jonathan M Davis
Gour D.
2010-10-10 08:09:33 UTC
Permalink
On Sun, 10 Oct 2010 00:46:47 -0700
Jonathan> Yes, a lack of positive feedback can be frustrating even if
Jonathan> you have the best code ever. And as much as the developers of
Jonathan> QtD likely want to use it for their own stuff, it's likely
Jonathan> not worth doing it just for themselves. It's just too much
Jonathan> work.

Well, I tried to do my little 'homework'...wrote to the QtD devs
explaining them that their project is essential to adopting D for our
project.

Moreover, informed them that only for the sake of trying QtD I've
created 32bit chroot on my machine(64bit dmd, do you hear me?) and got
some help on #qtd in order to build "hello world" (instructions at
http://www.dsource.org/projects/qtd/wiki/BuildLinux are now
up-to-date).

Lastly, I told devs that despite of current status of QtD, we have
decided to 'gamble' and will use D/QtD for our project.

The next step is to order Andrei's book, start learning the language,
experiment with non-GUI stuff and try to help (in any way) to push QtD
further.

Jonathan> QtD is a huge service to the D community.

Indeed!

Coming from Haskell community where all the GUI libs (bindings) are in
hands of just few devs, I sincerely hope that users of D will
recognize importance of QtD for the success of language itself and
help the project to become complete and fully usable asap.


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101010/0ac0bc91/attachment.pgp>
Walter Bright
2010-10-10 20:06:21 UTC
Permalink
Post by Gour D.
(64bit dmd, do you hear me?)
Don't I know it. 64 bit support is absolutely essential for D's future.
bioinfornatics
2010-10-10 23:50:31 UTC
Permalink
LDC support 64 bit ;)
Daniel Gibson
2010-10-11 00:27:55 UTC
Permalink
Post by bioinfornatics
LDC support 64 bit ;)
as well as GDC.

But both currently lack an up-to-date D2 compiler (but the GDC guys are
at least working on it, seems like they're currently at 2.029 - which is
great - about 3 months ago they were still at 2.018 and in between was
the big 2.020 update that introduced druntime).

I agree with walter that 64bit support in DMD is very important,
especially for D2: I started a project a few months ago that might have
benefited from D2s features (especially ranges), but I decided to use D1
because no 64bit compiler for D2 was in sight.

I am btw a bit worried about "upgrading" code from D1 to D2 some day
because of the heavy (still ongoing) changes, especially in phobos.
I read that for example std.stream is going to be deprecated and
replaced by either something with ranges or something like std.stdio
(whoever was right about that, maybe even both?). This means that all my
(file/network) IO-Code would have to be rewritten sooner or later..
But, currently lacking an alternative to std.stream/std.socketstream for
networking, I would have had the same problems even if I had started the
project with D2...
Jonathan M Davis
2010-10-11 01:59:51 UTC
Permalink
Post by Daniel Gibson
Post by bioinfornatics
LDC support 64 bit ;)
as well as GDC.
But both currently lack an up-to-date D2 compiler (but the GDC guys are
at least working on it, seems like they're currently at 2.029 - which is
great - about 3 months ago they were still at 2.018 and in between was
the big 2.020 update that introduced druntime).
I agree with walter that 64bit support in DMD is very important,
especially for D2: I started a project a few months ago that might have
benefited from D2s features (especially ranges), but I decided to use D1
because no 64bit compiler for D2 was in sight.
I am btw a bit worried about "upgrading" code from D1 to D2 some day
because of the heavy (still ongoing) changes, especially in phobos.
I read that for example std.stream is going to be deprecated and
replaced by either something with ranges or something like std.stdio
(whoever was right about that, maybe even both?). This means that all my
(file/network) IO-Code would have to be rewritten sooner or later..
But, currently lacking an alternative to std.stream/std.socketstream for
networking, I would have had the same problems even if I had started the
project with D2...
A stream solution is in the works (it's discussed periodically on the Phobos
list), but they haven't sorted out quite what they want to do with it yet. The
Phobos API in general is in flux, though pieces of it are likely to stay more or
less unchanged from what they currently are. But there's far from any kind of
guarantee that much of anything from the D1 Phobos is going to survive in the D2
Phobos. They're looking to make Phobos as good as they can, and they aren't yet
worried about keeping its API stable (though they don't make changes unless they
think that it's actually benificial, so things don't change willy-nilly). I'm
sure that the time will come, however, when Phobos' API will stabilize, and
projects will be able to rely on it staying the same.

I think that the reality of the matter is that porting D1 code to D2 code is
going to be just like, if not exactly like, porting code from one library to
another rather than an upgrade like you'd get between Qt3 and Qt4 (which had
plenty of changes). I'm sure that the split between D1 and D2 is going to cause
a lot of problems for people looking to port code from one to the other, but it
will be better for newly written code, so it's a definite tradeoff. I wouldn't
look forward to porting a project from D1 to D2 though.

- Jonathan M Davis
Andrei Alexandrescu
2010-10-11 04:26:28 UTC
Permalink
Post by Jonathan M Davis
Post by Daniel Gibson
Post by bioinfornatics
LDC support 64 bit ;)
as well as GDC.
But both currently lack an up-to-date D2 compiler (but the GDC guys are
at least working on it, seems like they're currently at 2.029 - which is
great - about 3 months ago they were still at 2.018 and in between was
the big 2.020 update that introduced druntime).
I agree with walter that 64bit support in DMD is very important,
especially for D2: I started a project a few months ago that might have
benefited from D2s features (especially ranges), but I decided to use D1
because no 64bit compiler for D2 was in sight.
I am btw a bit worried about "upgrading" code from D1 to D2 some day
because of the heavy (still ongoing) changes, especially in phobos.
I read that for example std.stream is going to be deprecated and
replaced by either something with ranges or something like std.stdio
(whoever was right about that, maybe even both?). This means that all my
(file/network) IO-Code would have to be rewritten sooner or later..
But, currently lacking an alternative to std.stream/std.socketstream for
networking, I would have had the same problems even if I had started the
project with D2...
A stream solution is in the works (it's discussed periodically on the Phobos
list), but they haven't sorted out quite what they want to do with it yet. The
Phobos API in general is in flux, though pieces of it are likely to stay more or
less unchanged from what they currently are. But there's far from any kind of
guarantee that much of anything from the D1 Phobos is going to survive in the D2
Phobos. They're looking to make Phobos as good as they can, and they aren't yet
worried about keeping its API stable (though they don't make changes unless they
think that it's actually benificial, so things don't change willy-nilly). I'm
sure that the time will come, however, when Phobos' API will stabilize, and
projects will be able to rely on it staying the same.
I think that the reality of the matter is that porting D1 code to D2 code is
going to be just like, if not exactly like, porting code from one library to
another rather than an upgrade like you'd get between Qt3 and Qt4 (which had
plenty of changes). I'm sure that the split between D1 and D2 is going to cause
a lot of problems for people looking to port code from one to the other, but it
will be better for newly written code, so it's a definite tradeoff. I wouldn't
look forward to porting a project from D1 to D2 though.
I think it's a bit hasty to speak on behalf of all of Phobos'
participants. Phobos 2 is indeed different from Phobos 1 but
backward-incompatible changes to Phobos 2 are increasingly rare.

Andrei
Jonathan M Davis
2010-10-11 04:54:03 UTC
Permalink
Post by Andrei Alexandrescu
Post by Jonathan M Davis
A stream solution is in the works (it's discussed periodically on the
Phobos list), but they haven't sorted out quite what they want to do
with it yet. The Phobos API in general is in flux, though pieces of it
are likely to stay more or less unchanged from what they currently are.
But there's far from any kind of guarantee that much of anything from
the D1 Phobos is going to survive in the D2 Phobos. They're looking to
make Phobos as good as they can, and they aren't yet worried about
keeping its API stable (though they don't make changes unless they think
that it's actually benificial, so things don't change willy-nilly). I'm
sure that the time will come, however, when Phobos' API will stabilize,
and projects will be able to rely on it staying the same.
I think that the reality of the matter is that porting D1 code to D2 code
is going to be just like, if not exactly like, porting code from one
library to another rather than an upgrade like you'd get between Qt3 and
Qt4 (which had plenty of changes). I'm sure that the split between D1
and D2 is going to cause a lot of problems for people looking to port
code from one to the other, but it will be better for newly written
code, so it's a definite tradeoff. I wouldn't look forward to porting a
project from D1 to D2 though.
I think it's a bit hasty to speak on behalf of all of Phobos'
participants. Phobos 2 is indeed different from Phobos 1 but
backward-incompatible changes to Phobos 2 are increasingly rare.
Sorry if I overstepped my bounds on that. It's just that from what I've seen,
the Phobos devs have been quite willing to make backwards incompatible changes
if they thought that they were an improvement, though they aren't done all that
frequently. Backwards compatability is considered, but improvements to the API
seem to override it.

Regardless, the result is that if you wrote your code for dmd 2.040 or something
similar and ended up trying to update it to 2.050, you'd likely have a number of
changes to make, though porting from Phobos 1 would be far worse. If Phobos were
completely stable or at least never made backwards-compatability breaking
changes, that wouldn't be the case. I fully expect that as Phobos matures, such
breaking changes will become quite rare if not outright nonexistent, but they do
still happen. Actually deprecating and replacing the modules that are intended
to be deprecated and replace will help a lot with that, but that obviously takes
time.

- Jonathan M Davis
Jacob Carlborg
2010-10-11 11:41:41 UTC
Permalink
Post by Jonathan M Davis
Post by Andrei Alexandrescu
I think it's a bit hasty to speak on behalf of all of Phobos'
participants. Phobos 2 is indeed different from Phobos 1 but
backward-incompatible changes to Phobos 2 are increasingly rare.
Sorry if I overstepped my bounds on that. It's just that from what I've seen,
the Phobos devs have been quite willing to make backwards incompatible changes
if they thought that they were an improvement, though they aren't done all that
frequently. Backwards compatability is considered, but improvements to the API
seem to override it.
Regardless, the result is that if you wrote your code for dmd 2.040 or something
similar and ended up trying to update it to 2.050, you'd likely have a number of
changes to make, though porting from Phobos 1 would be far worse. If Phobos were
completely stable or at least never made backwards-compatability breaking
changes, that wouldn't be the case. I fully expect that as Phobos matures, such
breaking changes will become quite rare if not outright nonexistent, but they do
still happen. Actually deprecating and replacing the modules that are intended
to be deprecated and replace will help a lot with that, but that obviously takes
time.
- Jonathan M Davis
I think it's time to separate the compiler, the language and phobos in
the releases. One idea is the use three numbers to indicate a release of
phobos, major, minor and build, like this: 2.4.3. Increment the build
number when a implementation detail is changed the doesn't change the
API. Increment the minor number when a backwards compatible API change
is made and increment the major number when a API change is made that
break backwards compatibility.
--
/Jacob Carlborg
Daniel Gibson
2010-10-11 17:38:39 UTC
Permalink
Post by Andrei Alexandrescu
Post by Jonathan M Davis
Post by Daniel Gibson
Post by bioinfornatics
LDC support 64 bit ;)
as well as GDC.
But both currently lack an up-to-date D2 compiler (but the GDC guys are
at least working on it, seems like they're currently at 2.029 - which is
great - about 3 months ago they were still at 2.018 and in between was
the big 2.020 update that introduced druntime).
I agree with walter that 64bit support in DMD is very important,
especially for D2: I started a project a few months ago that might have
benefited from D2s features (especially ranges), but I decided to use D1
because no 64bit compiler for D2 was in sight.
I am btw a bit worried about "upgrading" code from D1 to D2 some day
because of the heavy (still ongoing) changes, especially in phobos.
I read that for example std.stream is going to be deprecated and
replaced by either something with ranges or something like std.stdio
(whoever was right about that, maybe even both?). This means that all my
(file/network) IO-Code would have to be rewritten sooner or later..
But, currently lacking an alternative to std.stream/std.socketstream for
networking, I would have had the same problems even if I had started the
project with D2...
A stream solution is in the works (it's discussed periodically on the Phobos
list), but they haven't sorted out quite what they want to do with it yet. The
Phobos API in general is in flux, though pieces of it are likely to stay more or
less unchanged from what they currently are. But there's far from any kind of
guarantee that much of anything from the D1 Phobos is going to survive in the D2
Phobos. They're looking to make Phobos as good as they can, and they aren't yet
worried about keeping its API stable (though they don't make changes unless they
think that it's actually benificial, so things don't change
willy-nilly). I'm
sure that the time will come, however, when Phobos' API will
stabilize, and
projects will be able to rely on it staying the same.
I think that the reality of the matter is that porting D1 code to D2 code is
going to be just like, if not exactly like, porting code from one library to
another rather than an upgrade like you'd get between Qt3 and Qt4 (which had
plenty of changes). I'm sure that the split between D1 and D2 is going to cause
a lot of problems for people looking to port code from one to the other, but it
will be better for newly written code, so it's a definite tradeoff. I wouldn't
look forward to porting a project from D1 to D2 though.
I think it's a bit hasty to speak on behalf of all of Phobos'
participants. Phobos 2 is indeed different from Phobos 1 but
backward-incompatible changes to Phobos 2 are increasingly rare.
Andrei
But parts of phobos are deprecated or will be deprecated and there still
is no alternative for them.
That may prevent people from writing "real" projects in D2 (or D at all)
- who wants to use classes that will be deprecated soon?
Sure, that old stuff will not be removed and can still be used, but I
personally feel a bit uncomfortable with using deprecated code.

Cheers,
- Daniel
Andrei Alexandrescu
2010-10-11 22:32:55 UTC
Permalink
Post by Daniel Gibson
But parts of phobos are deprecated or will be deprecated and there still
is no alternative for them.
That may prevent people from writing "real" projects in D2 (or D at all)
- who wants to use classes that will be deprecated soon?
Sure, that old stuff will not be removed and can still be used, but I
personally feel a bit uncomfortable with using deprecated code.
Agreed. Maybe this is a good time to sart making a requirements list for
streams. What are the essential features/feature groups?

Andrei
Daniel Gibson
2010-10-12 00:49:15 UTC
Permalink
Post by Andrei Alexandrescu
Post by Daniel Gibson
But parts of phobos are deprecated or will be deprecated and there still
is no alternative for them.
That may prevent people from writing "real" projects in D2 (or D at all)
- who wants to use classes that will be deprecated soon?
Sure, that old stuff will not be removed and can still be used, but I
personally feel a bit uncomfortable with using deprecated code.
Agreed. Maybe this is a good time to sart making a requirements list for
streams. What are the essential features/feature groups?
Andrei
Maybe something like the following (I hope it's not too extensive):

* Input- Output- and InputAndOutput- Streams
- having InputStream and OutputStream as an interface like in the old design may be a good idea
- implementing the standard operations that are mostly independent from the data source/sink
like read/write for basic types, strings, ... in mixin templates is probably elegant to create
streams that are both Input and Output (one mixin that implements most of InputStream and
one that implements most of OutputStream)

* Two kinds of streams:
1. basic streams: reading/writing from/to:
* network (socket)
* files
* just memory (currently MemoryStream)
* Arrays/Ranges?
* ...
2. streams wrapping other streams:
* for buffering - buffer input/output/both
- with the possibility to peek?
* to modify data when it's read/written (e.g. change endianess - important for networking!)
* custom streams.. e.g. could parse/create CSV (comma seperated values) data or similar

* Also there are different types of streams: seekable, resettable (a network stream is neither), ...

* functionality/methods needed/desirable:
- low level access
* void read(void *buf, size_t len) // read *exactly* len bytes into buf
* void write(void *buf, size_t len) // write *exactly* len bytes from buf to stream
- convenient methods to read/write basic types in binary (!) from/to stream
* <type> read<Type>() (like int readInt()) or T read(T)() (like int read!int())
- with enforcing T is somehow basic (certainly no Object or pointer)
- could use read(void *buf, size_t len) like in old implementation
* void write( <basic type> val ) or void write(T)( T val ) - again T should be basic type
- could use write(void *buf, size_t len) like in old implementation
- convenient methods to read/write arrays of T (T should again be a basic type)
* T[] readArray(T)( size_t len) // return array of T's containing len T's
- probably both alternatives make sense - the first one to write into an existing
array (-slice), the second one for convenience if you want a new array anyway
* void read(T)( T[] array ) // read array.length T's into array
- maybe name this readArray(T)(..) as well for consistency?
* void writeArray(T)( T[] array )
- special cases for strings?
* void writeString(char[] str) // same for wchar and dchar
- could write str into the stream with its length (as ushort xor uint xor ulong,
_not_ size_t!) prepended
* char[] readString() // same for wchar and dchar
- read length of the string and then the string itself that will be returned

- all that array/string/low level stuff but reading *at most* len (or array.length) values
and returning the amount actually read ( readUpTo() ?)
* useful e.g. for parsing http (you don't know how long the header is etc)
* the same for write? don't see much use for that though..

- some way to determine whether the stream
* is at its definite end (eof on file, socket closed or something like that)
* currently empty (for input stream) - just doing a read() would block ?

- Output streams need flush()
- for Input streams skip(size_t noBytes) or even skip(T)(size_t noElems) may be
handy to just throw away data we're not interested in without having it
copied around - especially for non-seekable streams (network..)

Cheers,
- Daniel
Andrei Alexandrescu
2010-10-13 14:32:15 UTC
Permalink
Post by Daniel Gibson
Post by Andrei Alexandrescu
Agreed. Maybe this is a good time to sart making a requirements list
for streams. What are the essential features/feature groups?
Andrei
* Input- Output- and InputAndOutput- Streams
- having InputStream and OutputStream as an interface like in the old
design may be a good idea
- implementing the standard operations that are mostly independent from
the data source/sink
like read/write for basic types, strings, ... in mixin templates is
probably elegant to create
streams that are both Input and Output (one mixin that implements most of InputStream and
one that implements most of OutputStream)
So far so good. I will point out, however, that the classic read/write
routines are not all that good. For example if you want to implement a
line-buffered stream on top of a block-buffered stream you'll be forced
to write inefficient code.

Also, a requirement that I think is essential is separation between
formatting and transport. std.stream does not have that. At the top
level there are two types of transport: text and binary. On top of that
lie various formatters.
Post by Daniel Gibson
* network (socket)
* files
* just memory (currently MemoryStream)
* Arrays/Ranges?
* ...
* for buffering - buffer input/output/both
- with the possibility to peek?
* to modify data when it's read/written (e.g. change endianess - important for networking!)
* custom streams.. e.g. could parse/create CSV (comma seperated values) data or similar
Would these be streams be different in their interface?
Post by Daniel Gibson
* Also there are different types of streams: seekable, resettable (a
network stream is neither), ...
Agreed. Question: is there a file system that offers resettable but not
seekable files? I'm thinking of collapsing the two together.
Post by Daniel Gibson
- low level access
* void read(void *buf, size_t len) // read *exactly* len bytes into buf
* void write(void *buf, size_t len) // write *exactly* len bytes from buf to stream
- convenient methods to read/write basic types in binary (!) from/to stream
Again, binary vs. text is a capability of the stream. For example, a tty
can never transport binary data - programs like gzip refuse to write
binary data to a terminal. (Then of course a binary stream can always
accommodate text data.)
Post by Daniel Gibson
* <type> read<Type>() (like int readInt()) or T read(T)() (like int read!int())
Templates will be difficult for a class hierarchy.
Post by Daniel Gibson
- with enforcing T is somehow basic (certainly no Object or pointer)
- could use read(void *buf, size_t len) like in old implementation
* void write( <basic type> val ) or void write(T)( T val ) - again T should be basic type
- could use write(void *buf, size_t len) like in old implementation
- convenient methods to read/write arrays of T (T should again be a basic type)
* T[] readArray(T)( size_t len) // return array of T's containing len T's
- probably both alternatives make sense - the first one to write into an existing
array (-slice), the second one for convenience if you want a new array anyway
* void read(T)( T[] array ) // read array.length T's into array
- maybe name this readArray(T)(..) as well for consistency?
* void writeArray(T)( T[] array )
- special cases for strings?
* void writeString(char[] str) // same for wchar and dchar
- could write str into the stream with its length (as ushort xor uint xor ulong,
_not_ size_t!) prepended
* char[] readString() // same for wchar and dchar
- read length of the string and then the string itself that will be returned
Many of these capabilities involve template methods. Is a template-based
approach preferable to a straight class hierarchy? I tend to think that
in the case of streams, classic hierarchies are most adequate.
Post by Daniel Gibson
- all that array/string/low level stuff but reading *at most* len (or array.length) values
and returning the amount actually read ( readUpTo() ?)
* useful e.g. for parsing http (you don't know how long the header is etc)
* the same for write? don't see much use for that though..
- some way to determine whether the stream
* is at its definite end (eof on file, socket closed or something like that)
* currently empty (for input stream) - just doing a read() would block ?
- Output streams need flush()
- for Input streams skip(size_t noBytes) or even skip(T)(size_t noElems) may be
handy to just throw away data we're not interested in without having it
copied around - especially for non-seekable streams (network..)
OK, that's a good start. Let's toss this back and forth a few times and
see what sticks.


Andrei
Johannes Pfau
2010-10-13 15:37:06 UTC
Permalink
Post by Andrei Alexandrescu
Post by Daniel Gibson
* Also there are different types of streams: seekable, resettable (a
network stream is neither), ...
Agreed. Question: is there a file system that offers resettable but not
seekable files? I'm thinking of collapsing the two together.
Can't think of a file system. But for example a http stream is always
resettable, but not always seekable.
--
Johannes Pfau
Wayne Anderson
2010-10-13 16:07:30 UTC
Permalink
Don't forget data alignment. For example, when streaming structs it may be useful to stream out to the file
system or network in packed form, then unpack to the preferred alignment of the destination architecture
on streaming in.
Dmitry Olshansky
2010-10-13 16:08:02 UTC
Permalink
Post by Johannes Pfau
Post by Andrei Alexandrescu
Post by Daniel Gibson
* Also there are different types of streams: seekable, resettable (a
network stream is neither), ...
Agreed. Question: is there a file system that offers resettable but not
seekable files? I'm thinking of collapsing the two together.
Can't think of a file system. But for example a http stream is always
resettable, but not always seekable.
Another example, an archive file entry (at least in zip file) is
resettable, but not seekable.
To seek one would need to decompress stream and discard data (hardly any
faster then reading).
--
Dmitry Olshansky
Denis Koroskin
2010-10-13 16:16:38 UTC
Permalink
On Wed, 13 Oct 2010 18:32:15 +0400, Andrei Alexandrescu
Post by Andrei Alexandrescu
Post by Daniel Gibson
Post by Andrei Alexandrescu
Agreed. Maybe this is a good time to sart making a requirements list
for streams. What are the essential features/feature groups?
Andrei
* Input- Output- and InputAndOutput- Streams
- having InputStream and OutputStream as an interface like in the old
design may be a good idea
- implementing the standard operations that are mostly independent from
the data source/sink
like read/write for basic types, strings, ... in mixin templates is
probably elegant to create
streams that are both Input and Output (one mixin that implements most of InputStream and
one that implements most of OutputStream)
So far so good. I will point out, however, that the classic read/write
routines are not all that good. For example if you want to implement a
line-buffered stream on top of a block-buffered stream you'll be forced
to write inefficient code.
Never heard of filesystems that allow reading files in lines - they always
read in blocks, and that's what streams should do. That's because most of
the steams are binary streams, and there is no such thing as a "line" in
them (e.g. how often do you need to read a line from a SocketStream?).

I don't think streams should buffer anything either (what an underlying OS
I/O API caches should suffice), buffered streams adapters can do that in a
stream-independent way (why duplicate code when you can do that as
efficiently with external methods?).

Besides, as you noted, the buffering is redundant for byChunk/byLine
adapter ranges. It means that byChunk/byLine should operate on unbuffered
streams.

I'll explain my I/O streams implementation below in case you didn't read
my message (I've changed some stuff a little since then). My Stream
interface is very simple:

// A generic stream
interface Stream
{
@property InputStream input();
@property OutputStream output();
@property SeekableStream seekable();
@property bool endOfStream();
void close();
}

You may ask, why separate Input and Output streams? Well, that's because
you either read from them, write from them, or both.
Some streams are read-only (think Stdin), some write-only (Stdout), some
support both, like FileStream. Right?

Not exactly. Does FileStream support writing when you open file for
reading? Does it support reading when you open for writing?
So, you may or may not read from a generic stream, and you also may or may
not write to a generic stream. With a design like that you can make a
mistake: if a stream isn't readable, you have no reference to invoke
read() method on.

Similarly, a stream is either seekable, or not. SeekableStreams allow
stream cursor manipulation:

interface SeekableStream : Stream
{
long getPosition(Anchor whence = Anchor.begin);
void setPosition(long position, Anchor whence = Anchor.begin);
}

InputStream doesn't really has many methods:

interface InputStream
{
// reads up to buffer.length bytes from a stream
// returns number of bytes read
// throws on error
size_t read(ubyte[] buffer);

// reads from current position
AsyncReadRequest readAsync(ubyte[] buffer, Mailbox* mailbox = null);
}

So is OutputStream:

interface OutputStream
{
// returns number of bytes written
// throws on error
size_t write(const(ubyte)[] buffer);

// writes from current position
AsyncWriteRequest writeAsync(const(ubyte)[] buffer, Mailbox* mailbox =
null);
}

They basically support only reading and writing in blocks, nothing else.
However, they support asynchronous reads/writes, too (think of mailbox as
a std.concurrency's Tid).

Unlike Daniel's proposal, my design reads up to buffer size bytes for two
reasons:
- it avoids potential buffering and multiple sys calls
- it is the only way to go with SocketStreams. I mean, you often don't
know how many bytes an incoming socket message contains. You either have
to read it byte-by-byte, or your application might stall for potentially
infinite time (if message was shorter than your buffer, and no more
messages are being sent)

Why do my streams provide async methods? Because it's the modern approach
to I/O - blocking I/O (aka one thread per client) doesn't scale. E.g. Java
adds a second revision of Async I/O API in JDK7 (called NIO2, first
appeared in February, 2002), C# has asynchronous operations as part of
their Stream interface since .NET 1.1 (April, 2003).

With async I/O you can server many clients with one thread. Here is an
example (pseude-code, usings std.concurrency):

foreach (connection; networkConnections) {
connection.receiveMessage(getTid());
}

receiveOnly!( (NetworkMessage message) { /* do stuff */ }

This is still not the most performant solution, but it's still a lot
better than one thread per client.

Async I/O not only needed for network stuff. Here is a code snippet from
DMD (comments added):

#define ASYNCREAD 1
#if ASYNCREAD
AsyncRead *aw = AsyncRead::create(modules.dim);
for (i = 0; i < modules.dim; i++)
{
m = (Module *)modules.data[i];
aw->addFile(m->srcfile);
}
aw->start(); // executes async request, doesn't block
#else
// Single threaded
for (i = 0; i < modules.dim; i++)
{
m = (Module *)modules.data[i];
m->read(0); // blocks
}
#endif

// Do some other stuff

for (i = 0; i < modules.dim; i++)
{
...
#if ASYNCREAD
aw->read(i); // waits until async operation finishes
#endif

Walter told that this small change gave quite a speed up in compilation
time.

Also, my async methods return a reference to AsyncRequest interface that
allows waiting for completion (that's what Walter does in DMD), canceling,
querying a status (complete, in progress, failed), reporting an error, etc
and that's very useful, too.

I strongly believe we shouldn't ignore this type of API.

P.S. For threads this deep it's better fork a new one, especially when
changing the subject.
Andrei Alexandrescu
2010-10-13 16:55:04 UTC
Permalink
Post by Denis Koroskin
On Wed, 13 Oct 2010 18:32:15 +0400, Andrei Alexandrescu
Post by Andrei Alexandrescu
So far so good. I will point out, however, that the classic read/write
routines are not all that good. For example if you want to implement a
line-buffered stream on top of a block-buffered stream you'll be
forced to write inefficient code.
Never heard of filesystems that allow reading files in lines - they
always read in blocks, and that's what streams should do.
http://www.gnu.org/s/libc/manual/html_node/Buffering-Concepts.html

I don't think streams must mimic the low-level OS I/O interface.
Post by Denis Koroskin
That's because
most of the steams are binary streams, and there is no such thing as a
"line" in them (e.g. how often do you need to read a line from a
SocketStream?).
http://www.opengroup.org/onlinepubs/009695399/functions/isatty.html

You need a line when e.g. you parse a HTML header or a email header or
an FTP response. Again, if at a low level the transfer occurs in blocks,
that doesn't mean the API must do the same at all levels.
Post by Denis Koroskin
I don't think streams should buffer anything either (what an underlying
OS I/O API caches should suffice), buffered streams adapters can do that
in a stream-independent way (why duplicate code when you can do that as
efficiently with external methods?).
Most OS primitives don't give access to their own internal buffers.
Instead, they ask user code to provide a buffer and transfer data into
it. So clearly buffering on the client side is a must.
Post by Denis Koroskin
Besides, as you noted, the buffering is redundant for byChunk/byLine
adapter ranges. It means that byChunk/byLine should operate on
unbuffered streams.
Chunks keep their own buffer so indeed they could operate on streams
that don't do additional buffering. The story with lines is a fair
amount more complicated if it needs to be done efficiently.
Post by Denis Koroskin
I'll explain my I/O streams implementation below in case you didn't read
my message (I've changed some stuff a little since then).
Honest, I opened it to remember to read it but somehow your fonts are
small and make my eyes hurt.
Post by Denis Koroskin
My Stream
// A generic stream
interface Stream
{
@property InputStream input();
@property OutputStream output();
@property SeekableStream seekable();
@property bool endOfStream();
void close();
}
You may ask, why separate Input and Output streams?
I think my first question is: why doesn't Stream inherit InputStream and
OutputStream? My hypothesis: you want to sometimes return null. Nice.
Post by Denis Koroskin
Well, that's because
you either read from them, write from them, or both.
Some streams are read-only (think Stdin), some write-only (Stdout), some
support both, like FileStream. Right?
Sounds good. But then where's flush()? Must be in OutputStream.
Post by Denis Koroskin
Not exactly. Does FileStream support writing when you open file for
reading? Does it support reading when you open for writing?
So, you may or may not read from a generic stream, and you also may or
may not write to a generic stream. With a design like that you can make
a mistake: if a stream isn't readable, you have no reference to invoke
read() method on.
That is indeed pretty nifty. I hope you would allow us to copy that
feature in Phobos (unless you are considering submitting your library
wholesale). Let me know.
Post by Denis Koroskin
Similarly, a stream is either seekable, or not. SeekableStreams allow
interface SeekableStream : Stream
{
long getPosition(Anchor whence = Anchor.begin);
void setPosition(long position, Anchor whence = Anchor.begin);
}
Makes sense. Why is getPosition signed? Why do you need an anchor for
getPosition?
Post by Denis Koroskin
interface InputStream
{
// reads up to buffer.length bytes from a stream
// returns number of bytes read
// throws on error
size_t read(ubyte[] buffer);
That makes implementation of line buffering inefficient :o).
Post by Denis Koroskin
// reads from current position
AsyncReadRequest readAsync(ubyte[] buffer, Mailbox* mailbox = null);
}
Why doesn't Sean's concurrency API scale for your needs? Can that be
fixed? Would you consider submitting some informed bug reports?
Post by Denis Koroskin
interface OutputStream
{
// returns number of bytes written
// throws on error
size_t write(const(ubyte)[] buffer);
// writes from current position
AsyncWriteRequest writeAsync(const(ubyte)[] buffer, Mailbox* mailbox =
null);
}
They basically support only reading and writing in blocks, nothing else.
I'm surprised there's no flush().
Post by Denis Koroskin
However, they support asynchronous reads/writes, too (think of mailbox
as a std.concurrency's Tid).
Unlike Daniel's proposal, my design reads up to buffer size bytes for
- it avoids potential buffering and multiple sys calls
But there's a problem. It's very rare that the user knows what a good
buffer size is. And often there are size and alignment restrictions at
the low level. So somewhere there is still buffering going on, and also
there are potential inefficiencies (if a user reads small buffers).
Post by Denis Koroskin
- it is the only way to go with SocketStreams. I mean, you often don't
know how many bytes an incoming socket message contains. You either have
to read it byte-by-byte, or your application might stall for potentially
infinite time (if message was shorter than your buffer, and no more
messages are being sent)
But if you don't know how many bytes are in an incoming socket message,
a better design is to do this:

void read(ref ubyte[] buffer);

and resize the buffer to accommodate the incoming packet. Your design
_imposes_ that the socket does additional buffering.
Post by Denis Koroskin
Why do my streams provide async methods? Because it's the modern
approach to I/O - blocking I/O (aka one thread per client) doesn't
scale. E.g. Java adds a second revision of Async I/O API in JDK7 (called
NIO2, first appeared in February, 2002), C# has asynchronous operations
as part of their Stream interface since .NET 1.1 (April, 2003).
Async I/O is nice, no two ways about that. I have on my list to define
byChunkAsync that works exactly like byChunk from the client's
perspective, except it does I/O concurrently with client code.

[snip]
Post by Denis Koroskin
I strongly believe we shouldn't ignore this type of API.
P.S. For threads this deep it's better fork a new one, especially when
changing the subject.
I thought I did by changing the title...


Andrei
Daniel Gibson
2010-10-13 17:48:07 UTC
Permalink
Post by Andrei Alexandrescu
Post by Denis Koroskin
P.S. For threads this deep it's better fork a new one, especially when
changing the subject.
I thought I did by changing the title...
Andrei
At least on my Thunderbird/Icedove 2.0.0.24 it's still in the old Thread.
Denis Koroskin
2010-10-13 19:02:48 UTC
Permalink
On Wed, 13 Oct 2010 20:55:04 +0400, Andrei Alexandrescu
Post by Andrei Alexandrescu
Post by Denis Koroskin
On Wed, 13 Oct 2010 18:32:15 +0400, Andrei Alexandrescu
Post by Andrei Alexandrescu
So far so good. I will point out, however, that the classic read/write
routines are not all that good. For example if you want to implement a
line-buffered stream on top of a block-buffered stream you'll be
forced to write inefficient code.
Never heard of filesystems that allow reading files in lines - they
always read in blocks, and that's what streams should do.
http://www.gnu.org/s/libc/manual/html_node/Buffering-Concepts.html
I don't think streams must mimic the low-level OS I/O interface.
I in contrast think that Streams should be a lowest-level possible
platform-independent abstraction.
No buffering besides what an OS provides, no additional functionality. If
you need to be able to read something up to some character (besides, what
should be considered a new-line separator: \r, \n, \r\n?), this should be
done manually in "byLine".
Post by Andrei Alexandrescu
Post by Denis Koroskin
That's because
most of the steams are binary streams, and there is no such thing as a
"line" in them (e.g. how often do you need to read a line from a
SocketStream?).
http://www.opengroup.org/onlinepubs/009695399/functions/isatty.html
These are special cases I don't like. There is no such thing in Windows
anyway.
Post by Andrei Alexandrescu
You need a line when e.g. you parse a HTML header or a email header or
an FTP response. Again, if at a low level the transfer occurs in blocks,
that doesn't mean the API must do the same at all levels.
BSD sockets transmits in blocks. If you need to find a special sequence in
a socket stream, you are forced to fetch a chunk, and manually search for
a needed sequence. My position is that you should do it with an external
predicate (e.g. read until whitespace).
Post by Andrei Alexandrescu
Post by Denis Koroskin
I don't think streams should buffer anything either (what an underlying
OS I/O API caches should suffice), buffered streams adapters can do that
in a stream-independent way (why duplicate code when you can do that as
efficiently with external methods?).
Most OS primitives don't give access to their own internal buffers.
Instead, they ask user code to provide a buffer and transfer data into
it.
Right. This is why Stream may not cache.
Post by Andrei Alexandrescu
So clearly buffering on the client side is a must.
I don't see how is it implied from above.
Post by Andrei Alexandrescu
Post by Denis Koroskin
Besides, as you noted, the buffering is redundant for byChunk/byLine
adapter ranges. It means that byChunk/byLine should operate on
unbuffered streams.
Chunks keep their own buffer so indeed they could operate on streams
that don't do additional buffering. The story with lines is a fair
amount more complicated if it needs to be done efficiently.
Yes. But line-reading is a case that I don't see a need to be handled
specially.
Post by Andrei Alexandrescu
Post by Denis Koroskin
I'll explain my I/O streams implementation below in case you didn't read
my message (I've changed some stuff a little since then).
Honest, I opened it to remember to read it but somehow your fonts are
small and make my eyes hurt.
Post by Denis Koroskin
My Stream
// A generic stream
interface Stream
{
@property InputStream input();
@property OutputStream output();
@property SeekableStream seekable();
@property bool endOfStream();
void close();
}
You may ask, why separate Input and Output streams?
I think my first question is: why doesn't Stream inherit InputStream and
OutputStream? My hypothesis: you want to sometimes return null. Nice.
Right.
Post by Andrei Alexandrescu
Post by Denis Koroskin
Well, that's because
you either read from them, write from them, or both.
Some streams are read-only (think Stdin), some write-only (Stdout), some
support both, like FileStream. Right?
Sounds good. But then where's flush()? Must be in OutputStream.
That's probably because unbuffered streams don't need them.
Post by Andrei Alexandrescu
Post by Denis Koroskin
Not exactly. Does FileStream support writing when you open file for
reading? Does it support reading when you open for writing?
So, you may or may not read from a generic stream, and you also may or
may not write to a generic stream. With a design like that you can make
a mistake: if a stream isn't readable, you have no reference to invoke
read() method on.
That is indeed pretty nifty. I hope you would allow us to copy that
feature in Phobos (unless you are considering submitting your library
wholesale). Let me know.
Would love to contribute with design and implementation.
Post by Andrei Alexandrescu
Post by Denis Koroskin
Similarly, a stream is either seekable, or not. SeekableStreams allow
interface SeekableStream : Stream
{
long getPosition(Anchor whence = Anchor.begin);
void setPosition(long position, Anchor whence = Anchor.begin);
}
Makes sense. Why is getPosition signed? Why do you need an anchor for
getPosition?
long is chosen to be consistent with setPosition. Also getPosition may
return a negative value:

long pos = getPosition(Anchor.end); // how far is it till file end?

Also this is how you can get file size (need to invert though). This is
consistent with setPosition:

setPosition(getPosition(anchor), anchor); // a no-op for any kind of achor

I just thought why not? I'm okay with dropping it, but I find it nice.
Post by Andrei Alexandrescu
Post by Denis Koroskin
interface InputStream
{
// reads up to buffer.length bytes from a stream
// returns number of bytes read
// throws on error
size_t read(ubyte[] buffer);
That makes implementation of line buffering inefficient :o).
There is no way you can do it more efficient on Windows. Fetch a chunk;
search for a line end; found ? return : continue.
Post by Andrei Alexandrescu
Post by Denis Koroskin
// reads from current position
AsyncReadRequest readAsync(ubyte[] buffer, Mailbox* mailbox = null);
}
Why doesn't Sean's concurrency API scale for your needs? Can that be
fixed? Would you consider submitting some informed bug reports?
It's rather a design issue than a bug on its own. I'll write a separate
letter on that.
Post by Andrei Alexandrescu
Post by Denis Koroskin
interface OutputStream
{
// returns number of bytes written
// throws on error
size_t write(const(ubyte)[] buffer);
// writes from current position
AsyncWriteRequest writeAsync(const(ubyte)[] buffer, Mailbox* mailbox =
null);
}
They basically support only reading and writing in blocks, nothing else.
I'm surprised there's no flush().
No buffering - no flush.
Post by Andrei Alexandrescu
Post by Denis Koroskin
However, they support asynchronous reads/writes, too (think of mailbox
as a std.concurrency's Tid).
Unlike Daniel's proposal, my design reads up to buffer size bytes for
- it avoids potential buffering and multiple sys calls
But there's a problem. It's very rare that the user knows what a good
buffer size is. And often there are size and alignment restrictions at
the low level.
I agree, but he can guess. Or a library can give him a hint. E.g.
BUFFER_SIZE is a good buffer size to start with :)
Post by Andrei Alexandrescu
So somewhere there is still buffering going on, and also there are
potential inefficiencies (if a user reads small buffers).
Post by Denis Koroskin
- it is the only way to go with SocketStreams. I mean, you often don't
know how many bytes an incoming socket message contains. You either have
to read it byte-by-byte, or your application might stall for potentially
infinite time (if message was shorter than your buffer, and no more
messages are being sent)
But if you don't know how many bytes are in an incoming socket message,
void read(ref ubyte[] buffer);
That could work, too.
Post by Andrei Alexandrescu
and resize the buffer to accommodate the incoming packet. Your design
_imposes_ that the socket does additional buffering.
The socket API does it anyway. I just don't complicate it even further but
providing an additional layer of buffering.
Post by Andrei Alexandrescu
Post by Denis Koroskin
Why do my streams provide async methods? Because it's the modern
approach to I/O - blocking I/O (aka one thread per client) doesn't
scale. E.g. Java adds a second revision of Async I/O API in JDK7 (called
NIO2, first appeared in February, 2002), C# has asynchronous operations
as part of their Stream interface since .NET 1.1 (April, 2003).
Async I/O is nice, no two ways about that. I have on my list to define
byChunkAsync that works exactly like byChunk from the client's
perspective, except it does I/O concurrently with client code.
[snip]
Post by Denis Koroskin
I strongly believe we shouldn't ignore this type of API.
P.S. For threads this deep it's better fork a new one, especially when
changing the subject.
I thought I did by changing the title...
Andrei
No, changing title isn't enough.
Daniel Gibson
2010-10-13 20:10:00 UTC
Permalink
Post by Denis Koroskin
On Wed, 13 Oct 2010 20:55:04 +0400, Andrei Alexandrescu
Post by Andrei Alexandrescu
Post by Denis Koroskin
On Wed, 13 Oct 2010 18:32:15 +0400, Andrei Alexandrescu
Post by Andrei Alexandrescu
So far so good. I will point out, however, that the classic read/write
routines are not all that good. For example if you want to implement a
line-buffered stream on top of a block-buffered stream you'll be
forced to write inefficient code.
Never heard of filesystems that allow reading files in lines - they
always read in blocks, and that's what streams should do.
http://www.gnu.org/s/libc/manual/html_node/Buffering-Concepts.html
I don't think streams must mimic the low-level OS I/O interface.
I in contrast think that Streams should be a lowest-level possible
platform-independent abstraction.
No buffering besides what an OS provides, no additional functionality.
If you need to be able to read something up to some character (besides,
what should be considered a new-line separator: \r, \n, \r\n?), this
should be done manually in "byLine".
Platform-independent? OS-Independent, yes. But being independent of Endianess and availability of
80bit real etc is to much for a simple stream (of course we'd need an EndianStream that can wrap a
simple stream and take care of the endianess).
Post by Denis Koroskin
Post by Andrei Alexandrescu
Post by Denis Koroskin
That's because
most of the steams are binary streams, and there is no such thing as a
"line" in them (e.g. how often do you need to read a line from a
SocketStream?).
http://www.opengroup.org/onlinepubs/009695399/functions/isatty.html
These are special cases I don't like. There is no such thing in Windows
anyway.
Post by Andrei Alexandrescu
You need a line when e.g. you parse a HTML header or a email header or
an FTP response. Again, if at a low level the transfer occurs in
blocks, that doesn't mean the API must do the same at all levels.
BSD sockets transmits in blocks. If you need to find a special sequence
in a socket stream, you are forced to fetch a chunk, and manually search
for a needed sequence. My position is that you should do it with an
external predicate (e.g. read until whitespace).
Post by Andrei Alexandrescu
Post by Denis Koroskin
I don't think streams should buffer anything either (what an underlying
OS I/O API caches should suffice), buffered streams adapters can do that
in a stream-independent way (why duplicate code when you can do that as
efficiently with external methods?).
Most OS primitives don't give access to their own internal buffers.
Instead, they ask user code to provide a buffer and transfer data into
it.
Right. This is why Stream may not cache.
Simple streams should not cache, but there must be a BufferedStream wrapping simple streams.
When you read from a non-buffered SocketStream each read() (like readInt()) is a syscall - that's
really expensive.
In my project I got a speedup of about factor 4-5 by replacing std.Streams SocketStream with a
custom BufferedSocketStream. I have to do further testing, but I think that shifted the bottleneck
from socket-I/O to something else, so in other cases the speedup may be even bigger.
Post by Denis Koroskin
Post by Andrei Alexandrescu
So clearly buffering on the client side is a must.
I don't see how is it implied from above.
Post by Andrei Alexandrescu
Post by Denis Koroskin
Besides, as you noted, the buffering is redundant for byChunk/byLine
adapter ranges. It means that byChunk/byLine should operate on
unbuffered streams.
Chunks keep their own buffer so indeed they could operate on streams
that don't do additional buffering. The story with lines is a fair
amount more complicated if it needs to be done efficiently.
Yes. But line-reading is a case that I don't see a need to be handled
specially.
Post by Andrei Alexandrescu
Post by Denis Koroskin
I'll explain my I/O streams implementation below in case you didn't read
my message (I've changed some stuff a little since then).
Honest, I opened it to remember to read it but somehow your fonts are
small and make my eyes hurt.
Post by Denis Koroskin
My Stream
// A generic stream
interface Stream
{
@property InputStream input();
@property OutputStream output();
@property SeekableStream seekable();
@property bool endOfStream();
void close();
}
You may ask, why separate Input and Output streams?
I think my first question is: why doesn't Stream inherit InputStream
and OutputStream? My hypothesis: you want to sometimes return null. Nice.
Right.
Post by Andrei Alexandrescu
Post by Denis Koroskin
Well, that's because
you either read from them, write from them, or both.
Some streams are read-only (think Stdin), some write-only (Stdout), some
support both, like FileStream. Right?
Sounds good. But then where's flush()? Must be in OutputStream.
That's probably because unbuffered streams don't need them.
You may need to tell the OS to flush its buffer (fsync()).
Post by Denis Koroskin
Post by Andrei Alexandrescu
I'm surprised there's no flush().
No buffering - no flush.
see above


Cheers,
- Daniel
Andrei Alexandrescu
2010-10-13 20:19:45 UTC
Permalink
Post by Denis Koroskin
On Wed, 13 Oct 2010 20:55:04 +0400, Andrei Alexandrescu
Post by Andrei Alexandrescu
http://www.gnu.org/s/libc/manual/html_node/Buffering-Concepts.html
I don't think streams must mimic the low-level OS I/O interface.
I in contrast think that Streams should be a lowest-level possible
platform-independent abstraction.
No buffering besides what an OS provides, no additional functionality.
If you need to be able to read something up to some character (besides,
what should be considered a new-line separator: \r, \n, \r\n?), this
should be done manually in "byLine".
This aggravates client code for the sake of simplicity in a library that
was supposed to make streaming easy. I'm not seeing progress.
Post by Denis Koroskin
Post by Andrei Alexandrescu
Post by Denis Koroskin
That's because
most of the steams are binary streams, and there is no such thing as a
"line" in them (e.g. how often do you need to read a line from a
SocketStream?).
http://www.opengroup.org/onlinepubs/009695399/functions/isatty.html
These are special cases I don't like. There is no such thing in Windows
anyway.
I didn't say I like them. Windows has _isatty:
http://msdn.microsoft.com/en-us/library/f4s0ddew(v=VS.80).aspx
Post by Denis Koroskin
Post by Andrei Alexandrescu
You need a line when e.g. you parse a HTML header or a email header or
an FTP response. Again, if at a low level the transfer occurs in
blocks, that doesn't mean the API must do the same at all levels.
BSD sockets transmits in blocks. If you need to find a special sequence
in a socket stream, you are forced to fetch a chunk, and manually search
for a needed sequence. My position is that you should do it with an
external predicate (e.g. read until whitespace).
Problem is how you set up interfaces to avoid inefficiencies and
contortions in the client.
Post by Denis Koroskin
Post by Andrei Alexandrescu
Post by Denis Koroskin
I don't think streams should buffer anything either (what an underlying
OS I/O API caches should suffice), buffered streams adapters can do that
in a stream-independent way (why duplicate code when you can do that as
efficiently with external methods?).
Most OS primitives don't give access to their own internal buffers.
Instead, they ask user code to provide a buffer and transfer data into
it.
Right. This is why Stream may not cache.
This is a big misunderstanding. If the interface is:

size_t read(byte[] buffer);

then *I*, the client, need to provide the buffer. It's in client space.
This means willing or not I need to do buffering, regardless of whatever
internal buffering is going on under the wraps.
Post by Denis Koroskin
Post by Andrei Alexandrescu
So clearly buffering on the client side is a must.
I don't see how is it implied from above.
Please implement an abstraction that given this:

interface InputStream
{
size_t read(ubyte[] buf);
}

defines a line reader.


Andrei
Daniel Gibson
2010-10-13 18:45:14 UTC
Permalink
Post by Andrei Alexandrescu
Post by Daniel Gibson
Post by Andrei Alexandrescu
Agreed. Maybe this is a good time to sart making a requirements list
for streams. What are the essential features/feature groups?
Andrei
* Input- Output- and InputAndOutput- Streams
- having InputStream and OutputStream as an interface like in the old
design may be a good idea
- implementing the standard operations that are mostly independent from
the data source/sink
like read/write for basic types, strings, ... in mixin templates is
probably elegant to create
streams that are both Input and Output (one mixin that implements most of InputStream and
one that implements most of OutputStream)
So far so good. I will point out, however, that the classic read/write
routines are not all that good. For example if you want to implement a
line-buffered stream on top of a block-buffered stream you'll be forced
to write inefficient code.
So what's a possible alternative to the classic read/write routines?
Post by Andrei Alexandrescu
Also, a requirement that I think is essential is separation between
formatting and transport. std.stream does not have that. At the top
level there are two types of transport: text and binary. On top of that
lie various formatters.
Ok, one should differ between text and binary streams. I was mostly focused on binary streams
(because that's what I use).
So there might be a hierarchy like
* Input/Output-Stream (interface for all those read/write operations)
- BinaryStream // abstract class implementing writeInt() etc using write(void* buf, size_t len)
* BinarySocketStream
* BinaryFileStream
* ...
- TextStream // abstract class implementing writeInt() etc using something like to!string and
write(char[])
* TextFileStream
* ...

(This for both Input- and Outputstreams)
Post by Andrei Alexandrescu
Post by Daniel Gibson
* network (socket)
* files
* just memory (currently MemoryStream)
* Arrays/Ranges?
* ...
* for buffering - buffer input/output/both
- with the possibility to peek?
* to modify data when it's read/written (e.g. change endianess -
important for networking!)
* custom streams.. e.g. could parse/create CSV (comma seperated values) data or similar
Would these be streams be different in their interface?
No. I just wanted to point out that it must be possible (and should be easy) to wrap streams.
Post by Andrei Alexandrescu
Post by Daniel Gibson
* Also there are different types of streams: seekable, resettable (a
network stream is neither), ...
Agreed. Question: is there a file system that offers resettable but not
seekable files? I'm thinking of collapsing the two together.
As mentioned before in other branches of this thread: Probably no file system, but maybe archive
files (zip, ...)
Post by Andrei Alexandrescu
Post by Daniel Gibson
- low level access
* void read(void *buf, size_t len) // read *exactly* len bytes into buf
* void write(void *buf, size_t len) // write *exactly* len bytes from buf to stream
- convenient methods to read/write basic types in binary (!) from/to stream
Again, binary vs. text is a capability of the stream. For example, a tty
can never transport binary data - programs like gzip refuse to write
binary data to a terminal. (Then of course a binary stream can always
accommodate text data.)
Post by Daniel Gibson
* <type> read<Type>() (like int readInt()) or T read(T)() (like int read!int())
Templates will be difficult for a class hierarchy.
Ok.
Another issue, as you mentioned line based streams: of course read(T)()/write(T)() would be quite
messy on them (endless static if(is(T==int)) { ... } else static if(is(T==float)) {...} etc).

(here were a lot of templated methods)
Post by Andrei Alexandrescu
Post by Daniel Gibson
* void writeString(char[] str) // same for wchar and dchar
- could write str into the stream with its length (as ushort xor uint xor ulong,
_not_ size_t!) prepended
* char[] readString() // same for wchar and dchar
- read length of the string and then the string itself that will be returned
Many of these capabilities involve template methods. Is a template-based
approach preferable to a straight class hierarchy? I tend to think that
in the case of streams, classic hierarchies are most adequate.
Ok agreed.
I forgot that templated methods can't be overridden.

As writeArray(T)() etc is hardly possible, at least for strings there should be write(char[])
write(dchar[]) and write(wchar[]) (maybe with some const-stuff added). These should just write the
string to the stream, without its length.
(Analogous for read(...))
Post by Andrei Alexandrescu
Post by Daniel Gibson
- all that array/string/low level stuff but reading *at most* len (or
array.length) values
and returning the amount actually read ( readUpTo() ?)
* useful e.g. for parsing http (you don't know how long the header is etc)
* the same for write? don't see much use for that though..
- some way to determine whether the stream
* is at its definite end (eof on file, socket closed or something like that)
* currently empty (for input stream) - just doing a read() would block ?
- Output streams need flush()
- for Input streams skip(size_t noBytes) or even skip(T)(size_t noElems) may be
handy to just throw away data we're not interested in without having it
copied around - especially for non-seekable streams (network..)
OK, that's a good start. Let's toss this back and forth a few times and
see what sticks.
Andrei
Cheers,
- Daniel
Andrei Alexandrescu
2010-10-13 20:27:05 UTC
Permalink
Post by Daniel Gibson
Post by Andrei Alexandrescu
Post by Daniel Gibson
Post by Andrei Alexandrescu
Agreed. Maybe this is a good time to sart making a requirements list
for streams. What are the essential features/feature groups?
Andrei
* Input- Output- and InputAndOutput- Streams
- having InputStream and OutputStream as an interface like in the old
design may be a good idea
- implementing the standard operations that are mostly independent from
the data source/sink
like read/write for basic types, strings, ... in mixin templates is
probably elegant to create
streams that are both Input and Output (one mixin that implements most
of InputStream and
one that implements most of OutputStream)
So far so good. I will point out, however, that the classic read/write
routines are not all that good. For example if you want to implement a
line-buffered stream on top of a block-buffered stream you'll be
forced to write inefficient code.
So what's a possible alternative to the classic read/write routines?
It's something that ten people implement in eleven ways. Some starters:

1. Get one byte at a time. This may be inefficient.

2. Define this:

size_t append(ref ubyte[] buffer);

Such an interface allows the stream to append data to a user-maintained
buffer. Then the user manages terminators etc.

3. Define this:

size_t getDelim(ref ubyte[] buffer, in ubyte[] terminator);
Post by Daniel Gibson
No. I just wanted to point out that it must be possible (and should be
easy) to wrap streams.
And efficient too! That means no buffering friction :o).
Post by Daniel Gibson
Post by Andrei Alexandrescu
Agreed. Question: is there a file system that offers resettable but
not seekable files? I'm thinking of collapsing the two together.
As mentioned before in other branches of this thread: Probably no file
system, but maybe archive files (zip, ...)
I wonder to what extent resetting is really closing and reopening the
file/connection.


Andrei

pipe dream
2010-10-12 09:45:33 UTC
Permalink
Post by Andrei Alexandrescu
Post by Daniel Gibson
But parts of phobos are deprecated or will be deprecated and there still
is no alternative for them.
That may prevent people from writing "real" projects in D2 (or D at all)
- who wants to use classes that will be deprecated soon?
Sure, that old stuff will not be removed and can still be used, but I
personally feel a bit uncomfortable with using deprecated code.
Agreed. Maybe this is a good time to sart making a requirements list for
streams. What are the essential features/feature groups?
You could take a look at Tango's I/O by Kris Bell. It seems to have an efficient and clean design.
Jonathan M Davis
2010-10-12 09:56:05 UTC
Permalink
Post by pipe dream
Post by Andrei Alexandrescu
Post by Daniel Gibson
But parts of phobos are deprecated or will be deprecated and there
still is no alternative for them.
That may prevent people from writing "real" projects in D2 (or D at
all) - who wants to use classes that will be deprecated soon?
Sure, that old stuff will not be removed and can still be used, but I
personally feel a bit uncomfortable with using deprecated code.
Agreed. Maybe this is a good time to sart making a requirements list for
streams. What are the essential features/feature groups?
You could take a look at Tango's I/O by Kris Bell. It seems to have an
efficient and clean design.
Except that copying Tango is taboo. We want to avoid any possible accusation of
copying Tango's code or design. There have been issues in the past where Tango
devs thought that we might be doing that, and we just don't want to risk any
sort of problems with the Tango folks. So, whatever we put in Phobos, we do it
without looking at Tango.

- Jonathan M Davis
Simen kjaeraas
2010-10-12 11:08:13 UTC
Permalink
Post by Jonathan M Davis
Except that copying Tango is taboo. We want to avoid any possible
accusation of
copying Tango's code or design. There have been issues in the past where
Tango
devs thought that we might be doing that, and we just don't want to risk
any
sort of problems with the Tango folks. So, whatever we put in Phobos, we
do it
without looking at Tango.
You know, we might consider asking them for permission. That way, there
should be no problems.
--
Simen
Jonathan M Davis
2010-10-12 17:35:03 UTC
Permalink
Post by Simen kjaeraas
Post by Jonathan M Davis
Except that copying Tango is taboo. We want to avoid any possible
accusation of
copying Tango's code or design. There have been issues in the past where
Tango
devs thought that we might be doing that, and we just don't want to risk
any
sort of problems with the Tango folks. So, whatever we put in Phobos, we
do it
without looking at Tango.
You know, we might consider asking them for permission. That way, there
should be no problems.
That can be done, to be sure, but we definitely can't just look at their code -
or even API - and create something similar, and from what I recall, most cases
of trying to get permission have a been a problem (primarily due to there being
multiple authors, I think). If there's only one author for the stream code in
Tango, that would be easier.

Regardless, the point is that we can't just go and look at the Tango API and use
it to give ourselves ideas on what to do with Phobos.

- Jonathan M Davis
Walter Bright
2010-10-13 05:29:33 UTC
Permalink
Post by Simen kjaeraas
Post by Jonathan M Davis
So, whatever we put in Phobos,
we do it
without looking at Tango.
You know, we might consider asking them for permission. That way, there
should be no problems.
I have, on many occasions.
Jacob Carlborg
2010-10-13 10:42:20 UTC
Permalink
Post by Walter Bright
Post by Simen kjaeraas
Post by Jonathan M Davis
So, whatever we put in Phobos, we do it
without looking at Tango.
You know, we might consider asking them for permission. That way, there
should be no problems.
I have, on many occasions.
Druntime is basically the Tango runtime so apparently that worked out.
--
/Jacob Carlborg
Walter Bright
2010-10-13 17:42:02 UTC
Permalink
Post by Jacob Carlborg
Post by Walter Bright
Post by Simen kjaeraas
Post by Jonathan M Davis
So, whatever we put in Phobos, we do it
without looking at Tango.
You know, we might consider asking them for permission. That way, there
should be no problems.
I have, on many occasions.
Druntime is basically the Tango runtime so apparently that worked out.
The authors of some parts of Tango have graciously agreed to transfer their code
into Phobos. This includes the work done by Sean (druntime) and Don (math).
Daniel Gibson
2010-10-13 17:44:54 UTC
Permalink
Post by Walter Bright
Post by Jacob Carlborg
Post by Walter Bright
Post by Simen kjaeraas
Post by Jonathan M Davis
So, whatever we put in Phobos, we do it
without looking at Tango.
You know, we might consider asking them for permission. That way, there
should be no problems.
I have, on many occasions.
Druntime is basically the Tango runtime so apparently that worked out.
The authors of some parts of Tango have graciously agreed to transfer
their code into Phobos. This includes the work done by Sean (druntime)
and Don (math).
We should probably just ask the author if the stream code (Kris, according to their API doc) :)
Sean Kelly
2010-10-13 17:55:29 UTC
Permalink
Post by Daniel Gibson
Post by Walter Bright
Post by Jacob Carlborg
Post by Walter Bright
Post by Simen kjaeraas
Post by Jonathan M Davis
So, whatever we put in Phobos, we do it
without looking at Tango.
You know, we might consider asking them for permission. That way, there
should be no problems.
I have, on many occasions.
Druntime is basically the Tango runtime so apparently that worked out.
The authors of some parts of Tango have graciously agreed to transfer
their code into Phobos. This includes the work done by Sean (druntime)
and Don (math).
We should probably just ask the author if the stream code (Kris, according to their API doc) :)
Please excuse me if I don't hold my breath :-)
retard
2010-10-13 16:49:19 UTC
Permalink
Post by Jonathan M Davis
Post by Simen kjaeraas
Post by Jonathan M Davis
Except that copying Tango is taboo. We want to avoid any possible
accusation of
copying Tango's code or design. There have been issues in the past
where Tango
devs thought that we might be doing that, and we just don't want to
risk any
sort of problems with the Tango folks. So, whatever we put in Phobos,
we do it
without looking at Tango.
You know, we might consider asking them for permission. That way, there
should be no problems.
That can be done, to be sure, but we definitely can't just look at their
code - or even API - and create something similar, and from what I
recall, most cases of trying to get permission have a been a problem
(primarily due to there being multiple authors, I think). If there's
only one author for the stream code in Tango, that would be easier.
Regardless, the point is that we can't just go and look at the Tango API
and use it to give ourselves ideas on what to do with Phobos.
I doubt the copyright law can protect API definitions. In that case
projects like Wine (winehq.org) couldn't exist. What do you think?
Jonathan M Davis
2010-10-13 16:56:53 UTC
Permalink
Post by retard
Post by Jonathan M Davis
Post by Simen kjaeraas
Post by Jonathan M Davis
Except that copying Tango is taboo. We want to avoid any possible
accusation of
copying Tango's code or design. There have been issues in the past
where Tango
devs thought that we might be doing that, and we just don't want to
risk any
sort of problems with the Tango folks. So, whatever we put in Phobos,
we do it
without looking at Tango.
You know, we might consider asking them for permission. That way, there
should be no problems.
That can be done, to be sure, but we definitely can't just look at their
code - or even API - and create something similar, and from what I
recall, most cases of trying to get permission have a been a problem
(primarily due to there being multiple authors, I think). If there's
only one author for the stream code in Tango, that would be easier.
Regardless, the point is that we can't just go and look at the Tango API
and use it to give ourselves ideas on what to do with Phobos.
I doubt the copyright law can protect API definitions. In that case
projects like Wine (winehq.org) couldn't exist. What do you think?
I wouldn't think that it would be a problem, but I'm no expert, and we've had
problems in the past because Tango devs thought that proposed Phobos code was
too similar to Tango. So, as I understand it, unless we get specific permission
from the Tango devs which wrote a particular module, we're trying to not have
code in Phobos which is an API which is at all close to Tango's. That way we can
avoid potential conflicts with the Tango devs.

- Jonathan M Davis
Daniel Gibson
2010-10-13 17:13:47 UTC
Permalink
Post by retard
Post by Jonathan M Davis
Post by Simen kjaeraas
Post by Jonathan M Davis
Except that copying Tango is taboo. We want to avoid any possible
accusation of
copying Tango's code or design. There have been issues in the past
where Tango
devs thought that we might be doing that, and we just don't want to
risk any
sort of problems with the Tango folks. So, whatever we put in Phobos,
we do it
without looking at Tango.
You know, we might consider asking them for permission. That way, there
should be no problems.
That can be done, to be sure, but we definitely can't just look at their
code - or even API - and create something similar, and from what I
recall, most cases of trying to get permission have a been a problem
(primarily due to there being multiple authors, I think). If there's
only one author for the stream code in Tango, that would be easier.
Regardless, the point is that we can't just go and look at the Tango API
and use it to give ourselves ideas on what to do with Phobos.
I doubt the copyright law can protect API definitions. In that case
projects like Wine (winehq.org) couldn't exist. What do you think?
The problem is that for the windows API you don't see the implementation, so you can't have stolen
the code.
Tango's code however is available to anyone so it's a lot harder to prove that you only looked at
their API documention and not at their code.
And it's quite probable that your implementation looks similar to theirs - for standard stuff most
programmers (that may have seen any code doing something similar) will produce similar code - how do
you prove that you just did it the way that occurred natural to you and didn't copy their code?

So to be safe you have two possibilities:
1. don't model you API after Tango's
2. clone their API, look at their code and make sure yours is ridiculously different

Else it may turn out like the SHOO time code disaster...
klickverbot
2010-10-13 17:37:43 UTC
Permalink
Post by Jonathan M Davis
I wouldn't think that it would be a problem, but I'm no expert, and we've had
problems in the past because Tango devs thought that proposed Phobos code was
too similar to Tango. So, as I understand it, unless we get specific permission
from the Tango devs which wrote a particular module, we're trying to not have
code in Phobos which is an API which is at all close to Tango's. That way we can
avoid potential conflicts with the Tango devs.
- Jonathan M Davis
We had this over and over again, but I still think it should be noted
that the disaster around SOHO's code was not entirely made up by ?the
Tango devs?, but originated from a single developer's phone call to
Walter Bright and was then exaggerated by large parts of the D
community, including both ?sides? ? your statement(s) makes it look a
bit as if it was all Tango at fault there?
Jonathan M Davis
2010-10-13 18:21:53 UTC
Permalink
Post by klickverbot
Post by Jonathan M Davis
I wouldn't think that it would be a problem, but I'm no expert, and we've
had problems in the past because Tango devs thought that proposed Phobos
code was too similar to Tango. So, as I understand it, unless we get
specific permission from the Tango devs which wrote a particular module,
we're trying to not have code in Phobos which is an API which is at all
close to Tango's. That way we can avoid potential conflicts with the
Tango devs.
- Jonathan M Davis
We had this over and over again, but I still think it should be noted
that the disaster around SOHO's code was not entirely made up by ?the
Tango devs?, but originated from a single developer's phone call to
Walter Bright and was then exaggerated by large parts of the D
community, including both ?sides? ? your statement(s) makes it look a
bit as if it was all Tango at fault there?
I never "the" Tango devs, just Tango devs, so I'm not claiming anything about
all Tango devs. I'm not even really saying whether code was or wasn't copied,
but there are Tango devs who are very sensitive to anything that looks like it
might have been copied from Tango, and we want to avoid any misunderstandings or
issues that could arise from any Tango dev thinking that we're swiping their
code. So, we avoid doing anything that even looks similar, and many of us never
look at the Tango API at all, let alone the code. Whether any copying of any
kind has ever taken place is irrelevant at this point. What matters is that we
don't want to cause issues between the Phobos and Tango folks, so we need to
generally avoid anything that makes it look like we might be swiping anything
from Tango.

- Jonathan M Davis
Denis Koroskin
2010-10-12 11:04:28 UTC
Permalink
On Tue, 12 Oct 2010 02:32:55 +0400, Andrei Alexandrescu
Post by Andrei Alexandrescu
Post by Daniel Gibson
But parts of phobos are deprecated or will be deprecated and there still
is no alternative for them.
That may prevent people from writing "real" projects in D2 (or D at all)
- who wants to use classes that will be deprecated soon?
Sure, that old stuff will not be removed and can still be used, but I
personally feel a bit uncomfortable with using deprecated code.
Agreed. Maybe this is a good time to sart making a requirements list for
streams. What are the essential features/feature groups?
Andrei
For me, I/O should be scalable (and thus support async operations) so I
came up with my own implementation.
I've tried building it on top of std.concurrency, but it doesn't scale
either. So, once again, I had to implement my own message passing
mechanism. I can implement existing std.concurrency interface on top of my
own one without sacrificing anything, but not vice-versa).

Classes implemented so far: FileStream (file i/o), MemoryStream (e.g.
async memcpy) and SocketStream.

None of the streams support range interface explicitly (by design).
Instead, range interface can be achieved by StreamReader (InputStream) and
StreamWriter (OutputStream) adaptors.

Here it is in case you want to take a look and borrow ideas:

Stream: http://bitbucket.org/korDen/io/src/tip/io/stream.d
Mailbox: http://bitbucket.org/korDen/io/src/tip/io/mailbox.d
AsyncRequest: http://bitbucket.org/korDen/io/src/tip/io/async.d

Unlike std.concurrency, it is easy (and encouraged) to have as many
mailboxes as you need. Mailboxes can forward events to other mailboxes,
e.g. so that you can poll() only one mailbox (and not every single one of
them). And the main difference from std.concurrency is that it allows
event processing in a different thread context. For example, you can
process network message as soon as it arrives (which is done in a
background thread), parse it and then dispatch to main thread. This is how
my HttpRequest (which uses SocketStream) works.

Here is an example:

import io.http;
import io.mailbox;

import std.stdio;
import std.file;

void main()
{
auto host = "?????.??"; // supports international domain names, too

auto connection = new HttpConnection(host);

version (Wait) {
auto request = connection.execute(new HttpRequest(host, "/"));
request.wait(); // blocks until completed
std.file.write("out.html", request.response.contents);
} else {
// use thread-unique mailbox for event handing, similar to
std.concurrency.getTid()
connection.execute(new HttpRequest(host, "/"), mailbox);

bool done = false;
void onComplete(HttpResponseRequest request)
{
std.file.write("out.html", request.response.contents);
done = true;
}

mailbox.registerHandler!(HttpResponseRequest)(&onComplete);

version (Loop) {
while (!done) {
mailbox.poll(); // doesn't block
// do something useful (e.g. show progress bar) while not done
}
} else {
mailbox.poll(long.max);
assert(done);
}
}
connection.close();
}
Fawzi Mohamed
2010-10-12 12:01:30 UTC
Permalink
On Tue, 12 Oct 2010 02:32:55 +0400, Andrei Alexandrescu <SeeWebsiteForEmail at erdani.org
Post by Andrei Alexandrescu
Post by Daniel Gibson
But parts of phobos are deprecated or will be deprecated and there still
is no alternative for them.
That may prevent people from writing "real" projects in D2 (or D at all)
- who wants to use classes that will be deprecated soon?
Sure, that old stuff will not be removed and can still be used, but I
personally feel a bit uncomfortable with using deprecated code.
Agreed. Maybe this is a good time to sart making a requirements
list for streams. What are the essential features/feature groups?
Andrei
For me, I/O should be scalable (and thus support async operations)
so I came up with my own implementation.
I've tried building it on top of std.concurrency, but it doesn't
scale either. So, once again, I had to implement my own message
passing mechanism. I can implement existing std.concurrency
interface on top of my own one without sacrificing anything, but not
vice-versa).
I very much agree that IO should be scalable.
In my opinion this is possible if one has a robust framework for smp
parallelization.
This is what I have been working on with blip http://dsource.org/blip .
The API is fixed, and seem to work correctly in all the cases I
tested. It has not been optimized so it still has obvious
optimizations, but I wanted to have a bit more of code using it,
before thinking about optimization (so that I will be able to catch
wrong optimizations with a high probability).
Indeed I have already a rather large amount of code using it, and had
still cases where bugs were *very* rare and difficult to locate.
Still I already used it for example to build a socket server that uses
all available threads efficiently to implement rpc between processes.
One important thing in my opinion is that idle work should not use
resources: waiting for i/o, waiting for events uses basically no cpu.

All this is with D 1.0, and tango (even if I did try to reduce and
encapsulate the dependence on tango as much as possible).
For example when using sockets one does not see the blocking by
default (for a webserver one will likely want to add some timeout),
and the processor transparently switches to fibers that have work to do.
The programmer has to just think about tasks and things that can be
executed in parallel, if possible, the optimal scheduling and mapping
to threads is done automatically.

Fawzi
Denis Koroskin
2010-10-12 12:45:00 UTC
Permalink
Post by Fawzi Mohamed
Post by Denis Koroskin
On Tue, 12 Oct 2010 02:32:55 +0400, Andrei Alexandrescu
Post by Andrei Alexandrescu
Post by Daniel Gibson
But parts of phobos are deprecated or will be deprecated and there still
is no alternative for them.
That may prevent people from writing "real" projects in D2 (or D at all)
- who wants to use classes that will be deprecated soon?
Sure, that old stuff will not be removed and can still be used, but I
personally feel a bit uncomfortable with using deprecated code.
Agreed. Maybe this is a good time to sart making a requirements list
for streams. What are the essential features/feature groups?
Andrei
For me, I/O should be scalable (and thus support async operations) so I
came up with my own implementation.
I've tried building it on top of std.concurrency, but it doesn't scale
either. So, once again, I had to implement my own message passing
mechanism. I can implement existing std.concurrency interface on top of
my own one without sacrificing anything, but not vice-versa).
I very much agree that IO should be scalable.
In my opinion this is possible if one has a robust framework for smp
parallelization.
This is what I have been working on with blip http://dsource.org/blip .
The API is fixed, and seem to work correctly in all the cases I tested.
It has not been optimized so it still has obvious optimizations, but I
wanted to have a bit more of code using it, before thinking about
optimization (so that I will be able to catch wrong optimizations with a
high probability).
Indeed I have already a rather large amount of code using it, and had
still cases where bugs were *very* rare and difficult to locate.
Still I already used it for example to build a socket server that uses
all available threads efficiently to implement rpc between processes.
One important thing in my opinion is that idle work should not use
resources: waiting for i/o, waiting for events uses basically no cpu.
All this is with D 1.0, and tango (even if I did try to reduce and
encapsulate the dependence on tango as much as possible).
For example when using sockets one does not see the blocking by default
(for a webserver one will likely want to add some timeout), and the
processor transparently switches to fibers that have work to do.
The programmer has to just think about tasks and things that can be
executed in parallel, if possible, the optimal scheduling and mapping to
threads is done automatically.
Fawzi
I'm using a thread pool at this moment, however I was thinking about
implementing some kind of green threads recently.

In D2, there is a local/shared separation, and OS threads should be able
switch D local context, i.e. forward access to local variables through
custom TLS class reference, and swap that reference to reuse same thread
for many "green threads". Conceptually it is different from Fibers (i.e
you must return instead of just call yield() at any time), but it is also
easier to implement and pretty much portable to any platform.

Either way, it needs some language support, because currently TLS
implemented on compiler level rather than library level. I proposed this
change in past, but no one responded.
Walter Bright
2010-10-13 05:17:58 UTC
Permalink
Post by Denis Koroskin
Either way, it needs some language support, because currently TLS
implemented on compiler level rather than library level. I proposed this
change in past, but no one responded.
Using TLS needs operating system support, and there's special linker support for
it and the compiler has to generate TLS references in certain ways in order for
it to work.

There is no such support in OSX for TLS, and it is done manually by the library.
However, TLS access is 10 times slower than on Linux/Windows.

Without doing TLS in the operating system standard way, you tend to get hosed if
you link with a shared library/DLL that has TLS.
Denis Koroskin
2010-10-13 08:15:51 UTC
Permalink
On Wed, 13 Oct 2010 09:17:58 +0400, Walter Bright
Post by Walter Bright
Post by Denis Koroskin
Either way, it needs some language support, because currently TLS
implemented on compiler level rather than library level. I proposed
this change in past, but no one responded.
Using TLS needs operating system support, and there's special linker
support for it and the compiler has to generate TLS references in
certain ways in order for it to work.
There is no such support in OSX for TLS, and it is done manually by the
library. However, TLS access is 10 times slower than on Linux/Windows.
Without doing TLS in the operating system standard way, you tend to get
hosed if you link with a shared library/DLL that has TLS.
Aha. Thanks for pointing that out, it never occurred to me before. Means I
need to investigate Fibers for D2 then (thats the proper way anyway albeit
less portable).
Jacob Carlborg
2010-10-13 10:41:00 UTC
Permalink
Post by Walter Bright
Post by Denis Koroskin
Either way, it needs some language support, because currently TLS
implemented on compiler level rather than library level. I proposed
this change in past, but no one responded.
Using TLS needs operating system support, and there's special linker
support for it and the compiler has to generate TLS references in
certain ways in order for it to work.
There is no such support in OSX for TLS, and it is done manually by the
library. However, TLS access is 10 times slower than on Linux/Windows.
Without doing TLS in the operating system standard way, you tend to get
hosed if you link with a shared library/DLL that has TLS.
I don't know how you have implemented TLS on Mac OS X but it does
support TLS via the Posix API pthreads. This is the only page from
Apple's documentation I could find for now (I'm certain I've seen a
better page)
http://developer.apple.com/macosx/multithreadedprogramming.html .

According to these:
http://lifecs.likai.org/2010/05/mac-os-x-thread-local-storage.html and
http://lists.apple.com/archives/darwin-dev/2005/Sep/msg00005.html the
implementation of TLS in the Posix API on Mac OS X should be as fast as
the EFL implementation. As the blog post mentions, there is an inline
version of pthread_getspecific. I also have to add that I have no idea
if the pthreads can be used to implement TLS in the compiler.
--
/Jacob Carlborg
Walter Bright
2010-10-13 18:18:09 UTC
Permalink
Post by Jacob Carlborg
I don't know how you have implemented TLS on Mac OS X but it does
support TLS via the Posix API pthreads. This is the only page from
Apple's documentation I could find for now (I'm certain I've seen a
better page)
http://developer.apple.com/macosx/multithreadedprogramming.html .
Yeah, I know about pthreads TLS, but that's wholly inadequate.
Post by Jacob Carlborg
http://lifecs.likai.org/2010/05/mac-os-x-thread-local-storage.html and
http://lists.apple.com/archives/darwin-dev/2005/Sep/msg00005.html the
implementation of TLS in the Posix API on Mac OS X should be as fast as
the EFL implementation. As the blog post mentions, there is an inline
version of pthread_getspecific. I also have to add that I have no idea
if the pthreads can be used to implement TLS in the compiler.
With gcc on OSX, try this:

__thread int x;

It will fail. Furthermore, OSX has no documented way to allocate TLS static data
in the object file. I spent considerable effort figuring out a way to do this
and get around the numerous bugs in the OSX linker that tried to stop me.

There's good reason for Windows, Linux, FreeBSD, etc. to support the declaration
of TLS in the C source code.

BTW, the dates on the second article postdate the D TLS implementation by years.
Perhaps Apple has improved things. The third article just points out problems
with the TLS.

Anyhow, the source is here:

http://www.dsource.org/projects/druntime/browser/trunk/src/core/thread.d

and the function you're interested in is ___tls_get_addr. You're welcome to make
improvements to it.
Iain Buclaw
2010-10-11 10:15:45 UTC
Permalink
== Quote from Daniel Gibson (metalcaedes at gmail.com)'s article
Post by Daniel Gibson
Post by bioinfornatics
LDC support 64 bit ;)
as well as GDC.
But both currently lack an up-to-date D2 compiler (but the GDC guys are
at least working on it, seems like they're currently at 2.029 - which is
great - about 3 months ago they were still at 2.018 and in between was
the big 2.020 update that introduced druntime).
Merging frontend is one thing, implementing the new features the frontend offers
is another (though I think I've so far covered everything there ;).

I'm actually rather curious how DMD plans on doing varargs in 64bit. Not least
because I don't think their current method of "calculating" the address of the
_argptr takes 64bit parameter passing into account.
Walter Bright
2010-10-11 17:51:35 UTC
Permalink
Post by Iain Buclaw
I'm actually rather curious how DMD plans on doing varargs in 64bit. Not least
because I don't think their current method of "calculating" the address of the
_argptr takes 64bit parameter passing into account.
I'm working on it now. It's a bitch.

The way varargs works for the C ABI is inefficient, clumsy, and frankly won't
work for what D needs. So I'm going to have two varargs - one for extern (C)
which is compatible with the C ABI and one for extern (D) which will push things
on the stack as for 32 bits.

The only remaining nuisance for the D method is that some types have to be
aligned more strictly than the stack. That means TypeInfo will have to be
extended to provide an alignment size for the type.
Walter Bright
2010-10-10 20:13:40 UTC
Permalink
Post by Jonathan M Davis
Of course, projects like QtD suffer from the same sort of problem as a compiler
does in that it's not necessarily very useful until it's complete. Lots of
people may be interested in using QtD, but if it's not at least close to done,
it's not going to be useable enough to use in any major project, so people won't
use, they won't report bugs on it, and the won't give any kind of feedback on
the project. So, the poor QtD people then have to get a _lot_ of code done
before they see any kind of positive feedback from the community, and when they
_do_ start getting feedback, much of it is likely to be negative because feature
X hasn't been implemented yet or feature Y is buggy. A lot of people have given
up on D for similar reasons. Hopefully enough of the problems that they were
having with dmd get fixed soon enough that they're able to actually continue
working on the project without getting too frustrated over it.
Things sure have changed. Back in the 80's, people were able to get real
projects done with absolutely *terrible* compilers. Compilers have steadily
gotten better, and so have expectations.
Juanjo Alvarez
2010-10-11 07:09:06 UTC
Permalink
On Sun, 10 Oct 2010 00:46:47 -0700, Jonathan M Davis
Post by Jonathan M Davis
working on the project without getting too frustrated over it. QtD
is a huge
Post by Jonathan M Davis
service to the D community.
It is. Qt bindings were the first thing I looked for when I started
with the language, even with my current project not using any GUI.
Denis Koroskin
2010-10-05 19:59:07 UTC
Permalink
On Tue, 05 Oct 2010 20:29:03 +0400, Andrei Alexandrescu
Post by Andrei Alexandrescu
Post by Gour D.
On Tue, 05 Oct 2010 16:01:41 +0200
Don> I would estimate the truck factor as between 2.0 and 2.5. Two
Don> years ago, the truck factor was 1.0, but not any more.
Nice, nice...Still SO people say: "Neither Haskell nor D is popular
enough for it to be at all likely that you will ever attract a single
other developer to your project..." :-)
If developer attraction is a concern, you're likely better off with D.
Programmers who have used at least one Algol-like language (C, C++,
Java, C#) will have no problem feeling comfortable in D. With Haskell
you'd need to stick with "the choir".
Post by Gour D.
If just QtD hadn't been suspended...
I agree that's a bummer. I suggest you write the developers and ask what
would revive their interest. The perspective of a solid client is bound
to be noticeable.
Andrei
I've heard from one of the developers that one of the most frustrating
parts was inability of having struct default ctors
(http://d.puremagic.com/issues/show_bug.cgi?id=3852), and dtors that
aren't called (http://d.puremagic.com/issues/show_bug.cgi?id=3516). I also
know they also had *huge* issues with optlink
(http://d.puremagic.com/issues/show_bug.cgi?id=2436 and many others), but
those hopefully got fixed.

I'll try to talk eldar into sharing his development experience and the
issues they came across.

Until then you may be interested in reading these posts:

http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=103453
http://h3.gd/devlog/?p=22 - increasingly more people are unsatisfied with
D2 and talking about a fork so I wouldn't be surprised to see one sooner
or later (!)
bearophile
2010-10-05 20:15:39 UTC
Permalink
Post by Denis Koroskin
http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=103453
Yes, among the things D2 is designed to support, a there's a lack of built-in (or user-defined) basic features to create GUIs. But the good thing is that (I think) all the useful features here are additive changes.

Bye,
bearophile
Gour D.
2010-10-05 20:24:02 UTC
Permalink
On Tue, 05 Oct 2010 23:59:07 +0400
Denis> ...increasingly more people are unsatisfied with D2 and talking
Denis> about a fork so I wouldn't be surprised to see one sooner or
Denis> later (!)

Unsatisfied that D2 changed (too) much or with bugs and development in
general?

To me, it looks very early to fork D2...


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101005/3381a636/attachment.pgp>
Steven Schveighoffer
2010-10-06 10:36:29 UTC
Permalink
Post by Gour D.
On Tue, 05 Oct 2010 23:59:07 +0400
Denis> ...increasingly more people are unsatisfied with D2 and talking
Denis> about a fork so I wouldn't be surprised to see one sooner or
Denis> later (!)
Unsatisfied that D2 changed (too) much or with bugs and development in
general?
To me, it looks very early to fork D2...
I think the blog mentions forking D1 and adding certain useful features
from D2.

Forking D1 might be the only way to add new features to it. I think many
yearn for some of the useful, stable features of D2, but aren't willing to
put up with the not-yet-cooked features of D2. You don't have a choice at
the moment, if you want D2 features, you must use D2, and the only
up-to-date implementation of it -- dmd. On top of that, phobos is a long
way from being finished, but it is progressing rapidly, there are many
good developers working on it (as opposed to DMD which has 3).

I personally do not share that pessimistic view, and I don't think I'll
ever go back to D1 anyways, since the D1 phobos library is pretty
underpowered. D2/phobos2 will get there, perhaps in a year or so.

-Steve
Andrei Alexandrescu
2010-10-05 20:40:11 UTC
Permalink
Post by Denis Koroskin
http://h3.gd/devlog/?p=22 - increasingly more people are unsatisfied
with D2 and talking about a fork so I wouldn't be surprised to see one
sooner or later (!)
I'm confused. Reading through the link reveals exactly two people
mentioning that a fork would be good. That doesn't quite qualify for
"increasingly more people" in normal conversation.

Andrei
Jacob Carlborg
2010-10-06 07:29:57 UTC
Permalink
Post by Andrei Alexandrescu
Post by Gour D.
On Tue, 05 Oct 2010 16:01:41 +0200
Don> I would estimate the truck factor as between 2.0 and 2.5. Two
Don> years ago, the truck factor was 1.0, but not any more.
Nice, nice...Still SO people say: "Neither Haskell nor D is popular
enough for it to be at all likely that you will ever attract a single
other developer to your project..." :-)
If developer attraction is a concern, you're likely better off with D.
Programmers who have used at least one Algol-like language (C, C++,
Java, C#) will have no problem feeling comfortable in D. With Haskell
you'd need to stick with "the choir".
Post by Gour D.
If just QtD hadn't been suspended...
I agree that's a bummer. I suggest you write the developers and ask what
would revive their interest. The perspective of a solid client is bound
to be noticeable.
Andrei
Probably a compiler that is working better, one with fewer bugs.
--
/Jacob Carlborg
Simen kjaeraas
2010-10-06 09:43:04 UTC
Permalink
Post by Jacob Carlborg
Probably a compiler that is working better, one with fewer bugs.
Of course. And knowing which bugs those are, one could perhaps
fix them.
--
Simen
Jacob Carlborg
2010-10-06 11:25:47 UTC
Permalink
Post by Simen kjaeraas
Post by Jacob Carlborg
Probably a compiler that is working better, one with fewer bugs.
Of course. And knowing which bugs those are, one could perhaps
fix them.
Well that's the problem, fixing those bugs and you will encounter new
bugs. BTW, I wonder why Andrei asked anyway, he already asked this in
the thread "QtD is suspended" in the announce newsgroup and got an
answer:
http://www.digitalmars.com/d/archives/digitalmars/D/announce/QtD_is_suspended_19259.html#N19265
--
/Jacob Carlborg
Simen kjaeraas
2010-10-06 12:24:22 UTC
Permalink
Post by Jacob Carlborg
Well that's the problem, fixing those bugs and you will encounter new
bugs.
So we should all just give up, then?

Yes, there are bugs, and yes, fixing them will reveal new ones. But the
only reasonable thing to do is to try and fix those bugs that cause the
most headaches, and continue doing so.
--
Simen
Andrei Alexandrescu
2010-10-06 14:35:45 UTC
Permalink
Post by Simen kjaeraas
Post by Jacob Carlborg
Well that's the problem, fixing those bugs and you will encounter new
bugs.
So we should all just give up, then?
Yes, there are bugs, and yes, fixing them will reveal new ones. But the
only reasonable thing to do is to try and fix those bugs that cause the
most headaches, and continue doing so.
I agree that that looked like an unwinnable battle while D was also
evolving rapidly. Now that the language is stabilizing I expect the rate
and the absolute number of bugs will decrease.

Andrei
Justin Johansson
2010-10-07 13:24:52 UTC
Permalink
Post by Andrei Alexandrescu
Post by Simen kjaeraas
Post by Jacob Carlborg
Well that's the problem, fixing those bugs and you will encounter new
bugs.
So we should all just give up, then?
Yes, there are bugs, and yes, fixing them will reveal new ones. But the
only reasonable thing to do is to try and fix those bugs that cause the
most headaches, and continue doing so.
I agree that that looked like an unwinnable battle while D was also
evolving rapidly. Now that the language is stabilizing I expect the rate
and the absolute number of bugs will decrease.
Andrei
I think that long are the days that a single individual (or corporation)
can "own" a language. It will be an unwinnable battle so long as the
governance of D is under the auspices of a single mortal.

There is too much risk for investors of time let alone pecuniary
investors under the current regime.

Justin
Gour D.
2010-10-07 14:55:46 UTC
Permalink
On Fri, 08 Oct 2010 00:24:52 +1100
Justin> I think that long are the days that a single individual (or
Justin> corporation) can "own" a language. It will be an unwinnable
Justin> battle so long as the governance of D is under the auspices of
Justin> a single mortal.

I agree that, based on my short research about D, I see several
places with the label 'room for improvement' like (more) open
development process, using of DVCS, planning releases, better
organized web sites, up-to-date wiki, more docs etc.

Still, I believe that things are improving or, at least, there are
still people enthused with D. That's why I'm curious how to proceed my
evaluation phase in order to be able to properly decide for our
project? (Running 108 variations of "Hello world" is not adequate
test, neither of the language, nor for the tools.)

So, my question is whether is there some option to get more complete
coverage of the language (D2) besides TDPL book which will require
some time to arrive here in Croatia (I found that book is worthy
purchase no matter what we decide about D eventually)?

Justin> There is too much risk for investors of time let alone pecuniary
Justin> investors under the current regime.

I agree here...this is e.g. one of the reasons to cautious to invest
my time in learning & using ConTeXt (http://www.pragma-ade.com/)
seeing it practically as one-man-band mostly pushed by one developer
(Hans Hagen), no matter how talented he is and I'll continue using
LyX/LaTeX.

btw, my question on SO

(http://stackoverflow.com/questions/3863111/haskell-or-d-for-gui-desktop-application
or http://is.gd/fPK36)

got one nice response from Don Stewart who wrote: "Let's tease out
some requirements here, and I'll try to make the Haskell case. Perhaps
the D fans or others could try to do the same.", so it would be nice,
at least for other users, that some more experienced D user writes The
Case for D (no pun intended).


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101007/69aaa494/attachment.pgp>
bioinfornatics
2010-10-07 15:36:36 UTC
Permalink
For me i will use D1 while ldc do not support D2 or if GDC come a GCC
project and support D2. Until this is not done a big community part do
not go to D2.
Gour D.
2010-10-07 15:50:48 UTC
Permalink
On Thu, 7 Oct 2010 15:36:36 +0000 (UTC)
bioinfornatics> For me i will use D1 while ldc do not support D2 or if
bioinfornatics> GDC come a GCC project and support D2. Until this is
bioinfornatics> not done a big community part do not go to D2.

Hmm...based on that I saw, I consider that D2 features are much more
compelling when comparing D with Haskell...Hopefully, in a few months
some things may change in D2 arena...


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101007/0ce319b5/attachment.pgp>
Jacob Carlborg
2010-10-07 08:38:06 UTC
Permalink
Post by Simen kjaeraas
Post by Jacob Carlborg
Well that's the problem, fixing those bugs and you will encounter new
bugs.
So we should all just give up, then?
Yes, there are bugs, and yes, fixing them will reveal new ones. But the
only reasonable thing to do is to try and fix those bugs that cause the
most headaches, and continue doing so.
I just think it has been show several times that D2 is not ready to be
used just yet.
--
/Jacob Carlborg
bioinfornatics
2010-10-05 13:58:44 UTC
Permalink
LDC is free compiler last revision use dmdfe 1.063 and soon the ladt dmdfe
ldc support 32, 64 and arm and works on linux, mac OS X and windows
LDC is wrote in c++ using LLVM, all people who whant help this project are welcome.
They are a D2 experimental branch (you can too come for help this part )
Jesse Phillips
2010-10-05 20:14:49 UTC
Permalink
Every language you choose will have its issue, and your listed requirements are basically set up so the answer is 'none.' Not one language fulfills every requirement for various reasons.

It sounds to me like you want to use Haskel but due to the loss in potential help you are looking at D. You say there were a few people who didn't get on board because of Haskel, ask them if instead D was used would the join. If not go with Haskel, if yes than I wish to make on thing clear.

You will have to support the libraries you use or wait patiently. I am not saying there isn't code to use or people willing/working to support that code. I mean that we are a small community and have had some ruff hills to clime. Lots of libraries/tools were written for D1, lots of libraries were written using Tango (D1 only). Even when effort was taken to support D2, the result was a broken library on a future compiler release (it was expected as it was clearly stated, no longer the issue).

So even if something claims support for D2 it may require work to bring it up to the latest language specs. But the community is working, things are progressing. And I don't think D will be dieing. The more libraries D has, the more users will come and use the language resulting in more libraries. Right now we are missing some vital libraries that everyone wants, but no one wants to make/maintain (usually do to time issues).

In relation to compilers[1] I would not worry about it. By the time your project reaches a stage that must work an more platforms, it is likely that LDC and GDC, and maybe even DMD will be updated for D2 and a multitude of platforms. But for a GUI library, you would probably need to choose one and progress it as you do the project.

I do not wish to discourage you from choosing D, but feel it is always good to know what to expect. And really you could probably start your project, test the components you wish to use. It should be pretty obvious if you find something you couldn't live with.


1. http://www.wikiservice.at/d/wiki.cgi?D__Tutorial/StartingWithD/Compiler
Gour D.
2010-10-05 20:38:11 UTC
Permalink
On Tue, 05 Oct 2010 16:14:49 -0400
Jesse> Every language you choose will have its issue, and your listed
Jesse> requirements are basically set up so the answer is 'none.' Not
Jesse> one language fulfills every requirement for various reasons.

True.

Jesse> It sounds to me like you want to use Haskel but due to the loss
Jesse> in potential help you are looking at D.

Well, I'm somewhat familiar with Haskell and have zero experience with
D. Not even knowing which build system one use to develop.

Jesse> You will have to support the libraries you use or wait
Jesse> patiently. I am not saying there isn't code to use or people
Jesse> willing/working to support that code. I mean that we are a small
Jesse> community and have had some ruff hills to clime.

Right. I'm ready to help, according to my skills, the projects
providing bindings as I'd do to e.g. help qthaskell project, but with
Haskell one, at least, does not start from the scratch.

Here, I'd choose D, it is not certain that QtD will be revived at any
time in the future.

Jesse> Lots of libraries/tools were written for D1, lots of libraries
Jesse> were written using Tango (D1 only). Even when effort was taken
Jesse> to support D2, the result was a broken library on a future
Jesse> compiler release (it was expected as it was clearly stated, no
Jesse> longer the issue).

Otoh, GHC is quite stable and 7.0 is just around the corner...

Jesse> So even if something claims support for D2 it may require work
Jesse> to bring it up to the latest language specs. But the community
Jesse> is working, things are progressing. And I don't think D will be
Jesse> dieing.

But I already heard rumors about the fork of D2?

Jesse>The more libraries D has, the more users will come and
Jesse> use the language resulting in more libraries. Right now we are
Jesse> missing some vital libraries that everyone wants, but no one
Jesse> wants to make/maintain (usually do to time issues).

I'm following Haskell community for quite some time and I know when
there was no Hackage at all, iow I can understand what it means
'young' community. However, it looks that D1 --> D2 has brought
different winds, altgough it will probably pay off later.

Jesse> In relation to compilers[1] I would not worry about it. By the
Jesse> time your project reaches a stage that must work an more
Jesse> platforms, it is likely that LDC and GDC, and maybe even DMD
Jesse> will be updated for D2 and a multitude of platforms.

Correct. Having choice for compiler is great and makes one peaceful.

Jesse> But for a GUI library, you would probably need to choose one
Jesse> and progress it as you do the project.

This is a big problem for a desktop GUI project making the whole
project very uncertain right from the beginning...

Jesse> I do not wish to discourage you from choosing D, but feel it is
Jesse> always good to know what to expect. And really you could
Jesse> probably start your project, test the components you wish to
Jesse> use. It should be pretty obvious if you find something you
Jesse> couldn't live with.

As I wrote earlier, besides sqlite3 bindings and using existent C-lib
which needs to be bound to D, we need to write lot of native libs for
our app, but it all fails without proper GUI.


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101005/877b28c5/attachment.pgp>
Daniel Gibson
2010-10-05 20:46:26 UTC
Permalink
Post by Gour D.
On Tue, 05 Oct 2010 16:14:49 -0400
Jesse> You will have to support the libraries you use or wait
Jesse> patiently. I am not saying there isn't code to use or people
Jesse> willing/working to support that code. I mean that we are a small
Jesse> community and have had some ruff hills to clime.
Right. I'm ready to help, according to my skills, the projects
providing bindings as I'd do to e.g. help qthaskell project, but with
Haskell one, at least, does not start from the scratch.
Here, I'd choose D, it is not certain that QtD will be revived at any
time in the future.
What about using gtk instead of qt?
GtkD is, according to their homepage (
http://www.dsource.org/projects/gtkd ) and
http://www.prowiki.org/wiki4d/wiki.cgi?GuiLibraries stable and usable.

BTW: the quotation style of your email/ng-client is strange.
Gour D.
2010-10-06 06:18:57 UTC
Permalink
On Tue, 05 Oct 2010 22:46:26 +0200
Daniel> What about using gtk instead of qt?

I was considering gtk on haskell-side as well, but considering we'd
like to have 'lite' version of the application available on mobile
platform (Meego), it seems that it's better to have (more) common
codebase provided by Qt.

Moreover, we were told by dev working on gtk port for Mac that wx is
more suitable if Mac OS is important for us.

Since there is no wxQt, Qt emerged as clean winner here...

Well it's ? la Gnus. You have never seen it or you are thinking
something else?
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101006/cdcbef2b/attachment.pgp>
Anders F Björklund
2010-10-06 07:30:36 UTC
Permalink
Post by Gour D.
Moreover, we were told by dev working on gtk port for Mac that wx is
more suitable if Mac OS is important for us.
Probably right, even though wxWidgets for Cocoa is still in alpha.
GTK+ "works" (especially X11), but it doesn't really look "native".

wxWidgets for Carbon ("wxMac") is stable, but it is 32-bit only...
You can find old wx bindings for D at http://wxd.sourceforge.net/
Post by Gour D.
Since there is no wxQt, Qt emerged as clean winner here...
But see http://wxwidgets.blogspot.com/2010/07/gsoc-2010-mid-term.html

And http://wiki.wxwidgets.org/WxQt for the current state of the port.

--anders
Gour D.
2010-10-06 16:29:42 UTC
Permalink
On Wed, 06 Oct 2010 09:30:36 +0200
Anders> http://wxwidgets.blogspot.com/2010/07/gsoc-2010-mid-term.html
Anders>
Anders> And http://wiki.wxwidgets.org/WxQt for the current state of the
Anders> port.

Thank you for those links. It is very nice to see wxQT getting a nice
shape, but the problem is with wxD - it's D1 only and I'm sure that,
considering we don't have D1 legacy, there is no reasonable to think
about D1 (especially in comparison with Haskell's features).


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101006/2bf81811/attachment.pgp>
Anders F Björklund
2010-10-06 18:50:34 UTC
Permalink
Post by Gour D.
Anders> http://wxwidgets.blogspot.com/2010/07/gsoc-2010-mid-term.html
Anders>
Anders> And http://wiki.wxwidgets.org/WxQt for the current state of the
Anders> port.
Thank you for those links. It is very nice to see wxQT getting a nice
shape, but the problem is with wxD - it's D1 only and I'm sure that,
considering we don't have D1 legacy, there is no reasonable to think
about D1 (especially in comparison with Haskell's features).
wxD should build with D2 and Tango, just that I don't feel like
supporting them any more. It only does the 2.6 version API, too.

That mostly means it was written for GDC (with Phobos) and wx 2.8;
not that it couldn't be updated to D2 and wx3, if you wanted to ?

http://wxd.sourceforge.net/#d2
http://wxd.sourceforge.net/#tango

It's not a port (from C++), though. And that is a design decision.
Also means all languages can share the system wxWidgets libraries.

If you wanted to *port* it to D, you would need to also port all
the system headers too (some of which are in C++ and Objective-C)

DWT would be better, in that case:
http://wxd.sourceforge.net/#dwt

The idea was to have an open source compiler and GUI and IDE bundle.
Cross-platform, so that it would work on Windows, Macintosh, Unix.

I think if you are starting over with D2, that Qt is a better choice.
It is LGPL now, and looks native enough. And it doesn't need porting.

http://wxd.sourceforge.net/#screenshots
http://wxd.sourceforge.net/#codeblocks

--anders
Jacob Carlborg
2010-10-07 08:41:45 UTC
Permalink
Post by Anders F Björklund
http://wxd.sourceforge.net/#dwt
--anders
Since SWT 3.4 (currently stable version is 3.6) there has been a Cocoa
port available. DWT is a port of the Cocoa version.
--
/Jacob Carlborg
Anders F Björklund
2010-10-07 11:08:28 UTC
Permalink
Post by Jacob Carlborg
Post by Anders F Björklund
http://wxd.sourceforge.net/#dwt
Since SWT 3.4 (currently stable version is 3.6) there has been a Cocoa
port available. DWT is a port of the Cocoa version.
Yeah, that page is a bit outdated. But I'll try to fix.

When it was first written, it was Phobos and Carbon :-)

--anders
Jesse Phillips
2010-10-05 23:26:58 UTC
Permalink
Post by Gour D.
Jesse> But for a GUI library, you would probably need to choose one
Jesse> and progress it as you do the project.
This is a big problem for a desktop GUI project making the whole
project very uncertain right from the beginning...
What I meant was that there is something created and functioning (I really don't know how much) which may be just what you need. But I can not state that cross-platform GUI is available for D at this time.
Gour D.
2010-10-09 19:34:11 UTC
Permalink
On Tue, 05 Oct 2010 19:26:58 -0400
Jesse> But I can not state that cross-platform GUI is available for D
Jesse> at this time.

Despite of that we've decided to take a risk and choose with D &
(hopefully it will become ready) QtD.

Thanks to all who replied and see you in newer threads. ;)


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101009/a9cfd101/attachment.pgp>
Simen kjaeraas
2010-10-09 21:23:39 UTC
Permalink
Post by Gour D.
On Tue, 05 Oct 2010 19:26:58 -0400
Jesse> But I can not state that cross-platform GUI is available for D
Jesse> at this time.
Despite of that we've decided to take a risk and choose with D &
(hopefully it will become ready) QtD.
We're glad to hear that. Welcome aboard!
--
Simen
Jacob Carlborg
2010-10-06 07:35:13 UTC
Permalink
Post by Gour D.
As I wrote earlier, besides sqlite3 bindings and using existent C-lib
which needs to be bound to D, we need to write lot of native libs for
our app, but it all fails without proper GUI.
Sincerely,
Gour
Have you considered DWT? It's a port of the Eclipse project SWT, a
cross-platform GUI library. The windows and linux ports are usable,
although a little behind SWT. As for Mac OS X, the Cocoa port is a work
in progress, somethings are usable and somethings are not.

http://dsource.org/projects/dwt
--
/Jacob Carlborg
Gour D.
2010-10-06 16:31:17 UTC
Permalink
On Wed, 06 Oct 2010 09:35:13 +0200
Jacob> Have you considered DWT? It's a port of the Eclipse project SWT,
Jacob> a cross-platform GUI library. The windows and linux ports are
Jacob> usable, although a little behind SWT. As for Mac OS X, the Cocoa
Jacob> port is a work in progress, somethings are usable and somethings
Jacob> are not.

I got same suggestion on #d yesterday and went to see it
today. Interesting, but, afaics, no port for D2 which rules it out.


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101006/3402d1a4/attachment.pgp>
Denis Koroskin
2010-10-06 16:35:57 UTC
Permalink
Post by Gour D.
On Wed, 06 Oct 2010 09:35:13 +0200
Jacob> Have you considered DWT? It's a port of the Eclipse project SWT,
Jacob> a cross-platform GUI library. The windows and linux ports are
Jacob> usable, although a little behind SWT. As for Mac OS X, the Cocoa
Jacob> port is a work in progress, somethings are usable and somethings
Jacob> are not.
I got same suggestion on #d yesterday and went to see it
today. Interesting, but, afaics, no port for D2 which rules it out.
Sincerely,
Gour
Did you try DWT2? http://hg.dsource.org/projects/dwt2
Gour D.
2010-10-06 18:53:04 UTC
Permalink
On Wed, 06 Oct 2010 20:35:57 +0400
Denis> Did you try DWT2? http://hg.dsource.org/projects/dwt2

No, but 'doob' on #dwt (it seems this is Jacob Carlborg) replied my
question: "hi, is dwt available for d2?" with "no, not currently", so
I bet I saved some time trying it for myself.


Sincerely,
Gour
--
Gour | Hlapicina, Croatia | GPG key: CDBF17CA
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20101006/93e3aa58/attachment.pgp>
Jacob Carlborg
2010-10-07 08:42:22 UTC
Permalink
Post by Gour D.
On Wed, 06 Oct 2010 20:35:57 +0400
Denis> Did you try DWT2? http://hg.dsource.org/projects/dwt2
No, but 'doob' on #dwt (it seems this is Jacob Carlborg) replied my
question: "hi, is dwt available for d2?" with "no, not currently", so
I bet I saved some time trying it for myself.
Sincerely,
Gour
Exactly, hopefully I can get DWT to work with D2.
--
/Jacob Carlborg
bearophile
2010-10-11 18:13:31 UTC
Permalink
Post by Walter Bright
The way varargs works for the C ABI is inefficient, clumsy, and frankly won't
work for what D needs. So I'm going to have two varargs - one for extern (C)
which is compatible with the C ABI and one for extern (D) which will push things
on the stack as for 32 bits.
Recently I have seen something related on Reddit:

http://blog.nelhage.com/2010/10/amd64-and-va_arg/

http://www.reddit.com/r/programming/comments/dmm05/til_va_arg_on_amd64_is_way_more_complex_than_i/
Post by Walter Bright
The only remaining nuisance for the D method is that some types have to be
aligned more strictly than the stack. That means TypeInfo will have to be
extended to provide an alignment size for the type.
Is this somewhat related to this enhancement request "Guarantee alignment of stack-allocated variables on x86"?
http://d.puremagic.com/issues/show_bug.cgi?id=2278

Bye,
bearophile
Sean Kelly
2010-10-12 22:54:31 UTC
Permalink
Post by Denis Koroskin
On Tue, 12 Oct 2010 02:32:55 +0400, Andrei Alexandrescu
For me, I/O should be scalable (and thus support async operations) so I
came up with my own implementation.
I've tried building it on top of std.concurrency, but it doesn't scale
either. So, once again, I had to implement my own message passing
mechanism. I can implement existing std.concurrency interface on top of my
own one without sacrificing anything, but not vice-versa).
...
Post by Denis Koroskin
Unlike std.concurrency, it is easy (and encouraged) to have as many
mailboxes as you need. Mailboxes can forward events to other mailboxes,
e.g. so that you can poll() only one mailbox (and not every single one of
them).
Pretty cool. I'm not sure I would consider poll() scalable though. Or at least not to the levels where my concept of "scalable" applies. That aside, I think the correct approach is to have an IO API that stands by itself and then provide a few adaptors for integrating it with messaging in different ways. That will allow performance-minded folks to do their to-the-metal programming and convenience-minded folks to unify IO with the rest of their message-passing. This is one thing that I feel Erlang got wrong. The IO-messaging integration is too magic, and ends up being problematic (try handing control of a socket from one process to another, for example, without losing any in-flight messages in the process). In short, I think I agree with you than std.concurrency isn't exactly right for this. At least as the default interface at any rate.
Sean Kelly
2010-10-13 17:52:02 UTC
Permalink
Post by Jacob Carlborg
Post by Walter Bright
Post by Denis Koroskin
Either way, it needs some language support, because currently TLS
implemented on compiler level rather than library level. I proposed
this change in past, but no one responded.
Using TLS needs operating system support, and there's special linker
support for it and the compiler has to generate TLS references in
certain ways in order for it to work.
There is no such support in OSX for TLS, and it is done manually by the
library. However, TLS access is 10 times slower than on Linux/Windows.
Without doing TLS in the operating system standard way, you tend to get
hosed if you link with a shared library/DLL that has TLS.
I don't know how you have implemented TLS on Mac OS X but it does
support TLS via the Posix API pthreads. This is the only page from
Apple's documentation I could find for now (I'm certain I've seen a
better page)
http://developer.apple.com/macosx/multithreadedprogramming.html .
Using the Posix API here is tricky because it isn't always documented where the memory resides, so enabling GC scanning of it can be tricky. Also, default initialization gets a bit weird, etc. What we really need is for OSX to support the __thread storage class. We're not alone though, so I think it's likely we'll see it implemented in a future release.
Continue reading on narkive:
Loading...