Discussion:
RFC: scope and borrowing
via Digitalmars-d
2014-08-24 13:14:43 UTC
Permalink
In the "Opportunities for D" thread, Walter again mentioned the
topics ref counting, GC, uniqueness, and borrowing, from which a
lively discussion developed [1]. I took this thread as an
opportunity to write down some ideas about these topics. The
result is a rather extensive proposal for the implementation of
borrowing, and its implementations:

http://wiki.dlang.org/User:Schuetzm/scope

This is not a real DIP, but before I put more work into
formalizing it, I'd like to hear some thoughts from the languages
gurus here:

* Is this the general direction we want to go? Is it acceptable
in general?
* Is the proposal internally consistent?
* How big would the effort to implement it be? (I suspect it's a
large amount of work, but relatively straightforward.)

[1] http://forum.dlang.org/thread/lphnen$1ml7$1 at digitalmars.com
via Digitalmars-d
2014-08-25 01:09:58 UTC
Permalink
Cool initiative!

It is probably a good idea to look at what other languages with
linear type systems are doing:

http://en.m.wikipedia.org/wiki/Substructural_type_system

My gut feeling is that borrowing in D will suffer the same
problems as @safe without a high level IR that is proven correct.
Meaning, if the IR is correct you can add new language features
without breaking the type system.
via Digitalmars-d
2014-08-25 11:09:46 UTC
Permalink
On Monday, 25 August 2014 at 01:09:59 UTC, Ola Fosheim GrÞstad
Post by via Digitalmars-d
Cool initiative!
It is probably a good idea to look at what other languages with
http://en.m.wikipedia.org/wiki/Substructural_type_system
The idea is mostly inspired by Rust, but it came out very
different. Rust's borrowing rules are way more complicated. It
also has to deal with move semantics, for example. And it
annotates lifetimes, not owners, and thereby gets
non-hierarchical dependencies, as far as I understand it. I
believe this proposal is close to the best we can achieve without
resorting to data flow analysis.
Post by via Digitalmars-d
My gut feeling is that borrowing in D will suffer the same
correct. Meaning, if the IR is correct you can add new language
features without breaking the type system.
I'm unfortunately not familiar with the theoretical foundations
of type systems, and formal logic. So I'm not sure what exactly
you mean here. It seems to me the problems with @safe are with
the way it was implemented (allow everything _except_ certain
things that are known to be bad, instead of the opposite way:
allow nothing at first, but lift restrictions where it's safe to
do so), they are not conceptual.

But if you see potential problems, or know a way to avoid them,
this is exactly the kind of thing I'd like to see discussed.
via Digitalmars-d
2014-08-25 15:38:08 UTC
Permalink
I believe this proposal is close to the best we can achieve
without resorting to data flow analysis.
I agree with bearophile, perhaps data flow analysis would be
desirable. So I think it would be a good idea to hold the door
open on that.
I'm unfortunately not familiar with the theoretical foundations
of type systems, and formal logic. So I'm not sure what exactly
I don't know all that much about linear type systems, but this is
an opportunity to learn more for all of us! :-)
the way it was implemented (allow everything _except_ certain
allow nothing at first, but lift restrictions where it's safe
to do so), they are not conceptual.
Because real programming languages are hard to reason about it is
difficult to say where things break down. So one usually will map
the constructs of the language onto something simpler and more
homogeneous that is easier to reason about.

When it comes to @safe I think the main problem is that D makes
decisions on constructs and not on the "final semantics" of the
program. If you have a dedicated high level IR you can accept any
program segment that can be proven to be memory safe in terms of
the IR. The point is to have an IR that is less complicated than
the full language, but that retains needed information that is
lost in a low level IR and which you need to prove memory safety.

memset() is not unsafe per se, it is unsafe with the wrong
parameters. So you have to prove that the parameters are in the
safe region. Same thing with freeing memory and borrowed pointers
etc. You need a correctness proof even if you give up on
generality.

You may reject many safe programs but at least verify as many
simple safe programs as you can.

A bonus of having a high level IR is that you more easily can
combine languages with fewer interfacing problems. That would be
an advantage if you want a DSL to cooperate with D.
But if you see potential problems, or know a way to avoid them,
this is exactly the kind of thing I'd like to see discussed.
The problem is that D is too complex to reason about with any
reasonable level of confidence. So you need to reduce the
language to a level where you can reason about it with confidence
and build the other constructs on top of that.

(That way you don't have to reason about the combinatorial
explosion of constructs, just the building blocks).

Ola.
via Digitalmars-d
2014-08-25 18:14:02 UTC
Permalink
On Monday, 25 August 2014 at 15:38:09 UTC, Ola Fosheim GrÞstad
Post by via Digitalmars-d
I believe this proposal is close to the best we can achieve
without resorting to data flow analysis.
I agree with bearophile, perhaps data flow analysis would be
desirable. So I think it would be a good idea to hold the door
open on that.
I definitely don't want to exclude anything. But we need to find
out whether the additional complexity of full-blown DFA is really
necessary, see my reply to bearophile.
Post by via Digitalmars-d
I'm unfortunately not familiar with the theoretical
foundations of type systems, and formal logic. So I'm not sure
what exactly
I don't know all that much about linear type systems, but this
is an opportunity to learn more for all of us! :-)
the way it was implemented (allow everything _except_ certain
allow nothing at first, but lift restrictions where it's safe
to do so), they are not conceptual.
Because real programming languages are hard to reason about it
is difficult to say where things break down. So one usually
will map the constructs of the language onto something simpler
and more homogeneous that is easier to reason about.
decisions on constructs and not on the "final semantics" of the
program. If you have a dedicated high level IR you can accept
any program segment that can be proven to be memory safe in
terms of the IR. The point is to have an IR that is less
complicated than the full language, but that retains needed
information that is lost in a low level IR and which you need
to prove memory safety.
memset() is not unsafe per se, it is unsafe with the wrong
parameters. So you have to prove that the parameters are in the
safe region. Same thing with freeing memory and borrowed
pointers etc. You need a correctness proof even if you give up
on generality.
You may reject many safe programs but at least verify as many
simple safe programs as you can.
A bonus of having a high level IR is that you more easily can
combine languages with fewer interfacing problems. That would
be an advantage if you want a DSL to cooperate with D.
But this would require knowledge about the inner workings of
memset() to be part of the IR, or memset() to be implemented in
it. The same IR (or an equivalent one) would then need to be part
of language specification, otherwise different compilers would
allow different operations.

IMO the general idea of the current design is not okay, because
it's easy to implement, and easy to specify (a whitelist, or as
currently, blacklist approach). If something has been overlooked,
it can be added incrementally; at the same time there's always
@trusted to let the programmer specify what the compiler can't
prove.
via Digitalmars-d
2014-08-25 18:42:15 UTC
Permalink
Post by via Digitalmars-d
But this would require knowledge about the inner workings of
memset() to be part of the IR, or memset() to be implemented in
it.
The latter. You might have "zerofill(ptr,length)" and turn it
Post by via Digitalmars-d
The same IR (or an equivalent one) would then need to be part
of language specification, otherwise different compilers would
allow different operations.
I think you could have a low performance minimum requirement
reference implementation for this between the front-end and the
back-end.

As in, not written for performance, but as a baseline for testing
the real implementation.
Post by via Digitalmars-d
currently, blacklist approach). If something has been
overlooked, it can be added incrementally; at the same time
compiler can't prove.
I think borrow, @safe and other types of correctness oriented
analysis aspects should be seen as two aspects of the same thing.
So it is desirable to design it as a whole IMO.

Especially if assert() is going to turn into assume(). Which I
believe will only work reliably for preconditions on a code-scope
up to the postconditions, but I believe you need some heavy duty
analysis of the call-chain to get anywhere.

I presume you can do the same for allocations between
preconditions and postconditions and that could let you establish
@safe for a wider range of constructs this way.

I.e. the work on assert->assume for preconditions could lead to a
wider range of @safe and also perhaps automatic memory allocation
optimizations.

It sounds reasonable that constraining the input sometimes would
give information that could help on establishing @safe on code
that would otherwise have to be assume unsafe.

Ola.
Dominikus Dittes Scherkl via Digitalmars-d
2014-08-25 09:54:56 UTC
Permalink
Post by via Digitalmars-d
http://wiki.dlang.org/User:Schuetzm/scope
I like this.
It removes pretty much all of the need for manual memory
management for me. I would love to see this implemented.
Rikki Cattermole via Digitalmars-d
2014-08-25 10:06:08 UTC
Permalink
In the "Opportunities for D" thread, Walter again mentioned the topics
ref counting, GC, uniqueness, and borrowing, from which a lively
discussion developed [1]. I took this thread as an opportunity to write
down some ideas about these topics. The result is a rather extensive
http://wiki.dlang.org/User:Schuetzm/scope
This is not a real DIP, but before I put more work into formalizing it,
* Is this the general direction we want to go? Is it acceptable in general?
* Is the proposal internally consistent?
* How big would the effort to implement it be? (I suspect it's a large
amount of work, but relatively straightforward.)
[1] http://forum.dlang.org/thread/lphnen$1ml7$1 at digitalmars.com
Have you considered what happens when you cast away scope?
I didn't read anything about that, unless I missed something.

I do have to ask this, because what if you wanted to optionally return a
scoped reference?
via Digitalmars-d
2014-08-25 11:27:10 UTC
Permalink
Post by Rikki Cattermole via Digitalmars-d
Have you considered what happens when you cast away scope?
I didn't read anything about that, unless I missed something.
Right, this should be specified. But it's fairly obvious what
would happen: you lose the safety guarantees (=> @system), and
you're responsible to make sure you don't get dangling
references. You'd basically get what you have with normal
pointers. I don't think it has any deeper implications than that,
but I'm not sure there aren't any obscure optimizations the
compiler could make based on scope. If there are, it's no
different from casting away const, in this respect.
Post by Rikki Cattermole via Digitalmars-d
I do have to ask this, because what if you wanted to optionally
return a scoped reference?
Well, it's part of the return type, so it's either scoped, or it
isn't. You could return a tuple with a scoped and a non-scoped
pointer, though.

If it's only about returning a non-scoped pointer from a function
with a scoped return type, that's fine, because it's adding
scope, not removing it. (Use case: ScopeBuffer.)
Kagamin via Digitalmars-d
2014-08-25 10:27:52 UTC
Permalink
Can't scope(int*) on return type be equivalent to
scope!(a,b)(int*)? How often this is not desired?
via Digitalmars-d
2014-08-25 11:48:17 UTC
Permalink
Post by Kagamin via Digitalmars-d
Can't scope(int*) on return type be equivalent to
scope!(a,b)(int*)? How often this is not desired?
You mean as a default?

It would be desired in `chooseStringAtRandom`, but not in the
`findSubstring`, whose returned string shouldn't be limited by
the scope of the needle. If it is made the default, there would
need to be a way to opt out, such as removing an owner.

But note that in the `chooseStringAtRandom` example, it would be
sufficient to declare it as:

scope chooseStringAtRandom(scope(string) a, scope(string) b) {
return random() % 2 == 0 ? a : b;
}

Type deduction would then automatically add `a` and `b` as the
owners, by the rules under "Owner tracking".
Kagamin via Digitalmars-d
2014-09-21 09:15:44 UTC
Permalink
Post by via Digitalmars-d
It would be desired in `chooseStringAtRandom`, but not in the
`findSubstring`, whose returned string shouldn't be limited by
the scope of the needle. If it is made the default, there would
need to be a way to opt out, such as removing an owner.
If `in` will be defined as scope!callee, i.e. can't be returned,
then it can be removed from the return type.
bearophile via Digitalmars-d
2014-08-25 15:09:31 UTC
Permalink
Post by via Digitalmars-d
http://wiki.dlang.org/User:Schuetzm/scope
It looks nice. But perhaps it needs some kind of proof of
correctness.

Have you read the old blog posts (written before the creation of
Rust) by Bartosz Milewski regarding the borrowing in D?
Post by via Digitalmars-d
Implementation of this feature is possible without doing flow
control or interprocedural analysis.<
I remember that Walter has recently said that he's willing to add
some kind of flow analysis to the D front-end.

Bye,
bearophile
via Digitalmars-d
2014-08-25 18:03:20 UTC
Permalink
Post by bearophile via Digitalmars-d
Post by via Digitalmars-d
http://wiki.dlang.org/User:Schuetzm/scope
It looks nice. But perhaps it needs some kind of proof of
correctness.
Hmm... First there's the assignment rules. They make sure that
nothing with a shorter lifetime ends up in a variable with a
longer lifetime designation. The other part is to proof that the
type deduction and argument matching rules work.

Both parts are not difficult to reason about, but I don't know
what a formal proof needs to look like exactly. (How formal do we
need to go?)
Post by bearophile via Digitalmars-d
Have you read the old blog posts (written before the creation
of Rust) by Bartosz Milewski regarding the borrowing in D?
No, can you point me to them? I couldn't find them on his blog
under http://bartoszmilewski.com/category/d-programming-language/
There are some posts about ownership and regions, but only in the
context of multi-threading. I'm afraid this wouldn't easily fit
into a hierarchical system like I have in mind.
Post by bearophile via Digitalmars-d
Post by via Digitalmars-d
Implementation of this feature is possible without doing flow
control or interprocedural analysis.<
I remember that Walter has recently said that he's willing to
add some kind of flow analysis to the D front-end.
Interesting. The question is: is it worth it? Maybe we can
already cover 99% of the use cases with a simpler construct. The
concept needs to be understandable for the users of the language,
too. And maybe "some kind of flow analysis" just isn't enough to
get a significant improvement, maybe it would need whole-program
analysis...
bearophile via Digitalmars-d
2014-08-25 19:06:42 UTC
Permalink
(How formal do we need to go?)
It's not just a matter of how much formal to go, but also a
matter of how much D to formalize to avoid unwanted interactions
(surprises) later.

But before going formal, you need to wait for comments from
Walter & Andrei; and possibly also comments from a good Rust core
developer, because they have discussed such topics in detail for
lot of time :-)

Bye,
bearophile
Manu via Digitalmars-d
2014-08-26 11:47:30 UTC
Permalink
On 24 August 2014 23:14, via Digitalmars-d <digitalmars-d at puremagic.com>
In the "Opportunities for D" thread, Walter again mentioned the topics ref
counting, GC, uniqueness, and borrowing, from which a lively discussion
developed [1]. I took this thread as an opportunity to write down some
ideas about these topics. The result is a rather extensive proposal for the
http://wiki.dlang.org/User:Schuetzm/scope
This is not a real DIP, but before I put more work into formalizing it,
* Is this the general direction we want to go? Is it acceptable in general?
* Is the proposal internally consistent?
* How big would the effort to implement it be? (I suspect it's a large
amount of work, but relatively straightforward.)
[1] http://forum.dlang.org/thread/lphnen$1ml7$1 at digitalmars.com
This is the initiative I've been dreaming of for years!
I'll give it a critical review when I have some time. But in the mean time
at first glance, I just wanted to say, very nice work! :)
This is largely inline with my thoughts since I first came to D, except you
flesh out a few problem areas (return scope, which we discussed at length
at dconf2013), and the ideas appear to be quite sound.

This direction really opens up the language to support manual and
user-defined memory management at a much deeper and more useful level.
In my experience, manual/custom memory management depends extensively on
templates, and also implies that practically every api in any library
anywhere must also receive template args, such that it can receive objects
templated with custom allocation strategies.
D is a language that is acutely susceptible to over-templatisation and
extreme template bloat. This work on scope is critically important to
mitigate that tendency of D in library code.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20140826/6f8bf639/attachment-0001.html>
via Digitalmars-d
2014-08-26 19:52:54 UTC
Permalink
On Tuesday, 26 August 2014 at 11:47:39 UTC, Manu via
Post by Manu via Digitalmars-d
This is the initiative I've been dreaming of for years!
I'll give it a critical review when I have some time. But in
the mean time
at first glance, I just wanted to say, very nice work! :)
I had a suspicion you would like it ;-)
Post by Manu via Digitalmars-d
This is largely inline with my thoughts since I first came to
D, except you
flesh out a few problem areas (return scope, which we discussed
at length
at dconf2013), and the ideas appear to be quite sound.
I've heard that mentioned a few times. I guess it was an informal
discussion only, not an official talk, right? I only know about a
few discussions afterwards on the news group.
Post by Manu via Digitalmars-d
This direction really opens up the language to support manual
and
user-defined memory management at a much deeper and more useful
level.
In my experience, manual/custom memory management depends
extensively on
templates, and also implies that practically every api in any
library
anywhere must also receive template args, such that it can
receive objects
templated with custom allocation strategies.
D is a language that is acutely susceptible to
over-templatisation and
extreme template bloat. This work on scope is critically
important to
mitigate that tendency of D in library code.
I'm looking forward to hear your thoughts about it, because
you're one of the people who are involved in large complex
projects where this is relevant. For me, it is until now only a
theoretical exercise, it's important to hear from someone who can
assess how it would work out in practice.

I'm also curious about Walter and Andrei's opinion (I believe
they are quite busy with other things at the moment, so we'll
have to wait), and Kenji and others who can give an estimate
about the implementation.
Marco Leise via Digitalmars-d
2014-08-26 23:01:36 UTC
Permalink
Am Sun, 24 Aug 2014 13:14:43 +0000
Post by via Digitalmars-d
In the "Opportunities for D" thread, Walter again mentioned the
topics ref counting, GC, uniqueness, and borrowing, from which a
lively discussion developed [1]. I took this thread as an
opportunity to write down some ideas about these topics. The
result is a rather extensive proposal for the implementation of
http://wiki.dlang.org/User:Schuetzm/scope
The amount of possible use-cases you listed for this extension is
staggering. It surely carries its own weight.
scope!(ident1, ident2, ...) was quite clever. inout could
borrow from this.
Post by via Digitalmars-d
This is not a real DIP, but before I put more work into
formalizing it, I'd like to hear some thoughts from the languages
* Is this the general direction we want to go? Is it acceptable
in general?
* Is the proposal internally consistent?
Can anyone tell without actually implementing it? :)
You could try to formalize some error messages and how the
compiler's reasoning would go. What happens when I pass
identifiers to scope!(
) return types?
- Can the compiler look up the identifiers' types and scopes
in all cases?
- Will the compiler deduce the return type from these
identifiers? E.g. "scope!(someString) ref getString();" will
work like in your example? I don't think so, because the
"lifetime identifiers" could be structs containing the
returned type.
- What if we used incompatible types like
"scope!(someString, someIntPtr)" there?
- What about variables?

static int someInt = 32;
string someString;
scope!(someString, someInt) int* x;
x = &someInt;

Is the declaration of x in error? Strings don't contain
integers unless unsafe casts are used, so why would they
narrow the lifetime of an integer reference?

- Is it necessary to keep around all declared lifetime
identifiers? In the snippet above (assuming it is valid), it
looks like the shorter lived `someString' is enough to
establish the semantics.
Post by via Digitalmars-d
* How big would the effort to implement it be? (I suspect it's a
large amount of work, but relatively straightforward.)
[1] http://forum.dlang.org/thread/lphnen$1ml7$1 at digitalmars.com
--
Marco
Dicebot via Digitalmars-d
2014-08-27 02:36:40 UTC
Permalink
Post by Marco Leise via Digitalmars-d
Can anyone tell without actually implementing it? :)
I think this is the biggest problem with all scope proposals.
While concept is very natural figuring out all small details is
incredibly tricky and one can never be sure it works without
trying in practice on large-ish project. Of course implementing
it is no small feat either which is pretty much any progress in
this direction is so slow.
bearophile via Digitalmars-d
2014-08-27 11:47:20 UTC
Permalink
Post by Marco Leise via Digitalmars-d
The amount of possible use-cases you listed for this extension
is staggering.
With scope management in code like this:


import std.stdio, std.algorithm;
int foo(in int[] a) {
return sum([0, 1] ~ a[1 .. $ - 2] ~ 0 ~
[a[$ - 1] + 1, a[$ - 1] + 2]);
}
void main() {
int[5] b = [10, 20, 30, 40, 50];
b.foo.writeln;
}


The compiler in theory could lower the code to something like
this, removing all heap allocations for the array literals and
the concatenations, allowing 'foo' to be annotated with @nogc (I
don't know if Rust is able to do this, I presume it can):


import std.stdio, std.algorithm, core.stdc.stdlib;
T foo(T)(in T[] a) @nogc {
immutable size_t L = 2 + a.length - 3 + 1 + 2;
auto ptr = alloca(T.sizeof * L);
if (ptr == null)
exit(1); // Or throw a stack overflow error.
T[] b = (cast(T*)ptr)[0 .. L];
b[0] = 0;
b[1] = 1;
b[2 .. $ - 3] = a[1 .. $ - 2];
b[$ - 3] = 0;
b[$ - 2] = a[$ - 1] + 1;
b[$ - 1] = a[$ - 1] + 2;
return sum(b);

}
void main() {
int[5] c = [10, 20, 30, 40, 50];
c.foo.writeln;
}


But in some cases you don't want those arrays to be allocated on
the stack because you know they are very large. So the [...]s
array literal syntax becomes useful again:

import std.stdio, std.algorithm;
int foo(const scope int[] a) {
return sum([0, 1]s ~ a[1 .. $ - 2] ~ 0 ~
[a[$ - 1] + 1, a[$ - 1] + 2]s);
}


But this is still not enough, because even if those parts are all
safely stack-allocated, the result of the concatenation is still
not stack-allocated.


This could be enough:

import std.stdio, std.algorithm;
int foo(const scope int[] a) @nogc {
auto[$] a2 = [0, 1]s ~ a[1 .. $ - 2] ~ 0 ~
[a[$ - 1] + 1, a[$ - 1] + 2]s;
return sum(a2[]);
}

Bye,
bearophile
via Digitalmars-d
2014-08-27 19:32:23 UTC
Permalink
Post by Marco Leise via Digitalmars-d
You could try to formalize some error messages and how the
compiler's reasoning would go. What happens when I pass
identifiers to scope!(
) return types?
- Can the compiler look up the identifiers' types and scopes
in all cases?
It has to be able to. If a non-visible identifier is specified,
it is an error.
Post by Marco Leise via Digitalmars-d
- Will the compiler deduce the return type from these
identifiers? E.g. "scope!(someString) ref getString();" will
work like in your example? I don't think so, because the
"lifetime identifiers" could be structs containing the
returned type.
I see, this was a stupid mistake. Of course this function needs
to specify the full type, as no type deduction can happen when
the body isn't available. Fixed in on the Wiki.
Post by Marco Leise via Digitalmars-d
- What if we used incompatible types like
"scope!(someString, someIntPtr)" there?
- What about variables?
Just to be sure: You can only specify variables as owners, not
types. I've clarified this on the Wiki.
Post by Marco Leise via Digitalmars-d
static int someInt = 32;
string someString;
scope!(someString, someInt) int* x;
x = &someInt;
Is the declaration of x in error? Strings don't contain
integers unless unsafe casts are used, so why would they
narrow the lifetime of an integer reference?
This is an interesting thought... But I would still allow this
for a different reason. At the beginning, I only thought about
`scope` as a tool for memory management, but I believe it can be
applied for lifetime management of arbitrary resources. Let's
slightly modify your example, and use different types:

Task someProcess;
scope!someProcess HANDLE my_handle;

`someProcess` could represent an external process that is managed
by this program, and `my_handle` could refer to some kind of
object in this external process. This handle is only valid as
long as the process exists, even though it is not a memory
reference, and `Task` might not contain any members of type
`HANDLE`. (This is not an ideal example, of course, because the
process could terminate for reasons outside of our control.) A
similar example would be a connection to the X server, and a
handle to an object allocated from it.

I already wrote this idea into the section "scope for
non-reference types", with a simpler example. In fact, I believe
that the entire proposal will be a bit simpler if
references/pointers aren't treated specially.
Post by Marco Leise via Digitalmars-d
- Is it necessary to keep around all declared lifetime
identifiers? In the snippet above (assuming it is valid), it
looks like the shorter lived `someString' is enough to
establish the semantics.
Yes, it's possible to some degree, see the section
"Considerations for the implementation". Unfortunately, this
doesn't work with function declarations, and is incompatible with
the suggested `scope!(const ...)` extension.
Jacob Carlborg via Digitalmars-d
2014-08-28 06:52:30 UTC
Permalink
In the "Opportunities for D" thread, Walter again mentioned the topics
ref counting, GC, uniqueness, and borrowing, from which a lively
discussion developed [1]. I took this thread as an opportunity to write
down some ideas about these topics. The result is a rather extensive
http://wiki.dlang.org/User:Schuetzm/scope
I assume with this proposal it should be safe to do more stack
allocations and have the compiler verify references don't escape the
scope. Would there be a good idea to and a new function, besides the
destructor, that will be called for variables declared as "scope" when
they go out of scope.

The problem with destructors are that they can be called both when an
object is deleted by the GC and when an object goes of out scope.
--
/Jacob Carlborg
via Digitalmars-d
2014-08-28 09:16:36 UTC
Permalink
Post by Jacob Carlborg via Digitalmars-d
Post by via Digitalmars-d
In the "Opportunities for D" thread, Walter again mentioned
the topics
ref counting, GC, uniqueness, and borrowing, from which a
lively
discussion developed [1]. I took this thread as an opportunity to write
down some ideas about these topics. The result is a rather
extensive
proposal for the implementation of borrowing, and its
http://wiki.dlang.org/User:Schuetzm/scope
I assume with this proposal it should be safe to do more stack
allocations and have the compiler verify references don't
escape the scope. Would there be a good idea to and a new
function, besides the destructor, that will be called for
variables declared as "scope" when they go out of scope.
The problem with destructors are that they can be called both
when an object is deleted by the GC and when an object goes of
out scope.
I'd rather introduce a special method that is called only by the
GC. Cleaning up after an object that goes out of scope has always
been the task of the regular destructor, it's undeterministic
destruction that needs special treatment.
Jacob Carlborg via Digitalmars-d
2014-08-28 18:53:25 UTC
Permalink
I'd rather introduce a special method that is called only by the GC.
Cleaning up after an object that goes out of scope has always been the
task of the regular destructor, it's undeterministic destruction that
needs special treatment.
I was think about not breaking code. But introducing a new function that
is called by the GC which also calls the regular destructor might work.
--
/Jacob Carlborg
via Digitalmars-d
2014-08-28 19:27:42 UTC
Permalink
Post by Jacob Carlborg via Digitalmars-d
Post by via Digitalmars-d
I'd rather introduce a special method that is called only by
the GC.
Cleaning up after an object that goes out of scope has always
been the
task of the regular destructor, it's undeterministic
destruction that
needs special treatment.
I was think about not breaking code. But introducing a new
function that is called by the GC which also calls the regular
destructor might work.
The other way round would be safer: A destructor automatically
calls as its first step a finalizer (let's use that term for a
destructor called by the GC) if present, but a finalizer doesn't
call the destructor. Remember that the things that are forbidden
in a finalizer are usually fine in normal destructors. By calling
the destructor from inside the finalizer, you bring all the
problems back that you wanted to get rid of by introducing a
special finalizer, right?

And this would be backwards-compatible: There is already today no
guarantee that a destructor gets called by the GC, so never
calling it doesn't break any valid code, strictly speaking. Then
you could place "safe" cleanup actions (like closing a file) into
the finalizer, and "unsafe" ones (like removing yourself from a
linked list) into the destructor, and you don't need to duplicate
the actions from the finalizer in the destructor. The compiler
might then even detect unsafe operations in the finalizer and
refuse to compile them.
Jacob Carlborg via Digitalmars-d
2014-08-29 05:59:30 UTC
Permalink
The other way round would be safer: A destructor automatically calls as
its first step a finalizer (let's use that term for a destructor called
by the GC) if present, but a finalizer doesn't call the destructor.
Remember that the things that are forbidden in a finalizer are usually
fine in normal destructors. By calling the destructor from inside the
finalizer, you bring all the problems back that you wanted to get rid of
by introducing a special finalizer, right?
And this would be backwards-compatible: There is already today no
guarantee that a destructor gets called by the GC, so never calling it
doesn't break any valid code, strictly speaking. Then you could place
"safe" cleanup actions (like closing a file) into the finalizer, and
"unsafe" ones (like removing yourself from a linked list) into the
destructor, and you don't need to duplicate the actions from the
finalizer in the destructor. The compiler might then even detect unsafe
operations in the finalizer and refuse to compile them.
Yeah, you're probably right. I got it all backwards.
--
/Jacob Carlborg
via Digitalmars-d
2014-09-11 13:58:38 UTC
Permalink
PING

Now that there are again several GC related topics being
discussed, I thought I'd bump this thread.

Would be nice if Walter and/or Andrei could have a look and share
there opinions. Is this something worth pursuing further? Are
there fundamental objections against it?
Post by via Digitalmars-d
In the "Opportunities for D" thread, Walter again mentioned the
topics ref counting, GC, uniqueness, and borrowing, from which
a lively discussion developed [1]. I took this thread as an
opportunity to write down some ideas about these topics. The
result is a rather extensive proposal for the implementation of
http://wiki.dlang.org/User:Schuetzm/scope
This is not a real DIP, but before I put more work into
formalizing it, I'd like to hear some thoughts from the
* Is this the general direction we want to go? Is it acceptable
in general?
* Is the proposal internally consistent?
* How big would the effort to implement it be? (I suspect it's
a large amount of work, but relatively straightforward.)
[1] http://forum.dlang.org/thread/lphnen$1ml7$1 at digitalmars.com
bearophile via Digitalmars-d
2014-09-11 14:06:25 UTC
Permalink
Post by via Digitalmars-d
Now that there are again several GC related topics being
discussed, I thought I'd bump this thread.
Would be nice if Walter and/or Andrei could have a look and
share there opinions. Is this something worth pursuing further?
Are there fundamental objections against it?
At the moment the focus seems to be:
1) C++ interoperability
2) GC (in theory).

Bye,
bearophile
Andrei Alexandrescu via Digitalmars-d
2014-09-11 18:58:45 UTC
Permalink
Post by bearophile via Digitalmars-d
Now that there are again several GC related topics being discussed, I
thought I'd bump this thread.
Would be nice if Walter and/or Andrei could have a look and share
there opinions. Is this something worth pursuing further? Are there
fundamental objections against it?
1) C++ interoperability
2) GC (in theory).
scope is GC-related so looking at it is appropriate. -- Andrei
Ivan Timokhin via Digitalmars-d
2014-09-11 16:32:52 UTC
Permalink
I am in no way a language guru, but here are a few things that bother me
in your proposal. Thought I'd share.

1. AFAIK, all current D type modifiers can be safely removed from the
topmost level (i.e. it is OK to assign immutable(int[]) to
immutable(int)[]), because they currently apply to particular variable,
so there's no good reason to impose same restrictions on its copy.
Situation seems different with scope: it is absolutely not safe to cast
away and it applies to a *value*, not a variable holding it.

This is not only inconsistent, but may also cause trouble with
interaction with existing features. For example, what should be
std.traits.Unqual!(scope(int*)) ?

2. Consider findSubstring from your examples. What should be
typeof(findSubstring("", ""))? Is the following code legal?

scope(string) a = ..., b = ...;
...
typeof(findSubstring("", "")) c = findSubstring(a, b);

This is a bit troublesome, because this is how things like
std.range.ElementType work currently, so they may break. For example,
what would be ElementType!ByLineImpl (from the "scope(const...)" section)?

This troubles me the most, because currently return type of a function
may depend only on types of its arguments, and there is a lot of
templated code written in that assumption. With the current proposal it
ALL could break. Maybe there's no way around it if we want a solid
lifetime management system, but I think this is definitely a problem to
be aware of.

3. I believe it was mentioned before, but shouldn't scope propagate
*outwards*? This would not only make perfect sense, since the aggregate
obviously "holds the reference" just as well as its member does, it
would also make various range-wrappers and alike automatically
scope-aware, in that the wrapper would automatically become scoped if
the wrapped range is scoped.
PING
Now that there are again several GC related topics being discussed, I
thought I'd bump this thread.
Would be nice if Walter and/or Andrei could have a look and share there
opinions. Is this something worth pursuing further? Are there
fundamental objections against it?
In the "Opportunities for D" thread, Walter again mentioned the topics
ref counting, GC, uniqueness, and borrowing, from which a lively
discussion developed [1]. I took this thread as an opportunity to
write down some ideas about these topics. The result is a rather
extensive proposal for the implementation of borrowing, and its
http://wiki.dlang.org/User:Schuetzm/scope
This is not a real DIP, but before I put more work into formalizing
* Is this the general direction we want to go? Is it acceptable in
general?
* Is the proposal internally consistent?
* How big would the effort to implement it be? (I suspect it's a large
amount of work, but relatively straightforward.)
[1] http://forum.dlang.org/thread/lphnen$1ml7$1 at digitalmars.com
via Digitalmars-d
2014-09-11 20:45:08 UTC
Permalink
On Thursday, 11 September 2014 at 16:32:54 UTC, Ivan Timokhin
Post by Ivan Timokhin via Digitalmars-d
I am in no way a language guru, but here are a few things that
bother me in your proposal. Thought I'd share.
Neither am I :-)
Post by Ivan Timokhin via Digitalmars-d
1. AFAIK, all current D type modifiers can be safely removed
from the topmost level (i.e. it is OK to assign
immutable(int[]) to immutable(int)[]), because they currently
apply to particular variable, so there's no good reason to
impose same restrictions on its copy. Situation seems different
with scope: it is absolutely not safe to cast away and it
applies to a *value*, not a variable holding it.
The types in your example are implicitly convertable, indeed no
explicit cast is necessary. This is because when you copy a const
value, the result doesn't need to be const. But with scope, it
makes sense (and is of course necessary) to keep the ownership. I
don't see that as an inconsistency, but as a consequence of the
different things const and scope imply: mutability vs. ownership.
Post by Ivan Timokhin via Digitalmars-d
This is not only inconsistent, but may also cause trouble with
interaction with existing features. For example, what should be
std.traits.Unqual!(scope(int*)) ?
Good question. I would say it needs to keep scope, as it was
clearly designed with mutability in mind (although it also
removes shared, which is however related to mutability in a way).
Ownership is an orthogonal concept to mutability.
Post by Ivan Timokhin via Digitalmars-d
2. Consider findSubstring from your examples. What should be
typeof(findSubstring("", ""))? Is the following code legal?
scope(string) a = ..., b = ...;
...
typeof(findSubstring("", "")) c = findSubstring(a, b);
It's not legal. String literals live forever, so `c` has an owner
that lives longer than `a` and `b`.

An alternative interpretation would be that the literals are
temporary expressions; then it would have a very short lifetime,
thus the assignment would be accepted. But I guess there needs to
be a rule that says that the specified owners must not live
shorter than the variable itself.
Post by Ivan Timokhin via Digitalmars-d
This is a bit troublesome, because this is how things like
std.range.ElementType work currently, so they may break. For
example,
what would be ElementType!ByLineImpl (from the
"scope(const...)" section)?
I see... it can _not_ be:

scope!(const ByLineImpl!(char, "\n").init)(ByLineImpl!(char,
"\n"))

because the init value is copied and thus becomes a temporary.
This is ugly. It would however work if ElementType would take an
instance instead of a type.
Post by Ivan Timokhin via Digitalmars-d
This troubles me the most, because currently return type of a
function may depend only on types of its arguments, and there
is a lot of templated code written in that assumption.
I'm sorry, I don't understand what you mean here. This is clearly
not true, neither for normal functions, nor for templates.
Post by Ivan Timokhin via Digitalmars-d
With the current proposal it ALL could break. Maybe there's no
way around it if we want a solid lifetime management system,
but I think this is definitely a problem to be aware of.
The answer may be that scope needs to be something independent
from the type, indeed more like a storage class, rather than what
I suggested a type modifier. This would also solve the problem
about `std.traits.Unqual`, no?

I'd have to think this through, but I believe this is indeed the
way to go. It would make several other things cleaner. On the
other hand, it would then be impossible to have scoped member
fields, because storage classes aren't usable there, AFAIK. This
would need to be supported first.
Post by Ivan Timokhin via Digitalmars-d
3. I believe it was mentioned before, but shouldn't scope
propagate *outwards*? This would not only make perfect sense,
since the aggregate obviously "holds the reference" just as
well as its member does, it would also make various
range-wrappers and alike automatically scope-aware, in that the
wrapper would automatically become scoped if the wrapped range
is scoped.
You mean that any aggregate that contains a member with owner X
automatically gets X as its owner itself?

I don't think so, because assigning a struct is semantically
equivalent to assigning its members individually one after the
others (by default) or whatever opAssign() is implemented to do.
This means that an assignment that violates the rules would fail
anyway, because the ownership is codified as part of the member's
type. Instances of wrapper types would also need to be declared
as scope with the appropriate owner, because otherwise they could
not contain the scoped variables.

But I'm not sure how this is supposed to work with the storage
class version of scope.
via Digitalmars-d
2014-09-12 08:06:18 UTC
Permalink
Post by via Digitalmars-d
On Thursday, 11 September 2014 at 16:32:54 UTC, Ivan Timokhin
Post by Ivan Timokhin via Digitalmars-d
1. AFAIK, all current D type modifiers can be safely removed
from the topmost level (i.e. it is OK to assign
immutable(int[]) to immutable(int)[]), because they currently
apply to particular variable, so there's no good reason to
impose same restrictions on its copy. Situation seems
different with scope: it is absolutely not safe to cast away
and it applies to a *value*, not a variable holding it.
The types in your example are implicitly convertable, indeed no
explicit cast is necessary. This is because when you copy a
const value, the result doesn't need to be const. But with
scope, it makes sense (and is of course necessary) to keep the
ownership. I don't see that as an inconsistency, but as a
mutability vs. ownership.
I've addressed this in the Wiki now. There were only a few
changes to be made to move away from type modifiers. I had even
suggested it as an implementation detail, but didn't think of
making it part of the specification. Thank you for that insight,
it makes the proposal more consistent and avoids the troubles
with the types.
Post by via Digitalmars-d
Post by Ivan Timokhin via Digitalmars-d
This is not only inconsistent, but may also cause trouble with
interaction with existing features. For example, what should
be std.traits.Unqual!(scope(int*)) ?
Good question. I would say it needs to keep scope, as it was
clearly designed with mutability in mind (although it also
removes shared, which is however related to mutability in a
way). Ownership is an orthogonal concept to mutability.
After the changes, this is now the case.
Post by via Digitalmars-d
Post by Ivan Timokhin via Digitalmars-d
This is a bit troublesome, because this is how things like
std.range.ElementType work currently, so they may break. For
example,
what would be ElementType!ByLineImpl (from the
"scope(const...)" section)?
scope!(const ByLineImpl!(char,
"\n").init)(ByLineImpl!(char, "\n"))
because the init value is copied and thus becomes a temporary.
This is ugly. It would however work if ElementType would take
an instance instead of a type.
Ditto, this works now. The type is now simply `char[]` (or
whatever), the owner is tracked separately.
Manu via Digitalmars-d
2014-09-13 01:48:53 UTC
Permalink
On 12 September 2014 18:06, via Digitalmars-d <digitalmars-d at puremagic.com>
Post by Ivan Timokhin via Digitalmars-d
1. AFAIK, all current D type modifiers can be safely removed from the
topmost level (i.e. it is OK to assign immutable(int[]) to
immutable(int)[]), because they currently apply to particular variable, so
there's no good reason to impose same restrictions on its copy. Situation
seems different with scope: it is absolutely not safe to cast away and it
applies to a *value*, not a variable holding it.
The types in your example are implicitly convertable, indeed no explicit
cast is necessary. This is because when you copy a const value, the result
doesn't need to be const. But with scope, it makes sense (and is of course
necessary) to keep the ownership. I don't see that as an inconsistency, but
as a consequence of the different things const and scope imply: mutability
vs. ownership.
I've addressed this in the Wiki now. There were only a few changes to be
made to move away from type modifiers. I had even suggested it as an
implementation detail, but didn't think of making it part of the
specification. Thank you for that insight, it makes the proposal more
consistent and avoids the troubles with the types.
I'm not convinced this is a good change.
It sounds like you're just trading one problem with another more sinister
problem...

What happens when a scope() thing finds it's way into generic code? If the
type doesn't carry that information, then you end up in a situation like
ref. Have you ever had to wrestle with ref in generic code?
ref is the biggest disaster zone in D, and I think all it's problems will
translate straight to scope if you do this.


This is not only inconsistent, but may also cause trouble with interaction
Post by Ivan Timokhin via Digitalmars-d
with existing features. For example, what should be
std.traits.Unqual!(scope(int*)) ?
Good question. I would say it needs to keep scope, as it was clearly
designed with mutability in mind (although it also removes shared, which is
however related to mutability in a way). Ownership is an orthogonal concept
to mutability.
After the changes, this is now the case.
This is a bit troublesome, because this is how things like
Post by Ivan Timokhin via Digitalmars-d
std.range.ElementType work currently, so they may break. For example,
what would be ElementType!ByLineImpl (from the "scope(const...)" section)?
scope!(const ByLineImpl!(char, "\n").init)(ByLineImpl!(char, "\n"))
because the init value is copied and thus becomes a temporary. This is
ugly. It would however work if ElementType would take an instance instead
of a type.
Ditto, this works now. The type is now simply `char[]` (or whatever), the
owner is tracked separately.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20140913/fb080410/attachment-0001.html>
via Digitalmars-d
2014-09-13 11:02:41 UTC
Permalink
On Saturday, 13 September 2014 at 01:49:05 UTC, Manu via
Post by Manu via Digitalmars-d
On 12 September 2014 18:06, via Digitalmars-d
<digitalmars-d at puremagic.com>
Post by via Digitalmars-d
I've addressed this in the Wiki now. There were only a few
changes to be
made to move away from type modifiers. I had even suggested it
as an
implementation detail, but didn't think of making it part of
the
specification. Thank you for that insight, it makes the
proposal more
consistent and avoids the troubles with the types.
I'm not convinced this is a good change.
It sounds like you're just trading one problem with another
more sinister
problem...
What happens when a scope() thing finds it's way into generic
code? If the
type doesn't carry that information, then you end up in a
situation like
ref. Have you ever had to wrestle with ref in generic code?
ref is the biggest disaster zone in D, and I think all it's
problems will
translate straight to scope if you do this.
Could you give an example?
Walter Bright via Digitalmars-d
2014-09-21 03:47:36 UTC
Permalink
What happens when a scope() thing finds it's way into generic code? If the type
doesn't carry that information, then you end up in a situation like ref. Have
you ever had to wrestle with ref in generic code?
ref is the biggest disaster zone in D, and I think all it's problems will
translate straight to scope if you do this.
I'm unaware of this disaster zone.
deadalnix via Digitalmars-d
2014-09-21 06:02:45 UTC
Permalink
Post by Walter Bright via Digitalmars-d
Post by Manu via Digitalmars-d
What happens when a scope() thing finds it's way into generic
code? If the type
doesn't carry that information, then you end up in a situation like ref. Have
you ever had to wrestle with ref in generic code?
ref is the biggest disaster zone in D, and I think all it's
problems will
translate straight to scope if you do this.
I'm unaware of this disaster zone.
Well it is very real. I had to duplicate bunch of code in my
visitor generator recently because of it. Getting generic code
ref correct is very tedious, error prone, and guarantees code
duplication and/or various static ifs all over the place.
Manu via Digitalmars-d
2014-09-21 11:27:10 UTC
Permalink
On 21 September 2014 16:02, deadalnix via Digitalmars-d <
Post by Walter Bright via Digitalmars-d
What happens when a scope() thing finds it's way into generic code? If the type
doesn't carry that information, then you end up in a situation like ref. Have
you ever had to wrestle with ref in generic code?
ref is the biggest disaster zone in D, and I think all it's problems will
translate straight to scope if you do this.
I'm unaware of this disaster zone.
Well it is very real. I had to duplicate bunch of code in my visitor
generator recently because of it. Getting generic code ref correct is very
tedious, error prone, and guarantees code duplication and/or various static
ifs all over the place.
It's also extremely hard to unittest; explodes the number of static if
paths exponentially. I'm constantly finding bugs appear a year after
writing some code because I missed some static branch paths when originally
authoring.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20140921/5f257bf8/attachment.html>
Andrei Alexandrescu via Digitalmars-d
2014-09-21 15:10:25 UTC
Permalink
Post by Manu via Digitalmars-d
On 21 September 2014 16:02, deadalnix via Digitalmars-d
What happens when a scope() thing finds it's way into
generic code? If the type
doesn't carry that information, then you end up in a
situation like ref.. Have
you ever had to wrestle with ref in generic code?
ref is the biggest disaster zone in D, and I think all it's
problems will
translate straight to scope if you do this.
I'm unaware of this disaster zone.
Well it is very real. I had to duplicate bunch of code in my visitor
generator recently because of it. Getting generic code ref correct
is very tedious, error prone, and guarantees code duplication and/or
various static ifs all over the place.
It's also extremely hard to unittest; explodes the number of static if
paths exponentially.. I'm constantly finding bugs appear a year after
writing some code because I missed some static branch paths when
originally authoring.
Is this because of problems with ref's definition, or a natural
consequence of supporting ref parameters? -- Andrei
deadalnix via Digitalmars-d
2014-09-22 01:36:07 UTC
Permalink
On Sunday, 21 September 2014 at 15:10:23 UTC, Andrei Alexandrescu
Post by Andrei Alexandrescu via Digitalmars-d
Is this because of problems with ref's definition, or a natural
consequence of supporting ref parameters? -- Andrei
There is various reason why this is complex:
- classes must often be handled specifically.
- auto ref is not very controllable and tend to end up with an
combinatorial explosion of special cases.
- it is complex to find out if something is ref or not in
generic code.
- it is not possible to conditionally declare something as ref.
It is true of other qualifiers, but ref change the semantic in a
more deeper way than, say, dropping pure or @safe.
- you may want to use ref for very different semantics, which
lead to different policies about what should be ref or not. There
is no other way to implement these policies than spaghetti static
if with meatball code duplication.
Manu via Digitalmars-d
2014-09-22 10:23:21 UTC
Permalink
On 22 September 2014 01:10, Andrei Alexandrescu via Digitalmars-d <
Post by Andrei Alexandrescu via Digitalmars-d
Post by Manu via Digitalmars-d
On 21 September 2014 16:02, deadalnix via Digitalmars-d
What happens when a scope() thing finds it's way into
generic code? If the type
doesn't carry that information, then you end up in a
situation like ref.. Have
you ever had to wrestle with ref in generic code?
ref is the biggest disaster zone in D, and I think all it's
problems will
translate straight to scope if you do this.
I'm unaware of this disaster zone.
Well it is very real. I had to duplicate bunch of code in my visitor
generator recently because of it. Getting generic code ref correct
is very tedious, error prone, and guarantees code duplication and/or
various static ifs all over the place.
It's also extremely hard to unittest; explodes the number of static if
paths exponentially.. I'm constantly finding bugs appear a year after
writing some code because I missed some static branch paths when
originally authoring.
Is this because of problems with ref's definition, or a natural
consequence of supporting ref parameters? -- Andrei
It's all because ref is not part of the type.
You can't capture ref with typeof() or templates, you can't make ref
locals, it's hard to find if something is ref or not (detection is
different than everything else), etc.
The nature of it not being a type leads to static if's in every template
that ref appears, which must detect if things are ref (which is awkward),
and produce multiple paths for a ref and not-ref version. If we're dealing
with arguments, this might lead to num-arguments^^2 paths.

The only practical conclusion I (and others too) have reached, is to
eventually give up and invent Ref!T, but then we arrive at a new world of
problems. It's surprisingly hard to write a transparent Ref template which
interacts effectively with other generic code, and no 3rd party library
will support it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20140922/87735984/attachment.html>
Walter Bright via Digitalmars-d
2014-09-22 03:19:12 UTC
Permalink
It's also extremely hard to unittest; explodes the number of static if paths
exponentially. I'm constantly finding bugs appear a year after writing some code
because I missed some static branch paths when originally authoring.
If you throw -cov while running unittests, it'll give you a report on which code
was executed and which wasn't. Very simple and useful.
Manu via Digitalmars-d
2014-09-22 11:20:47 UTC
Permalink
On 22 September 2014 13:19, Walter Bright via Digitalmars-d <
Post by Walter Bright via Digitalmars-d
It's also extremely hard to unittest; explodes the number of static if paths
exponentially. I'm constantly finding bugs appear a year after writing some code
because I missed some static branch paths when originally authoring.
If you throw -cov while running unittests, it'll give you a report on
which code was executed and which wasn't. Very simple and useful.
It is a useful tool, but you can see how going to great lengths to write
this explosion of paths is a massive pain in the first place, let alone
additional overhead to comprehensively test that it works... it should
never have been a problem to start with.

You may argue that I didn't test my code effectively. I argue that my code
should never have existed in the first place. It's wildly unsanitary, and
very difficult to maintain; I can rarely understand the complexity of ref
handling code looking back after some months.

This was my very first complaint about D, on day 1... 6 years later, I'm
still struggling with it on a daily basis.

Here's some code I've been struggling with lately, tell me you think this
is right:
https://github.com/FeedBackDevs/LuaD/blob/master/luad/conversions/functions.d#L286
<- this one only deals with the return value. is broken, need to
special-case properties that receive arguments by ref, since you can't pass
rvalue->ref
https://github.com/FeedBackDevs/LuaD/blob/master/luad/conversions/functions.d#L115
<- Ref!RT workaround, because I need a ref local
https://github.com/FeedBackDevs/LuaD/blob/master/luad/conversions/functions.d#L172
<- Ref!T again, another instance where I need a ref local
https://github.com/FeedBackDevs/LuaD/blob/master/luad/conversions/functions.d#L180
https://github.com/FeedBackDevs/LuaD/blob/master/luad/stack.d#L129 <-
special case handling for Ref!T that I could eliminate if ref was part of
the type. Note: 3rd party code never has this concession...
https://github.com/FeedBackDevs/LuaD/blob/master/luad/stack.d#L448 <- more
invasion of Ref!, because getValue may or may not return ref, which would
be lost beyond this point, and it's not practical to static-if duplicate
this entire function with another version that returns ref.
https://github.com/FeedBackDevs/LuaD/blob/master/luad/stack.d#L157 <-
special-case function, again because ref isn't part of the type
There are many more instances of these throughout this code.

This is just one example, and not even a particularly bad one (I've had
worse), because there are never multiple args involved. These sorts of
things come up on most projects I've worked on.
The trouble with these examples, is that you can't try and imagine a direct
substitution of the static branches and Ref!T with 'ref T' if ref were part
of the type. This problem has deeply interfered with the code, and API
concessions have been made throughout to handle it. If ref were part of the
type, this code would be significantly re-worked and simplified. There
would probably be one little part somewhere that did logic on T, alias with
or without 'ref' as part of the type, and the cascading noise would mostly
disappear.

Add to all of that that I still can't pass an rvalue to a ref function (5
years later!!) >_<

It's also worth noting that, with regard to 'ref', the above code only
_just_ suits my needs. It's far from bug-free; there are places in there
where I was dealing with ref, but it got so complicated, and I didn't
actually make front-end use of the case in my project, that I gave up and
ignored those cases... which is not really ideal considering this is a
fairly popular library.

I have spent days trying to get this right. If I were being paid hourly, I
think I would have burned something like $2000.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20140922/3ff6e6a6/attachment-0001.html>
Kagamin via Digitalmars-d
2014-09-22 13:38:34 UTC
Permalink
On Monday, 22 September 2014 at 11:20:57 UTC, Manu via
Post by Manu via Digitalmars-d
It is a useful tool, but you can see how going to great lengths
to write
this explosion of paths is a massive pain in the first place,
let alone
additional overhead to comprehensively test that it works... it
should
never have been a problem to start with.
Hmm... even if the code is syntactically succinct, it doesn't
necessarily mean lower complexity or that it requires less
testing. You provided an example yourself: you have generic code,
which works for values, but not for references. You need a lot of
testing not because the features have different syntax, but
because they work differently, so code, which works for one
thing, may not work for another.
Manu via Digitalmars-d
2014-09-22 15:18:48 UTC
Permalink
On 22 September 2014 23:38, Kagamin via Digitalmars-d <
Post by Manu via Digitalmars-d
It is a useful tool, but you can see how going to great lengths to write
this explosion of paths is a massive pain in the first place, let alone
additional overhead to comprehensively test that it works... it should
never have been a problem to start with.
Hmm... even if the code is syntactically succinct, it doesn't necessarily
mean lower complexity or that it requires less testing. You provided an
example yourself: you have generic code, which works for values, but not
for references. You need a lot of testing not because the features have
different syntax, but because they work differently, so code, which works
for one thing, may not work for another.
Eliminating static branches containing different code has a very
significant reduction in complexity. It's also DRY.
I don't think I provided that example... although it's certainly true that
there are semantic differences that may lead to distinct code paths, it is
my experience that in the majority of cases, if I just had the ref-ness as
part of the type, the rest would follow naturally. I have never encountered
a situation where I would feel hindered by ref as part of the type.
I think it's also easier to get from ref in the type to the raw type than
the reverse (which we must do now); We are perfectly happy with Unqual!T
and things like that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20140923/89a73ae7/attachment-0001.html>
Walter Bright via Digitalmars-d
2014-09-23 09:45:39 UTC
Permalink
Post by Manu via Digitalmars-d
On 22 September 2014 13:19, Walter Bright via Digitalmars-d
It's also extremely hard to unittest; explodes the number of static if paths
exponentially. I'm constantly finding bugs appear a year after writing
some code
because I missed some static branch paths when originally authoring.
If you throw -cov while running unittests, it'll give you a report on which
code was executed and which wasn't. Very simple and useful.
It is a useful tool, but you can see how going to great lengths to write this
explosion of paths is a massive pain in the first place, let alone additional
overhead to comprehensively test that it works... it should never have been a
problem to start with.
There are two separate issues here - the first is knowing whether or not the
code is unittested. -cov solves that issue. The second is having the multiple
code paths in the first place.

Have you tried auto ref?

I don't really know what the code you show is supposed to do from a high level.
My first impression is you are trying to write it like you'd write C++ code.
Perhaps there's a more D idiomatic way of doing it that doesn't lead to the
tangle you've got.

BTW, ref (as you know) is part of the type in C++. However, I can vouch for it
being a special case everywhere in C++, and is a horrifying quagmire of strange
edge cases. That's why it's not part of the type in D.
via Digitalmars-d
2014-09-23 10:23:55 UTC
Permalink
On Tuesday, 23 September 2014 at 09:46:17 UTC, Walter Bright
Post by Walter Bright via Digitalmars-d
Have you tried auto ref?
For some purposes, auto ref does the wrong thing. Whether you get
a reference depends on whether you pass an lvalue or an rvalue.
But some templates need to take either a struct by reference, or
a class/interface (already being a reference) by value.

This is the case deadalnix mentioned further up in this thread. I
don't know whether it applies to Manu's code, too. AFAIUI he's
more concerned about forwarding parameters in wrapper types.
Manu via Digitalmars-d
2014-09-23 11:23:45 UTC
Permalink
On 23 September 2014 20:23, via Digitalmars-d
Post by Walter Bright via Digitalmars-d
Have you tried auto ref?
For some purposes, auto ref does the wrong thing. Whether you get a
reference depends on whether you pass an lvalue or an rvalue. But some
templates need to take either a struct by reference, or a class/interface
(already being a reference) by value.
This is the case deadalnix mentioned further up in this thread. I don't know
whether it applies to Manu's code, too. AFAIUI he's more concerned about
forwarding parameters in wrapper types.
That's just in this case. I'm concerned with many different cases over
time. This is a half decade running agony for me ;)
Manu via Digitalmars-d
2014-09-23 11:19:57 UTC
Permalink
On 23 September 2014 19:45, Walter Bright via Digitalmars-d
Post by Manu via Digitalmars-d
On 22 September 2014 13:19, Walter Bright via Digitalmars-d
It's also extremely hard to unittest; explodes the number of static if paths
exponentially. I'm constantly finding bugs appear a year after writing
some code
because I missed some static branch paths when originally authoring.
If you throw -cov while running unittests, it'll give you a report on which
code was executed and which wasn't. Very simple and useful.
It is a useful tool, but you can see how going to great lengths to write this
explosion of paths is a massive pain in the first place, let alone additional
overhead to comprehensively test that it works... it should never have been a
problem to start with.
There are two separate issues here - the first is knowing whether or not the code is unittested. -cov solves that issue. The second is having the multiple code paths in the first place.
Have you tried auto ref?
auto ref has never been what I've wanted. Semantically, it makes ref
out of all value-type lvalues, including int/float.
I don't want the compiler attempting to presume what I want in my
generic code. If it's part of the type, then there are no surprises;
it is exactly what I give it, which I gave deliberately.
I don't really know what the code you show is supposed to do from a high level. My first impression is you are trying to write it like you'd write C++ code. Perhaps there's a more D idiomatic way of doing it that doesn't lead to the tangle you've got.
The only way I can think to eliminate that is to write the lot in a
huge mixin which composes the text from bits. I avoid mixin.

D has extensive language to manipulate and compare types, it has very
little to deal with these few weird external concepts like 'storage
class'. D has the most powerful type system I'm aware of, I see no
reason to break from that.
I think 'storage class' is a source of extreme complexity in D, since
it breaks uniformity with the rest of the language.
BTW, ref (as you know) is part of the type in C++. However, I can vouch for it being a special case everywhere in C++, and is a horrifying quagmire of strange edge cases. That's why it's not part of the type in D.
I've never had any problems with ref in C++. D presents the horrible
quagmire of edge cases in my experience, some of which I've presented,
but I've had many more issues in the past.
Can you give me some examples of the problems in C++ you set out to
avoid? I've been programming C++ for 20 years, and D for 5-6 years.
I've never had this problem in C++, I have it so often in D, it drives
me crazy.

I also believe that D wouldn't suffer the same problems as C++,
because we have much better type manipulation. Things like
PointerTarget!T, Unqual!T, and all the other complex type manipulation
templates are common and work well in D, but that sort of thing almost
never works well in C++. The type comparison logic in C++ is too
feeble to be comparably useful.
Walter Bright via Digitalmars-d
2014-09-24 07:50:04 UTC
Permalink
Post by Walter Bright via Digitalmars-d
BTW, ref (as you know) is part of the type in C++. However, I can vouch for
it being a special case everywhere in C++, and is a horrifying quagmire of
strange edge cases. That's why it's not part of the type in D.
I've never had any problems with ref in C++. D presents the horrible quagmire
of edge cases in my experience, some of which I've presented, but I've had
many more issues in the past. Can you give me some examples of the problems
in C++ you set out to avoid?
Q: Given a T&, and type deduction, when do you get the T and when do you get the
T& ?

A: It's different for every situation. Nobody can remember or enumerate the
cases. Maybe Scott Meyers.
Manu via Digitalmars-d
2014-09-24 20:38:30 UTC
Permalink
On 24 September 2014 17:50, Walter Bright via Digitalmars-d
Post by Walter Bright via Digitalmars-d
Post by Walter Bright via Digitalmars-d
BTW, ref (as you know) is part of the type in C++. However, I can vouch for
it being a special case everywhere in C++, and is a horrifying quagmire of
strange edge cases. That's why it's not part of the type in D.
I've never had any problems with ref in C++. D presents the horrible quagmire
of edge cases in my experience, some of which I've presented, but I've had
many more issues in the past. Can you give me some examples of the problems
in C++ you set out to avoid?
Q: Given a T&, and type deduction, when do you get the T and when do you get
the T& ?
A: It's different for every situation. Nobody can remember or enumerate the
cases. Maybe Scott Meyers.
Scott Myers presented on this in extraordinary detail. By my
recollection (without referring to his talk), it was different in
*one* case, not every case. It's a binary situation, so it can't
possibly be different in 'every case', there are only 2 cases, and one
of them is the common case.

I suggest if the idea were explored in D, we would see if it works
where type deduction always gives what you expect (ie, gives ref(T)
for ref(T)). Unlike C++, we have an extensive suite of tools like
PointerTarget!T, Unqual!T, isPointer!T and friends; we can use those
tools explicitly in the places where we want T from deduction yielding
ref(T). I suspect we can avoid the C++ deduction hack given the
presence of these tools.

And if it proves that the C++ edge case is necessary (I don't think it
will), then so be it. I've never had problems with it, and always
found it intuitive in 20 years, whereas D's current setup is
immeasurably more complex than C++'s edge case, and causes me the
greatest source of pain in the language.
Walter Bright via Digitalmars-d
2014-09-23 09:48:54 UTC
Permalink
Manu, once again your posts have the message embedded twice in them, making for
very large posts. What's happening is the posts have the text in plain text,
then the text again in HTML.

Please configure your news editor to only produce the plain text messages. The
HTML versions are ignored and uselessly consume bandwidth and disk space.
Manu via Digitalmars-d
2014-09-23 11:21:18 UTC
Permalink
On 23 September 2014 19:48, Walter Bright via Digitalmars-d
Post by Walter Bright via Digitalmars-d
Manu, once again your posts have the message embedded twice in them, making
for very large posts. What's happening is the posts have the text in plain
text, then the text again in HTML.
Please configure your news editor to only produce the plain text messages.
The HTML versions are ignored and uselessly consume bandwidth and disk
space.
I use Gmail. It was configured... but it seems that it reverts to html
mode whenever I attach an image in any email I write, and then it
remembers the setting :/
Walter Bright via Digitalmars-d
2014-09-23 18:18:21 UTC
Permalink
Post by Manu via Digitalmars-d
On 23 September 2014 19:48, Walter Bright via Digitalmars-d
Post by Walter Bright via Digitalmars-d
Manu, once again your posts have the message embedded twice in them, making
for very large posts. What's happening is the posts have the text in plain
text, then the text again in HTML.
Please configure your news editor to only produce the plain text messages.
The HTML versions are ignored and uselessly consume bandwidth and disk
space.
I use Gmail. It was configured... but it seems that it reverts to html
mode whenever I attach an image in any email I write, and then it
remembers the setting :/
Well, that message was right!
Manu via Digitalmars-d
2014-09-24 05:03:04 UTC
Permalink
On 24 September 2014 04:18, Walter Bright via Digitalmars-d
Post by Walter Bright via Digitalmars-d
Post by Manu via Digitalmars-d
On 23 September 2014 19:48, Walter Bright via Digitalmars-d
Post by Walter Bright via Digitalmars-d
Manu, once again your posts have the message embedded twice in them, making
for very large posts. What's happening is the posts have the text in plain
text, then the text again in HTML.
Please configure your news editor to only produce the plain text messages.
The HTML versions are ignored and uselessly consume bandwidth and disk
space.
I use Gmail. It was configured... but it seems that it reverts to html
mode whenever I attach an image in any email I write, and then it
remembers the setting :/
Well, that message was right!
I changed the setting back ;)
via Digitalmars-d
2014-09-22 09:22:54 UTC
Permalink
On Sunday, 21 September 2014 at 11:37:19 UTC, Manu via
Post by Manu via Digitalmars-d
On 21 September 2014 16:02, deadalnix via Digitalmars-d <
On Sunday, 21 September 2014 at 03:48:36 UTC, Walter Bright
Post by Walter Bright via Digitalmars-d
Post by Manu via Digitalmars-d
What happens when a scope() thing finds it's way into
generic code? If
the type
doesn't carry that information, then you end up in a
situation like ref.
Have
you ever had to wrestle with ref in generic code?
ref is the biggest disaster zone in D, and I think all it's
problems will
translate straight to scope if you do this.
I'm unaware of this disaster zone.
Well it is very real. I had to duplicate bunch of code in my
visitor
generator recently because of it. Getting generic code ref
correct is very
tedious, error prone, and guarantees code duplication and/or
various static
ifs all over the place.
It's also extremely hard to unittest; explodes the number of
static if
paths exponentially. I'm constantly finding bugs appear a year
after
writing some code because I missed some static branch paths
when originally
authoring.
If I understand you right, your problems come from the fact that
sometimes in a template you want ref, and sometimes you don't.

But I think this mostly doesn't apply to scope: either you borrow
things, or you don't. In particular, when you do borrow
something, you're not interested in the owner your parameter has
inside the caller, you just take it by scope (narrowing the
lifetime). Thus there needs to be no information about it inside
the callee, and you don't need different instantiations depending
on it.

One special case where scope deduction might be desirable are
template functions that apply predicates (delegates, lambdas) to
passed-in parameters, like map and filter. For these, the
scope-ness of the input range can depend on whether the
predicates are able to take their parameters as scope.
Manu via Digitalmars-d
2014-09-22 11:45:31 UTC
Permalink
On 22 September 2014 19:22, via Digitalmars-d <digitalmars-d at puremagic.com>
Post by Manu via Digitalmars-d
On 21 September 2014 16:02, deadalnix via Digitalmars-d <
Post by Manu via Digitalmars-d
What happens when a scope() thing finds it's way into generic code? If
Post by via Digitalmars-d
the type
doesn't carry that information, then you end up in a situation like
ref.
Have
you ever had to wrestle with ref in generic code?
ref is the biggest disaster zone in D, and I think all it's problems will
translate straight to scope if you do this.
I'm unaware of this disaster zone.
Well it is very real. I had to duplicate bunch of code in my visitor
generator recently because of it. Getting generic code ref correct is
very
tedious, error prone, and guarantees code duplication and/or various
static
ifs all over the place.
It's also extremely hard to unittest; explodes the number of static if
paths exponentially. I'm constantly finding bugs appear a year after
writing some code because I missed some static branch paths when
originally
authoring.
If I understand you right, your problems come from the fact that sometimes
in a template you want ref, and sometimes you don't.
But I think this mostly doesn't apply to scope: either you borrow things,
or you don't. In particular, when you do borrow something, you're not
interested in the owner your parameter has inside the caller, you just take
it by scope (narrowing the lifetime). Thus there needs to be no information
about it inside the callee, and you don't need different instantiations
depending on it.
One special case where scope deduction might be desirable are template
functions that apply predicates (delegates, lambdas) to passed-in
parameters, like map and filter. For these, the scope-ness of the input
range can depend on whether the predicates are able to take their
parameters as scope.
Application to scope will be identical to ref. A function that returns or
receives scope that is inserted into generic code must have that property
cascaded outwards appropriately. If typeof() or alias loses 'scope', then
it will all go tits-up.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20140922/d5ce13f6/attachment.html>
via Digitalmars-d
2014-09-22 12:14:52 UTC
Permalink
On Monday, 22 September 2014 at 11:45:39 UTC, Manu via
Post by Manu via Digitalmars-d
Application to scope will be identical to ref. A function that
returns or
receives scope that is inserted into generic code must have
that property
cascaded outwards appropriately. If typeof() or alias loses
'scope', then
it will all go tits-up.
For receiving it's not necessary, because whether or not the
argument is scoped, the function can always borrow it. The
lifetime of its parameter is narrower than what it gets passed.

For return values, the situation is a bit different: They can of
course not be assigned to non-scoped variables. But the solution
for this simple: the generic code needs to use scope, too. A
function that returns scope does so for a reason after all. This
will work even if the return value of the called function turns
out not to be scoped for this particular instantiation. And all
this is an implementation of the generic code, it won't bleed
outside, unless the generic code wants to return the scoped
value. In this case, simply apply the same technique, just one
lever higher.

I don't see this as a problem for (new) code written with scope
in mind. For existing code, of course some adjustments are
necessary, but the same is true if you change existing code to be
const correct, for example, or to be compatible with `shared`.
Manu via Digitalmars-d
2014-09-22 12:37:38 UTC
Permalink
On 22 September 2014 22:14, via Digitalmars-d <digitalmars-d at puremagic.com>
Post by Manu via Digitalmars-d
Application to scope will be identical to ref. A function that returns or
receives scope that is inserted into generic code must have that property
cascaded outwards appropriately. If typeof() or alias loses 'scope', then
it will all go tits-up.
For receiving it's not necessary, because whether or not the argument is
scoped, the function can always borrow it. The lifetime of its parameter is
narrower than what it gets passed.
It's particularly common in D to produce templates that wrap functions.
If the wrapper doesn't propagate scope outwards, then it can no longer be
called by a caller who borrowed arguments which are to be forwarded to the
function being called. Likewise for return values.

For return values, the situation is a bit different: They can of course not
be assigned to non-scoped variables. But the solution for this simple: the
generic code needs to use scope, too.
This is precisely the problem with ref...
Are you saying that ALL generic code needs to be 'scope' always? That's not
semantically correct.

A function that returns scope does so for a reason after all.


And the generic code can't know what it is. That knowledge must be encoded
in the type system.

This will work even if the return value of the called function turns out
not to be scoped for this particular instantiation. And all this is an
implementation of the generic code, it won't bleed outside, unless the
generic code wants to return the scoped value. In this case, simply apply
the same technique, just one lever higher.
I can't see the solution you're trying to ilustrate, can you demonstrate?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20140922/c24547ec/attachment.html>
via Digitalmars-d
2014-09-22 15:00:51 UTC
Permalink
On Monday, 22 September 2014 at 12:37:47 UTC, Manu via
Post by Manu via Digitalmars-d
On 22 September 2014 22:14, via Digitalmars-d
<digitalmars-d at puremagic.com>
Post by via Digitalmars-d
On Monday, 22 September 2014 at 11:45:39 UTC, Manu via
Post by Manu via Digitalmars-d
Application to scope will be identical to ref. A function
that returns or
receives scope that is inserted into generic code must have
that property
cascaded outwards appropriately. If typeof() or alias loses
'scope', then
it will all go tits-up.
For receiving it's not necessary, because whether or not the
argument is
scoped, the function can always borrow it. The lifetime of its
parameter is
narrower than what it gets passed.
It's particularly common in D to produce templates that wrap
functions.
If the wrapper doesn't propagate scope outwards, then it can no
longer be
called by a caller who borrowed arguments which are to be
forwarded to the
function being called. Likewise for return values.
You have a point there.
Post by Manu via Digitalmars-d
For return values, the situation is a bit different: They can
of course not
Post by via Digitalmars-d
be assigned to non-scoped variables. But the solution for this
simple: the
generic code needs to use scope, too.
This is precisely the problem with ref...
Are you saying that ALL generic code needs to be 'scope'
always? That's not
semantically correct.
To be clear, I am referring to the implementation, the actual
code of the generic functions, not to its signature. The
signature of course needs to match the semantics of the generic
function.

I also over-generalized when I said that the return value cannot
be assigned to non-scope. It can theoretically depend on the
input, though I'm not sure whether it's a good idea to allow this:

scope!a T scopeFunc(scope T a, scope T b);

T* genericFunc(T)(T* input1, T* input2) {
...
// this is fine in theory: input1 points to GC or global
data
// (because it's not designated as scope)
string temp = scopeFunc(input1, input2);
...
return temp;
}

Evidently, this generic function cannot accept scoped pointers,
thus it can't take advantage of the fact that scopeFunc() does.
It's therefore a good idea, to make any generic (and non-generic,
too) function take its parameters by scope if at all possible:

scope!input1 T* genericFunc(T)(scope T* input1, scope T*
input2) {
...
scope temp = scopeFunc(input1, input2);
...
return temp;
}

This second version of the function will work with scope and
non-scope inputs alike. More importantly, it doesn't depend on
whether it's allowed to assign a scope return value to non-scope
if its owners aren't scoped (which I'd like to avoid).

Now, `genericFunc()` in turn returns a scoped reference, so any
other generic code that calls it must again be treated in the
same way. Everything else would be unsafe, after all. But note
that this only goes as far as an actual scoped value is returned
up the call-chain. Once you stop doing so (because you only need
to call the scope-returning functions internally for intermediate
results, for example), returning scope would no longer be
necessary. It still makes sense for these higher-up functions to
_accept_ scope, of course, if it's possible.

Of course, this is only true as long as the generic function
knows about the semantics of `scopeFunc()`. Once you're trying to
wrap functions (as alias predicates, opDispatch), there needs to
be another solution. I'm not sure what this could be though. I
see now why you mentioned ref. But the problem is not restricted
to ref and scope, it would also apply to UDAs. Maybe, because it
is a more general problem independent of scope, the solution
needs to be a more general one, too.

As far as I can see, there's always a variadic template parameter
involved (which is actually a list of aliases in most cases,
right?). Would it work if aliases would forward their storage
classes, too? Thinking about it, this seems natural, because
aliases mean "pass by name".
Post by Manu via Digitalmars-d
Post by via Digitalmars-d
A function that returns scope does so for a reason after all.
And the generic code can't know what it is. That knowledge must
be encoded
in the type system.
This will work even if the return value of the called function
turns out
Post by via Digitalmars-d
not to be scoped for this particular instantiation. And all
this is an
implementation of the generic code, it won't bleed outside,
unless the
generic code wants to return the scoped value. In this case,
simply apply
the same technique, just one lever higher.
I can't see the solution you're trying to ilustrate, can you
demonstrate?
I hope that the examples above illustrate what I mean. Of course,
this doesn't solve the "perfect forwarding" problem, which should
maybe be treated separately.

Maybe you can give counter examples too, if you think it doesn't
work.
Manu via Digitalmars-d
2014-09-22 15:54:12 UTC
Permalink
On 23 September 2014 01:00, via Digitalmars-d <digitalmars-d at puremagic.com>
Post by via Digitalmars-d
Post by Manu via Digitalmars-d
On 22 September 2014 22:14, via Digitalmars-d <
digitalmars-d at puremagic.com>
On Monday, 22 September 2014 at 11:45:39 UTC, Manu via Digitalmars-d
Post by Manu via Digitalmars-d
Application to scope will be identical to ref. A function that returns
Post by Manu via Digitalmars-d
or
receives scope that is inserted into generic code must have that
property
cascaded outwards appropriately. If typeof() or alias loses 'scope',
then
it will all go tits-up.
For receiving it's not necessary, because whether or not the argument is
scoped, the function can always borrow it. The lifetime of its parameter
is
narrower than what it gets passed.
It's particularly common in D to produce templates that wrap functions.
If the wrapper doesn't propagate scope outwards, then it can no longer be
called by a caller who borrowed arguments which are to be forwarded to the
function being called. Likewise for return values.
You have a point there.
It's massive. Trust me, when you're fabricating functions from
introspecting other functions, you NEED all these details in the type. If
they're not part of the types, then you need to craft lots of junk code to
detect that information explicitly, and then you need to branch out n^^2
distinct paths (massive DRY violation) to handle all the combinations.
Imagine if 'const' was a storage class in the context of generic code...
the practicality of that would be identical to 'ref', except that const
appears everywhere, and ref appears rarely (probably because people tend to
avoid it, because it's broken).

For return values, the situation is a bit different: They can of course not
Post by via Digitalmars-d
Post by Manu via Digitalmars-d
Post by Manu via Digitalmars-d
the
generic code needs to use scope, too.
This is precisely the problem with ref...
Are you saying that ALL generic code needs to be 'scope' always? That's
not
semantically correct.
To be clear, I am referring to the implementation, the actual code of the
generic functions, not to its signature. The signature of course needs to
match the semantics of the generic function.
So...?


I also over-generalized when I said that the return value cannot be
Post by via Digitalmars-d
assigned to non-scope. It can theoretically depend on the input, though I'm
scope!a T scopeFunc(scope T a, scope T b);
T* genericFunc(T)(T* input1, T* input2) {
...
// this is fine in theory: input1 points to GC or global data
// (because it's not designated as scope)
string temp = scopeFunc(input1, input2);
...
return temp;
}
Evidently, this generic function cannot accept scoped pointers, thus it
can't take advantage of the fact that scopeFunc() does. It's therefore a
good idea, to make any generic (and non-generic, too) function take its
scope!input1 T* genericFunc(T)(scope T* input1, scope T* input2) {
...
scope temp = scopeFunc(input1, input2);
...
return temp;
}
This second version of the function will work with scope and non-scope
inputs alike. More importantly, it doesn't depend on whether it's allowed
to assign a scope return value to non-scope if its owners aren't scoped
(which I'd like to avoid).
We arrive at yet another case of "it should have been that way from the
start" wrt 'scope'.
The baggage of annotation, and the lack of annotation to existing code is a
pretty big pill to swallow.
If it were just part of the type, there would be no problem, T would
already be 'scope T' in the cases where you expect. I can't see any
disadvantages there.

Now, `genericFunc()` in turn returns a scoped reference, so any other
Post by via Digitalmars-d
generic code that calls it must again be treated in the same way.
Everything else would be unsafe, after all. But note that this only goes as
far as an actual scoped value is returned up the call-chain. Once you stop
doing so (because you only need to call the scope-returning functions
internally for intermediate results, for example), returning scope would no
longer be necessary. It still makes sense for these higher-up functions to
_accept_ scope, of course, if it's possible.
Of course, this is only true as long as the generic function knows about
the semantics of `scopeFunc()`. Once you're trying to wrap functions (as
alias predicates, opDispatch), there needs to be another solution. I'm not
sure what this could be though. I see now why you mentioned ref. But the
problem is not restricted to ref and scope, it would also apply to UDAs.
Maybe, because it is a more general problem independent of scope, the
solution needs to be a more general one, too.
I think UDA's are clearly distinct from ref and scope. UDA's can attribute
types.
I strongly believe that the problem is the notion of a 'storage class',
it's a faulty concept. It has never yet proven itself be what I've ever
wanted in any case I'm aware of.
Your proposal even implies changes to the concept as it is; like being able
to create a local that is 'scope'. Is that a recognition of existing
problems? I've been asking for 'ref' local's for half a decade...


As far as I can see, there's always a variadic template parameter involved
Post by via Digitalmars-d
(which is actually a list of aliases in most cases, right?). Would it work
if aliases would forward their storage classes, too?
I'll say, no. It's equally important that is(), typeof(), and type aliasing
all work too.
I also think this kinda undermines the notion of a storage class in
principle...
Post by via Digitalmars-d
Thinking about it, this seems natural, because aliases mean "pass by name".
Types can't be passed to alias parameters. Alias refers to a symbol, not a
type.

Even if this were to be jigged somehow, I think it really just kicks the
can forward, and ref/scope is lost somewhere else. I can't imagine ANY
situation where I would ever want that information to be lost (unless it
was deliberate, like Unqual!), so what's the end goal? We end up in a
situation where it's properly passed along everywhere that the information
is currently lost... and we have the same thing as if it were just part of
the type in the first place?
Post by via Digitalmars-d
A function that returns scope does so for a reason after all.
Post by Manu via Digitalmars-d
And the generic code can't know what it is. That knowledge must be encoded
in the type system.
This will work even if the return value of the called function turns out
Post by Manu via Digitalmars-d
not to be scoped for this particular instantiation. And all this is an
implementation of the generic code, it won't bleed outside, unless the
generic code wants to return the scoped value. In this case, simply apply
the same technique, just one lever higher.
I can't see the solution you're trying to ilustrate, can you demonstrate?
I hope that the examples above illustrate what I mean. Of course, this
doesn't solve the "perfect forwarding" problem, which should maybe be
treated separately.
This would be another band-aid to a core problem. D already has plenty of
these.
I hear this "perfect forwarding" concept thrown around, and I think it's
another faulty concept. I rarely want 'perfect' forwarding... why would I
be forwarding in the first place if it's 'perfect' (there are cases, but
not so common)? I almost always want *imperfect* forwarding; that is, some
small detail(/s) about the forwarding are manipulated. I think this is the
primary case where storage class falls apart in concept.


Maybe you can give counter examples too, if you think it doesn't work.
It's complex and time consuming to do so. The situations where it all
breaks down are often fairly complex (probably why 'ref' as a storage class
seems like an okay idea at face value, although I still don't understand
the advantage conceptually), and they tend to appear when you don't expect
it. My examples with ref above are all practically applicable to scope too
though.

Let's turn this around... Why the complexity? Why would you make the change
to your proposal to make 'scope' something else outside of the type system?
What is the advantage to that complexity. D has no structured method for
dealing with that sort of meta, we only have types. Beyond that, it's just
spaghetti, as we learn from ref.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20140923/c611c85e/attachment.html>
via Digitalmars-d
2014-09-23 11:02:38 UTC
Permalink
On Monday, 22 September 2014 at 15:54:23 UTC, Manu via
Post by Manu via Digitalmars-d
We arrive at yet another case of "it should have been that way
from the
start" wrt 'scope'.
The baggage of annotation, and the lack of annotation to
existing code is a
pretty big pill to swallow.
If it were just part of the type, there would be no problem, T
would
already be 'scope T' in the cases where you expect. I can't see
any
disadvantages there.
But it has very different semantics. You cannot expect the same
code to work for scope and non-scope alike. And it is to be
expected that you have to adjust existing code if you want to
take advantage of a new feature. If by "it should have been that
way from the start" you mean that scope should be the default,
well yes, but we're in the same situation with immutable by
default, pure by default, @safe by default, nothrow by default,
... That ship has sailed, unfortunately.
Post by Manu via Digitalmars-d
I think UDA's are clearly distinct from ref and scope. UDA's
can attribute
types.
I strongly believe that the problem is the notion of a 'storage
class',
it's a faulty concept. It has never yet proven itself be what
I've ever
wanted in any case I'm aware of.
Your proposal even implies changes to the concept as it is;
like being able
to create a local that is 'scope'. Is that a recognition of
existing
problems? I've been asking for 'ref' local's for half a
decade...
That you can't declare ref locals is not inherent in the concept
of storage class. You can already today declare scope locals, it
just doesn't have an effect. And static is a storage class, too.
Post by Manu via Digitalmars-d
This would be another band-aid to a core problem. D already has
plenty of
these.
I hear this "perfect forwarding" concept thrown around, and I
think it's
another faulty concept. I rarely want 'perfect' forwarding...
why would I
be forwarding in the first place if it's 'perfect' (there are
cases, but
not so common)? I almost always want *imperfect* forwarding;
that is, some
small detail(/s) about the forwarding are manipulated. I think
this is the
primary case where storage class falls apart in concept.
That is a straw man. Of course, perfect forwarding needs to allow
for imperfect forwarding, too. It would indeed be not useful if
you couldn't inspect and modify the types (and storage classes)
of your parameters.
Post by Manu via Digitalmars-d
Maybe you can give counter examples too, if you think it
doesn't work.
It's complex and time consuming to do so. The situations where
it all
breaks down are often fairly complex (probably why 'ref' as a
storage class
seems like an okay idea at face value, although I still don't
understand
the advantage conceptually), and they tend to appear when you
don't expect
it. My examples with ref above are all practically applicable
to scope too
though.
A concrete example would still be very helpful to understand your
point of view. I.e., not only the concrete wrapper type/function,
but also how you would want to use it, and why it wouldn't work
without scope being part of the type.
Post by Manu via Digitalmars-d
Let's turn this around... Why the complexity? Why would you
make the change
to your proposal to make 'scope' something else outside of the
type system?
Well, I think Ivan gave an excellent example (ElementType) why it
should be separate from the type. Type modifiers currently only
deal with mutability (and the related shared). There can be some
nasty surprises if we add another concept there.
Post by Manu via Digitalmars-d
What is the advantage to that complexity. D has no structured
method for
dealing with that sort of meta, we only have types. Beyond
that, it's just
spaghetti, as we learn from ref.
Then it's better to introduce such a method, IMO.
Manu via Digitalmars-d
2014-09-23 12:02:15 UTC
Permalink
On 23 September 2014 21:02, via Digitalmars-d
Post by Manu via Digitalmars-d
We arrive at yet another case of "it should have been that way from the
start" wrt 'scope'.
The baggage of annotation, and the lack of annotation to existing code is
a
pretty big pill to swallow.
If it were just part of the type, there would be no problem, T would
already be 'scope T' in the cases where you expect. I can't see any
disadvantages there.
But it has very different semantics. You cannot expect the same code to work
for scope and non-scope alike. And it is to be expected that you have to
adjust existing code if you want to take advantage of a new feature. If by
"it should have been that way from the start" you mean that scope should be
the default, well yes, but we're in the same situation with immutable by
ship has sailed, unfortunately.
You can't expect the same code to work for int and immutable(int)
alike either... or int and int[], they all have to be handled somewhat
explicitly.
Generic code shouldn't try to work with is(T == scope(U), U) if the
code doesn't support scope. If the generic code is to support scope,
then the generic code either specifies it's argument to be scope(T),
which works the same as const(T) wrt mutability, or it detects and
handles is(T == scope(U), U) internally.

And yes, that's what I meant by 'should have been that way from the start' :)
Post by Manu via Digitalmars-d
This would be another band-aid to a core problem. D already has plenty of
these.
I hear this "perfect forwarding" concept thrown around, and I think it's
another faulty concept. I rarely want 'perfect' forwarding... why would I
be forwarding in the first place if it's 'perfect' (there are cases, but
not so common)? I almost always want *imperfect* forwarding; that is, some
small detail(/s) about the forwarding are manipulated. I think this is the
primary case where storage class falls apart in concept.
That is a straw man. Of course, perfect forwarding needs to allow for
imperfect forwarding, too. It would indeed be not useful if you couldn't
inspect and modify the types (and storage classes) of your parameters.
Well you suggested like perfect forwarding was a discrete problem to solve.
Imperfect forwarding implies composing a signature from a complex set
of typeof(), alias, template arguments, traits, etc. We can't be
losing ref-ness in any of these cases, or the forwarding is broken.
scope likewise.
(im)perfect forwarding would work just fine right now if there wasn't
random bits of information that were separated from the type system.
Post by Manu via Digitalmars-d
Maybe you can give counter examples too, if you think it doesn't work.
It's complex and time consuming to do so. The situations where it all
breaks down are often fairly complex (probably why 'ref' as a storage
class
seems like an okay idea at face value, although I still don't understand
the advantage conceptually), and they tend to appear when you don't expect
it. My examples with ref above are all practically applicable to scope too
though.
A concrete example would still be very helpful to understand your point of
view. I.e., not only the concrete wrapper type/function, but also how you
would want to use it, and why it wouldn't work without scope being part of
the type.
Generating function signatures is certainly the simplest and also the
most common example of these problems.
The problem is simple; if it's not part of the type, then the only
solution is a static if with explicit detection for the thing, and
code duplication with/without the non-type attribute.
Templates are rarely flat either, they are usually a composition of
other templates. Everything in the chain needs to have special case
handling for ref (and scope) that lives outside the type system.
Post by Manu via Digitalmars-d
Let's turn this around... Why the complexity? Why would you make the
change
to your proposal to make 'scope' something else outside of the type
system?
Well, I think Ivan gave an excellent example (ElementType) why it should be
separate from the type. Type modifiers currently only deal with mutability
(and the related shared). There can be some nasty surprises if we add
another concept there.
I don't think shared is really related, also what about *, [], [n],
etc. these are all type modifiers that cause the same sort of
'surprises'.
We have tools for dealing with all of these: Unqual!T,
PointerTarget!T, isPointer!T, isArray!T, etc... they all work well, I
haven't had any complaints about them personally. scope detection
would have no significant impact on the mix.

We also have tools like is(T : U), which should be extended to work as
expected with scope.
In the same way as "is(X : const(T)) == true" works for T == mutable,
immutable, or const, we would have the same is(X : scope(T)) to detect
if the lifetimes are compatible.
We need scope in the type system so we can make use of all the
powerful type manipulation features that D has. As far as I'm
concerned, D's type system *IS* D. (this is also true for ref)
Post by Manu via Digitalmars-d
What is the advantage to that complexity. D has no structured method for
dealing with that sort of meta, we only have types. Beyond that, it's just
spaghetti, as we learn from ref.
Then it's better to introduce such a method, IMO.
Please no. D does NOT need a parallel suite of syntax to deal with
'storage class' meta.
via Digitalmars-d
2014-09-29 14:17:22 UTC
Permalink
On Saturday, 13 September 2014 at 01:49:05 UTC, Manu via
Post by Manu via Digitalmars-d
I'm not convinced this is a good change.
It sounds like you're just trading one problem with another
more sinister
problem...
Ok, I thought about it some more. I'm still not convinced
completely, but I'm warming up to the idea of making scope a type
modifier. However, Ivan's objections need to be addressed.

Maybe let's start with a list of problems of the type modifier
way. So far:

* What is ElementType!(ByLineImpl!(char, "\n")) in the example
from the wiki page [1]?
* Should Unqual!T strip `scope`?

Anything else? How can we solve these problems?

Another argument against storage class is a syntactical ambiguity
in conncection with methods:

struct S {
scope!this int* foo() scope;
}

It's ambiguous whether any given scope keyword applies to the
return value or `this`. One could argue though that this is a
consequence of the general function attribute problem and should
preferably be fixed there. (`ref` doesn't have this problem
because it cannot apply to `this`.)

[1]
http://wiki.dlang.org/User:Schuetzm/scope#scope.21.28const_....29
Ivan Timokhin via Digitalmars-d
2014-10-03 19:05:24 UTC
Permalink
* What is ElementType!(ByLineImpl!(char, "\n")) in the example from the
wiki page [1]?
OK, I think I have an idea, but it's not overly elegant.

First of all, I would like to note that the same issue exists with, for
example, findSubstring function (from "scope with owners" section),
which does not have a definite return type (so it's unclear what
ReturnType!findSubstring should be).

Now, an idea that I have is that bare scope should be just a syntactic
sugar for self-owned scope; i.e., `scope(int*) x;` would be just a short
way of writing `scope!x(int*) x;`. This would imply that every scoped
variable has its own unique type, which is probably not that terrible,
considering that they can't be assigned freely to each other either way.
This is also somewhat more natural, because it means that there is no
such thing as just "scoped" variable, it is always connected to a
particular lifetime.

Then, the following:
---
string findSubstring(scope(string) haystack,
scope(string) needle)
---
would be equivalent to
---
string findSubstring(scope!haystack(string) haystack,
scope!needle(string) needle)
---
so each argument is self-owned and all ownership information is lost
(but function is still callable with scoped arguments, because scope
narrowing is allowed).

To preserve ownership information, which is encoded in an arguments'
types, template is needed:
---
scope!haystackOwner(string)
findSubstring(alias haystackOwner)
(scope!haystackOwner(string) haystack,
scope(string) needle)
---
So, findSubstring *still* doesn't have a definite return type, but now
it's because it is a function template, not a function.

As for passing unscoped strings to findSubstring, I see two alternatives:
1) Declare that all unscoped references are implicitly convertible to
scope!GC or something like that (and this conversion is used in such
cases). This one is probably better.
2) Require a separate overload for an unscoped string.

Now, to address ElementType problem, I would like to reconsider `this`
lifetime. Since
---
struct S {
void f() {}
}
---
is more or less equivalent to
---
// not a valid D code
struct S {}
void f(ref S this) {}
---
(not strictly equivalent, of course, but the general idea is like that),
`this` inside a method body would behave like a ref parameter and have
approximately the same lifetime as normal parameters.

Then this code:
---
@property scope!(const this)(Char[]) front() const
---
would become illegal, since the return value is declared as having
function-local scope. However, this code:
---
@property scope!(const this)(Char[])
front(alias thisOwner)() const scope!thisOwner
---
would work just fine, because it explicitly propagates current object's
scope (the return type seems the same, but `this` itself now has a
different type). The downside is that `front` is now callable only for
scoped variables (it seems inappropriate to convert structs to scope!GC).

Now, ByLineImpl!(char, "\n") wouldn't be an input range, because front
would not be defined in an unscoped case, and `scope ByLineImpl!(char,
"\n")` is not a type (because bare scope would be purely a syntactic
sugar in declarations), so not a range.

However, with this declaration:
---
scope(ByLineImpl!(char, "\n")) x = ...;
---
typeof(x) would be scope!x(ByLineImpl!(char, "\n")), and
ElementType!(typeof(x)) would be scope!(const x)(char[]).

As I have said in the beginning, not too elegant, but I think it may
work. As a bonus, this is a little bit more straightforward and requires
less special-casing from the compiler side: has only one scope type
instead of two and no magic tricks with scoped return values. The only
problems that I can see right away are that the code now is a little
verbose, and that 'front' is restricted to scoped variables in a new
version.

Any thoughts?
via Digitalmars-d
2014-10-04 13:38:12 UTC
Permalink
29.09.2014 18:17, "Marc =?UTF-8?B?U2Now7x0eiI=?=
* What is ElementType!(ByLineImpl!(char, "\n")) in the example from the
wiki page [1]?
OK, I think I have an idea, but it's not overly elegant.
First of all, I would like to note that the same issue exists
with, for example, findSubstring function (from "scope with
owners" section), which does not have a definite return type
(so it's unclear what ReturnType!findSubstring should be).
Indeed. It's everywhere that an owner depends on another function
argument (including `this`), but AFAICS nowhere else.
Now, an idea that I have is that bare scope should be just a
syntactic sugar for self-owned scope; i.e., `scope(int*) x;`
would be just a short way of writing `scope!x(int*) x;`. This
would imply that every scoped variable has its own unique type,
which is probably not that terrible, considering that they
can't be assigned freely to each other either way. This is also
somewhat more natural, because it means that there is no such
thing as just "scoped" variable, it is always connected to a
particular lifetime.
I've already suggested this as an implementation detail.
---
string findSubstring(scope(string) haystack,
scope(string) needle)
---
would be equivalent to
---
string findSubstring(scope!haystack(string) haystack,
scope!needle(string) needle)
---
so each argument is self-owned and all ownership information is
lost (but function is still callable with scoped arguments,
because scope narrowing is allowed).
To preserve ownership information, which is encoded in an
---
scope!haystackOwner(string)
findSubstring(alias haystackOwner)
(scope!haystackOwner(string) haystack,
scope(string) needle)
---
So, findSubstring *still* doesn't have a definite return type,
but now it's because it is a function template, not a function.
This of course has the unfortunate side effect of incredible
template bloat: For any distinct passed argument (which in
practice means for almost every call), we'd get a new template
instance. IMO this is not acceptable.
As for passing unscoped strings to findSubstring, I see two
1) Declare that all unscoped references are implicitly
convertible to scope!GC or something like that (and this
conversion is used in such cases). This one is probably better.
OTOH it would preclude automatic demoting of GC allocations to
stack allocations. And it would marry us to the GC is "default"
allocation strategy, which we might want to move away from
(probably in favor of generic allocators changeable at any point
in time).

We also need to allow borrowing of temporaries if we want to have
rvalue references. And this runs into problems if we are dealing
with scoped non-references, for which I'm seeing more use cases
now than just file descriptors (in particular reference
counting). In short, I'd like to avoid it.
2) Require a separate overload for an unscoped string.
Ugly, of course, and I believe it's not necessary.
Now, to address ElementType problem, I would like to reconsider
`this` lifetime. Since
---
struct S {
void f() {}
}
---
is more or less equivalent to
---
// not a valid D code
struct S {}
void f(ref S this) {}
---
(not strictly equivalent, of course, but the general idea is
like that),
`this` inside a method body would behave like a ref parameter
and have approximately the same lifetime as normal parameters.
---
@property scope!(const this)(Char[]) front() const
---
would become illegal, since the return value is declared as
---
@property scope!(const this)(Char[])
front(alias thisOwner)() const scope!thisOwner
---
would work just fine, because it explicitly propagates current
object's scope (the return type seems the same, but `this`
itself now has a different type). The downside is that `front`
is now callable only for scoped variables (it seems
inappropriate to convert structs to scope!GC).
And of course, again, the template bloat :-( This time, a new
instance of `front` for every instance of the range.
Now, ByLineImpl!(char, "\n") wouldn't be an input range,
because front would not be defined in an unscoped case, and
`scope ByLineImpl!(char, "\n")` is not a type (because bare
scope would be purely a syntactic sugar in declarations), so
not a range.
---
scope(ByLineImpl!(char, "\n")) x = ...;
---
typeof(x) would be scope!x(ByLineImpl!(char, "\n")), and
ElementType!(typeof(x)) would be scope!(const x)(char[]).
As I have said in the beginning, not too elegant, but I think
it may work. As a bonus, this is a little bit more
straightforward and requires less special-casing from the
compiler side: has only one scope type instead of two and no
magic tricks with scoped return values. The only problems that
I can see right away are that the code now is a little verbose,
and that 'front' is restricted to scoped variables in a new
version.
Any thoughts?
I think the key is in separating the scope attribute and the
owner. The former needs to be part of the type, the latter
doesn't. In this vein, it's probably a good idea to restrict the
`scope!owner` syntax to function signatures, where it may only
refer to other parameters and `this`. The use cases for it
elsewhere are very marginal (if they exist at all).

This naturally makes the return type of `findSubstring()` just
`scope(string)`; declaring a variable of it simply works:

typeof(findSubstring("", "")) s = findSubstring("Hello,
world", "world");

is equivalent to:

scope(string) s = findSubstring("Hello, world", "world");

This is a valid assignment, and owner propagation would even take
care of preserving the owners (though only on declaration, but
that's natural because the owners are only known at the call
site):

string haystack, needle;
scope(string) s = findSubstring(haystack, needle);
// type of `s` is scope(string), owner is `haystack`

In other words, the type part of `scope!a(T)` is just `scope(T)`,
the owner is not part of the type and tracked separately.

Let's look at isInputRange:

template isInputRange(R)
{
enum bool isInputRange = is(typeof(
(inout int = 0)
{
R r = R.init; // can define a range object
if (r.empty) {} // can test for empty
r.popFront(); // can invoke popFront()
auto h = r.front; // can get the front of the
range
}));
}

auto byline = stdin.byLine();

`isInputRange!(typeof(byline))` expands to:

scope(ByLineImpl...) r = scope(ByLineImpl...).init;
if (r.empty) {}
r.popFront();
auto h = r.front;

The first line is valid, the last line too, because `scope` is
now part of the type and will of course be deduced by `auto`.

So this solves the template bloat problem, along with making
ElementType usable. That still leaves us with the problem of
forwarding and wrappers.

auto trace(alias func, Args...)(Args args) {
writeln("Calling " ~ func.stringof ~ "(" ~ args.stringof
~ ")");
return func(args);
}

auto s = trace!findSubstring(haystack, needle);

Here, `Args` becomes `(scope(string), scope(string))`. This
cannot work, of course, because it drops the owners. How can we
a) preserve the owners on the parameters, and b) propagate the
result's owner back outward?

@Manu: Under the condition that we'd find a solution for "perfect
forwarding", or imperfect if you like ;-), preferrably one also
applicable to `ref`, would you be okay with the above?
Ivan Timokhin via Digitalmars-d
2014-10-04 17:01:57 UTC
Permalink
Post by via Digitalmars-d
29.09.2014 18:17, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm at gmx.net>"
...
Now, an idea that I have is that bare scope should be just a syntactic
sugar for self-owned scope; i.e., `scope(int*) x;` would be just a
short way of writing `scope!x(int*) x;`. This would imply that every
scoped variable has its own unique type, which is probably not that
terrible, considering that they can't be assigned freely to each other
either way. This is also somewhat more natural, because it means that
there is no such thing as just "scoped" variable, it is always
connected to a particular lifetime.
I've already suggested this as an implementation detail.
Sorry then, must have missed it.
Post by via Digitalmars-d
...
This of course has the unfortunate side effect of incredible template
bloat: For any distinct passed argument (which in practice means for
almost every call), we'd get a new template instance. IMO this is not
acceptable.
They could be merged in an executable, but a symbol table would be
cluttered. That does sound like a major issue.
Post by via Digitalmars-d
1) Declare that all unscoped references are implicitly convertible to
scope!GC or something like that (and this conversion is used in such
cases). This one is probably better.
OTOH it would preclude automatic demoting of GC allocations to stack
allocations. And it would marry us to the GC is "default" allocation
strategy, which we might want to move away from (probably in favor of
generic allocators changeable at any point in time).
It doesn't necessarily have to be GC, just any object that we can safely
treat as an owner.
Post by via Digitalmars-d
...
I think the key is in separating the scope attribute and the owner. The
former needs to be part of the type, the latter doesn't. In this vein,
it's probably a good idea to restrict the `scope!owner` syntax to
function signatures, where it may only refer to other parameters and
`this`. The use cases for it elsewhere are very marginal (if they exist
at all).
This naturally makes the return type of `findSubstring()` just
typeof(findSubstring("", "")) s = findSubstring("Hello, world",
"world");
scope(string) s = findSubstring("Hello, world", "world");
This is a valid assignment, and owner propagation would even take care
of preserving the owners (though only on declaration, but that's natural
string haystack, needle;
scope(string) s = findSubstring(haystack, needle);
// type of `s` is scope(string), owner is `haystack`
In other words, the type part of `scope!a(T)` is just `scope(T)`, the
owner is not part of the type and tracked separately.
That's ok with me (in fact, it looks very nice), but I think Manu's
point was that dealing with anything that isn't part of the type is
troublesome.
Post by via Digitalmars-d
...
Ivan Timokhin via Digitalmars-d
2014-10-04 18:12:57 UTC
Permalink
Post by Ivan Timokhin via Digitalmars-d
Post by via Digitalmars-d
I think the key is in separating the scope attribute and the owner. The
former needs to be part of the type, the latter doesn't. In this vein,
it's probably a good idea to restrict the `scope!owner` syntax to
function signatures, where it may only refer to other parameters and
`this`. The use cases for it elsewhere are very marginal (if they exist
at all).
This naturally makes the return type of `findSubstring()` just
typeof(findSubstring("", "")) s = findSubstring("Hello, world",
"world");
scope(string) s = findSubstring("Hello, world", "world");
This is a valid assignment, and owner propagation would even take care
of preserving the owners (though only on declaration, but that's natural
string haystack, needle;
scope(string) s = findSubstring(haystack, needle);
// type of `s` is scope(string), owner is `haystack`
In other words, the type part of `scope!a(T)` is just `scope(T)`, the
owner is not part of the type and tracked separately.
That's ok with me (in fact, it looks very nice), but I think Manu's
point was that dealing with anything that isn't part of the type is
troublesome.
On the second thought, doesn't syntax look a bit awkward now? I mean,
`scope!a(T)` certainly looks like `scope!a` as a whole is a type
modifier. Since this syntax is pretty much limited to function return
types with this proposal, maybe a better solution would be to have an
owner as a separate annotation on a function?

Also, would it really make much sense to track the owner further than
the assignment of a function's return value? That seems to complicate
things a lot by adding a hidden attribute to a variable that is not only
invisible at the declaration (even though the full type is spelled out),
but, in fact, cannot be specified explicitly (because there's no syntax
for that, now that scope with owners is limited to function signatures).

How about this:
---
scope(string) haystack, needle;
// next assignment is okay, because `s` is guaranteed not to outlive
// `haystack`.
scope(string) s = findSubstring(haystack, needle);
// type of `s` is now scope(string), no additional information
// attached

// so the next assignment is disallowed:
//needle = s; // error!
---

This could be unnecessarily limiting, but would it really cause much
trouble?
via Digitalmars-d
2014-10-04 19:22:44 UTC
Permalink
Post by Ivan Timokhin via Digitalmars-d
On the second thought, doesn't syntax look a bit awkward now? I
mean, `scope!a(T)` certainly looks like `scope!a` as a whole is
a type modifier. Since this syntax is pretty much limited to
function return types with this proposal, maybe a better
solution would be to have an owner as a separate annotation on
a function?
I agree, but it should still stay closely associated to the
return type.
Post by Ivan Timokhin via Digitalmars-d
Also, would it really make much sense to track the owner
further than the assignment of a function's return value? That
seems to complicate things a lot by adding a hidden attribute
to a variable that is not only invisible at the declaration
(even though the full type is spelled out), but, in fact,
cannot be specified explicitly (because there's no syntax for
that, now that scope with owners is limited to function
signatures).
---
scope(string) haystack, needle;
// next assignment is okay, because `s` is guaranteed not
to outlive
// `haystack`.
scope(string) s = findSubstring(haystack, needle);
// type of `s` is now scope(string), no additional
information
// attached
//needle = s; // error!
---
This could be unnecessarily limiting, but would it really cause
much trouble?
I think you're right, I thought about this after I replied to
you. It would be the logical next step. On the other hand, I
wouldn't want to lose const borrowing, because it turned out to
be a requirement for safe moving. But I think it can still be
tracked internally (owner tracking is necessary for
implementation anyway).
via Digitalmars-d
2014-10-04 19:25:59 UTC
Permalink
On Saturday, 4 October 2014 at 18:13:03 UTC, Ivan Timokhin
Post by Ivan Timokhin via Digitalmars-d
Also, would it really make much sense to track the owner
further than the assignment of a function's return value? That
seems to complicate things a lot by adding a hidden attribute
to a variable that is not only invisible at the declaration
(even though the full type is spelled out), but, in fact,
cannot be specified explicitly (because there's no syntax for
that, now that scope with owners is limited to function
signatures).
---
scope(string) haystack, needle;
// next assignment is okay, because `s` is guaranteed not
to outlive
// `haystack`.
scope(string) s = findSubstring(haystack, needle);
// type of `s` is now scope(string), no additional
information
// attached
//needle = s; // error!
---
This could be unnecessarily limiting, but would it really
cause much trouble?
I think you're right, I thought about this after I replied to
you. It would be the logical next step. On the other hand, I
wouldn't want to lose const borrowing, because it turned out to
be a requirement for safe moving. But I think it can still be
tracked internally (owner tracking is necessary for
implementation anyway).
Owner tracking is then completely limited to one expression. I
think this will simplify the implementation a lot. Besides, it
has precedences: uniqueness is also only tracked inside an
expression, AFAIK.

Ivan Timokhin via Digitalmars-d
2014-09-12 20:24:50 UTC
Permalink
...
Post by Ivan Timokhin via Digitalmars-d
This troubles me the most, because currently return type of a function
may depend only on types of its arguments, and there is a lot of
templated code written in that assumption.
I'm sorry, I don't understand what you mean here. This is clearly not
true, neither for normal functions, nor for templates.
Well, me and my bad English. I was trying to say that currently, AFAIK,
for any symbols f, a, b, if f(a) is valid and the types of a and b
match, then f(b) is also valid and types of f(a) and f(b) match. OTOH,
if the types of a and b are different, types of f(a) and f(b) may also
be different (because of overloading or templates like T f(T)(T x)).

This would change if scope became a type qualifier and f was something
like scope!x(T) f(T)(scope(T) x).
...
Marco Leise via Digitalmars-d
2014-09-11 18:06:39 UTC
Permalink
Am Thu, 11 Sep 2014 13:58:38 +0000
Post by via Digitalmars-d
PING
Now that there are again several GC related topics being
discussed, I thought I'd bump this thread.
Would be nice if Walter and/or Andrei could have a look and share
there opinions. Is this something worth pursuing further? Are
there fundamental objections against it?
I just needed this again for a stack based allocator. It would
make such idioms safer where you return a pointer into an RAII
struct and need to make sure it doesn't outlive the struct.
It got me a nasty overwritten stack. I cannot comment on the
implementation, just that I have long felt it is missing.
--
Marco
via Digitalmars-d
2014-09-19 11:34:26 UTC
Permalink
PING again
Post by via Digitalmars-d
PING
Now that there are again several GC related topics being
discussed, I thought I'd bump this thread.
Would be nice if Walter and/or Andrei could have a look and
share there opinions. Is this something worth pursuing further?
Are there fundamental objections against it?
Post by via Digitalmars-d
In the "Opportunities for D" thread, Walter again mentioned
the topics ref counting, GC, uniqueness, and borrowing, from
which a lively discussion developed [1]. I took this thread as
an opportunity to write down some ideas about these topics.
The result is a rather extensive proposal for the
http://wiki.dlang.org/User:Schuetzm/scope
This is not a real DIP, but before I put more work into
formalizing it, I'd like to hear some thoughts from the
* Is this the general direction we want to go? Is it
acceptable in general?
* Is the proposal internally consistent?
* How big would the effort to implement it be? (I suspect it's
a large amount of work, but relatively straightforward.)
[1] http://forum.dlang.org/thread/lphnen$1ml7$1 at digitalmars.com
Andrei Alexandrescu via Digitalmars-d
2014-09-19 15:14:55 UTC
Permalink
Post by via Digitalmars-d
PING again
Thanks for your work. I've put it on my todo list. -- Andrei
Walter Bright via Digitalmars-d
2014-09-21 01:27:27 UTC
Permalink
Previous discussions:

http://www.digitalmars.com/d/archives/digitalmars/D/borrowed_pointers_vs_ref_232090.html

http://www.digitalmars.com/d/archives/digitalmars/D/RFC_scope_and_borrowing_240834.html

http://www.digitalmars.com/d/archives/digitalmars/D/scope_escaping_222858.html

http://www.digitalmars.com/d/archives/digitalmars/D/ref_is_unsafe_184935.html
Vladimir Panteleev via Digitalmars-d
2014-09-21 01:39:23 UTC
Permalink
Forum links:

http://forum.dlang.org/?group=digitalmars.D&artnum=232090
http://forum.dlang.org/?group=digitalmars.D&artnum=240834
http://forum.dlang.org/?group=digitalmars.D&artnum=222858
http://forum.dlang.org/?group=digitalmars.D&artnum=184935
Walter Bright via Digitalmars-d
2014-09-21 03:38:29 UTC
Permalink
I think it's a well thought out proposal. Thanks for doing this!

A couple thoughts:

1. const can be both a storage class and a type constructor. Scope is only a
storage class. The scope(int) syntax implies scope is a type constructor, too.

const int* a; // const used as storage class
const(int*) b; // const used as type constructor

The type constructor syntax should be disallowed for const.


2. I think there is quite a bit of overlap between scope and ref. Essentially,
ref does everything scope does, except deal with classes. I'm not terribly
comfortable with such a large overlap, it implies something is wrong. I don't
have an answer at the moment.
Jacob Carlborg via Digitalmars-d
2014-09-21 08:25:28 UTC
Permalink
Post by Walter Bright via Digitalmars-d
2. I think there is quite a bit of overlap between scope and ref.
Essentially, ref does everything scope does, except deal with classes.
I'm not terribly comfortable with such a large overlap, it implies
something is wrong. I don't have an answer at the moment.
Am I missing something but isn't "ref" for passing something by
reference instead of by value. "scope", in this proposal, is for dealing
with lifetime? Or do you have any other proposal for what "ref" might
become?
--
/Jacob Carlborg
Walter Bright via Digitalmars-d
2014-09-21 08:34:09 UTC
Permalink
Am I missing something but isn't "ref" for passing something by reference
instead of by value. "scope", in this proposal, is for dealing with lifetime? Or
do you have any other proposal for what "ref" might become?
See this discussion:

http://www.digitalmars.com/d/archives/digitalmars/D/borrowed_pointers_vs_ref_232090.html
via Digitalmars-d
2014-09-21 09:11:03 UTC
Permalink
Post by Walter Bright via Digitalmars-d
I think it's a well thought out proposal. Thanks for doing this!
1. const can be both a storage class and a type constructor.
Scope is only a storage class. The scope(int) syntax implies
scope is a type constructor, too.
const int* a; // const used as storage class
const(int*) b; // const used as type constructor
The type constructor syntax should be disallowed for const.
(... disallowed for _scope_, I assume)

I originally intended it to be part of the type. Ivan Timokhin
pointed out severe problems with that [1], so I removed it from
the proposal. The syntax is a remainder of that.

But before I remove it too, I have a question: Will it still be
possible to use the storage class syntax for members of
aggregates?

struct S {
scope!myAllocator int* p;
}
Post by Walter Bright via Digitalmars-d
2. I think there is quite a bit of overlap between scope and
ref. Essentially, ref does everything scope does, except deal
with classes. I'm not terribly comfortable with such a large
overlap, it implies something is wrong. I don't have an answer
at the moment.
I also see the overlap, but there are also large differences to
the point that I cannot see how the two concepts could be
unified. For one, `ref` with classes (and reference types in
general) is troublesome, and would introduce a double indirection
for borrowing, when a simple copy of the reference would be
sufficient. Then, `scope` could also for non-reference types.
File handles come to mind, which are often integers, and just
need to be copied.

[1]
http://forum.dlang.org/thread/etjuormplgfbomwdrurp at forum.dlang.org?page=3#post-lusirm:2421d9:241:40digitalmars.com
Walter Bright via Digitalmars-d
2014-09-22 06:00:04 UTC
Permalink
Post by via Digitalmars-d
Post by Walter Bright via Digitalmars-d
I think it's a well thought out proposal. Thanks for doing this!
1. const can be both a storage class and a type constructor. Scope is only a
storage class. The scope(int) syntax implies scope is a type constructor, too.
const int* a; // const used as storage class
const(int*) b; // const used as type constructor
The type constructor syntax should be disallowed for const.
(... disallowed for _scope_, I assume)
Yes, my mistake.
Post by via Digitalmars-d
I originally intended it to be part of the type. Ivan Timokhin pointed out
severe problems with that [1], so I removed it from the proposal. The syntax is
a remainder of that.
But before I remove it too, I have a question: Will it still be possible to use
the storage class syntax for members of aggregates?
struct S {
scope!myAllocator int* p;
}
Possible, but exactly how that would work remains to be seen.
bearophile via Digitalmars-d
2014-09-23 09:17:47 UTC
Permalink
Post by via Digitalmars-d
http://wiki.dlang.org/User:Schuetzm/scope
If a mutable argument of a function is tagged as unique, the type
system guarantees that there are no other references to it. So
can a function 'foo' like this be "strongly pure"?


int[] foo(unique int[] a) pure {
a[0]++;
return a;
}

Bye,
bearophile
via Digitalmars-d
2014-09-23 10:16:25 UTC
Permalink
Post by bearophile via Digitalmars-d
Post by via Digitalmars-d
http://wiki.dlang.org/User:Schuetzm/scope
If a mutable argument of a function is tagged as unique, the
type system guarantees that there are no other references to
it. So can a function 'foo' like this be "strongly pure"?
int[] foo(unique int[] a) pure {
a[0]++;
return a;
}
I think so. But note that `unique` is not part of my proposal, I
merely used it in the example. I think it could be implemented
relatively easily, because DMD internally already has a concept
of uniqueness that is used for converting things to immutable
implicitly. But this would be a different proposal.
via Digitalmars-d
2014-09-23 10:26:27 UTC
Permalink
Post by via Digitalmars-d
Post by bearophile via Digitalmars-d
Post by via Digitalmars-d
http://wiki.dlang.org/User:Schuetzm/scope
If a mutable argument of a function is tagged as unique, the
type system guarantees that there are no other references to
it. So can a function 'foo' like this be "strongly pure"?
int[] foo(unique int[] a) pure {
a[0]++;
return a;
}
I think so.
Ok, I take it back ;-) Steven is right. It is however the case
that this function's return value would still be unique.
Steven Schveighoffer via Digitalmars-d
2014-09-23 10:29:25 UTC
Permalink
On 9/23/14 6:26 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm at gmx.net>"
Ok, I take it back ;-) Steven is right. It is however the case that this
function's return value would still be unique.
Yes, it could be unique. I haven't read this thread really, so I don't
know what has been proposed, but looking at the snippet, wouldn't you
have to tag the return value? You tagged the parameter with unique.

-Steve
via Digitalmars-d
2014-09-23 11:05:49 UTC
Permalink
On Tuesday, 23 September 2014 at 10:29:25 UTC, Steven
Post by Steven Schveighoffer via Digitalmars-d
On 9/23/14 6:26 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?=
Ok, I take it back ;-) Steven is right. It is however the case that this
function's return value would still be unique.
Yes, it could be unique. I haven't read this thread really, so
I don't know what has been proposed, but looking at the
snippet, wouldn't you have to tag the return value? You tagged
the parameter with unique.
Bearophile did, not me. But yes, you would have to, absent an
extension to return type inference. As I already replied to him,
uniqueness was really just used in an example because it made it
cleaner; it's mostly unrelated to my proposal.
deadalnix via Digitalmars-d
2014-09-23 20:54:12 UTC
Permalink
On Tuesday, 23 September 2014 at 10:29:25 UTC, Steven
Post by Steven Schveighoffer via Digitalmars-d
On 9/23/14 6:26 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?=
Ok, I take it back ;-) Steven is right. It is however the case that this
function's return value would still be unique.
Yes, it could be unique. I haven't read this thread really, so
I don't know what has been proposed, but looking at the
snippet, wouldn't you have to tag the return value? You tagged
the parameter with unique.
-Steve
Unique is a bad name. You want to have various reference locally,
and the whole discussion is about scope.

Which bring us to the main point here, before discussing
borrowing, we'd better define ownership.
Steven Schveighoffer via Digitalmars-d
2014-09-23 10:21:15 UTC
Permalink
Post by via Digitalmars-d
http://wiki.dlang.org/User:Schuetzm/scope
If a mutable argument of a function is tagged as unique, the type system
guarantees that there are no other references to it. So can a function
'foo' like this be "strongly pure"?
int[] foo(unique int[] a) pure {
a[0]++;
return a;
}
I don't think so. Strong pure function optimizations would not work for
something like:

auto x = foo(a) ~ foo(a);

Which for a strong pure function could be optimized to:

auto r = foo(a);
auto x = r ~ r;

-Steve
bearophile via Digitalmars-d
2014-09-23 11:11:36 UTC
Permalink
Post by Steven Schveighoffer via Digitalmars-d
Post by bearophile via Digitalmars-d
int[] foo(unique int[] a) pure {
...
I don't think so. Strong pure function optimizations would not
auto x = foo(a) ~ foo(a);
This is similar to:

unique x1 = foo(a);
unique x2 = foo(a);
unique x = x1 ~ x2;

When the call to the first foo ends you have a x1 reference to
the array data. Such reference x1 is unique, so now 'a' is not
usable any more, you can't pass 'a' to foo once more.
I need to learn more about such stuff of linear typing.

Bye,
bearophile
Steven Schveighoffer via Digitalmars-d
2014-09-23 11:56:38 UTC
Permalink
Post by bearophile via Digitalmars-d
Post by Steven Schveighoffer via Digitalmars-d
Post by bearophile via Digitalmars-d
int[] foo(unique int[] a) pure {
...
I don't think so. Strong pure function optimizations would not work
auto x = foo(a) ~ foo(a);
unique x1 = foo(a);
unique x2 = foo(a);
unique x = x1 ~ x2;
When the call to the first foo ends you have a x1 reference to the array
data. Such reference x1 is unique, so now 'a' is not usable any more,
you can't pass 'a' to foo once more.
I need to learn more about such stuff of linear typing.
This begs the question, what is the point of having "strong purity" if
you can't optimize based on it?

-Steve
bearophile via Digitalmars-d
2014-09-23 12:12:48 UTC
Permalink
Post by Steven Schveighoffer via Digitalmars-d
This begs the question, what is the point of having "strong
purity" if you can't optimize based on it?
Linear typing gives some guarantees that help the GC a lot, and
avoid some coding mistakes. And some people could answer you that
having (strongly) pure functions is a quality regardless of
optimizations, because it makes both unit testing and code
understanding simpler :-) I am not very expert on this stuff, I
need to learn more.

Bye,
bearophile
Continue reading on narkive:
Search results for 'RFC: scope and borrowing' (Questions and Answers)
16
replies
The great depression- What year did it end?!?
started 2007-01-15 12:10:36 UTC
homework help
Loading...