@safety of Array

[Brad Roberts via Digitalmars-d]

I've been spot checking parts of phobos today to see what all isn't
@safe ready. I'm not shocked that Array isn't, but doesn't it need to be?

For instance, not even the most basic of uses works:

$ git diff -U5
diff --git a/std/container/array.d b/std/container/array.d
index 4a1bfb4..2672bc6 100644
--- a/std/container/array.d
+++ b/std/container/array.d
@@ -826,11 +826,11 @@ $(D r)
length = offset1 + tailLength;
return this[length - tailLength .. length];
}
}

-unittest
+ at safe unittest
{
Array!int a;
assert(a.empty);
}

==> std/container/array.d(833): Error: safe function
'std.container.array.__unittestL831_1' cannot call system function
'std.container.array.Array!int.Array.~this'

I know it's a tricky implementation, but let's focus on the goal..
should Array be usable in @safe code?

[Jakob Ovrum via Digitalmars-d]

On Monday, 13 October 2014 at 00:42:09 UTC, Brad Roberts via

Post by Brad Roberts via Digitalmars-d
I know it's a tricky implementation, but let's focus on the

Looks like the `RefCounted` dependency has to be made @safe first.

[Andrei Alexandrescu via Digitalmars-d]

Post by Brad Roberts via Digitalmars-d
I know it's a tricky implementation, but let's focus on the goal..

Yes. In order for that to be 100% automatically checkable, we need the
rules restricting escape of addresses of returns by reference. -- Andrei

[Brad Roberts via Digitalmars-d]

Post by Andrei Alexandrescu via Digitalmars-d

Post by Brad Roberts via Digitalmars-d
I know it's a tricky implementation, but let's focus on the goal..

Yes. In order for that to be 100% automatically checkable, we need the
rules restricting escape of addresses of returns by reference. -- Andrei

100% checkable isn't required right now. For it to be used in an @safe
context all that's needed is liberal use of @trusted. That can be
refined over time to a more checked version. We shouldn't wait for
checkability.

Will one of you experts in the impl of Array volunteer to make the
appropriate changes?

[monarch_dodra via Digitalmars-d]

On Monday, 13 October 2014 at 17:16:40 UTC, Brad Roberts via

On 10/13/2014 7:47 AM, Andrei Alexandrescu via Digitalmars-d

Post by Andrei Alexandrescu via Digitalmars-d

Post by Brad Roberts via Digitalmars-d
I know it's a tricky implementation, but let's focus on the
goal..

Yes. In order for that to be 100% automatically checkable, we
need the
rules restricting escape of addresses of returns by reference. -- Andrei

100% checkable isn't required right now. For it to be used in
That can be refined over time to a more checked version. We
shouldn't wait for checkability.
Will one of you experts in the impl of Array volunteer to make
the appropriate changes?

The issue is that it's *not* safe though. You can escape the
reference, destroy it, and end up with a dangling pointer.
Arbitrarily marking things as trusted seriously undermines what
safe means. @trusted should be used with extreme caution.

[monarch_dodra via Digitalmars-d]

Example:

ref int getAsRef(int a) @safe //Magic!
{
RefCounted!int rc = a;
return rc.getPayload();
}

I wouldn't want to be on the calling end of this "safe"
function...

[Brad Roberts via Digitalmars-d]

Post by monarch_dodra via Digitalmars-d
On Monday, 13 October 2014 at 17:16:40 UTC, Brad Roberts via

Post by Brad Roberts via Digitalmars-d

Post by Andrei Alexandrescu via Digitalmars-d

Post by Brad Roberts via Digitalmars-d
I know it's a tricky implementation, but let's focus on the goal..

Yes. In order for that to be 100% automatically checkable, we need the
rules restricting escape of addresses of returns by reference. -- Andrei

100% checkable isn't required right now. For it to be used in an
@safe context all that's needed is liberal use of @trusted. That can
be refined over time to a more checked version. We shouldn't wait for
checkability.
Will one of you experts in the impl of Array volunteer to make the
appropriate changes?

The issue is that it's *not* safe though. You can escape the reference,
destroy it, and end up with a dangling pointer. Arbitrarily marking
be used with extreme caution.

That's why I asked the question I did. The core question isn't about
what the current implementation is or does but about where it should end
up. Should Array be usable in @safe code. So far:

Jakob: focused on impl
Andrei: yes
Monarch: focused on impl

I totally agree that @trusted must be used with lots of caution. But my
point in that post was that impl isn't the issue and requiring that
everything be fixed and perfect also isn't the issue. If we don't know
and understand where we want to be, the chances of accidentally landing
there are rather low.

More and more code is being created in Phobos all the time, and it's use
in @safe code is largely an afterthought. Please don't derail this
thread and talk about process.. keep this thread focused on Array.

Thanks,
Brad

[Dicebot via Digitalmars-d]

On Tuesday, 14 October 2014 at 01:47:10 UTC, Brad Roberts via

Post by Brad Roberts via Digitalmars-d
That's why I asked the question I did. The core question isn't
about what the current implementation is or does but about

It should but it can't unless something like `scope` is
implemented. We can't have @safe in standard library that doesn't
hold @safe promises.

[monarch_dodra via Digitalmars-d]

On Tuesday, 14 October 2014 at 01:47:10 UTC, Brad Roberts via

Post by Brad Roberts via Digitalmars-d

Post by monarch_dodra via Digitalmars-d
On Monday, 13 October 2014 at 17:16:40 UTC, Brad Roberts via

On 10/13/2014 7:47 AM, Andrei Alexandrescu via Digitalmars-d

Post by Andrei Alexandrescu via Digitalmars-d

Post by Brad Roberts via Digitalmars-d
I know it's a tricky implementation, but let's focus on the goal..

Yes. In order for that to be 100% automatically checkable,
we need the
rules restricting escape of addresses of returns by
reference. -- Andrei

100% checkable isn't required right now. For it to be used
in an
@safe context all that's needed is liberal use of @trusted.
That can
be refined over time to a more checked version. We shouldn't
wait for
checkability.
Will one of you experts in the impl of Array volunteer to
make the
appropriate changes?

The issue is that it's *not* safe though. You can escape the
reference,
destroy it, and end up with a dangling pointer. Arbitrarily
marking
things as trusted seriously undermines what safe means.
@trusted should
be used with extreme caution.

That's why I asked the question I did. The core question isn't
about what the current implementation is or does but about
Jakob: focused on impl
Andrei: yes
Monarch: focused on impl
caution. But my point in that post was that impl isn't the
issue and requiring that everything be fixed and perfect also
isn't the issue. If we don't know and understand where we want
to be, the chances of accidentally landing there are rather low.
More and more code is being created in Phobos all the time, and
don't derail this thread and talk about process.. keep this
thread focused on Array.
Thanks,
Brad

You say I'm focused on impl, but @safe *is* an implementation
certification.

I'm not derailing the thread or talking about process. If Array
can't be certified memory safe, then it can't be marked as @safe.
That's really all there is to it.

[Brad Roberts via Digitalmars-d]

Post by monarch_dodra via Digitalmars-d
certification.
I'm not derailing the thread or talking about process. If Array can't be
all there is to it.

Sorry, the request to not derail was for future posts to the thread, not
a reaction to your comment. I was/am worried that the "more and more
code being created" comment would spiral the discussion sideways.

Your response wasn't making any statements of what should happen but
rather why it can't based on the current state. Useful, but still
irrelevant to the Should question. Unless you were saying that it
shouldn't become usable due to that part of the api. To that I'd
respond that your thinking is too narrow in scope or too black and white.

As to the rest, once we decide if Array should be usable in the @safe
subset of the language, then we can start to make choices about how to
achieve that. Some obvious choices:

1) remove the parts that aren't (unlikely to be a good choice)

2) partition the api into parts that are and parts that aren't (only
some parts get the @safe attribute, maybe some gets @trusted)

3) improve the implementation of @safe to cover all the parts that can't
right now (likely to result in significant delay before any useful
progress is made)

4) force the parts that aren't anyway (probably violates the basic
precepts of @safety but including for the sake of completeness)

5) ?

6) some combination of the above

My personal thinking is that #2 is the way to go in the short term as
long as a reasonably large subset of the functionality can be made
usable (right now we can't even construct one). With a likely very
strategic sprinkling of @trusted where absolutely necessary. As #3
progresses on it's own merits, the set that falls into #2 might expand.

[monarch_dodra via Digitalmars-d]

On Tuesday, 14 October 2014 at 17:59:43 UTC, Brad Roberts via

Post by Brad Roberts via Digitalmars-d

Post by monarch_dodra via Digitalmars-d
certification.
I'm not derailing the thread or talking about process. If
Array can't be
That's really
all there is to it.

Sorry, the request to not derail was for future posts to the
thread, not a reaction to your comment. I was/am worried that
the "more and more code being created" comment would spiral the
discussion sideways.
Your response wasn't making any statements of what should
happen but rather why it can't based on the current state.
Useful, but still irrelevant to the Should question. Unless
you were saying that it shouldn't become usable due to that
part of the api. To that I'd respond that your thinking is too
narrow in scope or too black and white.
As to the rest, once we decide if Array should be usable in the
@safe subset of the language, then we can start to make choices
1) remove the parts that aren't (unlikely to be a good choice)
2) partition the api into parts that are and parts that aren't
@trusted)
that can't right now (likely to result in significant delay
before any useful progress is made)
4) force the parts that aren't anyway (probably violates the
completeness)
5) ?
6) some combination of the above
My personal thinking is that #2 is the way to go in the short
term as long as a reasonably large subset of the functionality
can be made usable (right now we can't even construct one).
absolutely necessary. As #3 progresses on it's own merits, the
set that falls into #2 might expand.

I'm on vacation on a phone, so I'll be brief for now.

I replied what I said because I felt some arguments were leading
to: let's make it trusted and worry about implementation layer.

I'la ignore the comment that my vision is narrow and discuss
improvement possibilities.

The issue witH safe/unsafe split is that the functions themselves
aren't actually unsafe, but rather their vombination: deletion is
only unsafe *if* an escape has occurred.

The funny thing is that Array used to be sealed (specifically to
avoid escapes) abd could have been safe. I unsealed it with
Andrei because sealed containers come with their own problems.

To the topic at hand though, I don't think safety should dictate
our implementations. In particular, dmd improvements can and will
mean that something unsafe can become safe later.

The real question here is when will we implement the long
promised scope?

[Dicebot via Digitalmars-d]

On Tuesday, 14 October 2014 at 17:59:43 UTC, Brad Roberts via

Post by Brad Roberts via Digitalmars-d

Post by monarch_dodra via Digitalmars-d
certification.
I'm not derailing the thread or talking about process. If
Array can't be
That's really
all there is to it.

Sorry, the request to not derail was for future posts to the
thread, not a reaction to your comment. I was/am worried that
the "more and more code being created" comment would spiral the
discussion sideways.
Your response wasn't making any statements of what should
happen but rather why it can't based on the current state.
Useful, but still irrelevant to the Should question. Unless
you were saying that it shouldn't become usable due to that
part of the api. To that I'd respond that your thinking is too
narrow in scope or too black and white.
As to the rest, once we decide if Array should be usable in the
@safe subset of the language, then we can start to make choices
1) remove the parts that aren't (unlikely to be a good choice)
2) partition the api into parts that are and parts that aren't
@trusted)

There are no distinct @system parts in Array API, it is a general
reference escaping issue with manually managed memory.

Post by Brad Roberts via Digitalmars-d
that can't right now (likely to result in significant delay
before any useful progress is made)

There no issues in @safe itself, language is simply missing the
tools to ensure no reference escaping (== scope) right now.

Post by Brad Roberts via Digitalmars-d
My personal thinking is that #2 is the way to go in the short
term as long as a reasonably large subset of the functionality
can be made usable (right now we can't even construct one).
absolutely necessary. As #3 progresses on it's own merits, the
set that falls into #2 might expand.

There is nothing that can be done here right now as far as I see
it. If you think otherwise PR is welcome :)

[Jakob Ovrum via Digitalmars-d]

On Tuesday, 14 October 2014 at 01:47:10 UTC, Brad Roberts via

Post by Brad Roberts via Digitalmars-d
That's why I asked the question I did. The core question isn't
about what the current implementation is or does but about

Well, I think it goes without saying that every piece of
interface that is memory safe should be @safe. Granted, for
abstract interfaces, like the range and container concepts, or
virtual functions in classes and `interface`'s, some care is
needed to determine whether or not memory safety should be a
requirement, but for concrete implementations like `Array` we
should provide the strongest guarantees possible.

To elaborate on my thinking for my previous reply: I don't know
if Array's interface in its entirety is memory safe (element
access looks murky, as was pointed out), but construction is
definitely a memory safe part of it; so yes, I think there's work
to be done here, and to that end, the safety of `RefCounted` has
to be considered first. My initial thoughts are that at least
construction of `RefCounted` should also be memory safe.

@safe vs @trusted is a red herring, it's just a matter of whether
memory safety is automatically or manually checked.

(That said, the trap here is to disregard @safe because of
current implementation deficits and just *assume* memory safety
without proper checking and applying @trusted everywhere as a
stop-gap measure. Just because it's clear that the interface is
SafeD doesn't mean the implementation is (yet), and this is what
minimized @trusted helps figure out. For example - although I'm
not sure if it technically violates SafeD - if malloc signals OOM
in Array.this, the code will steam ahead and try to emplace
elements at an offset from null. Just applying @trusted and
moving on would not catch cases like this.)

I don't think it's true that @safe is an afterthought for new
code. New generic code uses @safe unit tests to test
safe-readiness, and new non-generic code without regard for
attributes is also pointed out in review. The problem with
attributes is overwhelmingly with old code, not new code.