Parameterized unit testing and benchmarking of phobos

[Robert burner Schadek via Digitalmars-d]

Lately, I find myself wondering, if I should add parameterized
unit tests to std.string, because the last few bugs I fixed where
not caught by tests, as the test-data was not good enough. I know
random data is not perfect either, but it would be good addition
IMO.

Additionally, I thought these unit tests could be used to
benchmark the performance of the functions and the compilers.
Thus allowing continues monitoring of performance changes.

I'm thinking of something like:

version(unittest_benchmark) {
unittest {
auto ben = Benchmark("bench_result_file", numberOfRuns);
auto values = ValueGen!(StringGen(0, 12), IntGen(2, 24))(ben);

foreach(string rStr, int rInt; values) {
auto rslt = functionToTest(rStr, rInt);
// some asserts
}
}
}

The bench_result_file would be a csv with e.g. date,runtime,...

ideas, suggestions?

[Andrei Alexandrescu via Digitalmars-d]

Post by Robert burner Schadek via Digitalmars-d
version(unittest_benchmark) {
unittest {

No need for the outer braces :o). -- Andrei

[Robert burner Schadek via Digitalmars-d]

On Wednesday, 8 October 2014 at 23:31:59 UTC, Andrei Alexandrescu

Post by Andrei Alexandrescu via Digitalmars-d

Post by Robert burner Schadek via Digitalmars-d
version(unittest_benchmark) {
unittest {

No need for the outer braces :o). -- Andrei

I just love my braces.
If that's gone be the most negative comment, I will have a PR
ready before next week.

[Andrei Alexandrescu via Digitalmars-d]

Post by Robert burner Schadek via Digitalmars-d

Post by Andrei Alexandrescu via Digitalmars-d

Post by Robert burner Schadek via Digitalmars-d
version(unittest_benchmark) {
unittest {

No need for the outer braces :o). -- Andrei

I just love my braces.

If you love your braces you gotta love your indentation. They come
together... -- Andrei

[Robert burner Schadek via Digitalmars-d]

On Thursday, 9 October 2014 at 00:13:25 UTC, Andrei Alexandrescu

Post by Andrei Alexandrescu via Digitalmars-d
If you love your braces you gotta love your indentation. They
come together... -- Andrei

I guess you're right, but back to the topic.
Has anyone tried something similar for phobos? (I couldn't find
anything)
Does anyone else consider the idea worth pursuing?

[Andrei Alexandrescu via Digitalmars-d]

Post by Robert burner Schadek via Digitalmars-d

Post by Andrei Alexandrescu via Digitalmars-d
If you love your braces you gotta love your indentation. They come
together... -- Andrei

I guess you're right, but back to the topic.
Has anyone tried something similar for phobos? (I couldn't find anything)
Does anyone else consider the idea worth pursuing?

Not yet but I think we need to create a task and a plan for adding the
old std.benchmark to std.experimental.benchmark, and use it internally
for Phobos. -- Andrei

[Johannes Pfau via Digitalmars-d]

Am Thu, 09 Oct 2014 08:11:33 -0700

Post by Andrei Alexandrescu via Digitalmars-d

Post by Robert burner Schadek via Digitalmars-d

Post by Andrei Alexandrescu via Digitalmars-d
If you love your braces you gotta love your indentation. They come
together... -- Andrei

I guess you're right, but back to the topic.
Has anyone tried something similar for phobos? (I couldn't find
anything) Does anyone else consider the idea worth pursuing?

Not yet but I think we need to create a task and a plan for adding
the old std.benchmark to std.experimental.benchmark, and use it
internally for Phobos. -- Andrei

+1, std.benchmark should be revived. There's still the constructor
cycles issue but we can probably find workarounds. In the worst
case we do not generate constructors but runBenchmark functions and
call these manually via some runAllBenchmarks("std.xyz", "std.abc")
template magic)

[Andrei Alexandrescu via Digitalmars-d]

Post by Johannes Pfau via Digitalmars-d
Am Thu, 09 Oct 2014 08:11:33 -0700

Post by Andrei Alexandrescu via Digitalmars-d

Post by Robert burner Schadek via Digitalmars-d

Post by Andrei Alexandrescu via Digitalmars-d
If you love your braces you gotta love your indentation. They come
together... -- Andrei

I guess you're right, but back to the topic.
Has anyone tried something similar for phobos? (I couldn't find
anything) Does anyone else consider the idea worth pursuing?

Not yet but I think we need to create a task and a plan for adding
the old std.benchmark to std.experimental.benchmark, and use it
internally for Phobos. -- Andrei

+1, std.benchmark should be revived. There's still the constructor
cycles issue but we can probably find workarounds. In the worst
case we do not generate constructors but runBenchmark functions and
call these manually via some runAllBenchmarks("std.xyz", "std.abc")
template magic)

My suggestion is to first use it in std.internal for a couple of months,
and only then propose it for inclusion. -- Andrei

[Robert burner Schadek via Digitalmars-d]

On Thursday, 9 October 2014 at 16:36:35 UTC, Johannes Pfau wrote:>

Post by Johannes Pfau via Digitalmars-d
+1, std.benchmark should be revived.

-1, This really does not solve my problem. I need a parameterized
unit test facility aka. Haskell QuickCheck with the additional
benchmarking feature that allows me/us to track performance over
time aka. benchmark result dump to file in csv style manner.

[Andrei Alexandrescu via Digitalmars-d]

Post by Robert burner Schadek via Digitalmars-d
On Thursday, 9 October 2014 at 16:36:35 UTC, Johannes Pfau wrote:>

Post by Johannes Pfau via Digitalmars-d
+1, std.benchmark should be revived.

-1, This really does not solve my problem. I need a parameterized unit
test facility aka. Haskell QuickCheck with the additional benchmarking
feature that allows me/us to track performance over time aka. benchmark
result dump to file in csv style manner.

version (xyz) unittest
{
...
}

is a good idiom. I noticed you feel strongly about bracing but I
unrecommend it (and we should disallow it for phobos) as it forces you
to either double-indent everything (net loss) or create an exception
with regard to indentation. Also the opening extra brace has a "garden
path sentence" liability, it suggests more stuff will be encompassed by
the brace but then only at the closing brace there's the realization
there's only encompassed entity.

The idiom is already used, see e.g.
https://github.com/D-Programming-Language/phobos/blob/master/std/file.d#L322.


Andrei

[Robert burner Schadek via Digitalmars-d]

I get the bracing point, but how does this translate to my unit
testing (quickcheck) problem?

[Jacob Carlborg via Digitalmars-d]

Lately, I find myself wondering, if I should add parameterized unit
tests to std.string, because the last few bugs I fixed where not caught
by tests, as the test-data was not good enough. I know random data is
not perfect either, but it would be good addition IMO.
Additionally, I thought these unit tests could be used to benchmark the
performance of the functions and the compilers. Thus allowing continues
monitoring of performance changes.
version(unittest_benchmark) {
unittest {
auto ben = Benchmark("bench_result_file", numberOfRuns);
auto values = ValueGen!(StringGen(0, 12), IntGen(2, 24))(ben);
foreach(string rStr, int rInt; values) {
auto rslt = functionToTest(rStr, rInt);
// some asserts
}
}
}
The bench_result_file would be a csv with e.g. date,runtime,...
ideas, suggestions?

I think you should use a UDA's instead. The a unit test framework can,
hopefully, handle this automatically.

[Robert burner Schadek via Digitalmars-d]

Post by Jacob Carlborg via Digitalmars-d
I think you should use a UDA's instead. The a unit test
framework can, hopefully, handle this automatically.

This will work for me private project, but I want this in
std.string.

[Jacob Carlborg via Digitalmars-d]

This will work for me private project, but I want this in std.string.

Why wouldn't this working std.string?

[Robert burner Schadek via Digitalmars-d]

Post by Jacob Carlborg via Digitalmars-d

Post by Robert burner Schadek via Digitalmars-d
This will work for me private project, but I want this in
std.string.

Why wouldn't this working std.string?

because, this would require changing the phobos unittester, doing
it non intrusive inside the unittest is way more likely to
succeed. add is more flexible IMO.

[Martin Nowak via Digitalmars-d]

Lately, I find myself wondering, if I should add parameterized unit
tests to std.string, because the last few bugs I fixed where not caught
by tests, as the test-data was not good enough. I know random data is
not perfect either, but it would be good addition IMO.

https://github.com/MartinNowak/qcheck
Haven't used it for quite a while, file a bug report if anything doesn't
work.

[Robert burner Schadek via Digitalmars-d]

Post by Martin Nowak via Digitalmars-d

Lately, I find myself wondering, if I should add parameterized unit
tests to std.string, because the last few bugs I fixed where
not caught
by tests, as the test-data was not good enough. I know random
data is
not perfect either, but it would be good addition IMO.

https://github.com/MartinNowak/qcheck
Haven't used it for quite a while, file a bug report if
anything doesn't work.

well quite a nice and big library. You add the benchmark feature,
get a merged into phobos and I will gladly use it to test
std.string.

[Martin Nowak via Digitalmars-d]

well quite a nice and big library. You add the benchmark feature, get a
merged into phobos and I will gladly use it to test std.string.

Not sure whether a random testing library belongs into phobos.

[Robert burner Schadek via Digitalmars-d]

Post by Martin Nowak via Digitalmars-d

Post by Robert burner Schadek via Digitalmars-d
well quite a nice and big library. You add the benchmark
feature, get a
merged into phobos and I will gladly use it to test std.string.

Not sure whether a random testing library belongs into phobos.

After taking a look at the bug I found in std.string, I think
that this is a good first step.