Pre-release testing

[Tony van der Hoff]

On Fri, 14 Apr 2000, at 11:06:00, Jonathan Coxhead
<jonathan at doves.demon.co.uk> wrote on the subject "Pre-release testing":

>   See examples.c.test for (an out-of-date) one of those.
>
Or source.test.c.test for an up to date one :-)
>
>   When I was the maintainer, at each release I used to check that 
>all the examples compiled (c.test was the best one), and that any 
>differences between the s files aand h files as generated by the 
>previous version were for reasons I clearly understood (regression 
>testing). 

C.test compiled at the alpha stage; prior to that it picked up the name
duplications in the headers and bodies of the variable structures. I
admit to have forgotten to re-compile it at the later stages. However,
ironically, I *did* diff the headers and the s files prior to release.
Unfortunately, I missed the _W problem, and I probably assumed the
macros to be correct. They certainly didn't look badly wrong :-(

Jonathan mentions diffing the s and the h files, but does not mention
the H. Does C correctness imply ASM correctness?

Although I'm quite good at proof-reading text, I'm unable to spot
problems from a large diff; the amount of data is simply too great to
concentrate for long. I don't think the problem is peculiar to me.
 
I believe anything relying on human interaction is doomed to failure.
That is, of course, no reason not to try it, in the absence of anything
else. 

However, some sort of automation, per Stewart's proposal looks like a
step in the right direction, although according to Tom, it would not
have been any better in catching the macro error than the simple test.c.  

Is it worth doing? If so, how to do it? My guess is, that DefMod could
be persuaded to generate test files whilst it is generating the C
headers, containing a declaration of each type. These would then be
compiled when the library build is complete. I guess I'll have to grab
the bull by the horns, and delve into DefMod; something I've avoided so
far. Unless someone else wants to volunteer?

In any case, I will add test.c to the main library build; that will
overcome the problem of my absent mindedness in future. This is really
the crux of the present issue.

>I also made sure that whatever programme I was working on 
>at the time still worked, to give a bit of confidence about run-time 
>behaviour.
>
I guess that's how Tom found the remaining post-release errors. It is a
shame they didn't get picked up at the beta stage, but that's life. My
TestFW (which is what I was concentrating on) worked fine, but then it
does not #include the faulty headers (and, therefore, arguably, is an
unsatisfactory test). 
Unfortunately, what I'm *really* working on - Borland C++ Builder/SQL
Links - has little relevance in this case. I'd rather be working on RISC
OS, but it doesn't pay the bills :-( 
-- 
Tony van der Hoff         |  Mailto:tony at mk-net.demon.co.uk
                          |  Mailto:avanderhoff at iee.org
Buckinghamshire, England  |  http:www.mk-net.demon.co.uk