I think that the crux of the problem is here. For reviews to be
effective, they have to be a part of the corporate culture, and the
developers have to believe in them. If reviewers consider the review "a
waste of time", they will be. If reviewers look forward to the
possibility of reviewing code (and having their own code reviewed, of
course), as a positive experience, then reviews can be effective.

I'll admit that if reviews are an accepted policy, I don't see how
sending the code to review before testing could be considered "a waste
of time". I would argue just the reverse -- the preliminary testing is
a waste of time. After all, the review WILL take place, and it would
find the errors anyway. The cost of doing a review is roughly
proportional to the number of lines of code, and largely independant of
the number of errors found (within reason, of course).

Note that reviews have another important psychological effect. You know
that others will be seeing your code, so you take more pride in it. You
make more effort to make it readable, and who says readable says
maintainable. (For that matter, unreadable code is generally rejected
out of hand, as not being ready for review.)


Agreed. If it is a question of fixing one small bug, we would often
even skip the review. Provided it is a *small* bug; the size is judged
by the impact, and as often as not, errors occur higher up. If the
error is a design problem, then you correct the design first.

In practice, once the review procedure is ingrained in the process, the
question doesn't come up. I've seen less than one error per 100.000
lines of code going into integration, and I've heard of companies doing
a lot better than that. That doesn't leave too many small bugs to fix.
In this respect, we found design review to be even more important than
code review -- coding errors are small bugs, an oversight in design can
have large repercussions (even if the case triggering it is rare).


Exactly. The error which took me the most time to find was a simple
case where I used the expression "i := i + 0" to increment an index.
(This was in my pre-C days, and long before I'd even heard of code
review.) I had gotten to the point of looking at the generated
assembler, and swearing at the compiler because it hadn't generated any
code for my line, when someone, looking over my shoulder and seeing the
code for the first time, asked me what the purpose of that line could
possibly be.

It is, I fear, typical, at least for me.


Code review is pretty good for that, too. With the advantage that it
also makes them realize that the comments aren't perfect either, and
that what was "obvious" when he was deep in the coding isn't obvious at
all to someone approaching the code from the outside.

Do both, but if the tests do reveal any errors, investigate what went
wrong in the code review.

My experience is that when you first introduce reviews, they don't find
that many errors. And the errors show up in the tests (after code
review -- at this point, it is *very* important that no tests have taken
place beforehand). The review team (the developer, one to three
reviewers, and the moderator) then sits down together to ****yse why the
errors got that far, and how they could have done the review (and the
coding, for that matter) differently so that the errors wouldn't have
gotten past the review. After about six months of such a policy, if the
participants take it seriously, you'll find that testing finds
pratically no errors.

Interestingly enough, because of the psychological effect I mentioned
before (you're writing for your collegues, and not for the compiler),
you'll find that the total number of errors, whether found by code
review or by testing, drops, typically by well over half, and sometimes
a lot more.

--
James Kanze GABI Software mailto:kanze@gabi-soft.fr
Conseils en informatique orientée objet/ http://www.gabi-soft.fr
Beratung in objektorientierter Datenverarbeitung
11 rue de Rambouillet, 78460 Chevreuse, France, Tél. : +33 (0)130234516

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

tom houlder

I doubt NT 1.0 was particularly stable. If it was, it would certainly
not be in Microsoft's tradition.

Not at all a proof, but I consider it a fairly strong indication.


It's about how developers interact, not about a cathedral building
vs bazaar junk. Even I know that, and I think you do too.


You can catch a bug either before or after an official release is out.
According to you, practically all bugs should be caught before the
release. I pointed out that gcc has a very active process for
catching
bugs after the release, and thus does not follow your guidelines at
all. I also tried to make it probable that many or perhaps most of
the compilations of the new code were buggy.


I take what you've said on face value. The problem is that you don't
seem to mean what you say. Now, it's not compiling a part of the
program but linking the final product. Could you then please
elaborate
a little? Do you consider it OK to have a unit test that doesn't work
after linking? And, do you consider it OK to have a beta program
that doesn't work after linking?

Perhaps you should stop saying that "compiled code works"? It
does not seem accurate for what you actually mean. That's probably
also why Francis had to go out and remind us that you're working on
systems where a single bug may have big consequenses.


I base it on the egcs team's statement that they wanted more
interaction with users, more frequent official versions and a "faster
debugging cycle". Between the last two good versions (2.95 and
3.2) there have been numerous buggy versions. Before egcs, I
have the impression that every function should pass through
one of the selected few before being incorporated into the main source tree.

I think most of gcc's developers have learned by doing. If every
new function had been under the scrutiny you suggest, I suspect
the process had been bogged down waiting for people who could
to do proper code reviews. In the end it could put off developers
with ideas but without the needed experience. Better release it
with possibly a few minor bugs. I've no problems with this, if I
use some new functionality in some program, I expect it to be buggy.


I'm happy we can finally pinpoint our different opinions. You base
your
opinion on your experience and on what you've read. I base my
opinions
on my experience and the fact that most of the desktop programs I use
have had numerous buggy versions in the past.


--
Tom Houlder
thoulder@houlder.net

Chic - C++ without hassles
http://www.houlder.net


[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

I agree in general, although I don't think that the g++ team have any
rules forbidding several developers to be geographically close, or
actually meeting. This is one of the reasons that a purely open
development cannot attain the same quality levels that are possible in a
commercial organization. (On the other hand, at least one commercial
organization I talked with boasted of their tele-working, and the fact
that their employees sometimes lived a thousand miles from the office.
The result is, of course, that their products are of even less quality
than many free products. The fact that commercial organizations can, in
theory, do better doesn't mean that they do:-(.)


The speed with which Linux crashes, too:-(.

Seriously, there are many factors involved. The most obvious is that
many commercial organizations don't use efficient processes either. The
"thousands of reviewers" claimed by the "bazaar" process may be along
the same lines as thousands of monkeys typing to produce Shakespeare,
but it certainly isn't worse than absolutely no one ever looking at the
code. There is also a question of specifications -- of what you are
trying to do. Both Windows NT and SVR4 were developped in much less
time than Linux, with many more features. Not because its development
process was necessarily better, but because there was a concrete
decision that this was what was needed; in the early days of Linux,
things got added more or less on an ad hoc basis, because someone
capable of writing the software needed it.


Prototyping has a role, as does doing trade show versions. No problem,
provided one knows that that is what one is doing.

I've said it before, and I'll say it again. Programming is a team
activity. A programmer working alone cannot produce code of acceptable quality.


Today, mostly large scale servers. My current project tracks orders on
the European stock markets. My last application allocated dynamic IP
addresses for Germany's largest WAP portal. I've also done a lot of
work on telephone routing systems. In the past, I've also worked on
compilers, and on lower level embedded systems -- a locomotive brake system, for example.


When you're programming very close to the hardware, there are additional
parameters to consider. My experience, from the time I did that sort of
thing, is that more than half the time, the "error" is due to the
hardware not reacting like the hardware people said it would.

My experience in this field dates to before I learned anything about
software processes. I suspect that in such cases, you'd have to draw
the hardware team into the process as well -- I suspect, in fact, that
something like a code review for the hardware circuit would improve
things on that side as well. But I have no experience to confirm or
contradict it, and I don't know of any work in development processes at that level.

There I disagree. There are a certain number of things where testing is
almost useless, and only code review can guarantee any reasonable
quality. Memory management is one of them -- you'll never be able to
forsee the exact sequences of allocations and frees which will cause a
problem, in order to write a test to exercise it.


But that's exactly the way things work in most shops I've seen. Whether
you are doing things that way, or using the waterfall model, is
orthogonal to whether you have an effective development process or not.

There are a lot of people working in companies without any real
development process. That is certain; you just have to look at the
quality of most of the software you be to realize that. The result is
that tailoring your compiler to please these markets will probably
result in increased sales -- there is definitly a demand for IDE's with
integrated debuggers, for example, even though we know that integrated
debuggers have nothing to do in a process which produces quality
software.

Still, that doesn't change reality. A lot of companies are spending
more than they need to to develop software with more errors than there
should be. For various reasons, "process" has become associated with
quality (only), and the fact that it also reduces cost is overlooked.
In fact, there is a general belief that because it imcreases quality, it
must also increase cost, where as just the opposite is true.

--
James Kanze GABI Software mailto:kanze@gabi-soft.fr
Conseils en informatique orientée objet/ http://www.gabi-soft.fr
Beratung in objektorientierter Datenverarbeitung
11 rue de Rambouillet, 78460 Chevreuse, France, Tél. : +33 (0)130234516

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

[I pretty much agree with most of what you are saying. However...]


I have to disagree here. Face to face interaction is important, for
many things. The problem isn't that commercial companies can't do
better, it's that they don't. Many don't have formalized review
meetings, and a surprising number don't do any code review at all.
There are exceptions, but many commercial companies seem to be doing
their best to prove that free software is better.

I've been fortunate with my customers. Formal code review has been the
case in most of the larger projects, and even in the smaller ones, we
organized informal, face-to-face idea exchanges, after having read each
others' code, to achieve much the same effect. The resulting quality
was better than anything you will be able to download from the net.

It's rare that you actually discuss things at that level. And if you
do, there's always the white board.

It is a very poor process if face-to-face reviews leave no written
record. All of the defects found must be recorded; those which required
correction must be checked off before the developper can check his code
in.

Some of the free software is run in a very professional manner. I still
think that commercial operations have the advantages, but if they fail
to use them... Indirect peer reviews in a mailing list is still a long
sight better than no code review at all.

--
James Kanze GABI Software mailto:kanze@gabi-soft.fr
Conseils en informatique orientée objet/ http://www.gabi-soft.fr
Beratung in objektorientierter Datenverarbeitung
11 rue de Rambouillet, 78460 Chevreuse, France, Tél. : +33 (0)130234516

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

I can say; I'm just reporting what I have heard. From my experience,
however, what excapes my eye doesn't escape the reviewer's eye (and vice
versa), except when the language makes a special effort (like the C++
function declaration trap, where a variable with initialization looks
like a function declarations).


OK. That's really what I would do, even with a "clean" language.
Compile, then review, then check in for the integration team (which, in
a small project, might actually be the developers wearing different
hats).

--
James Kanze GABI Software mailto:kanze@gabi-soft.fr
Conseils en informatique orientée objet/ http://www.gabi-soft.fr
Beratung in objektorientierter Datenverarbeitung
11 rue de Rambouillet, 78460 Chevreuse, France, Tél. : +33 (0)130234516

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

llewelly

"Tom Houlder" <thoulder@houlder.net> writes:
[snip]

[snip]

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

bart van ingen schenau

On 8 Jul 2003 20:00:32 -0400, "Tom Houlder" <thoulder@houlder.net>


You can, but if you want to have bugs found after the release depends
a lot on the costs involved.
I work in the mobile-phone industry, and here any bug that is still in
the product after the release could mean the end of the company,
because the consumers just don't trust your products anymore.
As you can see, that gives a very strong incentive for a development
process that does not leave any bugs in the final product.

If, on the other hand, you work mainly on software targeted for
desktop computers, the incentive for producing bug-free software seems
to be a lot less, because it seems generally accepted that programs on
desktop computers can crash now and then.

As I pointer out above, this depends largely on the costs involved
with having the customers find bugs for you. And that is not just the
cost of solving the problem, but also the loss in goodwill.
<snip>

And this difference of opinion seems to come from the fact that in
some environments buggy versions are accepted as a fact of life and in
other environments buggy versions are totally unacceptable.

B. v Ingen Schenau

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

ben hutchings

Given how much work Cygnus (now Red Hat) has been doing on GCC for
the last decade or so, I think it's a fair bet that many of the
developers have desks a few metres or tens of metres apart...


How is GCC not commercial? It is largely developed and supported by
companies like Red Hat, SuSE, IBM...
<snip>

SVR4 (1990?) is based on SVR3 and BSD and its code can be traced back
to version 4 of Unix (1975) (that's the first version written in C).
Linux began development in 1991 and today has far more functionality
than SVR4 did. With Windows NT you may have a point; that began at
around the same time as Linux and is ahead in some areas.
<snip>

Obvious counter-example: Don Knuth. But he is perhaps a rare exception.
<snip>


<snip>

This may account for the differing impressions people get of the
stability of operating systems. Both Windows NT and Linux can be
stable on good hardware but very unreliable elsewhere because of a
hardware bug (or a bug in the specific driver - it may be hard to
pin blame to one or the other).

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

ed avis

kanze@gabi-soft.fr writes:

I am very sceptical of this claim. A reviewer may be a highly skilled
programmer but he or she is not infallible. It's just as crazy to
claim that a review will find all bugs as to claim that testing will.

As Knuth famously said:

'Beware of bugs in the above code; I have only proved it correct,
not tried it.'

So what, then, is the cost of the programmer doing a test? I think it
depends very much on the application, but the time taken is probably
proportional to the complexity of the interface being implemented.


that change is just a refactoring and shouldn't change the existing
behaviour, then there is no reason not to run the existing test suite
and check for regressions. Why wait for a review first? You can test
now and still get your review later.


This is an argument for having code reviews. It is not an argument
against programmers testing their code before review time.


Yes, and myself, I make more effort to make the code work. If I know
code is going to be seen by others I am more likely to test it (or
test it more thoroughly) before they see it.


Hmm, this is an interesting approach. Sometimes after finding a bug
during testing I will look back over existing code to find other
instances of the same bug, or something that could be like it. But
often what went wrong is some silly mistake like i += 0. I don't
think those need a full investigation during review; I would however
investigate what went wrong if the test suite failed to catch that bug.

Surely you could apply the same approach to testing. After each test
failure the developer (or developers, if you are using XP and two
people wrote the code together) ****yse what went wrong. If the bug
wasn't caught by early tests but only in production, you also have to
work out why the test suite didn't catch it. --
Ed Avis <ed@membled.com>

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

heliosc

I should have chosen my words more carefully. I thinking more of a
formal peer review, where a bunch of developers get down and go over
code line by line. It does happen in the open source world, but there,
code that gets to the peer review stage is usually already
well-tested. Communication in a distributed environment is too
expensive to clutter these peer reviews with issues that could have
easily been caught by compiling and doing some basic testing and
debugging.


I'm talking about developing formal design plans here. Writing out
every object the software will have, defining exact interfaces between
all the objects, etc. High level design is easy enough to do over the
net (though, as a verbal person, I find it nearly impossible to do it
without face-to-face communication) but throughly formal design plans
are not. Even very well managed projects like GCC don't do such strict
formal design, (at least, its not evident from their mailing lists).
If you're going to be working in an environment were debugging isn't
necessary (read kanze's "hack and hope" blurb higher up in this
thread) you need that kind of strict pre-planning.


Its evolution as opposed to design. kanze, the original poster, was
espousing a model of software design in which all the specs are
written in stone before hand, code is written without testing to see
if it compiles, and a build means the product is 100% bug-free. Of
course, I'm exaggerating, but you get the idea of what I mean by
design. Evolution means that the software improves in a gradual
manner. Some clever guy gets an idea of a really great algorithm,
writes some high-level design on paper he pulls out of his printer
tray, and codes it. If it passes muster, it gets into the official
source tree. See Linus's spiel about this on kernel traffic.

You've got some very good measures. Just take a look at what aspects
of the Linux kernel were dramatically improved between 2.4 and 2.6.
The virtual memory layer, block-I/O layer, and scheduler were
rewritten. The kernel was made preemptible. The audio layer was
swapped out. SMP was greatly improved. All those changes are much
bigger than the total changes between NT 4.0 kernel and the XP kernel,
and occurred on a much smaller time scale.

On the contrary. Its probably a lot slower to develop this way. Its
much easier (and this is from personal experience) to code something
when you have all the specs ahead of time. However its a sad fact of
life that often times, you can't design the project ahead of time.
This happens a lot when a software project is a secondary aspect of a
hardware project. Until the hardware project gets to an advanced state
of development, the software project is fuzzy on the exact interfaces
to code too. However, because of time constraints, development on the
two has to proceed in parallel. Say you have 10 months to do both
projects. The hardware development time is 7 months. Properly done,
the software development time is 5 months. If you do it serially, that
puts you over. If you do it in parallel, it might take you 7 months to
do the software, and another 3 months to integrate everything and fix
issues caused by the development process, but at least you get done on
time.

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]