Not debugging?

java-programmer, Not debugging? Phlip wrote:
>
> Duane Bozarth wrote:
>
> >> The define "legacy" as "requires debugging".
> >
> > That's a bizarre (at best) definition of "legacy"...
>
> That's why Greg didn't understand why I used it like that.

I didn't either (and still don't) because it has nothing whatsoever to
to w/ "legacy" or not...

> Me: Strive to never debug.
>
> Greg: What about blah blah blah.
>
> Me: You are using something that you can't design
> fresh from scratch to resist bugs. So you must
> run the debugger more often than greenfield code
>
> Greg: It's not "legacy" it's embedded blah blah blah

In that sense everything is "legacy" -- I can't redesign a commercial
compiler, either.

...

> Just don't leave the emulator out of the loop. Greg implied using it would
> slow down the tail end of development.

At some point in most embedded systems, that <is> true...you get to a
point at which the depth of emulation required isn't worth the effort
that would be required. Once at that point, reverting is rarely
productive use of resources.

java-programmer >> Not debugging? Phlip wrote:
>
> Duane Bozarth wrote:
>
> >> The define "legacy" as "requires debugging".
> >
> > That's a bizarre (at best) definition of "legacy"...
>
> That's why Greg didn't understand why I used it like that.

I didn't either (and still don't) because it has nothing whatsoever to
to w/ "legacy" or not...

> Me: Strive to never debug.
>
> Greg: What about blah blah blah.
>
> Me: You are using something that you can't design
> fresh from scratch to resist bugs. So you must
> run the debugger more often than greenfield code
>
> Greg: It's not "legacy" it's embedded blah blah blah

In that sense everything is "legacy" -- I can't redesign a commercial
compiler, either.

...

> Just don't leave the emulator out of the loop. Greg implied using it would
> slow down the tail end of development.

At some point in most embedded systems, that <is> true...you get to a
point at which the depth of emulation required isn't worth the effort
that would be required. Once at that point, reverting is rarely
productive use of resources.

java-programmer >> Not debugging? George Neuner wrote:

> In principle you're right ... but it's rarely that simple.

Yes it is. You are considering the harder task of retrofitting
acceptance-level tests to the outside of finished APIs. Not the simpler and
more direct task of writing tests that fail so you can write a line or two
of behavior to pass them.

> I support testing and assertions wherever possible, including
> intra-procedure if there is some meaningful test that can be done on a
> partial result - but testing line by line is ridiculous. It may be
> extremely difficult or quite impossible to figure out what would
> happen if a particular line of code is wrong or missing.

That's why you run the test and ensure it fails for the correct reason,
before writing the code to pass the test. The test doesn't have to be
exact - it doesn't need to constrain your code to only one possible new line
[or two]. You write more tests until all of them constrain.

> It's also
> unwieldy because such tests frequently can't be batched but must be
> executed in-line due to following code altering the test conditions.

Design-for-testing implies you have just a little more API, and you use it
to detect these intermediate states. The requirement to write tests first
makes such API adjustments easier.

> I didn't agree with this "line by line proof" approach in 1981 when
> Gries proposed it in "The Science of Programming" and I don't agree
> with it now. YMMV.

That was probably something different, and TDD is not a "proof" system.

> I use to do image processing in which the processing was a dependent
> chain of fuzzy logic. All the possible answers were wrong in some
> absolute sense - and the object was to find the answer that was least
> wrong. If I missed a minor step or had a bug in a calculation
> somewhere, the chances are I wouldn't be able to tell by looking at
> the intermediate results.

Right. Such tests can be very fragile and hyperactive. Hence, run them
_more_ often, and if they fail unexpectedly use Undo to back out your
change, then think of a smaller change that doesn't change an irrelevant
side-effect. Tests that constrain too much are better than ones you run
infrequently.

--
Phlip
http://www.greencheese.org/ZeekLand <-- NOT a blog!!!

java-programmer >> Not debugging? Phlip wrote:
> Tim X wrote:
>
>
>>I'm guessing he was referring to the irritating development of
>>programmers who are not able to debug code without a high level IDE
>>which includes a debugger which allows them to step/trace through the
>>code one line at a time and watch what happens to variables in a watch
>>window.
>
>
> No. I'm talking about developers who don't write unit tests as they write
> code. These provide the option of using Undo, instead of debugging, when the
> tests fail unexpectedly.
>
> This leads to a development cycle with highly bug-resistant code, and
> without proactive debugging to implement new functions.
>
> Yes, you still need the debugger - typically for legacy situations - and you
> still need elaborate debugging skills. New code stays ahead of them.
>
> The idea that we can implement without debugging is incomprehensible to most
> programmers. But that really is what I meant.

I've been waiting for someone to make this claim about unit testing.
Guess I'll pick on you :)

I don't believe that unit testing eliminates debugging. I'll give a
real life example.

We had a bug where an update to our library disabled some features of a
client program. After some investigating, it was determined that the
client program was doing this:

if (library_version == 3)

instead of this:

if (library_version >= 3)

Our fix for this bug was to detect the client and report 3 for the
version if the client was the offending program, and otherwise report
the true version.

This is the sort of bug that gets caught in integration testing. I
can't think of any way that unit testing would have helped this
situation. I'd be very interested in hearing how this sort of bug would
be approached in the test-first "no debugging" philosophy.

-Peter

--
Pull out a splinter to reply.

java-programmer >> Not debugging? Peter Ammon wrote:

> I've been waiting for someone to make this claim about unit testing. Guess
> I'll pick on you :)

Oh goody 'cause I didn't claim it. [I think.]

> I don't believe that unit testing eliminates debugging. I'll give a real
> life example.

Some TDD authors prevaricate and say "/virtually/ bug free". My emphasis. I
prefer to prevaricate more accurately and usefully.

TDD produces bug-resistant code, with reduced the odds of long bug hunts.
The kind that typically require debugging and/or trace statements.

> We had a bug where an update to our library disabled some features of a
> client program. After some investigating, it was determined that the
> client program was doing this:
>
> if (library_version == 3)
>
> instead of this:
>
> if (library_version >= 3)
>
> Our fix for this bug was to detect the client and report 3 for the version
> if the client was the offending program, and otherwise report the true
> version.

TDD works in tiny steps. Upgrading a library is a big step (and another
"legacy" situation). Fortunately, your tests let you roll back to the
previous library version for frequent sanity checks (and emergency
releases), until you debug the situation.

> be approached in the test-first "no debugging" philosophy.

There is no test-first "no debugging" philosophy. Test cases make an
exquisite platform for debugging. For example, in VB[A] I use Debug.Assert
in a test case, and failure raises the debugger. Then I move the current
execution point back to the called method, step inside it, fix the code to
pass the test, and resume running the remaining tests.

Without TDD, this is the Unholiest of Unholies - programming in the debugger
to generate spaghetti code. With TDD, it's nothing more than leveraging your
tools effectively.

When people use TDD, they generally report surprise that the incidences of
_punitive_ debugging, with absolutely no other recourse, go way down. And
some teams (in greenfield projects with user-level code) indeed never turn
on their debugger, and code for years without it.

--
Phlip
http://www.greencheese.org/ZeekLand <-- NOT a blog!!!

java-programmer >> Not debugging? In article <email***@***.com>,
George Neuner <gneuner2/@comcast.net> wrote:
>On Thu, 25 Aug 2005 01:24:01 GMT, "Phlip" <email***@***.com> wrote:
>
>>
>>If you can think of the next _line_ of code to write, you must perforce be
>>able to think of a complementing test case that would fail if the line were
>>not there. Just make writing the test case part of writing that line.
>>
>
>In principle you're right ... but it's rarely that simple.
>
>I support testing and assertions wherever possible, including
>intra-procedure if there is some meaningful test that can be done on a
>partial result - but testing line by line is ridiculous. It may be
>extremely difficult or quite impossible to figure out what would
>happen if a particular line of code is wrong or missing. It's also
>unwieldy because such tests frequently can't be batched but must be
>executed in-line due to following code altering the test conditions.

You left out the important sentence in Phlip's original where he said "It's
not white box testing." From this I deduce he's talking about providing
inputs to the unit (method), not adding lines of code to what's being tested.

There remains the problem of "figuring out what would happen if a particular
line of code is wrong or missing". Once again I plead ignorance of TDD
(thankfully Phlip is kind to me when I make mistakes!), but it seems to me
that Phlip is talking about the simplest form of "test coverage" which gives
us two tests for "if A and B then S" -- one for if true, one for if false.
There are more complex forms of coverage such as checking all possible
combinations of truth values, and others which look at number of repetitions
of loops (somewhere I have an old reference to an article defining a hierarchy
of test coverage approaches, which I will try to find if anyone needs it). It
seems to me that the line-by-line approach would have to suffer from the same
limitations as test coverage: number of test cases grows exponentially as the
program gets longer and more complex. I suppose "small methods" helps limit
that problem, but isn't it still present?
--
"Yo' ideas need to be thinked befo' they are say'd" - Ian Lamb, age 3.5
http://www.cs.queensu.ca/~dalamb/ qucis->cs to reply (it's a long story...)

java-programmer >> Not debugging? "Phlip" <email***@***.com> writes:

> Greg Menke wrote:
>
> >> You describe a legacy situation - someone else invented this bizarro
> >> hardware, and legacy situations require debugging to learn their
> >> characteristics.
> >
> > No I don't. I describe a realtime system- or even a simple device
> > driver. There are lots of those, new and old.
>
> The define "legacy" as "requires debugging".

Bizarre.

> > Say you wrote some subsystem on this embedded system, and you
> > implemented some kind of test framework to help you get it as
> > functionally debugged as possible. What you've tested is your inputs,
> > outputs and algorithms work in an fairly abstract and simplistic
> > situation. Thats helpful for first order easy debugging, but you're
> > going to be doing it the hard way along with everybody else once the
> > software is on the hardware and some of the unknown idiosyncracies of
> > the system start showing up. More of the unknown idiosyncracies will
> > appear over time. And at this stage, your emulator is pretty much
> > useless because nobody cares about what it says when the real thing is
> > sitting in the lab & thats were the bugs are.
>
> This sounds like the code went through an emulation phase and then a real
> thing phase.

Nope- went through breadboard designs of the hardware using inputs &
outputs first from emulation hardware and eventually from the real
hardware. The breadboard designs are used to work up hardware and
software designs and implementations, feeding back changes & bugfixes,
etc into the evolving specs. Ideally, the working software meets the
working hardware later on in the project. Low fidelity emulation is
pretty handy for easier classes of bugs early on; mismatched or unpacked
structs, endian stuff, etc.. But race conditions and pathological i/o
states (the harder problems) often depend on the real hardware in the
real operating environment (maybe interrupt timing changes because
capacitors & clocks drift when the system goes below freezing, for
example).

> Ideally, I would configure one button on my editor to run all tests against
> the emulator, then run all possible tests against the hardware. If the code
> fails in the hardware I would trivially attempt to upgrade the emulator to
> match the fault, so the code also fails with the emulator. But this is
> speculation, and of course it won't prevent the need to debug against the
> hardware.

Neat. Who's going to pay for the emulator (build it & prove that it
matches the hardware- and keep it matching) when we also have to do the
real system?

Now one place that emulators are really handy are for working up smaller
subsystem elements; ie logic in a single fpga or related fpga's- then
excercising the i/o is a bit easier.

>
> Realistically, you are apparently relentlessly testing, and admittedly in a
> "legacy" situation that prevents you from using tests to make the hardware
> more bug resistant. Carry on!
>

We don't get many opportunites to make hardware more bug resistant. If
software discovers a discrepancy in the documented interface we can
sometimes get fixes back into hardware, but it has to be well proven.
Early in the hardware lifecycle its easier to get changes to the
interface specs, but later on the software people pretty much have to
lump it.

Gregm

java-programmer >> Not debugging? Greg Menke wrote:

>> Then define "legacy" as "requires debugging".
>
> Bizarre.

Then define "foo" as "frequent punitive open-ended debugging causing
occassional schedule risk". Some teams use "legacy" as a casual shorthand
way to say "foo". Please substitute my definition for "foo" wherever I wrote
"legacy", so we can address the actual points raised.

> Nope- went through breadboard designs of the hardware using inputs &
> outputs first from emulation hardware and eventually from the real
> hardware. The breadboard designs are used to work up hardware and
> software designs and implementations, feeding back changes & bugfixes,
> etc into the evolving specs. Ideally, the working software meets the
> working hardware later on in the project.

Sorry to interrupt; why is that "ideal"? Does the software match up even
later sometimes?

> Low fidelity emulation is
> pretty handy for easier classes of bugs early on; mismatched or unpacked
> structs, endian stuff, etc.. But race conditions and pathological i/o
> states (the harder problems) often depend on the real hardware in the
> real operating environment (maybe interrupt timing changes because
> capacitors & clocks drift when the system goes below freezing, for
> example).

As you stated before, in that space you often must research the nature of
faults, and that research will occupy significant development time.

>> Ideally, I would configure one button on my editor to run all tests
>> against
>> the emulator, then run all possible tests against the hardware. If the
>> code
>> fails in the hardware I would trivially attempt to upgrade the emulator
>> to
>> match the fault, so the code also fails with the emulator. But this is
>> speculation, and of course it won't prevent the need to debug against the
>> hardware.
>
> Neat. Who's going to pay for the emulator (build it & prove that it
> matches the hardware- and keep it matching) when we also have to do the
> real system?

In my experience, many teams are too busy doing "foo" because they did not
carefully plan a development environment that reduces the odds of "foo". I
suspect your team does have a good development environment, so we may be
talking past each other now.

But that's no reason to stop!

--
Phlip
http://www.greencheese.org/ZeekLand <-- NOT a blog!!!

java-programmer >> Not debugging? "Phlip" <email***@***.com> writes:

> Greg Menke wrote:
>
>
> > Nope- went through breadboard designs of the hardware using inputs &
> > outputs first from emulation hardware and eventually from the real
> > hardware. The breadboard designs are used to work up hardware and
> > software designs and implementations, feeding back changes & bugfixes,
> > etc into the evolving specs. Ideally, the working software meets the
> > working hardware later on in the project.
>
> Sorry to interrupt; why is that "ideal"? Does the software match up even
> later sometimes?

I'm not proposing that it is. But theres not much alternative to
something like that when both the hardware and software are being
developed.

> > Low fidelity emulation is
> > pretty handy for easier classes of bugs early on; mismatched or unpacked
> > structs, endian stuff, etc.. But race conditions and pathological i/o
> > states (the harder problems) often depend on the real hardware in the
> > real operating environment (maybe interrupt timing changes because
> > capacitors & clocks drift when the system goes below freezing, for
> > example).
>
> As you stated before, in that space you often must research the nature of
> faults, and that research will occupy significant development time.

And for these hard problem, you have to test with the real thing because
any emulator is going to miss some (many) of these subtle issues. For
the many easy bugs, an emulator (or maybe simulator) makes a good deal
of sense, but its no panacea.

> >> Ideally, I would configure one button on my editor to run all tests
> >> against
> >> the emulator, then run all possible tests against the hardware. If the
> >> code
> >> fails in the hardware I would trivially attempt to upgrade the emulator
> >> to
> >> match the fault, so the code also fails with the emulator. But this is
> >> speculation, and of course it won't prevent the need to debug against the
> >> hardware.
> >
> > Neat. Who's going to pay for the emulator (build it & prove that it
> > matches the hardware- and keep it matching) when we also have to do the
> > real system?
>
> In my experience, many teams are too busy doing "foo" because they did not
> carefully plan a development environment that reduces the odds of "foo". I
> suspect your team does have a good development environment, so we may be
> talking past each other now.

How do you reduce the chances of bugs by writing an emulator thats as
least as complicated as the real system, and that is essentially
guaranteed to have different & unknown bugs which will limit its
fidelity at some ill-defined point? Do you really think that teams
working on systems like this don't spend a lot of time trying to
minimize the opportunities of bugs showing up? There is lots of
code-reuse, walkthroughs, reviews, etc... on both the hardware and
software- emulators and simulators where feasible; for example, opnet
may be used to evaluate task and network schedulability in all sorts of
circumstances before the software is even started.

As we speak, I'm helping deal with a crash-on-reset problem on the
production hardware that doesn't happen on the development hardware,
plus weird edac behavior to go along with it. Analyzing this problem
requires a top to bottom analysis of the system will all sorts of
inter-team documentation of whats going on. How is an emulator going to
help here? At best it will exhibit the same problems (but we still
don't know what the problems are), at worst it will not duplicate the
behavior. It hasn't gotten us any closer in the former case, in the
latter its worse because project management is now nervous because its
presenting another difference from the real system yet they're (supposed
to be) relying on it to help them test.

Now if you're talking about a high-level app that can afford to ignore
hardware issues, then a low-fidelity emulator doesn't have to address
the hard problems and so offers greater coverage.

Gregm

java-programmer >> Not debugging? Greg Menke wrote:

> As we speak, I'm helping deal with a crash-on-reset problem on the
> production hardware that doesn't happen on the development hardware,
> plus weird edac behavior to go along with it. Analyzing this problem
> requires a top to bottom analysis of the system will all sorts of
> inter-team documentation of whats going on. How is an emulator going to
> help here?

The outer topic is your development environment. Yours is resisting this bug
as we "speak".

(You also had few opportunities to _prevent_ the bug.)

The inner topic is an emulator.

> At best it will exhibit the same problems (but we still
> don't know what the problems are), at worst it will not duplicate the
> behavior.

Here's a completely unrelated topic. When you use "make menuconfig" to build
Linux, you get a zillion options for all kinds of hardware peripherals. You
can freely mix and match all the options, and build a stable Linux.

How could Linux possibly be stable if you activate some combination of
options that nobody ever activated before? The odds increase as you plug in
weird hardware combinations.

The answer is those combinations are not a zillion opportunities for bugs,
they are a zillion forces against bugs. Each cross-constraint, passing its
tests, reduces the odds of bugs in parallel situations. Constraint spaces
with a high number of dimensions are naturally sparse, so a viable Linux
kernel can only exist in very few states.

I would prefer code that passed both emulation tests and tests in real
hardware. Cross-testing is good, even when the emulator doesn't exactly
match the hardware. Sometimes it's good _because_ they don't match. Run such
tests more often.

There are indeed many reasons why an emulator might not be ideal for every
kind of embedded situation.

In _theory_, for each bug you debug in the hardware, you should force the
emulator to exhibit the same bug before killing it.

The decision to do that should be per-bug, not per-emulator.

> It hasn't gotten us any closer in the former case, in the
> latter its worse because project management is now nervous because its
> presenting another difference from the real system yet they're (supposed
> to be) relying on it to help them test.

Ah, so you are exposing your managers to implementation details.

Pay no attention to the little emulator behind the curtain!

--
Phlip
http://www.greencheese.org/ZeekLand <-- NOT a blog!!!

java-programmer >> Not debugging? Greg Menke wrote:
> "Phlip" <email***@***.com> writes:

>>Greg Menke wrote:

>>>Ideally, the working software meets the
>>>working hardware later on in the project.

>>Sorry to interrupt; why is that "ideal"? Does the software match up even
>>later sometimes?

> I'm not proposing that it is. But theres not much alternative to
> something like that when both the hardware and software are being
> developed.

Forgive me for asking, but if you are part of a team that is
developing both specialized hardware *and* software to run on
it, don't you have an accurate emulator in the first place in
your HDL file for the new hardware? I mean, if you're
designing the hardware, you've got the design in a Hardware
Description Language (such as VeriLog) and your HDL
development environment will support some excruciatingly
exact simulations that you can test software against.

At least, I've never seen a Verilog environment that didn't
have a simulation mode you could test software against.

Bear

java-programmer >> Not debugging? > From: Russell Shaw <email***@***.com>
> You can only write tests before the code if you are very familiar
> with what you want and have thought about it for hours/days/weeks.

I disagree with most of that.

When unit-testing single lines of code, the test consists of two parts,
the data going in, and the data going out. The data going in is already
there before I write that one line of code to deal with it. The data
going out is immediately checked by eye upon executing that one line of
code. No hours/days/weeks of prior thought are necessary to perform
that single-line unit test.

When unit-testing single functions, the input part is already there
before even the first line of code is written, which is long before the
function is put together. The output part is already there immediately
after the last line of code is tested, which is immediately before
putting the function together. That's for the first unit-test for that
function. Other unit tests are then constructed immediately after
running the first unit test. The whole bunch of them are finished a few
minutes after the function is put together. No hours/days/weeks of
prior thought are necessary to set up and perform those single-function
unit tests.

> In the process of making the solution of a very tedious problem clear
> to me, i can do a *lot* of throwing away of code and refactoring for
> weeks at a time. The amount of extra churn from rewriting unit tests
> would be hopeless.

I don't see it that way. I test just about every single line of code,
and every function, regardless of whether it's a brand-new function or
a refactoring of something. Unit testing is no burden, it's how I make
sure my code works before I move to writing the next part of my code.

> Write code that works, then write unit tests to make sure it
> keeps working.

So you don't test your code until after you're sure it's working?
How can you be sure it works when you havne't yet tested it??