[vtk-developers] C++ unit tests

Tue Jul 28 14:45:40 EDT 2015

DC,

I thought about writing a wiki/blog/source/paper on vtk testing (once
winter sets in). It would need to focus on best practices, because vtk
has many different approaches to testing.

I intended to start on non-rendering unit testing. This is a real
weakness in vtk.  A solid unit testing framework is needed. David G's
testing framework might be a good start, but perhaps a more thorough
requirements analysis is required.

I think a good description of rendering-based testing is possible and
might be a short putt.

Bill

On Tue, Jul 28, 2015 at 2:32 PM, David Cole via vtk-developers
<vtk-developers at vtk.org> wrote:
> Sounds like we "just" need a wiki page / blog post / Source article /
> white-papery-thingy about C++ VTK testing.
>
> (And then a follow-on one for python.)
>
> Having a way to add a C++ unit test of a newly added VTK class will be
> great! (Adding them for all the existing classes will certainly be a
> chore, but it is finite, and will increase code coverage, although
> perhaps not as easily as any of us would like...)
>
>
>
> On Tue, Jul 28, 2015 at 2:27 PM, Shawn Waldon <shawn.waldon at kitware.com> wrote:
>> +1 to the test recipe.
>>
>> On Tue, Jul 28, 2015 at 2:22 PM, David E DeMarle <dave.demarle at kitware.com>
>> wrote:
>>>
>>> We've got years of layers of great but poorly described regression test
>>> infrastructure in the code today that makes it harder than it could be to
>>> make new tests. If this is yet another layer, I'm not in favor.
>>>
>>> But if this makes adding tests tractable for newbies and we put together
>>> and post an easy but comprehensive recipe for adding tests along with this,
>>> then I'm very much in favor.
>>>
>>>
>>>
>>>
>>> David E DeMarle
>>> Kitware, Inc.
>>> R&D Engineer
>>> 21 Corporate Drive
>>> Clifton Park, NY 12065-8662
>>> Phone: 518-881-4909
>>>
>>> On Tue, Jul 28, 2015 at 2:10 PM, Bill Lorensen <bill.lorensen at gmail.com>
>>> wrote:
>>>>
>>>> I have been adding unit tests lately using old school vtk testing.
>>>> I'll be willing to kick the tires on David's framework.
>>>>
>>>> My goal in these unit tests to get as close as 100% coverage as
>>>> possible, verifying results when possible. This includes exercising
>>>> Error and Warning code and checking the messages themselves.
>>>>
>>>> I'll admit that I'm skeptical, but hopeful that adding a new test
>>>> framework will increase code coverage.
>>>>
>>>> On Tue, Jul 28, 2015 at 1:34 PM, David Thompson
>>>> <david.thompson at kitware.com> wrote:
>>>> > Hi David(s),
>>>> >
>>>> > I really like the single-header idea, especially the macros for testing
>>>> > whether values match to within a tolerance.
>>>> >
>>>> > My concerns are
>>>> >
>>>> > 1. The C++ Testing/Rendering/vtkTesting.h class which provides (or now
>>>> > that we have ExternalData, provided?) access to baseline images doesn't seem
>>>> > to fit into the FIXTURE macro. Is there a way to integrate it? Or a way to
>>>> > extend the vtkTesting class to be more usable by including methods similar
>>>> > to your macros?
>>>> >
>>>> > 2. One thing I dislike about most/all of the C++ test frameworks is
>>>> > that the error messages are assumed to be fixed and the methods often do not
>>>> > present a way to include the incorrect result as part of the failure
>>>> > message. It is nice that your macros include that. However, given that tests
>>>> > often fail on machines I don't have access to, it is really nice to
>>>> > encourage people to write useful messages. Macros like CHECK_CLOSE would be
>>>> > better if they accepted a message as a final argument, especially if the
>>>> > argument was passed to a std::ostream object the way
>>>> > vtkGenericWarningMacro() does so that people can (are forced to) provide
>>>> > more context about where the two numbers being compared come from.
>>>> >
>>>> >         David
>>>> >
>>>> >> On Jul 28, 2015, at 10:43 AM, David Gobbi <david.gobbi at gmail.com>
>>>> >> wrote:
>>>> >>
>>>> >> Hi David (and David),
>>>> >>
>>>> >> The only advantage of my unit test header is that it is a single
>>>> >> header
>>>> >> (a short one, too) with zero dependencies.  So it's easy to add to VTK
>>>> >> and it's trivial to maintain.
>>>> >>
>>>> >> If either of you has a better solution, then go ahead and make a merge
>>>> >> request.  I'll review it for you ;)
>>>> >>
>>>> >>  - David
>>>> >>
>>>> >>
>>>> >> On Tue, Jul 28, 2015 at 8:27 AM, David Thompson
>>>> >> <david.thompson at kitware.com> wrote:
>>>> >> Another project I work on has been using Bandit[1] to good effect.
>>>> >>
>>>> >>         David
>>>> >>
>>>> >> [1]:http://banditcpp.org/reference.html
>>>> >>
>>>> >> On Jul 28, 2015, at 10:01 AM, David Cole via vtk-developers
>>>> >> <vtk-developers at vtk.org> wrote:
>>>> >>
>>>> >> > A single header file unit testing solution sounds like a good thing.
>>>> >> >
>>>> >> > ... but ... seems like we could also take advantage of a
>>>> >> > well-established generally used unit testing framework (like gtest
>>>> >> > or
>>>> >> > CppUnit) just as easily, and not clutter the VTK source with
>>>> >> > yet-another-custom-thing-to-learn-about-VTKs-infrastructure.
>>>> >> >
>>>> >> > Would it be just as easy to add unit testing using an external unit
>>>> >> > testing package? Or are there things about that scenario that are
>>>> >> > overly complicated that having our own makes simpler...?
>>>> >> >
>>>> >> >
>>>> >> > David C.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Mon, Jul 27, 2015 at 5:25 PM, David Gobbi <david.gobbi at gmail.com>
>>>> >> > wrote:
>>>> >> >> Hi All,
>>>> >> >>
>>>> >> >> I've always enjoyed using the python unit testing framework in
>>>> >> >> "vtk.test".
>>>> >> >> It's been brought up on the list before, but it would be nice if
>>>> >> >> VTK
>>>> >> >> included a unit test framework for C++, as well.
>>>> >> >>
>>>> >> >> For my work projects, I've been using a unit test header that I
>>>> >> >> wrote a year
>>>> >> >> and a half ago:
>>>> >> >> https://github.com/dgobbi/UnitTest/blob/master/UnitTest.h
>>>> >> >>
>>>> >> >> It's a single header file that provides a collection of macros for
>>>> >> >> building
>>>> >> >> unit tests, it's meant to be innocuous and easy to use.  It could
>>>> >> >> go into
>>>> >> >> VTK as Testing/Core/vtkUnitTest.h and people could use it at their
>>>> >> >> leisure.
>>>> >> >>
>>>> >> >> Any thoughts?
>>>> >> >>
>>>> >> >> - David
>>>> >>
>>>> >>
>>>> >
>>>> > _______________________________________________
>>>> > Powered by www.kitware.com
>>>> >
>>>> > Visit other Kitware open-source projects at
>>>> > http://www.kitware.com/opensource/opensource.html
>>>> >
>>>> > Search the list archives at:
>>>> > http://markmail.org/search/?q=vtk-developers
>>>> >
>>>> > Follow this link to subscribe/unsubscribe:
>>>> > http://public.kitware.com/mailman/listinfo/vtk-developers
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Unpaid intern in BillsBasement at noware dot com
>>>> _______________________________________________
>>>> Powered by www.kitware.com
>>>>
>>>> Visit other Kitware open-source projects at
>>>> http://www.kitware.com/opensource/opensource.html
>>>>
>>>> Search the list archives at: http://markmail.org/search/?q=vtk-developers
>>>>
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://public.kitware.com/mailman/listinfo/vtk-developers
>>>>
>>>
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Search the list archives at: http://markmail.org/search/?q=vtk-developers
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://public.kitware.com/mailman/listinfo/vtk-developers
>>>
>>>
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Search the list archives at: http://markmail.org/search/?q=vtk-developers
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/vtk-developers
>>
>>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Search the list archives at: http://markmail.org/search/?q=vtk-developers
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/vtk-developers
>

-- 
Unpaid intern in BillsBasement at noware dot com