Testing Interfaces

[From UnitTestsReconsidered...]

Should we be TestingInterfaces?

UnitTests help define interfaces.

In one sense, this is true. In the case where the interface is predefined, the UnitTests can define the reference verification and validation for the interface. This like the Java Compliance Tests. Quoting PhilGoodwin,

I think that UnitTests are a necessary part of the definition of an interface. At least from the perspective of the LiskovSubstitutionPrinciple. A UnitTest shows us what an interface is supposed to do when it's used. I think that UnitTests coupled with asserts are a very powerful way to add semantic information to an interface definition. This is basically the same as the DesignByContract concept. -- PhilGoodwin

On the other hand, the JCTs are often wrong with respect to the API specification, and hacked to work with the current version of Sun's JDK. In this respect, we should remember that because UnitTests as the definition is perfectly logically strict, yet can still be semantically wrong, they can be dangerous tools if not fixable. (Especially in a legal environment.)

In another sense, many people claim that TestDrivenProgramming results in better designed interfaces. For most people, it's difficult to foresee how a class interface will be used, so exercising it in the testing situation helps pounds out the roughness in the interface design. This is much the same as using a graphical user interface to remove usability warts. GeorgeGruschow and DaveHarris comment on this practice:

When I do UnitTests first, they force me to think about "What do I want this class to do?" more than "How am I going to do this?". I think most people agree that that's the correct order to think about those things. I end up documenting what the class is supposed to do in the form of a UnitTest, rather than in prose, UML, or some other non-executable form. The benefits for me are that I write the specification OnceAndOnlyOnce (which avoids documentation getting out of date), and that same specification is directly testable. -- GeorgeGruschow
Yes... and they do that more than writing the calling code does. For example, I might be writing code that needs to insert something into some special collection. So I write a message-send to do that. When I switch into UnitTest mode, I start to think more carefully about what that actually means. What the preconditions and postconditions are, how it effects the collection's class invariants. Some of these are things which the caller doesn't need to know about; they are more to do with the collection. But I still want to think about them in an abstract way, from outside of the collection. -- DaveHarris

However, there is another sense to the interface that moves beyond the mere signatures of the interface. Here, the interface is not the common usage, akin to Java interfaces or C++ class declarations. The interface is the quasi semantic membrane between the problem and the solution, vs. the syntactic signatures. Some people might call this the architecture of the system; others might call it the "strategy" of the interface vs. the "tactics" of the signatures.

In this respect, a UnitTest cannot help you. A UnitTest doesn't tell you what units to make, only how to to implement those units. So, in this respect, the calling code is more useful. Moreover, the nature of the methods that a class has are directly influenced by the calling situations--those situations provide the context in which the class exists, the forces that instigated its creation. A UnitTest is a completely different context. If you modify your interface to simplify testing, but increase the difficultly of calling it in the proper context, you've failed. The UnitTests have hurt you. Lesson: blackbox tests should never force a special interface.

Finally, there is the notion of TestFirstDesign:

TestFirstDesign is a design technique that uses UnitTests as a way to think about features just before you implement them. It's the up-front design in XP. The key is that it isn't done very far up front and it isn't the design for anything terribly big. The key is that it tells you what to think about first. The first time you talk about the code that you are about to write you talk about it in terms of what you want it to do - not how it should work. Often you find that you need to change that language in order to actually write the code, but at least with test-first the client gets the first word. -- PhilGoodwin
The reason why I can map to individual public interface member is that I tend not to design abstractions that require users to call many member in a particular order - just to carry out some aggregate behavior. If I discover members that need to be used together or in a particular order, I will try to wrap the call sequence in a public member and then make the contained calls private or protected. After all, if a user needs to call "a", then "b" and then "c" in order to get "d", why not just give them "d". So, in effect, I do test in a feature driven way - its just that I try to create methods that model composite features. I have found that if I allow many methods that must be used in combination to perform some single feature that my class becomes more complex, more difficult to use, and (worse yet) imposes method data and flow interdependencies for the client of my class for which I have no programmatic way of enforcing. --RobertDiFalco

That is a bad example. Clearly in the original calling context, if the aggregate behaviour is repeated often, this will force the wrapping of the aggregate behaviour. The tests themselves don't really add much (maybe speed up the discovery of the design failure). Actually, your design strategy follows the design principle of NarrowTheInterface.

In my opinion, it's superior to flow with the natural forces of the system, and create the design that naturally fits. For each interface layer, NarrowTheInterface. Then test the resulting units in isolation if you have to. Never modify design to fit an external force like coding standards or UnitTests because that will only result in design warts.

I'm not sure where you are disagreeing or what bad example you are referring to.

I'm saying the example provided wasn't the best you could offer. I think you could do better, as you even slashed it down yourself a little bit. It would be helpful to find a better example to represent your position.

[JeffGrigg takes issue with RobertDiFalco's statement "method data and flow interdependencies [...] for which I have no programmatic way of enforcing." above.]

Required sequence of calls can be enforced by with a StateMachine in the called class, and using Assert. (But you'd still want callers to call "d" instead of "a, b, & c", because the CodeSmells if you don't. -- JeffGrigg ;-)

In general, simple code makes for fewer bugs. Another strategy to limit bugs is not to test, but to limit complexity, such as by NarrowingTheInterface or using a StateMachine in a GUI, or by using TrafficCops in a multithreaded application. UnitTests increase complexity by adding to the number of lines of code to deal with even if they have other benefits. The system thus has different forces acting on it, meaning that an extreme approach to using UnitTests is suboptimal. In Artificial Intelligence, there is a term called "MUD" which stands for "mostly useless definitions." The more predicates you have in the knowledge base, the quicker the program gets stuck in MUD. RegressionTests can become mud if overdone because they are designed to keep the system from moving (in any direction).

See CompleteCoverageIsExpensive.

UnitTests help define interfaces.

In the case where the interface is predefined, the UnitTests can define the reference verification and validation for the interface. This like the Java Compliance Tests. Quoting PhilGoodwin,

I think that UnitTests are a necessary part of the definition of an interface. At least from the perspective of the LiskovSubstitutionPrinciple. A UnitTest shows us what an interface is supposed to do when it's used. I think that UnitTests coupled with asserts are a very powerful way to add semantic information to an interface definition. This is basically the same as the DesignByContract concept. -- PhilGoodwin

In cases where there is also a written specification, the designer needs to decide whether the tests form part of it or are used as an external tool to validate it: for example, the JCTs are often wrong with respect to the API specification - is the correct behaviour to follow the spec or to follow the tests?

Many people find that TestDrivingProgramming helps them create better designed interfaces. For most people, it's difficult to foresee how a class interface will be used, so exercising it in the testing situation helps pounds out the roughness in the interface design. This is much the same as using a graphical user interface to remove usability warts. GeorgeGruschow and DaveHarris comment on this practice:

When I do UnitTests first, they force me to think about "What do I want this class to do?" more than "How am I going to do this?". I think most people agree that that's the correct order to think about those things. I end up documenting what the class is supposed to do in the form of a UnitTest, rather than in prose, UML, or some other non-executable form. The benefits for me are that I write the specification OnceAndOnlyOnce (which avoids documentation getting out of date), and that same specification is directly testable. -- GeorgeGruschow
Yes... and they do that more than writing the calling code does. For example, I might be writing code that needs to insert something into some special collection. So I write a message-send to do that. When I switch into UnitTest mode, I start to think more carefully about what that actually means. What the preconditions and postconditions are, how it effects the collection's class invariants. Some of these are things which the caller doesn't need to know about; they are more to do with the collection. But I still want to think about them in an abstract way, from outside of the collection. -- DaveHarris

Of course, the UnitTest is not all that you need for interface design, because it operates at too low a level. It can't tell you what classes you should have; the best it can do is tell you what methods should be in them. You'll probably use the calling code too - that provides the context in which the class exists, the forces that instigated its creation. A UnitTest is a completely different context.

Sunir Shah writes: "If you modify your interface to simplify testing, but increase the difficulty of calling it in the proper context, you've failed. The UnitTests have hurt you. Lesson: blackbox tests should never force a special interface." As a personal aside, I'd be interested in exploring situations where this is the case, because my experience is that making it simpler to test the interface usually simplifies it for other purposes too. Working to minimize the need for setup/teardown, removing state from the unit, getting rid of side-effects - these practices should all steer us to simpler units. -- DanBarlow

Finally, there is the notion of TestFirstDesign:

TestFirstDesign is a design technique that uses UnitTests as a way to think about features just before you implement them. It's the up-front design in XP. The key is that it isn't done very far up front and it isn't the design for anything terribly big. The key is that it tells you what to think about first. The first time you talk about the code that you are about to write you talk about it in terms of what you want it to do - not how it should work. Often you find that you need to change that language in order to actually write the code, but at least with test-first the client gets the first word. -- PhilGoodwin

I'm not sure that the following three/four paras are relevant to this point. I suspect that the original page had some more context that didn't make it through onto UnitTestsReconsidered. As unreferenced code, I'm going to delete them from this copy unless someone obliges with the linking material.

The reason why I can map to individual public interface member is that I tend not to design abstractions that require users to call many member in a particular order - just to carry out some aggregate behavior. If I discover members that need to be used together or in a particular order, I will try to wrap the call sequence in a public member and then make the contained calls private or protected. After all, if a user needs to call "a", then "b" and then "c" in order to get "d", why not just give them "d". So, in effect, I do test in a feature driven way - it's just that I try to create methods that model composite features. I have found that if I allow many methods that must be used in combination to perform some single feature that my class becomes more complex, more difficult to use, and (worse yet) imposes method data and flow interdependencies for the client of my class for which I have no programmatic way of enforcing. -- RobertDiFalco
(I usually enforce calling sequence rules by implementing a StateMachine in the called class, and using Assert. So they can be enforced. But you'd still want callers to call d instead of a, b, c, because the CodeSmells if you don't. NarrowTheInterface) -- JeffGrigg ;-)

That is a bad example. (I think your point could be made with a stronger example; feel free to improve the text.) Clearly in the original calling context, if the aggregate behaviour is repeated often, this will force the wrapping of the aggregate behaviour. The tests themselves don't really add much (maybe speed up the discovery of the design failure). Actually, your design strategy follows the design principle of NarrowTheInterface.

In my opinion, it's superior to flow with the natural forces of the system, and create the design that naturally fits. For each interface layer, NarrowTheInterface. Then test the resulting units in isolation if you have to. Never modify design to fit an external force like coding standards or UnitTests because that will only result in design warts.