Rearchitecting ATF: Road trip to 1.0
This document describes the major steps to be taken before getting to the first stable release of ATF, 1.0. Some of these steps describe the transition from the old code base to the new ideas documented in the specification (TODO: Add link somehow).
Add atf-sh interface to build-time tests.
Properly document the libraries: i.e. one page per module, detailed information of each function and type, etc. At the very least atf-c.
Add a tool to collect multiple outputs of atf-run (from different machines) and generate a single (XML-only?) log with everything. Must allow easy conversion to HTML for online publishing.
Allow grouping of test programs in tiers in an Atffile. This is to permit the user specify "dependencies" between test programs: e.g. do not run a specific test program if its dependencies have failed, because it will certainly fail also.
Provide a kernel-level unit testing API (for NetBSD only, at the moment). This should come in the form of an atf.ko module that provides functions to define and register test cases, functions for results reporting and an interface (a trivial file system?) that transports the application/X-atf-tcs output to user-space, provides information to user-space about the available test cases (a list) and allows user-space to launch the execution of test cases.
Add a module to atf-c to manage dynamic memory. Should provide a "mem chunk" object that can only be managed through functions (i.e. not directly) so that access to buffers can be safely controlled. Dealing with strdup and similar functions, for example, might be complex.
See these old revisions for a start, but these did not work correctly because the use of (void **) casts brought aliasing problems:
78eeacf3b9284493e5e96b7866fc1d163a13bc02 8e200683a2b70eadca728c2e0347d4790777b3fc 872393ed0177cbcc30ffacede228eb3986c42ab7
Fix all occurrences of XXX, TODO and FIXME.
Split the distfile into multiple components. We should have a component for each language binding and a component providing the ATF tools, at the very least. If we had this, external programs using ATF wouldn't need to depend on the tools and/or the C++ binding, because they could just require the user to build the atf-c binding.
Think of a way to properly add tests for (almost?) all error paths. Most of them are probably broken already.
Improve error reporting: aside from clarifying error messages, this also implies adding more error cases to give them more semantic meaning at the source code level..
Make the shell library work with 'set -e'?
Shell test programs dynamically locate where the shell library is by calling atf-config (done by atf.init.subr). Contrarywise, binary test programs are directly linked against the final location of libatf. It may be nice if the latter loaded the library dynamically based on what atf-config said, so the user could switch atf installations by simply changing its PATH (and effectively making atf relocatable on the file system). Why could this be nice? To painlessly run an older atf test suite against a more recent version of the code base to ensure there are no regressions even with older tests. Just a crazy idea, as maybe what the shell test programs currently do is stupid.
Allow users to customize the build of atf by defining additional meta-data for test cases. At the moment this is possible because the meta-data is not sanity-checked, but I think it should be. Following the previous item, NetBSD could add a 'netbsd.pr' variable and then use this data when generating reports to add direct links to the appropriate PRs.
Make sure that the tests in tests/atf have, at the very least, the same coverage as the ones in tests/bootstrap.
Document the code.
Possibly add a way to automatically gain or drop privileges when require.user is set.
Add a way to specify which bug/issue/whatever a given test case is stress-testing. This information is useful when detecting regressions.
Build libatf as a shared library and set -version-info accordingly.
Set the DTDs' versions to 1.0.
Allow the parallel execution of tests. Achieving this with a test program granularity is easy: only need to change atf-run. Lowering it to a finer granularity (test cases) is harder and maybe not worth it.