Automated Testing Framework (atf)
Copyright (c) 2007 The NetBSD Foundation, Inc.
All rights reserved.


Introduction
============

This document contains a *non-exhaustive* list of things that should be
addressed in ATF before it can be considered stable.  Some of the items
are planned for specific future releases, but take this with a big grain
of salt: the focus of a specific release may change completely depending
on how things develop.


Plans for 0.3
=============

* Add a way to specify which bug/issue/whatever a given test case is
  stress-testing.  This information is useful when detecting regressions.

* Implement custom XML output in atf-report.  Need to create a DTD too.

* Maybe implement direct HTML output in atf-report to avoid a dependency
  on xsltproc.  Or maybe not.  Evaluate this idea and decide later.


Plans for 0.4
=============

* Add a way to add an upper-bound limit for test run times.  Otherwise a
  test that enters an infinite loop can stall the whole process.

* Add a class, similar in idea to boost::noncopyable, to easily mark some
  classes as not copyable.  Or, failing to do that, any other way to do
  the same.

* Add a .pc file for libatf.


Plans for pre-1.0
=================

* Fix all occurrences of XXX, TODO and FIXME.

* Improve error reporting: aside from clarifying error messages, this also
  implies adding more error cases to give them more semantic meaning at the
  source code level..

* The test suite should not use atf-compile to generate helper test cases.
  Reason: NetBSD will include atf-compile in the comp.tgz set.  If it is
  not installed, a lot of tests will be skipped without a "real" reason.
  Use real helpers instead.

* Make the shell library work with 'set -e'?

* Shell test programs dynamically locate where the shell library is by
  calling atf-config (done by atf.init.subr).  Contrarywise, binary test
  programs are directly linked against the final location of libatf.
  It may be nice if the latter loaded the library dynamically based on
  what atf-config said, so the user could switch atf installations by
  simply changing its PATH (and effectively making atf relocatable on the
  file system).  Why could this be nice?  To painlessly run an older atf
  test suite against a more recent version of the code base to ensure
  there are no regressions even with older tests.  Just a crazy idea, as
  maybe what the shell test programs currently do is stupid.

* Allow users to customize the build of atf by defining additional meta-data
  for test cases.  At the moment this is possible because the meta-data is
  not sanity-checked, but I think it should be.  Following the previous
  item, NetBSD could add a 'netbsd.pr' variable and then use this data when
  generating reports to add direct links to the appropriate PRs.

* Make sure that the tests in tests/atf have, at the very least, the same
  coverage as the ones in tests/bootstrap.

* Add manual pages for the library interfaces.

* Document the code.

* Possibly add a way to automatically gain or drop privileges when
  require.user is set.


Plans for 1.0-RC1
=================

* Build libatf as a shared library and set -version-info accordingly.


Plans for post-1.0
==================

* Allow the parallel execution of tests.  Achieving this with a test
  program granularity is easy: only need to change atf-run.  Lowering it
  to a finer granularity (test cases) is harder and maybe not worth it.


Things to we will not do
========================

This is a list of things that will *not* be addressed anytime soon in the
project.  Of course most of them would be nice to have in the future, but
they will not block releases nor drive development.  We can obviously
change our mind in the future and add move some of these to the above list.

* Native Win32 support: we are not talking about building atf with tools
  such as Cygwin or MinGW, which should already be possible.  Native Win32
  support means modifying the code to use Win32 library calls and easily
  build out of the box (i.e. the GNU Autotools are not useful in that
  case).

* Extreme efficiency: even though we will focus on using the most suitable
  data structures in each situation, we will not attempt to get extreme
  efficiency by adding hacks that make the code uglier.  Testing is a task
  that requires a lot of resources on its own, and some tests will be
  specially intensive, so there is no point in micro-optimizing atf if the
  global gains are negligible.

* Extreme portability: the initial goal of this project was to provide a
  testing framework for the NetBSD Operating System.  Given that this
  system has a very nice build framework, we can be sure that atf will be
  built and used with sane tools (basically a decent C++ compiler and a
  POSIX-compliant shell interpreter).  We will definitely not aim for
  portability to broken systems by tweaking our code to not follow the
  standards.


-- EOF
