src_test

Default src_test

The default test phase in EAPI 6 is equivalent to the following:

src_test() {
	if nonfatal emake check -n &> /dev/null; then
		emake check
	elif nonfatal emake test -n &> /dev/null; then
		emake test
	fi
}

Sample src_test

src_test() {
	cd "${S}"/src/testdir || die

	# Test 49 won't work inside a Portage environment
	sed -i -e 's~test49.out~~g' Makefile || die

	# Try to run the non-gui tests only
	# pass -j1 if tests do not support being run in parallel
	emake -j1 test-nongui
}

Supporting test suites in packages

If the packaged software is equipped with a test suite, it is sensible to run it. Sometimes the package will need additional dependencies for this, i.e., dependencies that are only required to run the test suite. Such test-only dependencies should be specified in DEPEND or BDEPEND behind a USE flag; often, the test USE flag will be used for this. Please refer to the section on Test dependencies for more information.

Note that the test USE flag is only necessary if there are test dependencies, tests are being built conditionally, or it is desirable to vary behaviour based on whether tests will be run (e.g. downloading large files in SRC_URI). It is not appropriate to use the flag to simply indicate that tests exist and are expected to pass.

Often the default src_test is fine. Sometimes it is necessary to remove certain tests from the list if they cannot be used with a Portage environment. Reasons for such a failure could include:

• Needing to work with files which are disallowed by the sandbox.
• Requiring user input (src_test must not be interactive).
• Requiring root privileges.

Usually, removing the relevant test from the Makefile using sed or skipping a particular make target is sufficient.

Try to ensure that tests work properly for your ebuild. A good test suite is extremely helpful for arch maintainers. Sometimes it is necessary to skip tests entirely. This can be done by setting RESTRICT="test" in the ebuild.

Tests that require network or service access

Sometimes test suites (and other build-time programs) attempt to use remote or local network, or production servers running on the host. All of these are strictly forbidden. Developers should either fix such tests to work in an isolated environment, or disable them completely unless explicitly allowed by the user. At the bare minimum, the tests must not fail with FEATURES=network-sandbox being enabled.

Internet access within the build procedure is forbidden for the following reasons:

• the build may be running in an environment with no or restricted Internet access, and this must not cause the tests (build) to fail;
• the Internet connection may be unstable (e.g. poor reception) in which case an interrupted connection or packet loss must not cause the tests to fail or hang, and it should not cause unnecessary delays;
• the Internet connection may be running on a limited data plan in which case the additional network use may cause additional charges or other inconveniences to the user;
• the remote network services used by the tests may become unavailable temporarily or permanently, causing unexpected test failures;
• accessing remote sites always poses a privacy issue, and possibly a threat to security (e.g. through inadvertently exposing information about the system).

Fixing tests that require Internet access usually requires cooperation with upstream, and porting the tests to use test techniques such as mocking or using replay data. For this reason, developers report the issue upstream and skip tests that require network access. It is recommended to explicitly leave a note as to why the tests are skipped, so that other developers can re-enable them locally to run a more complete test suite.

It is generally considered acceptable to rely on IPv4 localhost being resolvable and available for binding. Tests should only connect to services that are started as part of the testsuite. It is not acceptable to connect to daemons run outside the test environment.

Local server access within the build procedure is forbidden for the following reasons:

• tests must run reliably independently of whether a particular server is running throughout the build process or not,
• using production services for running tests is extremely dangerous as it may inadvertently expose bugs in those services, causing instability, data loss or even exposing security vulnerabilities.

Fixing tests that require access to local services is usually done via starting additional isolated instances of those services during the test phase. Those services must either be running on a UNIX socket or on the loopback interface, to reliably prevent remote access.

For all networked services exposed during the test phase (either by the ebuild or the tests themselves), UNIX sockets are strongly preferred over IP sockets as they provide better means for unique naming and access control mechanisms. IP sockets can be subject to port collisions with other local services and they can be accessed by local system users who may exploit a vulnerability through the tests.

Additional protection against those issues is provided through FEATURES=network-sandbox. However, this is only an optional Portage feature relying on specific Linux kernel namespace mechanisms and developers should not rely on it being enabled.

Tests that require X11

Some packages include tests (or other build-time applications) that attempt to use the user's X11 session and fail being unable to connect to it. Those tests need to be fixed to work independently of the X11 server that might or might not be running when packages are being built.

If the program in question does not strictly need X11 but merely attempts to take opportunity of the DISPLAY variable being set, the best solution is to simply unset this variable in the ebuild.

src_test() {
	# tests attempt to connect to X11 and fail when it is set
	# however, they work just fine without X11
	unset DISPLAY

	default
}

If the package actually requires a running X11 server to run the complete test suite, you can use the virtualx eclass to provide an isolated Xvfb environment for the tests to use. It provides a virtual X11 display that is not connected to any physical device and that programs can use reliably.

inherit virtualx

src_test() {
	virtx default
}