Commit Graph

56 Commits

Author SHA1 Message Date
Daniel Hahler 6aee6c2053 ci: overhaul coverage processing
- Execute the tests without compiling, and don't mess with the source
   files when coverage is enabled.
   This ensures that the coverage report lines are correct.
   This disables the doc tests, as their results would be unused.
   Hack: it still expands macros on util.lua, because of
   `util.get_themes_dir()` and `util.get_awesome_icon_dir()`, which might
   be moved later.  Ref:
   https://github.com/awesomeWM/awesome/pull/1239#discussion_r93828178.
 - ensure that BUILD_APIDOC and DO_COVERAGE are not used together
 - awesomeConfig.cmake: add DO_COVERAGE as an option
 - Travis: only install codecov with DO_COVERAGE=codecov
 - Travis: do not use set -v; use set -x with DO_COVERAGE
 - do not use trailing slashes with dirs in tests/examples/CMakeLists.txt / .luacov
 - Use latest luacov (0.12.0-1) again
   This reverts parts of 4cc6a815.
   I think it is better to fix any failure that 4cc6a815 tried to work around.
 - Travis: simplify/fix require('luacov') for functionaltests
 - tests/examples/CMakeLists.txt: resolve ../.. in SOURCE_DIR
 - tests/examples/CMakeLists.txt: add DO_COVERAGE to env
 - Cleanup/simplify .luacov: work with SOURCE_DIRECTORY alone
 - tests/run.sh: pass through / set SOURCE_DIRECTORY when running awesome
 - tests/run.sh: resolve source_dir
 - use DO_COVERAGE from env only
2017-01-08 20:34:13 +01:00
Uli Schlachter f7025e2f73 Fix tests/run.sh
Commit a944636c02 "bashified" tests/run.sh for some reason.
Afterwards, commit 4abd820051 fixed some of the fall-out.
However, there is still a problem left.

We have "set -e" in this script. Thus, whenever some command exits with
status 0, the script abort. When the variable errors is zero/unset, the
command "((errors++))" has exit status zero. Thus, this instruction
caused the shell script to abort. This was not intended.

Fix this by using "((++errors))" instead.

Signed-off-by: Uli Schlachter <psychon@znc.in>
2017-01-02 19:35:13 +01:00
Daniel Hahler 58773488e4 tests: do not set -x by default on Travis (#1318)
This is rather noisy, and the test runner should not require this anymore for debugging.

[ci skip]
2016-12-28 12:43:00 +01:00
Daniel Hahler 8f8094b57e tests/run.sh: allow to pass full file name
This is easier with shell completion.
2016-12-27 02:43:05 +01:00
Daniel Hahler 8e97ae51c3 tests/run.sh: remove exclamation mark for better copy'n'paste experience
Closes https://github.com/awesomeWM/awesome/pull/1293.
2016-12-27 02:43:05 +01:00
Daniel Hahler 4abd820051 tests: fix incrementing errors (#1297) 2016-12-25 15:13:22 +01:00
Daniel Hahler fb874122ba tests/run.sh: SHELL=/bin/sh, HOME=/dev/null (#1291) 2016-12-25 03:46:28 +01:00
Daniel Hahler d5b3fa8be0 Merge pull request #1290 from blueyed/tests-run-shellcheck-etc
tests/run.sh: shellcheck and other fixes
2016-12-25 03:43:50 +01:00
Daniel Hahler 58067834ad tests: remove TEST_QUIT_ON_TIMEOUT/quit_awesome_on_timeout (#1288)
It was set to 1 unconditionally anyway, and does not make sense to parametrize after all.
2016-12-25 03:43:03 +01:00
Daniel Hahler 8c9a1b5491 fixup! tests/run.sh: shellcheck and other fixes 2016-12-25 03:09:43 +01:00
Daniel Hahler a944636c02 tests/run.sh: shellcheck and other fixes 2016-12-25 03:08:35 +01:00
Emmanuel Lepage Vallée 8935ed2a30 Remove traces of the old URL (#1280)
**WARNING**: This breaks the API
2016-12-21 02:03:03 +01:00
Emmanuel Lepage Vallee 771f5a13c6 tests: Bump the timeout to 3 minutes.
There we go again... When hitting a slow CI node, there is again
timeouts when doing the multi-screen tests. As no solution to
bug leading to this has been found, the only thing to do is
increase the timeout.

Hopefully this commit will be reverted soon.
2016-12-01 19:34:46 -05:00
Uli Schlachter b2e0e55fc0 tests/run.sh: Inherit $HEADLESS (#1201)
Via this, I can set HEADLESS=1 in my wrapper-GNUMakefile that I use and
"make check" will no longer open a new window that gets in my way.

Signed-off-by: Uli Schlachter <psychon@znc.in>
2016-10-31 23:22:31 +01:00
Uli Schlachter cea5d41518 tests/run.sh: Use dbus-send directly with low timeouts (#1116)
run.sh waits for awesome's startup to be done by having awesome execute "return
1" via its dbus interface. However, by default dbus has a 25 second timeout
before it fails a dbus-send invocation. This defeats the purpose of this
exercise.

So instead of using awesome-client, this commit makes the code use dbus-send
directly and specifies a relative low reply timeout (which should still be
plenty so that this doesn't erroneously fail).

Signed-off-by: Uli Schlachter <psychon@znc.in>
2016-09-25 20:28:27 +02:00
Uli Schlachter a430ac1e23 tests/run.sh: Print special message for timeout
The timeout utility guarantees an exit code of 124 when the process died due to
timeout. Since awesome only ever exits with 0 or 1, we can use this to reliably
detect timeouts and print a matching message.

Signed-off-by: Uli Schlachter <psychon@znc.in>
2016-09-25 06:00:15 +02:00
Uli Schlachter 60ee10cfff tests/run.sh: Use the "timeout" command
coreutils provides a timeout command. Use that instead of (badly) inventing our
own version of it. This "timeout" command seems to be new. Let's hope everyone
has it and think about alternative solutions only when needed.

Fixes: https://github.com/awesomeWM/awesome/issues/1075
Signed-off-by: Uli Schlachter <psychon@znc.in>
2016-09-24 15:56:36 +02:00
Uli Schlachter 62d0961994 tests/run.sh: Show awesome's exit code
When awesome exits with a non-zero code, this is something interesting that we
should log. Do so.

The "set +e" / "set -e" dance is required so that we do not abort because the
wait builtin returns a non-zero code.

Signed-off-by: Uli Schlachter <psychon@znc.in>
2016-09-24 15:55:24 +02:00
Uli Schlachter 86ee5bb750 tests/run.sh: Don't fail because of grep
This script runs under "set -e", so any command exiting with a non-zero status
makes it abort. However, we do not care about failures from grep to find
anything, so handle that case gracefully.

Signed-off-by: Uli Schlachter <psychon@znc.in>
2016-09-24 15:53:35 +02:00
Daniel Hahler b4d6bfa4db Travis: fix codecov (#1105)
tests/run.sh: no slash with --search lib

Fixes https://github.com/awesomeWM/awesome/issues/1049.
2016-09-20 19:53:10 +02:00
Emmanuel Lepage Vallee 35c0476c86 tests: Increase time limit to 120 seconds
The "test launching client on a specific screen" suit is very slow.

However, it is also necessary to avoid issues such as #1069 or #154
from regressing again.

This is a temporary fix until a faster test client "daemon" is
developped.
2016-09-07 00:40:30 -04:00
Uli Schlachter cf89f108ac Make use of awesome-client commands
This uses the new support introduced in f0f31bc305 in the docs and in
tests/run.sh, removing an useless use of cat/echo.

Signed-off-by: Uli Schlachter <psychon@znc.in>
2016-08-31 17:45:37 +02:00
Uli Schlachter 71049a67f7 tests/run.sh: Use --search instead of $LUA_PATH
Signed-off-by: Uli Schlachter <psychon@znc.in>
2016-08-20 16:04:11 +02:00
Daniel Hahler e8dfe36c59 tests/run: change to script dir first
Then use `$source_dir` for detecting the build dir: this way you can run
`./run.sh some-test-file.lua` from `./tests/`.
2016-07-21 19:08:43 +02:00
Emmanuel Lepage Vallée 36e47fb1a9 tests: Don't use arbitrary build dir path (#898)
Calling the test script by hand from the build dir stopped working.
2016-06-12 23:28:32 +02:00
Daniel Hahler eaa80fef9a tests/run: clarify timeout in wait_until_success
Ref: https://github.com/awesomeWM/awesome/pull/937#discussion-diff-65313945
2016-06-05 17:05:23 +02:00
Daniel Hahler 8e221522f7 Travis: kill awesome after max. 60s per test file
This is required for when the test setup fails already, e.g. because of
an assertion error at the top of a test file.
2016-06-05 17:05:23 +02:00
Uli Schlachter 9d30bf87ae Merge branch 'more-travis-variance' of https://github.com/psychon/awesome 2016-05-09 18:42:09 +02:00
Uli Schlachter 1ab3f43575 Merge branch 'useful-env-vars' of https://github.com/psychon/awesome 2016-05-09 18:41:02 +02:00
Daniel Hahler 58209cd89e Use an uneven screen width/height (1921x1079) in one Travis job (#870)
This is meant to trigger issues that would need `math.ceil()` fixes etc.

Ref: https://github.com/awesomeWM/awesome/issues/848#issuecomment-216068831
2016-05-08 20:58:19 +02:00
Uli Schlachter 7ea816045d "make check": Support out-of-tree builds
Currently, tests/run.sh expects the directory layout that our wrapper Makefile
sets up before running CMake. This commit adds support for any other directory
configuration as well.

Signed-off-by: Uli Schlachter <psychon@znc.in>
2016-05-08 18:52:18 +02:00
Uli Schlachter beb9b7f950 tests/run.sh: Set up some useful env vars
GDK_SCALE=1 is needed to overwrite other settings that people might have which
would make geometry-related tests fail.

NO_AT_BRIDGE=1 gets rid of the following message that I am seeing:

** (lua:8321): WARNING **: Error retrieving accessibility bus address:
org.freedesktop.DBus.Error.ServiceUnknown: The name org.a11y.Bus was not
provided by any .service files

Signed-off-by: Uli Schlachter <psychon@znc.in>
2016-05-08 16:48:24 +02:00
Daniel Hahler 087d578755 tests: fail if $AWESOME does not exist / is not executable (#872)
Without this the stdout/stderr rediraction when launching it makes the
error regarding the "program not found" disappear(?!).

[ci skip]
2016-05-03 12:08:00 +02:00
Daniel Hahler b6810576b0 tests/run.sh: display number of errors [ci skip] 2016-04-06 01:54:18 +02:00
Daniel Hahler 3730086bcc tests/run.sh: avoid warnings by setting up fg/bg/colors in xrdb 2016-03-04 01:41:43 +01:00
Uli Schlachter f87da55dfc Make the functional test runner more robust
Currently, an error in the default config in the right place isn't noticed. Fix
this by doing two things:

- Also grep for "error" (this catches runtime errors with a stack trace)
- Make _runner print a "success" message at the end and also grep for that

Fixes: https://github.com/awesomeWM/awesome/issues/689
Signed-off-by: Uli Schlachter <psychon@znc.in>
2016-02-13 07:36:59 +01:00
Uli Schlachter 781f8d30bb test/run.sh: Handle updated beauitful.init call in default config
While setting up an environment to run the integration tests in, the run.sh
script uses sed to generate versions of several files that refer to the
not-installed version of files. One of these needs to replace the call to
beautiful.init().

Before commit 20c9723c5b, the corresponding line was:

  beautiful.init("@AWESOME_THEMES_PATH@/default/theme.lua")

Now this wants to find and replace the following:

  beautiful.init(awful.util.get_themes_dir() .. "default/theme.lua")

To handle both versions, this commit adds some wildcards to the sed-expression
so that any line containing a call to beautiful.init is found and replaced.

Signed-off-by: Uli Schlachter <psychon@znc.in>
2016-01-17 16:16:39 +01:00
Uli Schlachter 5db6c04c42 Add vim modelines to all shell scripts
Signed-off-by: Uli Schlachter <psychon@znc.in>
2015-12-12 17:36:35 +01:00
Daniel Hahler fa1dea6df4 Improve test runner to be usable for git-bisect 2015-10-03 14:54:17 +02:00
Uli Schlachter 4bfddbb3f8 tests/run.sh: Start Xvfb with -noreset
When using Xephyr, this already starts the server with -noreset, but for Xvfb we
also need this flag. Without this flag, the DPI value that is set via xrdb gets
lost at server reset.

This wasn't a problem before commit 6d4837a53a. That commit moved the launch
of the dbus session after the setting of the DPI. So previously, waiting for
server startup was half broken (the dbus session already tried to connect to the
server to check for when it shuts down), but due to this no server reset
occurred and thus the DPI was correctly applied. After this commit, the server
immediately resets after xrdb is done setting the DPI and the value set is lost.

Signed-off-by: Uli Schlachter <psychon@znc.in>
2015-08-13 12:59:57 +02:00
Uli Schlachter cbba615748 tests/run.sh: Show full error messages
If an error occurs during startup, tail might not show the full error message.

Signed-off-by: Uli Schlachter <psychon@znc.in>
2015-08-12 09:49:30 +02:00
Uli Schlachter 0ee24317c7 tests/run.sh: Temporarily disable -x in wait_until_success
Signed-off-by: Uli Schlachter <psychon@znc.in>
2015-08-12 09:49:30 +02:00
Uli Schlachter 17e8827212 tests/run.sh: Factor out a helper function
This factors out a function wait_until_success that runs some command until it
succeeds (with a timeout) and uses this function in the two places where this
was already done before.

Note that this removes the "kill -0" trick for early exit again and instead will
use the timeout in case awesome dies during startup.

Signed-off-by: Uli Schlachter <psychon@znc.in>
2015-08-12 09:49:28 +02:00
Uli Schlachter 0688e8899e tests/run.sh: Allow execution without installing
When this script is not run under Travis, it will prepare a temporary config
file and a theme file that point to the files that were not yet installed.

Signed-off-by: Uli Schlachter <psychon@znc.in>
2015-08-12 09:49:08 +02:00
Uli Schlachter d8a1b563e7 tests/run.sh: Use a temporary directory
Signed-off-by: Uli Schlachter <psychon@znc.in>
2015-08-12 09:48:28 +02:00
Uli Schlachter 41d81a4501 tests/run.sh: Move definition of AWESOME_CLIENT up
Signed-off-by: Uli Schlachter <psychon@znc.in>
2015-08-12 09:48:27 +02:00
Uli Schlachter 6d4837a53a tests/run.sh: Start dbus after X server startup
The code does some dances with xrdb to ensure that the server finished starting
up. However, before this it already tries to access the server via dbus-launch.

Since nothing uses dbus in this part of the code, we can just move this down.

Signed-off-by: Uli Schlachter <psychon@znc.in>
2015-08-12 09:48:27 +02:00
Daniel Hahler 85facc0391 tests/run: handle 'expr 1 - 1' (exitcode 1); check for running awesome 2015-08-11 19:08:35 +02:00
Daniel Hahler f070c4e060 tests: wait until DISPLAY is ready; xrdb might complain/abort 2015-07-21 13:09:07 +02:00
Daniel Hahler 2c3c6b5ca8 tests: cleanup, remove sleep/print 2015-07-18 15:08:27 +02:00