Commit Graph

44 Commits

Author SHA1 Message Date
Fabian Homborg
23e6698dc8 cmake: Force color in the tests
For littlecheck/pexpect this just unconditionally enables color.

I have no idea what happens if you run cmake outside of a terminal
, but the worst that can happen is that *errors* have color
escapes in them.

If someone figures out how to get cmake to tell us if it's running in
a terminal, we can add a check.
2021-08-30 17:16:19 +02:00
Mahmoud Al-Qudsi
1f4d16cb07 Add support for bubbling up skipped tests to cmake
This prevents tests that were skipped (e.g. because of a missing
REQUIRES) from being reported as successes in the CTest overall run
results list.
2021-08-29 08:56:12 +02:00
Mahmoud Al-Qudsi
35975d83af Run each test fully independently in own environment 2021-08-29 08:56:12 +02:00
Fabian Homborg
61d322d403 Cleanup test setup a bit
Put the env setup into test_util, which does some additional work.

Also use some more builtins and stuff.
2020-11-15 09:17:23 +01:00
Fabian Homborg
fbe56a84c7 tests: Export $TERM
I think this might be causing problems on Github CI.
2020-09-06 11:15:54 +02:00
Fabian Homborg
fec3a593a2 test.fish: Erase more environment variables
Unfortunately this doesn't quite fix the issue with Pantheon
Terminal (#7913), as that somehow manages to re-set $VTE_VERSION by
the time littlecheck runs.
2020-07-13 20:55:39 +02:00
Fabian Homborg
52a6574751 tests: Use status dirname 2020-06-14 19:30:03 +02:00
Johannes Altmanninger
49c5f96470 Use set -l to force use of a local variable
Bare set overwrites a global/universal variable if it exists.
2020-05-15 08:25:07 +02:00
Fabian Homborg
1c880e5888 tests: Specify which tests completed successfully
It's a bit weird to just see "All tests completed successfully" and
then another set of tests follows.
2020-03-28 18:41:44 +01:00
Fabian Homborg
82977f07a0 test.fish: Remove mention of ".in" files 2020-03-28 18:29:21 +01:00
Fabian Homborg
4d052da922 Remove classic ".in" test machinery
We have now entirely switched the script tests to littlecheck.

Note: This adjusts the complete_directories test, because it removes a
directory that was created before by a .in test. There's no real
change in behavior.

This does require the test directory be cleaned, or the tests will fail.

test_util gets to stay for a while longer, because it sets up the
testing env (locale and such).
2020-03-17 21:34:36 +01:00
ridiculousfish
5f0c698601 Pass all check files to littlecheck at once
Instead of invoking littlecheck.py independently for each file, pass
all files at once. This amortizes the Python startup cost, and reduces
the total test time by ~15 seconds (!).
2020-03-15 18:16:14 -07:00
ridiculousfish
3040486968 Use -S when invoking Python for littlecheck
This prevents needless imports. It speeds up the test time by ~2 seconds.
2020-03-09 15:08:33 -07:00
Fabian Homborg
9367d4ff71 Reindent functions to remove useless quotes
This does not include checks/function.fish because that currently
includes a "; end" in a message that indent would remove, breaking the test.
2020-03-09 19:46:43 +01:00
Fabian Homborg
69b464bc37 Run fish_indent on all our fish scripts
It's now good enough to do so.

We don't allow grid-alignment:

```fish
complete -c foo -s b -l barnanana -a '(something)'
complete -c foo -s z              -a '(something)'
```

becomes

```fish
complete -c foo -s b -l barnanana -a '(something)'
complete -c foo -s z -a '(something)'
```

It's just more trouble than it is worth.

The one part I'd change:

We align and/or'd parts of an if-condition with the in-block code:

```fish
if true
   and false
    dosomething
end
```

becomes

```fish
if true
    and false
    dosomething
end
```

but it's not used terribly much and if we ever fix it we can just
reindent.
2020-01-13 20:34:22 +01:00
ridiculousfish
ca11b5edc8 Add a simple test for SIGINT out of loops 2019-10-15 22:33:53 -07:00
Aaron Gyes
5c06e68115 tests: add a lame gdate-based ms-precision timer if installed
I got tired of seeing ' ... ok (0 sec)' so now with GNU date/gdate
installed there is millisecond output shown. One can get rough
nanoseconds from gdate.
2019-09-19 14:26:53 -07:00
Fabian Homborg
7d3bec1297 test.fish: Notice littlecheck failures
This failed to propagate the exit status. Sorry!
2019-07-02 22:50:16 +02:00
Fabian Homborg
21667850b3 tests/test.fish: Normalize littlecheck output
This prints a green "ok" with the duration, just like the rest of the
tests.

Note that this clashes a bit with
https://github.com/ridiculousfish/littlecheck/pull/3.

(also don't check for python again and again and again)
2019-06-25 20:57:37 +02:00
Fabian Homborg
93d9fa4551 tests/test.fish: Run littlecheck with any python
Some operating systems don't have a "python" symlink, so we just run
it with the first python we find.
2019-06-25 19:25:36 +02:00
ridiculousfish
fcf0593dfb Port printf tests to littlecheck and teach the tester how to run it
This adds support for .check files inside the tests directory. .check
files are tests designed to be run with littlecheck.

Port printf test to littlecheck and remove the printf.in test.
2019-06-16 14:10:41 -07:00
Mahmoud Al-Qudsi
0c5015d467 Correct the reversed diff output for all tests
This has been driving nuts for years. The output of the diff emitted
when a test fails was always reversed, because the diff tool is called
with `${difftool} ${new} ${old}` so all the `-` and `+` contexts are
reversed, and the highlights are all screwed up.

The output of a `make test` run should show what has changed from the
baseline/expected, not how the expected differs from the actual. When
considered from both the perspective of intentional changes to the test
outputs and failed test outputs, it is desirable to see how the test
output has changed from the previously expected, and not the other way
around.

(If you were used to the previous behavior, I apologize. But it was
wrong.)
2019-03-28 18:23:32 -05:00
ridiculousfish
8e2d207869 Report duration of each test 2018-09-03 14:13:31 -07:00
Peter Ammon
5b489ca30f Remove caret redirection
This removes the caret as a shorthand for redirecting stderr.

Note that stderr may be redirected to a file via 2>/some/path...
and may be redirected with a pipe via 2>|.

Fixes #4394
2018-04-01 13:48:21 -07:00
Kurtis Rader
f3d025bd75 always define argv for scripts
Found when testing what breaks when dereferencing undefined vars is an
errors (#4163).

Fixes #4181
2017-07-03 16:58:14 -07:00
Kurtis Rader
763063e741 more changes to better handle undef vars
This is only needed if we make undefined vars an error as proposed in
issue #4163 but it's a good idea anyway.
2017-07-03 10:16:31 -07:00
Kurtis Rader
06282f02fd don't run fish_update_completions in unit tests
Running `fish_update_completions` in unit tests is not needed and
hideously expensive. To the point it can cause flakey test behavior.
2017-01-29 18:00:48 -08:00
Kurtis Rader
93bea5b321 make unit tests more hermetic regarding env vars
Specifically, the `TERM` and `ITERM_PROFILE` env vars. If we don't set
them to known values the unit tests can fail in unexpected ways.

Fixes #3395
2016-09-19 22:13:55 -07:00
Kurtis Rader
2f0cb2a32b remove the tests/*.status files
The tests/*.status files aren't useful so eliminate them. Just verify whether
a given test module exited with a success status. There isn't any point in
having a "status" file that indicates the test module should exit with a
success (zero) status.

Closes #3208
Closes #3209
2016-07-06 21:37:29 -07:00
Kurtis Rader
8b67a1b26f make testing on local servers hermetic
I noticed while fixing issue #2702 that the fish program being tested
was sourcing config.fish files outside of the current build. This also
happens when Travis CI runs the tests but isn't an issue there because
of how Travis is configured to execute the tests.

I also noticed that running `make test` was polluting my personal fish
history; which will become a bigger problem if and when the fishd universal
var file is moved from $XDG_CONFIG_HOME to $XDG_DATA_HOME.

This change makes it possible for an individual to run the tests on
their local machine secure in the knowledge that only the config.fish and
related files from their git repository will be used and doing so won't
pollute their personal fish history.

Resolves #469
2016-02-25 17:16:36 -08:00
Kurtis Rader
6969cfab3d fix unit tests related to the escape timeout 2015-12-23 21:41:48 -08:00
ridiculousfish
96589920d5 Allow running specific high-level tests by passing the name to test.fish
Example:
  tests/test.fish expansion
This will run the tests in expansion.in only
2014-12-23 12:20:44 -08:00
Kevin Ballard
e13d423b68 Tweak test runner to set up environment better
Update the test runners so they set up their own environment in
test_util.fish. This simplifies the Makefile and paves the way for
adding utility functions for use in the tests themselves.
2014-11-24 01:51:07 -08:00
Kevin Ballard
ae7b6156ac Rewrite fishscript testrunner for better output
Update the fishscript testrunner to use the same output style as the
interactive testrunner.
2014-10-02 12:33:42 -07:00
Kevin Ballard
5f82f721d2 Rejigger test suite
Split test_interactive off from test_fishscript and add a new target
test_high_level that tests both.

Add some Makefile magic so the tests can be run serially without using
sub-make, which gets rid of a little noise from the make output.

Rewrite interactive tests to look better.
2014-09-23 22:39:23 -07:00
Kevin Ballard
190712d4b0 Add a test harness for interactive behavior
Add a test harness that uses `expect` to drive Fish to test interactive
behavior.

Include some tests for `read`.
2014-09-07 23:56:59 -07:00
Kevin Ballard
efb1467e4e Cleanup fish tests a bit
Split `make test` into two targets `make test_low_level` and `make
test_fishscript`, primarily so fishscript tests can be rechecked quickly
after edits.

Reformat the test.fish file and update some of the code to be a little
more straightforward (e.g. `if not cmd` instead of `if cmd; else`).
2014-09-07 23:50:44 -07:00
Kevin Ballard
1c4223889b Fix test output for complete -e tests
GNU sort behaves stupidly when LC_ALL is not C. This caused the test
output to be sorted wrong.
2014-09-02 15:52:56 -07:00
ridiculousfish
699d78bcfe Make the test harness output file diffs on failure 2014-02-04 11:28:29 -08:00
David Adam
3106cffb65 Makefile/tests: use return values to communicate errors 2014-01-15 23:50:42 +08:00
Jan Kanis
640beafa8c fix a memory leak in profiling, add a profiling test 2013-01-14 01:38:05 +01:00
Grissiom
27df727ca1 don't parse the options that should be passed to fish scripts
This commit makes fish_parse_opt stop at first non-option argument,
which expected as fish script name and pass the remaining argument to
that script. I also updated the test cases.
2010-10-03 11:46:26 +08:00
Grissiom
70322077d2 remove trialing spaces #2 2010-09-18 10:18:26 +08:00
axel
7ddecde543 Add subdirectories
darcs-hash:20050920133155-ac50b-9a14c6c664dd03afbe8e15e7c7998fcfb5c3c750.gz
2005-09-20 23:31:55 +10:00