Commit Graph

176 Commits

Author SHA1 Message Date
Alan Guo Xiang Tan
865f7a9852
DEV: Reduce number of parallel test processors to 3 for system tests (#22423)
This is an experiment to see if not saturating resources on a runner
with 8 vCPU leads to more stability on our CI.
2023-07-05 08:08:46 +08:00
David Taylor
251d6f0627
DEV: Introduce Zeitwerk reloading check in CI (#22151)
This should help us to catch the most catastrophic cases where plugins fail to auto-reload in development mode.
2023-06-16 14:33:14 +01:00
Alan Guo Xiang Tan
5897709a90
DEV: Use runtime info to split test files for parallel testing (#22060)
Using the runtime information, we will be able to more efficiently group
the test files across the test processes hence leading to better
utilization of resources.
2023-06-12 09:07:17 +08:00
Daniel Waterworth
67afd85aae
Revert "DEV: Use runtime info to split test files for parallel testing (#21896)" (#22016)
This reverts commit 14ed971db6.

This prevented the core backend tests from running in GitHub CI
2023-06-08 15:13:26 -05:00
Alan Guo Xiang Tan
14ed971db6
DEV: Use runtime info to split test files for parallel testing (#21896)
Using the runtime information, we will be able to more efficiently group
the test files across the test processes hence leading to better
utilization of resources.
2023-06-05 08:01:41 +08:00
Alan Guo Xiang Tan
ba251dec6b
DEV: Increase number of parallel process for system test to 5 on CI (#21878)
4dd053a69c addressed most of the
instability we were seeing with system tests on CI and locally. Let's
try pushing the number of parallel processes up to squeeze as much time
savings as possible from the runner.
2023-06-01 10:41:11 +08:00
Alan Guo Xiang Tan
2c1c7d1e4f DEV: Use --profile when running system tests in CI 2023-05-30 13:46:14 +09:00
Alan Guo Xiang Tan
123a77a2bc
DEV: Correct Capybara default max wait time setting in CI (#21801)
We were not setting capybara's default max wait time correctly in CI due
to a spelling error.

This regressd in fc17045876
2023-05-29 10:41:24 +08:00
Alan Guo Xiang Tan
038318bffc
DEV: Bump CAPYBARA_DEFAULT_MAX_WAIT_TIME to 10 seconds on CI (#21711)
We're running on pretty crappy hardware on Github's CI and this has an
impact on the stability of our system tests on CI. Therefore, we are
bumping `CABPYARA_DEFAULT_MAX_WAIT_TIME` to 10 seconds to account for
the less than ideal hardware we're running the system tests on.

This change trades off speed for stability but speed is already bad on
CI so stability is more important for our case.
2023-05-25 09:25:08 +08:00
Jarek Radosz
fc17045876
DEV: Clean up workflow files (#21526) 2023-05-12 14:00:04 +02:00
Jarek Radosz
19ac90536f
DEV: Restore the documentation format in system tests (#21471) 2023-05-12 11:13:52 +02:00
David Taylor
d6f94e0916
DEV: Revert buildjet (#21475)
This reverts commits 17603794b5 and 41bdf8cfcd.
2023-05-11 22:25:30 +02:00
Natalie Tay
17603794b5
DEV: Reduce plugin system test parallel processors (#21466) 2023-05-10 15:43:41 +08:00
Natalie Tay
41bdf8cfcd
DEV: Use BuildJet for some test jobs (#21404) 2023-05-10 10:39:27 +08:00
Alan Guo Xiang Tan
1f6d57ab93
DEV: Run turbo rspecs with verbose output on CI (#21379)
In CI, we the output to be as verbose as possible so that when it fails we have the necessary information to debug the failures.
2023-05-04 10:34:02 +08:00
David Taylor
c6f5b4297d
DEV: Use --frozen-lockfile in GitHub CI (#21338)
This will help us to catch missing lockfile changes before they are merged into `main`
2023-05-02 16:25:22 +01:00
Joffrey JAFFEUX
e495a2fc3f
DEV: Enable parallel system specs in GitHub actions CI (#21251)
Also skips/improves few flakey specs
2023-04-26 13:02:19 +02:00
Jarek Radosz
00630e4c74
DEV: Remove RUBY_GLOBAL_METHOD_CACHE_SIZE (#21249)
It doesn't do anything since ruby 3.0.0.preview1. It was removed in https://github.com/ruby/ruby/pull/2888
2023-04-26 10:39:39 +02:00
Joffrey JAFFEUX
d4c6457065
DEV: increase plugin system tests timeout (#21247) 2023-04-26 10:08:10 +02:00
Alan Guo Xiang Tan
3cb9fd739a
DEV: Run system tests with documentation format on github actions (#21069)
Allows us to see the tests which have timed out
2023-04-12 14:47:05 +08:00
Daniel Waterworth
52999d1422
DEV: Show which system tests are being executed in CI (#20795)
... to see if it's always the same test causing timeouts
2023-03-23 15:12:14 -05:00
David Taylor
150a6601c0
DEV: Check Zeitwerk eager loading in GitHub CI (#20699)
In production, `eager_load=true`. This sometimes leads to boot errors which are not present in dev/test environments. Running `zeitwerk:check` in CI will help us to pick up on any errors early.

This commit also introduces a `DISCOURSE_ZEITWERK_EAGER_LOAD` environment variable to make it easier to toggle the behaviour when developing locally.
2023-03-16 14:22:16 +00:00
Daniel Waterworth
5324216740 DEV: Use rspec instead of turbo_rspec with one core 2023-03-15 16:46:48 -05:00
Rafael dos Santos Silva
c3ae555ac2
DEV: Show correct version of Ruby we are using on CI (#20415) 2023-02-22 15:39:49 -03:00
David Taylor
a6b680f4fe
DEV: Fix GitHub CI permissions issues (#20069)
The `git` version in our discourse_test docker image was recently updated to include a permissions check before running any git commands. For this to pass, the owner of the discourse directory needs to match the user running any git commands.

Under GitHub actions, by default the working directory is created with uid=1000 as the owner. We run all our tests as `root`, so this mismatch causes git to raise the permissions error. We can't switch to run the entire workflow as the `discourse (uid=1000)` user because our discourse_test image is not configured to allow `discourse` access to postgres/redis directories. For now, this commit updates the working directory's owner to match the user running the workflow.
2023-01-30 15:39:43 +00:00
David Taylor
b96869d5fb
DEV: Disable parallel system specs in GitHub actions (#20023)
We have some flakiness which needs to be resolved. Followup to e717529d80
2023-01-26 14:37:47 +00:00
David Taylor
e717529d80
DEV: Enable parallel system specs in GitHub actions CI (#19584) 2023-01-26 13:26:02 +00:00
Rafael dos Santos Silva
60ebbfd7e7
DEV: Stop testing with Ruby 3.2 for now (#19909) 2023-01-18 12:04:49 -03:00
Rafael dos Santos Silva
076b3a6514
DEV: Key bundler CI cache on Ruby version (#19868) 2023-01-13 11:39:49 -03:00
Rafael dos Santos Silva
8e7e6e14c7
DEV: Add Ruby 3.2 to test matrix (#19862)
* DEV: Add Ruby 3.2 to test matrix

* DEV: Update test name
2023-01-13 09:22:33 -03:00
Alan Guo Xiang Tan
e58277adf3
DEV: Increase Capybara.default_max_wait_time on github actions (#19750)
Our working theory is that system tests on Github run on much less
powerful hardware as compared to running the tests on our work machines.
Hopefully, increasing the wait time now will help reduce some flakes
that we're seeing on Github.
2023-01-05 08:50:35 +08:00
David Taylor
335893ae91
DEV: Correct private-fork -> private-mirror (#19560)
Followup to 7eb9482ba9
2022-12-21 16:52:35 +00:00
David Taylor
7eb9482ba9
DEV: Skip 'push' workflow events for discourse-private-mirror (#19556)
We don't want 'push' workflows to run on this private fork (which is used for developing security-fixes before public disclosure)
2022-12-21 16:44:38 +00:00
Martin Brennan
8b3c6cd396
DEV: Fix github workflow system spec screenshot location (#19435)
These screenshots are located at paths like:

/__w/discourse/discourse/tmp/capybara/failures_r_spec_example_groups_quoting_chat_message_transcripts_copying_quote_transcripts_with_the_clipboard_quotes_multiple_chat_messages_into_a_topic_134.png

not /tmp/screenshots. This should fix the issue. Also makes plugin system specs
use documentation format and profile.
2022-12-13 15:36:30 +10:00
Alan Guo Xiang Tan
5103268312
DEV: Run system tests with documentation and profiling on actions (#19271)
When a test takes too long, we want to know which test and at what step
2022-12-01 05:54:17 +08:00
David Taylor
f8939bd294
DEV: Bump @actions/checkout to v3 in frontend tests (#18989)
v2 uses Node 12, which is deprecated
2022-11-11 13:31:28 +00:00
David Taylor
70a990da03
DEV: Update GitHub actions set-output uses (#18988)
https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-11 13:12:08 +00:00
Joffrey JAFFEUX
11f3618b80
DEV: initial system tests for chat and plugins (#18881)
This is a very basic to ensure it's working and open future possible work
2022-11-04 15:06:24 +01:00
Martin Brennan
57caf08e13
DEV: Minimal first pass of rails system test setup (#16311)
This commit introduces rails system tests run with chromedriver, selenium,
and headless chrome to our testing toolbox.

We use the `webdrivers` gem and `selenium-webdriver` which is what
the latest Rails uses so the tests run locally and in CI out of the box.

You can use `SELENIUM_VERBOSE_DRIVER_LOGS=1` to show extra
verbose logs of what selenium is doing to communicate with the system
tests.

By default JS logs are verbose so errors from JS are shown when
running system tests, you can disable this with
`SELENIUM_DISABLE_VERBOSE_JS_LOGS=1`

You can use `SELENIUM_HEADLESS=0` to run the system
tests inside a chrome browser instead of headless, which can be useful to debug things
and see what the spec sees. See note above about `bin/ember-cli` to avoid
surprises.

I have modified `bin/turbo_rspec` to exclude `spec/system` by default,
support for parallel system specs is a little shaky right now and we don't
want them slowing down the turbo by default either.

### PageObjects and System Tests

To make querying and inspecting parts of the page easier
and more reusable inbetween system tests, we are using the
concept of [PageObjects](https://www.selenium.dev/documentation/test_practices/encouraged/page_object_models/) in
our system tests. A "Page" here is generally corresponds to
an overarching ember route, e.g. "Topic" for `/t/324345/some-topic`,
and this contains logic for querying components within the topic
such as "Posts".

I have also split "Modals" into their own entity. Further down the
line we may want to explore creating independent "Component"
contexts.

Capybara DSL should be included in each PageObject class,
reference for this can be found at https://rubydoc.info/github/teamcapybara/capybara/master#the-dsl

For system tests, since they are so slow, we want to focus on
the "happy path" and not do every different possible context
and branch check using them. They are meant to be overarching
tests that check a number of things are correct using the full stack
from JS and ember to rails to ruby and then the database.

### CI Setup

Whenever a system spec fails, a screenshot
is taken and a build artifact is produced _after the entire CI run is complete_,
which can be downloaded from the Actions UI in the repo.

Most importantly, a step to build the Ember app using Ember CLI
is needed, otherwise the JS assets cannot be found by capybara:

```
- name: Build Ember CLI
  run: bin/ember-cli --build
```

A new `--build` argument has been added to `bin/ember-cli` for this
case, which is not needed locally if you already have the discourse
rails server running via `bin/ember-cli -u` since the whole server is built and
set up by default.

Co-authored-by: David Taylor <david@taylorhq.com>
2022-09-28 11:48:16 +10:00
David Taylor
885e133cac
DEV: Cache turbo_rspec_runtime.log in CI (#18318) 2022-09-21 22:13:25 +01:00
David Taylor
a11aea4fe2
DEV: Update github workflow for 8-core workers (#18271) 2022-09-21 18:13:13 +01:00
David Taylor
b0a9d8b761
DEV: Improve labelling of Firefox Evergreen/ESR CI runs (#18313)
Both versions are used with `--headless`, so labelling one "Firefox" and the other "Firefox Headless" doesn't really make sense. Evergreen / ESR are better descriptions.
2022-09-21 15:34:26 +01:00
David Taylor
42d226f727
DEV: Ensure GitHub workflows cancel cleanly (#18314)
We added `always()` on some steps so that they run even if previous steps fail. That helps give us a picture of all failures in one run, rather than having to re-run the workflow after fixing the first failure.

However, when we explicitly cancel a job, we should skip running these steps. `!cancelled()` is a better substitute for `always()` in this case.
2022-09-21 14:32:21 +01:00
David Taylor
e06b9d4a52
DEV: Remove support for legacy plugin JS compilation pipeline (#18293)
This became the default in b1755137
2022-09-21 12:38:02 +01:00
David Taylor
ef39193a06
DEV: Add rake plugins:turbo_spec task (#18289)
This leans on our existing `turbo_rspec` implementation to run plugin specs in parallel on all available cores
2022-09-20 15:42:54 +01:00
David Taylor
33a2624f09
DEV: Introduce flag for compiling Plugin JS with Ember CLI (#17965)
When `EMBER_CLI_PLUGIN_ASSETS=1`, plugin application JS will be compiled via Ember CLI. In this mode, the existing `register_asset` API will cause any registered JS files to be made available in `/plugins/{plugin-name}_extra.js`. These 'extra' files will be loaded immediately after the plugin app JS file, so this should not affect functionality.

Plugin compilation in Ember CLI is implemented as an addon, similar to the existing 'admin' addon. We bypass the normal Ember CLI compilation process (which would add the JS to the main app bundle), and reroute the addon Broccoli tree into a separate JS file per-plugin. Previously, Sprockets would add compiled templates directly to `Ember.TEMPLATES`. Under Ember CLI, they are compiled into es6 modules. Some new logic in `discourse-boot.js` takes care of remapping the new module names into the old-style `Ember.TEMPLATES`.

This change has been designed to be a like-for-like replacement of the old plugin compilation system, so we do not expect any breakage. Even so, the environment variable flag will allow us to test this in a range of environments before enabling it by default.

A manual silence implementation is added for the build-time `ember-glimmer.link-to.positional-arguments` deprecation while we work on a better story for plugins.
2022-08-22 09:56:39 +01:00
Jarek Radosz
5d1cf006ab
DEV: Tweak core_frontend_tests timeouts (#17902)
Each test chunk takes about 10 minutes, so those timeouts can be decreased from 20 to 15.

And there are three of those chunks so total can be a bit over 30 minutes, hence the bump to 35.
2022-08-14 17:30:15 +02:00
Jarek Radosz
89d43235c4
DEV: Set permissions for GitHub actions (#17733)
Included permissions for the action.

https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions
https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs

Co-authored-by: naveensrinivasan <172697+naveensrinivasan@users.noreply.github.com>
2022-07-30 17:22:03 +02:00
Jarek Radosz
eef358dc13
DEV: Remove frontend/core-plugins job (#17533)
Frontend tests for core plugins already run in frontend/plugins job
2022-07-16 22:55:39 +02:00
Jarek Radosz
2c1fc28d00
DEV: Remove ember-cli flags from the backend (#17147)
…and other auxiliary code

* Restore `QUNIT_EMBER_CLI` flag warning
* Add `ALLOW_EMBER_CLI_PROXY_BYPASS`
2022-06-20 16:33:05 +02:00
David Taylor
12980418ae
DEV: Disable the use of 'legacy' Ember assets (#17127)
Anyone still using `EMBER_CLI_PROD_ASSETS=0` in development or production will be gracefully switched to Ember CLI. In development, a repeated message will be logged to STDERR.

Similarly, passing `QUNIT_EMBER_CLI=0` to the qunit rake task will now do nothing. A warning will be printed, and ember-cli mode will be used. Note that we've chosen not to fail the task, so that existing plugin/theme CI jobs don't immediately start failing. We may switch to a hard fail in the coming days/weeks.
2022-06-17 16:51:28 +01:00
Jarek Radosz
fcb4e5a1a1
DEV: Make wizard an ember addon (#17027)
Co-authored-by: David Taylor <david@taylorhq.com>
2022-06-17 14:50:21 +02:00
Alan Guo Xiang Tan
072faa08bb DEV: Reenable wizard test after fixing auto start for wizard qunit tests
`run-qunit.js` does not expect QUnit tests to start automatically but
our wizard QUnit setup did not respect the `qunit_disable_auto_start`
URL param. Hence, tests would start running automatically and when a
subsequent `QUnit.start()` function call is made, we ended up getting a
`QUnit.start cannot be called inside a test context.` error.

This error can be consistently reproduced in the `discourse:discourse_test` container but not in
the local development environment. I do not know why and did not feel
like it is important at this point in time to know why.
2022-05-25 15:12:27 +08:00
Alan Guo Xiang Tan
05212f6cc3 DEV: Skip wizard tests while we investigate why it fails on CI 2022-05-25 13:48:09 +08:00
Jarek Radosz
de19003bad
DEV: Minor workflow updates (#16583) 2022-04-28 15:51:48 +02:00
David Taylor
e9c1e3d022
DEV: Always run all three qunit partitions, even with earlier failure (#16411)
Previously, if Core QUnit 1 failed, then QUnit 2/3 wouldn't even be attempted. When dealing with multiple failures, this can make the feedback cycle. Setting `if: always()` ensures that the steps run regardless of any earlier failures. This is the same approach we take in the linting workflow.
2022-04-07 14:44:52 +01:00
dependabot[bot]
a9ea43ee67
Build(deps): Bump actions/cache from 2 to 3 (#16296)
Bumps [actions/cache](https://github.com/actions/cache) from 2 to 3.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](https://github.com/actions/cache/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-31 12:51:42 +02:00
Alan Guo Xiang Tan
66cf866b4a
DEV: Pull compatible version for plugins in Github test workflow. (#16219)
We have 3 branches which we care about, `main`, `beta` and `stable`.
However, each of this branch has different compatibilties with plugins
and we want to respect that.
2022-03-18 10:49:58 +08:00
Alan Guo Xiang Tan
937d79405d
DEV: Run tests on push to beta and stable branch too. (#16209) 2022-03-18 10:15:48 +08:00
David Taylor
8a69de7964
DEV: Enable Ember CLI tests in GitHub Actions for official plugins (#16148) 2022-03-10 12:36:58 +00:00
Robin Ward
14d31417de Revert "Revert "DEV: Run Ember CLI tests in random order" (#15717)"
The worst of the flaky tests should be fixed now, so let's re-enable
this.
2022-01-27 12:22:20 -05:00
David Taylor
2464839cbf
Revert "DEV: Run Ember CLI tests in random order" (#15717)
This reverts commit f43bba8d59.

Adding randomness has introduced a lot of flakiness in our ember-cli tests. We should fix those issues at the source. However, given the upcoming stable release, this randomness has been reverted so that the stable release includes a stable test suite. Having a stable test suite on stable will make backporting future commits much easier.
2022-01-26 15:30:03 +00:00
Robin Ward
f43bba8d59 DEV: Run Ember CLI tests in random order
In browser this uses the `seed` config, in ember exam it adds `--random`
as a parameter.
2022-01-25 14:49:40 -05:00
David Taylor
ffd0f5b500
DEV: Update GitHub actions config (#15636)
- Move ember-cli tests into the main test workflow, so they're listed alongside other tests
- Remove the 'experimental' label
- Add the 'legacy' label to old-style qunit tests
- Add core-plugin EmberCLI tests
- Add scaffolding for all-plugin EmberCLI tests, but disable in matrix for now
2022-01-19 10:41:52 +00:00
David Taylor
da6f837572
DEV: Bump bundler from 2.2.26 to 2.3.4 (#15549) 2022-01-13 08:50:04 +11:00
David Taylor
948574b978
DEV: Update GitHub workflows to use slim discourse_test images (#15298) 2021-12-14 18:30:25 +00:00
David Taylor
0e87f882a7
DEV: Use discourse image for postgres in GitHub Actions (#15291)
The discourse base image already contains a postgres installation, so pulling a separate postgres image is a little wasteful. Using the copy of Postgres in the discourse image saves about 20 seconds on every GitHub actions run.

This commit sets up Postgres with a few performance-improving flags, which we were already using for the `rake docker:test` task (used on our internal CI system).
2021-12-14 17:20:06 +00:00
David Taylor
1c42b36c89
DEV: Use slim container for backend tests in GitHub Actions (#15290) 2021-12-14 11:32:35 +00:00
David Taylor
eb3eae2fcc
DEV: Cache database in GitHub actions tests (#15279)
A cached database (and its uploads) will only be used if the current run has exactly the same set of migration files. Otherwise, the database will be migrated from scratch

This saves approximately 75s on the core backend specs and 45s on other runs.
2021-12-14 09:40:16 +00:00
Penar Musaraj
1b5f26e0ab
DEV: Avoid cancelling in-progress tests for branches (#15101) 2021-11-25 17:31:05 -05:00
Penar Musaraj
78830d14b2
DEV: Cancel in-progress test runs on PRs (#15096) 2021-11-25 15:44:40 -05:00
Jarek Radosz
e1581f6dfd
DEV: Use discourse image redis in CI (#14280) 2021-09-08 14:01:37 +02:00
Alan Guo Xiang Tan
9b7c17b925
DEV: Remove references to stale branches in github actions. (#14053) 2021-08-16 13:29:07 +08:00
David Taylor
8c370c3fe3 DEV: Add annotate rake tasks, and enforce via GitHub actions
`bin/rake annotate` is an alias of `bin/annotate --models`
`bin/rake annotate:clean` generates annotations by using a temporary, freshly migrated database. This should help us to produce more consistent annotations, even if development databases have been polluted by plugin migrations.

A GitHub actions task is also added which generates annotations on a clean database, and raises an error if they differ from the committed annotations.
2021-07-06 10:11:06 +01:00
Jarek Radosz
78dcf4ad08 DEV: Run turbo_rspec in the verbose mode on CI 2021-06-08 09:27:23 +08:00
Jarek Radosz
a60e26e799
DEV: Clean up and refactor CI workflow(s) (#12144)
Includes:

* DEV: Remove external plugin linting (that's covered by CI in their repositories)
* DEV: Move lint stages to a separate workflow (partial de-`if`-ication of workflows)
* DEV: Run CI on `main` branch too
* DEV: Update postgres to 13
* DEV: Update redis to 6.x

Other changes:
* DEV: Remove matrix.os
* DEV: Remove env.BUILD_TYPE
* DEV: Remove env.TARGET
* DEV: Rename `build_types` config option to `build_type`
* DEV: Lowercase `target` and `build_type` names
* DEV: Rename `ci` to `tests`
* DEV: Rename `lint` to `linting`
* DEV: Lower the wizard qunit timeout (30 min -> 10)
* DEV: Ruby version is no longer configurable
* DEV: Run plugin tests only in the `plugins` target
* DEV: Use binstubs where applicable
* DEV: We don't open PRs to `tests-passed`
2021-02-22 10:28:32 +01:00