mirror of
https://github.com/discourse/discourse.git
synced 2024-11-23 22:26:26 +08:00
39da9106ba
What motivated this change? Our builds on Github actions have been extremely flaky mostly due to system tests. This has led to a drop in confidence in our test suite where our developers tend to assume that a failed job is due to a flaky system test. As a result, we have had occurrences where changes that resulted in legitimate test failures are merged into the `main` branch because developers assumed it was a flaky test. What does this change do? This change seeks to reduce the flakiness of our builds on Github Actions by automatically re-running RSpec tests once when they fail. If a failed test passes subsequently in the re-run, we mark the test as flaky by logging it into a file on disk which is then uploaded as an artifact of the Github workflow run. We understand that automatically re-runs will lead to lower accuracy of our tests but we accept this as an acceptable trade-off since a fragile build has a much greater impact on our developers' time. Internally, the Discourse development team will be running a service to fetch the flaky tests which have been logged for internal monitoring. How is the change implemented? 1. A `--retry-and-log-flaky-tests` CLI flag is added to the `bin/turbo_rspec` CLI which will then initialize `TurboTests::Runner` with the `retry_and_log_flaky_tests` kwarg set to `true`. 2. When the `retry_and_log_flaky_tests` kwarg is set to `true` for `TurboTests::Runner`, we will register an additional formatter `Flaky::FailuresLoggerFormatter` to the `TurboTests::Reporter` in the `TurboTests::Runner#run` method. The `Flaky::FailuresLoggerFormatter` has a simple job of logging all failed examples to a file on disk when running all the tests. The details of the failed example which are logged can be found in `TurboTests::Flaky::FailedExample.to_h`. 3. Once all the tests have been run once, we check the result for any failed examples and if there are, we read the file on disk to fetch the `location_rerun_location` of the failed examples which is then used to run the tests in a new RSpec process. In the rerun, we configure a `TurboTests::Flaky::FlakyDetectorFormatter` with RSpec which removes all failed examples from the log file on disk since those examples are not flaky tests. Note that if there are too many failed examples on the first run, we will deem the failures to likely not be due to flaky tests and not re-run the test failures. As of writing, the threshold of failed examples is set to 10. If there are more than 10 failed examples, we will not re-run the failures.
415 lines
15 KiB
YAML
415 lines
15 KiB
YAML
name: Tests
|
|
|
|
on:
|
|
pull_request:
|
|
paths-ignore:
|
|
- "migrations/**"
|
|
push:
|
|
branches:
|
|
- main
|
|
- beta
|
|
- stable
|
|
paths-ignore:
|
|
- "migrations/**"
|
|
|
|
concurrency:
|
|
group: tests-${{ format('{0}-{1}', github.head_ref || github.run_number, github.job) }}
|
|
cancel-in-progress: true
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
build:
|
|
if: github.event_name == 'pull_request' || github.repository != 'discourse/discourse-private-mirror'
|
|
name: ${{ matrix.target }} ${{ matrix.build_type }}${{ matrix.updated_ember && ' (Ember 5)' || '' }}
|
|
runs-on: ${{ (matrix.build_type == 'annotations') && 'ubuntu-latest' || 'ubuntu-20.04-8core' }}
|
|
container: discourse/discourse_test:slim${{ (matrix.build_type == 'frontend' || matrix.build_type == 'system') && '-browsers' || '' }}${{ (matrix.ruby == '3.1') && '-ruby-3.1.0' || '' }}
|
|
timeout-minutes: 20
|
|
|
|
env:
|
|
DISCOURSE_HOSTNAME: www.example.com
|
|
RAILS_ENV: test
|
|
PGUSER: discourse
|
|
PGPASSWORD: discourse
|
|
USES_PARALLEL_DATABASES: ${{ matrix.build_type == 'backend' || matrix.build_type == 'system' }}
|
|
CAPYBARA_DEFAULT_MAX_WAIT_TIME: 10
|
|
MINIO_RUNNER_LOG_LEVEL: DEBUG
|
|
EMBER_VERSION: ${{ (matrix.updated_ember && '5') || '3' }}
|
|
|
|
strategy:
|
|
fail-fast: false
|
|
|
|
matrix:
|
|
build_type: [backend, frontend, system, annotations]
|
|
target: [core, plugins, themes]
|
|
ruby: ["3.2"]
|
|
updated_ember: [false]
|
|
exclude:
|
|
- build_type: annotations
|
|
target: plugins
|
|
- build_type: annotations
|
|
target: themes
|
|
- build_type: backend
|
|
target: themes
|
|
- build_type: frontend
|
|
target: core # Handled by core_frontend_tests job (below)
|
|
include:
|
|
- build_type: system
|
|
target: chat
|
|
- build_type: system
|
|
target: chat
|
|
updated_ember: true
|
|
- build_type: system
|
|
target: core
|
|
updated_ember: true
|
|
- build_type: frontend
|
|
target: plugins
|
|
updated_ember: true
|
|
- build_type: frontend
|
|
target: themes
|
|
updated_ember: true
|
|
|
|
steps:
|
|
- name: Set working directory owner
|
|
run: chown root:root .
|
|
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
fetch-depth: 1
|
|
|
|
- name: Setup Git
|
|
run: |
|
|
git config --global user.email "ci@ci.invalid"
|
|
git config --global user.name "Discourse CI"
|
|
|
|
- name: Start redis
|
|
run: |
|
|
redis-server /etc/redis/redis.conf &
|
|
|
|
- name: Start Postgres
|
|
run: |
|
|
chown -R postgres /var/run/postgresql
|
|
sudo -E -u postgres script/start_test_db.rb
|
|
sudo -u postgres psql -c "CREATE ROLE $PGUSER LOGIN SUPERUSER PASSWORD '$PGPASSWORD';"
|
|
|
|
- name: Bundler cache
|
|
uses: actions/cache@v3
|
|
with:
|
|
path: vendor/bundle
|
|
key: ${{ runner.os }}-${{ matrix.ruby }}-gem-${{ hashFiles('**/Gemfile.lock') }}-cachev2
|
|
|
|
- name: Setup gems
|
|
run: |
|
|
gem install bundler --conservative -v $(awk '/BUNDLED WITH/ { getline; gsub(/ /,""); print $0 }' Gemfile.lock)
|
|
bundle config --local path vendor/bundle
|
|
bundle config --local deployment true
|
|
bundle config --local without development
|
|
bundle install --jobs 4
|
|
bundle clean
|
|
|
|
- name: Get yarn cache directory
|
|
id: yarn-cache-dir
|
|
run: echo "dir=$(yarn cache dir)" >> $GITHUB_OUTPUT
|
|
|
|
- name: Yarn cache
|
|
uses: actions/cache@v3
|
|
id: yarn-cache
|
|
with:
|
|
path: ${{ steps.yarn-cache-dir.outputs.dir }}
|
|
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}-cachev2
|
|
|
|
- name: Upgrade Ember
|
|
if: matrix.updated_ember == true
|
|
run: script/switch_ember_version 5
|
|
|
|
- name: Yarn install
|
|
run: yarn install --frozen-lockfile
|
|
|
|
- name: Checkout official plugins
|
|
if: matrix.target == 'plugins'
|
|
run: bin/rake plugin:install_all_official
|
|
|
|
- name: Pull compatible versions of plugins
|
|
if: matrix.target == 'plugins'
|
|
run: bin/rake plugin:pull_compatible_all
|
|
|
|
- name: Plugin gems cache
|
|
uses: actions/cache@v3
|
|
with:
|
|
path: plugins/*/gems
|
|
key: ${{ runner.os }}-plugin-gems-${{matrix.ruby}}-${{ hashFiles('plugins/*/plugin.rb') }}
|
|
|
|
- name: Checkout official themes
|
|
if: matrix.target == 'themes' && matrix.build_type == 'system'
|
|
run: bin/rake themes:clone_all_official
|
|
|
|
- name: Add hosts to /etc/hosts, otherwise Chrome cannot reach minio
|
|
if: matrix.build_type == 'system' && matrix.target == 'core'
|
|
run: |
|
|
echo "127.0.0.1 minio.local" | sudo tee -a /etc/hosts
|
|
echo "127.0.0.1 discoursetest.minio.local" | sudo tee -a /etc/hosts
|
|
|
|
- name: Fetch app state cache
|
|
uses: actions/cache@v3
|
|
id: app-cache
|
|
with:
|
|
path: tmp/app-cache
|
|
key: >-
|
|
${{ runner.os }}-
|
|
${{ hashFiles('.github/workflows/tests.yml') }}-
|
|
${{ hashFiles('db/**/*', 'plugins/**/db/**/*') }}-
|
|
${{ env.USES_PARALLEL_DATABASES }}
|
|
|
|
- name: Restore database from cache
|
|
if: steps.app-cache.outputs.cache-hit == 'true'
|
|
run: script/silence_successful_output psql --quiet -o /dev/null -f tmp/app-cache/cache.sql postgres
|
|
|
|
- name: Restore uploads from cache
|
|
if: steps.app-cache.outputs.cache-hit == 'true'
|
|
run: rm -rf public/uploads && cp -r tmp/app-cache/uploads public/uploads
|
|
|
|
- name: Create and migrate database
|
|
if: steps.app-cache.outputs.cache-hit != 'true'
|
|
run: |
|
|
bin/rake db:create
|
|
script/silence_successful_output bin/rake db:migrate
|
|
|
|
- name: Create and migrate parallel databases
|
|
if: >-
|
|
env.USES_PARALLEL_DATABASES == 'true' &&
|
|
steps.app-cache.outputs.cache-hit != 'true'
|
|
run: |
|
|
bin/rake parallel:create
|
|
script/silence_successful_output bin/rake parallel:migrate
|
|
|
|
- name: Dump database for cache
|
|
if: steps.app-cache.outputs.cache-hit != 'true'
|
|
run: mkdir -p tmp/app-cache && pg_dumpall > tmp/app-cache/cache.sql
|
|
|
|
- name: Dump uploads for cache
|
|
if: steps.app-cache.outputs.cache-hit != 'true'
|
|
run: rm -rf tmp/app-cache/uploads && cp -r public/uploads tmp/app-cache/uploads
|
|
|
|
- name: Fetch turbo_rspec_runtime.log cache
|
|
uses: actions/cache@v3
|
|
id: test-runtime-cache
|
|
if: matrix.build_type == 'backend' || matrix.build_type == 'system'
|
|
with:
|
|
path: tmp/turbo_rspec_runtime.log
|
|
key: rspec-runtime-${{ matrix.build_type }}-${{ matrix.target }}-${{ github.run_id }}
|
|
restore-keys: rspec-runtime-${{ matrix.build_type }}-${{ matrix.target }}-
|
|
|
|
- name: Check Zeitwerk eager_load
|
|
if: matrix.build_type == 'backend'
|
|
env:
|
|
LOAD_PLUGINS: ${{ (matrix.target == 'plugins') && '1' || '0' }}
|
|
run: |
|
|
if ! bin/rails zeitwerk:check --trace; then
|
|
echo
|
|
echo "---------------------------------------------"
|
|
echo
|
|
echo "::error::'bin/rails zeitwerk:check' failed - the app will fail to boot with 'eager_load=true' (e.g. in production)."
|
|
echo "To reproduce locally, run 'bin/rails zeitwerk:check'."
|
|
echo "Alternatively, you can run your local server/tests with the 'DISCOURSE_ZEITWERK_EAGER_LOAD=1' environment variable."
|
|
echo
|
|
exit 1
|
|
fi
|
|
|
|
- name: Check Zeitwerk reloading
|
|
if: matrix.build_type == 'backend'
|
|
env:
|
|
LOAD_PLUGINS: ${{ (matrix.target == 'plugins') && '1' || '0' }}
|
|
run: |
|
|
if ! bin/rails runner 'Rails.application.reloader.reload!'; then
|
|
echo
|
|
echo "---------------------------------------------"
|
|
echo
|
|
echo "::error::Zeitwerk reload failed - the app will not be able to reload properly in development."
|
|
echo "To reproduce locally, run \`bin/rails runner 'Rails.application.reloader.reload!'\`."
|
|
echo
|
|
exit 1
|
|
fi
|
|
|
|
- name: Core RSpec
|
|
if: matrix.build_type == 'backend' && matrix.target == 'core'
|
|
run: bin/turbo_rspec --use-runtime-info --verbose --format documentation --retry-and-log-flaky-tests
|
|
|
|
- name: Plugin RSpec
|
|
if: matrix.build_type == 'backend' && matrix.target == 'plugins'
|
|
run: bin/rake plugin:turbo_spec['*','--verbose --format documentation --use-runtime-info --retry-and-log-flaky-tests']
|
|
|
|
- name: Plugin QUnit
|
|
if: matrix.build_type == 'frontend' && matrix.target == 'plugins'
|
|
run: QUNIT_WRITE_EXECUTION_FILE=1 QUNIT_PARALLEL=3 bin/rake plugin:qunit['*','1200000']
|
|
timeout-minutes: 30
|
|
|
|
- name: Theme QUnit
|
|
if: matrix.build_type == 'frontend' && matrix.target == 'themes'
|
|
run: DISCOURSE_DEV_DB=discourse_test QUNIT_PARALLEL=3 bin/rake themes:qunit_all_official
|
|
timeout-minutes: 15
|
|
|
|
- uses: actions/upload-artifact@v3
|
|
if: always() && matrix.build_type == 'frontend' && matrix.target == 'plugins'
|
|
with:
|
|
name: ember-exam-execution-plugins-frontend
|
|
path: ./app/assets/javascripts/discourse/test-execution-*.json
|
|
|
|
- name: Ember Build for System Tests
|
|
if: matrix.build_type == 'system'
|
|
run: bin/ember-cli --build
|
|
|
|
- name: Ensure latest minio binary installed for Core System Tests
|
|
if: matrix.build_type == 'system' && matrix.target == 'core'
|
|
run: bundle exec ruby script/install_minio_binaries.rb
|
|
|
|
- name: Core System Tests
|
|
if: matrix.build_type == 'system' && matrix.target == 'core'
|
|
run: RAILS_ENABLE_TEST_LOG=1 RAILS_TEST_LOG_LEVEL=error PARALLEL_TEST_PROCESSORS=4 bin/turbo_rspec --use-runtime-info --profile=50 --verbose --format documentation --retry-and-log-flaky-tests spec/system
|
|
|
|
- name: Plugin System Tests
|
|
if: matrix.build_type == 'system' && matrix.target == 'plugins'
|
|
run: |
|
|
GLOBIGNORE="plugins/chat/*";
|
|
LOAD_PLUGINS=1 RAILS_ENABLE_TEST_LOG=1 RAILS_TEST_LOG_LEVEL=error PARALLEL_TEST_PROCESSORS=4 bin/turbo_rspec --use-runtime-info --profile=50 --verbose --format documentation --retry-and-log-flaky-tests plugins/*/spec/system
|
|
shell: bash
|
|
timeout-minutes: 30
|
|
|
|
- name: Chat System Tests
|
|
if: matrix.build_type == 'system' && matrix.target == 'chat'
|
|
run: LOAD_PLUGINS=1 RAILS_ENABLE_TEST_LOG=1 RAILS_TEST_LOG_LEVEL=error PARALLEL_TEST_PROCESSORS=4 bin/turbo_rspec --use-runtime-info --profile=50 --verbose --format documentation --retry-and-log-flaky-tests plugins/chat/spec/system
|
|
timeout-minutes: 30
|
|
|
|
- name: Theme System Tests
|
|
if: matrix.build_type == 'system' && matrix.target == 'themes'
|
|
run: |
|
|
RAILS_ENABLE_TEST_LOG=1 RAILS_TEST_LOG_LEVEL=error PARALLEL_TEST_PROCESSORS=4 bin/turbo_rspec --profile=50 --verbose --format documentation --retry-and-log-flaky-tests tmp/themes/*/spec/system
|
|
shell: bash
|
|
timeout-minutes: 30
|
|
|
|
- name: Check for failed system test screenshots
|
|
id: check-failed-system-test-screenshots
|
|
if: always() && matrix.build_type == 'system'
|
|
run: |
|
|
if [ -d tmp/capybara ] && [ "$(ls -A tmp/capybara/)" ]; then
|
|
echo "exists=true" >> $GITHUB_OUTPUT
|
|
else
|
|
echo "exists=false" >> $GITHUB_OUTPUT
|
|
fi
|
|
shell: bash
|
|
|
|
- name: Upload failed system test screenshots
|
|
uses: actions/upload-artifact@v3
|
|
if: always() && steps.check-failed-system-test-screenshots.outputs.exists == 'true'
|
|
with:
|
|
name: failed-system-test-screenshots
|
|
path: tmp/capybara/*.png
|
|
|
|
- name: Check for flaky tests report
|
|
id: check-flaky-spec-report
|
|
if: always()
|
|
run: |
|
|
if [ -f tmp/turbo_rspec_flaky_tests.json ]; then
|
|
echo "exists=true" >> $GITHUB_OUTPUT
|
|
else
|
|
echo "exists=false" >> $GITHUB_OUTPUT
|
|
fi
|
|
shell: bash
|
|
|
|
- name: Create flaky tests report artifact
|
|
if: always() && steps.check-flaky-spec-report.outputs.exists == 'true'
|
|
run: cp tmp/turbo_rspec_flaky_tests.json tmp/turbo_rspec_flaky_tests-${{ matrix.build_type }}-${{ matrix.target }}.json
|
|
|
|
- name: Upload flaky tests report
|
|
uses: actions/upload-artifact@v3
|
|
if: always() && steps.check-flaky-spec-report.outputs.exists == 'true'
|
|
with:
|
|
name: flaky-test-reports
|
|
path: tmp/turbo_rspec_flaky_tests-${{ matrix.build_type }}-${{ matrix.target }}.json
|
|
|
|
- name: Check Annotations
|
|
if: matrix.build_type == 'annotations'
|
|
run: |
|
|
bin/rake annotate:ensure_all_indexes
|
|
bin/annotate --models --model-dir app/models
|
|
|
|
if [ ! -z "$(git status --porcelain app/models/)" ]; then
|
|
echo "Core annotations are not up to date. To resolve, run:"
|
|
echo " bin/rake annotate:clean"
|
|
echo
|
|
echo "Or manually apply the diff printed below:"
|
|
echo "---------------------------------------------"
|
|
git -c color.ui=always diff app/models/
|
|
exit 1
|
|
fi
|
|
timeout-minutes: 30
|
|
|
|
core_frontend_tests:
|
|
if: github.event_name == 'pull_request' || github.repository != 'discourse/discourse-private-mirror'
|
|
name: core frontend (${{ matrix.browser }})${{ matrix.updated_ember && ' (Ember 5)' || '' }}
|
|
runs-on: ubuntu-20.04-8core
|
|
container:
|
|
image: discourse/discourse_test:slim-browsers
|
|
options: --user discourse
|
|
|
|
timeout-minutes: 35
|
|
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
browser: ["Chrome", "Firefox ESR", "Firefox Evergreen"]
|
|
updated_ember: [false]
|
|
include:
|
|
- browser: Chrome
|
|
updated_ember: true
|
|
|
|
env:
|
|
TESTEM_BROWSER: ${{ (startsWith(matrix.browser, 'Firefox') && 'Firefox') || matrix.browser }}
|
|
TESTEM_FIREFOX_PATH: ${{ (matrix.browser == 'Firefox Evergreen') && '/opt/firefox-evergreen/firefox' }}
|
|
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
fetch-depth: 1
|
|
|
|
- name: Setup Git
|
|
run: |
|
|
git config --global user.email "ci@ci.invalid"
|
|
git config --global user.name "Discourse CI"
|
|
|
|
- name: Get yarn cache directory
|
|
id: yarn-cache-dir
|
|
run: echo "dir=$(yarn cache dir)" >> $GITHUB_OUTPUT
|
|
|
|
- name: Yarn cache
|
|
uses: actions/cache@v3
|
|
id: yarn-cache
|
|
with:
|
|
path: ${{ steps.yarn-cache-dir.outputs.dir }}
|
|
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}-cachev2
|
|
|
|
- name: Upgrade Ember
|
|
if: matrix.updated_ember == true
|
|
run: script/switch_ember_version 5
|
|
|
|
- name: Yarn install
|
|
working-directory: ./app/assets/javascripts/discourse
|
|
run: yarn install --frozen-lockfile
|
|
|
|
- name: Ember Build
|
|
working-directory: ./app/assets/javascripts/discourse
|
|
run: |
|
|
mkdir /tmp/emberbuild
|
|
yarn ember build --environment=test -o /tmp/emberbuild
|
|
|
|
- name: Core QUnit
|
|
working-directory: ./app/assets/javascripts/discourse
|
|
run: yarn ember exam --path /tmp/emberbuild --load-balance --parallel=5 --launch "${{ env.TESTEM_BROWSER }}" --write-execution-file --random
|
|
timeout-minutes: 15
|
|
|
|
- uses: actions/upload-artifact@v3
|
|
if: ${{ always() }}
|
|
with:
|
|
name: ember-exam-execution-${{ matrix.browser }}
|
|
path: ./app/assets/javascripts/discourse/test-execution-*.json
|