discourse/bin/turbo_rspec

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

134 lines
3.4 KiB
Plaintext
Raw Normal View History

#!/usr/bin/env ruby
# frozen_string_literal: true
ENV["RAILS_ENV"] ||= "test"
require "./lib/turbo_tests"
require "optparse"
requires = []
formatters = []
verbose = false
fail_fast = nil
seed = rand(2**16)
profile = false
profile_print_slowest_examples_count = 10
use_runtime_info = nil
enable_system_tests = nil
disable_system_tests = nil
DEV: Introduce automatic reruns to RSpec tests on Github actions (#24811) What motivated this change? Our builds on Github actions have been extremely flaky mostly due to system tests. This has led to a drop in confidence in our test suite where our developers tend to assume that a failed job is due to a flaky system test. As a result, we have had occurrences where changes that resulted in legitimate test failures are merged into the `main` branch because developers assumed it was a flaky test. What does this change do? This change seeks to reduce the flakiness of our builds on Github Actions by automatically re-running RSpec tests once when they fail. If a failed test passes subsequently in the re-run, we mark the test as flaky by logging it into a file on disk which is then uploaded as an artifact of the Github workflow run. We understand that automatically re-runs will lead to lower accuracy of our tests but we accept this as an acceptable trade-off since a fragile build has a much greater impact on our developers' time. Internally, the Discourse development team will be running a service to fetch the flaky tests which have been logged for internal monitoring. How is the change implemented? 1. A `--retry-and-log-flaky-tests` CLI flag is added to the `bin/turbo_rspec` CLI which will then initialize `TurboTests::Runner` with the `retry_and_log_flaky_tests` kwarg set to `true`. 2. When the `retry_and_log_flaky_tests` kwarg is set to `true` for `TurboTests::Runner`, we will register an additional formatter `Flaky::FailuresLoggerFormatter` to the `TurboTests::Reporter` in the `TurboTests::Runner#run` method. The `Flaky::FailuresLoggerFormatter` has a simple job of logging all failed examples to a file on disk when running all the tests. The details of the failed example which are logged can be found in `TurboTests::Flaky::FailedExample.to_h`. 3. Once all the tests have been run once, we check the result for any failed examples and if there are, we read the file on disk to fetch the `location_rerun_location` of the failed examples which is then used to run the tests in a new RSpec process. In the rerun, we configure a `TurboTests::Flaky::FlakyDetectorFormatter` with RSpec which removes all failed examples from the log file on disk since those examples are not flaky tests. Note that if there are too many failed examples on the first run, we will deem the failures to likely not be due to flaky tests and not re-run the test failures. As of writing, the threshold of failed examples is set to 10. If there are more than 10 failed examples, we will not re-run the failures.
2023-12-13 07:18:27 +08:00
retry_and_log_flaky_tests = false
OptionParser
.new do |opts|
opts.on("-r", "--require PATH", "Require a file.") { |filename| requires << filename }
opts.on("-f", "--format FORMATTER", "Choose a formatter.") do |name|
formatters << { name: name, outputs: [] }
end
opts.on("-o", "--out FILE", "Write output to a file instead of $stdout") do |filename|
formatters << { name: "progress", outputs: [] } if formatters.empty?
formatters.last[:outputs] << filename
end
opts.on("-v", "--verbose", "More output") { verbose = true }
opts.on(
"-p",
"--profile=[COUNT]",
"Benchmark the runtime of each example and list the slowest examples (default: 10)",
) do |count|
profile = true
profile_print_slowest_examples_count = count.to_i if count
end
opts.on("--fail-fast=[N]") do |n|
n =
begin
Integer(n)
rescue StandardError
nil
end
fail_fast = (n.nil? || n < 1) ? 1 : n
end
DEV: Introduce automatic reruns to RSpec tests on Github actions (#24811) What motivated this change? Our builds on Github actions have been extremely flaky mostly due to system tests. This has led to a drop in confidence in our test suite where our developers tend to assume that a failed job is due to a flaky system test. As a result, we have had occurrences where changes that resulted in legitimate test failures are merged into the `main` branch because developers assumed it was a flaky test. What does this change do? This change seeks to reduce the flakiness of our builds on Github Actions by automatically re-running RSpec tests once when they fail. If a failed test passes subsequently in the re-run, we mark the test as flaky by logging it into a file on disk which is then uploaded as an artifact of the Github workflow run. We understand that automatically re-runs will lead to lower accuracy of our tests but we accept this as an acceptable trade-off since a fragile build has a much greater impact on our developers' time. Internally, the Discourse development team will be running a service to fetch the flaky tests which have been logged for internal monitoring. How is the change implemented? 1. A `--retry-and-log-flaky-tests` CLI flag is added to the `bin/turbo_rspec` CLI which will then initialize `TurboTests::Runner` with the `retry_and_log_flaky_tests` kwarg set to `true`. 2. When the `retry_and_log_flaky_tests` kwarg is set to `true` for `TurboTests::Runner`, we will register an additional formatter `Flaky::FailuresLoggerFormatter` to the `TurboTests::Reporter` in the `TurboTests::Runner#run` method. The `Flaky::FailuresLoggerFormatter` has a simple job of logging all failed examples to a file on disk when running all the tests. The details of the failed example which are logged can be found in `TurboTests::Flaky::FailedExample.to_h`. 3. Once all the tests have been run once, we check the result for any failed examples and if there are, we read the file on disk to fetch the `location_rerun_location` of the failed examples which is then used to run the tests in a new RSpec process. In the rerun, we configure a `TurboTests::Flaky::FlakyDetectorFormatter` with RSpec which removes all failed examples from the log file on disk since those examples are not flaky tests. Note that if there are too many failed examples on the first run, we will deem the failures to likely not be due to flaky tests and not re-run the test failures. As of writing, the threshold of failed examples is set to 10. If there are more than 10 failed examples, we will not re-run the failures.
2023-12-13 07:18:27 +08:00
opts.on(
"--retry-and-log-flaky-tests",
"Retry failed tests and log if test is deemed to be flaky",
) { retry_and_log_flaky_tests = true }
opts.on("--seed SEED", "The seed for the random order") { |s| seed = s.to_i }
opts.on("--use-runtime-info", "Use runtime info for tests group splitting") do
use_runtime_info = true
end
opts.on("--enable-system-tests", "Run the system tests (defaults false)") do
enable_system_tests = true
end
opts.on("--disable-system-tests", "Don't run the system tests (defaults true)") do
disable_system_tests = true
end
end
.parse!(ARGV)
if enable_system_tests && disable_system_tests
STDERR.puts "Error: I'm not sure how to enable and disable system tests simultaneously"
exit 1
end
# OptionParser modifies ARGV, leaving the leftover arguments in ARGV
if ARGV.empty?
if !enable_system_tests && !disable_system_tests
STDERR.puts "Not running system tests since it wasn't explicitly specified"
end
run_system_tests = !!enable_system_tests
files =
if run_system_tests
["#{Rails.root}/spec"]
else
TurboTests::Runner.default_spec_folders
end
use_runtime_info = true if use_runtime_info.nil?
else
if enable_system_tests || disable_system_tests
STDERR.puts "Ignoring system test options since files were specified"
end
files = ARGV
use_runtime_info = false if use_runtime_info.nil?
end
requires.each { |f| require(f) }
formatters << { name: "progress", outputs: [] } if formatters.empty?
formatters.each { |formatter| formatter[:outputs] << "-" if formatter[:outputs].empty? }
puts "::group::Run turbo_rspec" if ENV["GITHUB_ACTIONS"]
puts "Running turbo_rspec (seed: #{seed}) using files in: #{files}"
puts "::endgroup::" if ENV["GITHUB_ACTIONS"]
2019-07-09 15:51:23 +08:00
success =
TurboTests::Runner.run(
formatters: formatters,
DEV: Minimal first pass of rails system test setup (#16311) This commit introduces rails system tests run with chromedriver, selenium, and headless chrome to our testing toolbox. We use the `webdrivers` gem and `selenium-webdriver` which is what the latest Rails uses so the tests run locally and in CI out of the box. You can use `SELENIUM_VERBOSE_DRIVER_LOGS=1` to show extra verbose logs of what selenium is doing to communicate with the system tests. By default JS logs are verbose so errors from JS are shown when running system tests, you can disable this with `SELENIUM_DISABLE_VERBOSE_JS_LOGS=1` You can use `SELENIUM_HEADLESS=0` to run the system tests inside a chrome browser instead of headless, which can be useful to debug things and see what the spec sees. See note above about `bin/ember-cli` to avoid surprises. I have modified `bin/turbo_rspec` to exclude `spec/system` by default, support for parallel system specs is a little shaky right now and we don't want them slowing down the turbo by default either. ### PageObjects and System Tests To make querying and inspecting parts of the page easier and more reusable inbetween system tests, we are using the concept of [PageObjects](https://www.selenium.dev/documentation/test_practices/encouraged/page_object_models/) in our system tests. A "Page" here is generally corresponds to an overarching ember route, e.g. "Topic" for `/t/324345/some-topic`, and this contains logic for querying components within the topic such as "Posts". I have also split "Modals" into their own entity. Further down the line we may want to explore creating independent "Component" contexts. Capybara DSL should be included in each PageObject class, reference for this can be found at https://rubydoc.info/github/teamcapybara/capybara/master#the-dsl For system tests, since they are so slow, we want to focus on the "happy path" and not do every different possible context and branch check using them. They are meant to be overarching tests that check a number of things are correct using the full stack from JS and ember to rails to ruby and then the database. ### CI Setup Whenever a system spec fails, a screenshot is taken and a build artifact is produced _after the entire CI run is complete_, which can be downloaded from the Actions UI in the repo. Most importantly, a step to build the Ember app using Ember CLI is needed, otherwise the JS assets cannot be found by capybara: ``` - name: Build Ember CLI run: bin/ember-cli --build ``` A new `--build` argument has been added to `bin/ember-cli` for this case, which is not needed locally if you already have the discourse rails server running via `bin/ember-cli -u` since the whole server is built and set up by default. Co-authored-by: David Taylor <david@taylorhq.com>
2022-09-28 09:48:16 +08:00
files: files,
verbose: verbose,
fail_fast: fail_fast,
use_runtime_info: use_runtime_info,
seed: seed.to_s,
profile:,
profile_print_slowest_examples_count:,
DEV: Introduce automatic reruns to RSpec tests on Github actions (#24811) What motivated this change? Our builds on Github actions have been extremely flaky mostly due to system tests. This has led to a drop in confidence in our test suite where our developers tend to assume that a failed job is due to a flaky system test. As a result, we have had occurrences where changes that resulted in legitimate test failures are merged into the `main` branch because developers assumed it was a flaky test. What does this change do? This change seeks to reduce the flakiness of our builds on Github Actions by automatically re-running RSpec tests once when they fail. If a failed test passes subsequently in the re-run, we mark the test as flaky by logging it into a file on disk which is then uploaded as an artifact of the Github workflow run. We understand that automatically re-runs will lead to lower accuracy of our tests but we accept this as an acceptable trade-off since a fragile build has a much greater impact on our developers' time. Internally, the Discourse development team will be running a service to fetch the flaky tests which have been logged for internal monitoring. How is the change implemented? 1. A `--retry-and-log-flaky-tests` CLI flag is added to the `bin/turbo_rspec` CLI which will then initialize `TurboTests::Runner` with the `retry_and_log_flaky_tests` kwarg set to `true`. 2. When the `retry_and_log_flaky_tests` kwarg is set to `true` for `TurboTests::Runner`, we will register an additional formatter `Flaky::FailuresLoggerFormatter` to the `TurboTests::Reporter` in the `TurboTests::Runner#run` method. The `Flaky::FailuresLoggerFormatter` has a simple job of logging all failed examples to a file on disk when running all the tests. The details of the failed example which are logged can be found in `TurboTests::Flaky::FailedExample.to_h`. 3. Once all the tests have been run once, we check the result for any failed examples and if there are, we read the file on disk to fetch the `location_rerun_location` of the failed examples which is then used to run the tests in a new RSpec process. In the rerun, we configure a `TurboTests::Flaky::FlakyDetectorFormatter` with RSpec which removes all failed examples from the log file on disk since those examples are not flaky tests. Note that if there are too many failed examples on the first run, we will deem the failures to likely not be due to flaky tests and not re-run the test failures. As of writing, the threshold of failed examples is set to 10. If there are more than 10 failed examples, we will not re-run the failures.
2023-12-13 07:18:27 +08:00
retry_and_log_flaky_tests:,
2019-07-09 15:51:23 +08:00
)
if success
exit 0
else
exit 1
end