discourse/lib/turbo_tests/flaky/manager.rb
Alan Guo Xiang Tan cfec408bc1
DEV: Remove flaky tests report when there are too many failures (#25031)
Why this change?

Currently we only rerun failing tests to check if they are flaky tests
when there are 10 or less failing tests. When there are more than 10
failing tests in the first run, we assume that the odds of those tests
being flaky are low and do not rerun the tests. However, there was a bug
where we do not clean up the potential flaky tests being logged when
there are too many test failures. This resulted in those test failures
being treated as flaky tests.

What does this change do?

Clean up the flaky tests report when we do not rerun the tests.
2023-12-26 09:23:17 +08:00

55 lines
1.8 KiB
Ruby

# frozen_string_literal: true
module TurboTests
module Flaky
class Manager
PATH = Rails.root.join("tmp/turbo_rspec_flaky_tests.json")
def self.potential_flaky_tests
JSON
.parse(File.read(PATH))
.map { |failed_example| failed_example["location_rerun_argument"] }
end
def self.remove_flaky_tests
File.delete(PATH) if File.exist?(PATH)
end
# This method should only be called by a formatter registered with `TurboTests::Runner` and logs the failed examples
# to `PATH`. See `FailedExample#to_h` for the details of each example that is logged.
#
# @param [Array<TurboTests::FakeExample>] failed_examples
def self.log_potential_flaky_tests(failed_examples)
return if failed_examples.empty?
File.open(PATH, "w") do |file|
file.puts(
failed_examples.map { |failed_example| FailedExample.new(failed_example).to_h }.to_json,
)
end
end
# This method should only be called by a formatter registered with `RSpec::Core::Formatters.register` and removes
# the given examples from the log file at `PATH` by matching the `location_rerun_argument` of each example.
#
# @param [Array<RSpec::Core::Example>] failed_examples
def self.remove_example(failed_examples)
flaky_tests =
JSON
.parse(File.read(PATH))
.reject do |failed_example|
failed_examples.any? do |example|
failed_example["location_rerun_argument"] == example.location_rerun_argument
end
end
if flaky_tests.present?
File.write(PATH, flaky_tests.to_json)
else
File.delete(PATH)
end
end
end
end
end