From cfec408bc13a0593bc6fb952a6ca9205fed7e102 Mon Sep 17 00:00:00 2001 From: Alan Guo Xiang Tan Date: Tue, 26 Dec 2023 09:23:17 +0800 Subject: [PATCH] DEV: Remove flaky tests report when there are too many failures (#25031) Why this change? Currently we only rerun failing tests to check if they are flaky tests when there are 10 or less failing tests. When there are more than 10 failing tests in the first run, we assume that the odds of those tests being flaky are low and do not rerun the tests. However, there was a bug where we do not clean up the potential flaky tests being logged when there are too many test failures. This resulted in those test failures being treated as flaky tests. What does this change do? Clean up the flaky tests report when we do not rerun the tests. --- lib/turbo_tests/flaky/manager.rb | 4 ++++ lib/turbo_tests/runner.rb | 1 + 2 files changed, 5 insertions(+) diff --git a/lib/turbo_tests/flaky/manager.rb b/lib/turbo_tests/flaky/manager.rb index d1efd4f5b9a..c71e5146dcd 100644 --- a/lib/turbo_tests/flaky/manager.rb +++ b/lib/turbo_tests/flaky/manager.rb @@ -11,6 +11,10 @@ module TurboTests .map { |failed_example| failed_example["location_rerun_argument"] } end + def self.remove_flaky_tests + File.delete(PATH) if File.exist?(PATH) + end + # This method should only be called by a formatter registered with `TurboTests::Runner` and logs the failed examples # to `PATH`. See `FailedExample#to_h` for the details of each example that is logged. # diff --git a/lib/turbo_tests/runner.rb b/lib/turbo_tests/runner.rb index d04b3b44c33..b4de43eed84 100644 --- a/lib/turbo_tests/runner.rb +++ b/lib/turbo_tests/runner.rb @@ -103,6 +103,7 @@ module TurboTests return rerun_failed_examples(@reporter.failed_examples) else STDOUT.puts "Retry and log flaky tests was enabled but ignored because there are more than #{retry_failed_examples_threshold} failures." + Flaky::Manager.remove_flaky_tests end end