PERF: Switch plugin:install_all_official to clone plugins concurrently (#24511)

Why this change?

`plugin:install_all_official` is quite slow at the moment taking roughly
1 minute and 51 seconds on my machine. Since most of the time is spent
waiting on the network, we can actually speed up the Rake task
significantly by executing the cloning concurrently. With a 8 cores
machine, cloning all plugins will only take 15 seconds.

What does this change do?

This change wraps the `git clone` operation in the
`plugin:install_all_official` Rake task in a `Concurrent::Promise` which
basically runs the `git clone` operation in a Thread. The `--quiet`
option has also been added to `git clone` since running stuff
concurrently messes up the output. That could be fixed but it has been
determined to be not worth it since the output from `git clone` is
meaningless to us.
This commit is contained in:
Alan Guo Xiang Tan 2023-11-22 21:43:42 +08:00 committed by GitHub
parent 17033d46c3
commit d0117ff6e3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -8,14 +8,18 @@ task "plugin:install_all_official" do
STDERR.puts "Allowing write to all repos!" if ENV["GIT_WRITE"]
promises = []
failures = []
Plugin::Metadata::OFFICIAL_PLUGINS.each do |name|
next if skip.include? name
repo = "https://github.com/discourse/#{name}"
dir = repo.split("/").last
path = File.expand_path("plugins/" + dir)
if Dir.exist? path
STDERR.puts "Skipping #{dir} cause it already exists!"
STDOUT.puts "Skipping #{dir} cause it already exists!"
next
end
@ -24,16 +28,24 @@ task "plugin:install_all_official" do
repo += ".git"
end
attempts = 0
begin
attempts += 1
system("git clone #{repo} #{path}", exception: true)
promises << Concurrent::Promise.execute do
attempts ||= 1
STDOUT.puts("Cloning '#{repo}' to '#{path}'...")
system("git clone --quiet #{repo} #{path}", exception: true)
rescue StandardError
abort("Failed to clone #{repo}") if attempts >= 3
STDERR.puts "Failed to clone #{repo}... trying again..."
if attempts == 3
failures << repo
abort
end
STDOUT.puts "Failed to clone #{repo}... trying again..."
attempts += 1
retry
end
end
Concurrent::Promise.zip(*promises).value!
failures.each { |repo| STDOUT.puts "Failed to clone #{repo}" } if failures.present?
end
desc "install plugin"