Commit Graph

91 Commits

Author SHA1 Message Date
Gerhard Schlager
6b3e28216c
FEATURE: Allow pausing of restore before DB migration and uploads are restored (#30269)
This can be helpful if you need to fix problems in the DB before the DB gets migrated as well as before uploads are restored.
2024-12-16 12:50:08 +01:00
Alan Guo Xiang Tan
44a81069ac
DEV: Avoid creating system message when system user initiates restore (#30027)
There is no point creating a message for the system user since it is a
non-human user.
2024-12-02 16:13:38 +08:00
David Taylor
6417173082
DEV: Apply syntax_tree formatting to lib/* 2023-01-09 12:10:19 +00:00
Bianca Nenciu
e2c415457c
FEATURE: Attach backup log as upload (#13849)
Discourse automatically sends a private message after backup or
restore finished. The private message used to contain the log inline
even when it was very long. A very long log can create issues because
the length of the post will be over the maximum allowed length of a
post. When that happens, Discourse will try to create an upload with
the logs. If that fails, it will trim the log and inline it.
2021-08-03 20:06:50 +03:00
Gerhard Schlager
a37b5b177b
FIX: Remapping of uploads could fail during restore of backup (#13897)
Sidekiq should be paused until the end of the restore, otherwise it might interfere (deadlock) with the remap of upload URLs in posts.
2021-07-30 11:25:28 +02:00
Gerhard Schlager
4b05fc2d2d
FIX: Restoring backup via UI was broken (#12089) 2021-02-15 16:46:44 +01:00
Gerhard Schlager
4f5ea4fbde FIX: Restoring backup could fail due to missing uploads
Clearing theme and emoji cache might require uploaded files.
2021-02-09 17:28:03 +01:00
Gerhard Schlager
4d719725c8
FEATURE: Allow overriding the backup location when restoring via CLI (#12015)
You can use `discourse restore --location=local FILENAME` if you want to restore a backup that is stored locally even though the `backup_location` has the value `s3`.
2021-02-09 16:02:44 +01:00
Gerhard Schlager
d5ef6188ed
PERF: Disable Sidekiq only during database restore (#10857)
It pauses Sidekiq, clears Redis (namespaced to the current site), clears Sidekiq jobs for the current site, restores the database and unpauses Sidekiq. Previously it stayed paused until the end of the restore.

Redis is cleared because we don't want any old data lying around (e.g. old Sidekiq jobs). Most data in Redis is prefixed with the name of the multisite, but Sidekiq jobs in a multisite are all stored in the same keys. So, deleting those jobs requires a little bit more logic.
2020-10-16 15:19:02 +02:00
Gerhard Schlager
ac70c48be4 FIX: Prevent "uploads are missing in S3" alerts after restoring a backup
After restoring a backup it takes up to 48 hours for uploads stored on S3 to appear in the S3 inventory. This change prevents alerts about missing uploads by preventing the EnsureS3UploadsExistence job from running in the first 48 hours after a restore. During the restore it  deletes the count of missing uploads from the PluginStore, so that an alert isn't triggered by an old number.
2020-09-10 21:37:48 +02:00
Gerhard Schlager
6d5e9db883 FIX: Restoring backup didn't clear cached translation overrides 2020-05-18 18:51:51 +02:00
Dan Ungureanu
3d9c320aab
PERF: Cache Category.subcategory_ids (#9350)
Also reset category cache after backup restore.
2020-04-09 15:42:24 +03:00
Gerhard Schlager
7c30986b5e FIX: Failed to notify user after restoring backup 2020-01-25 22:07:41 +01:00
Gerhard Schlager
e474cda321 REFACTOR: Restoring of backups and migration of uploads to S3 2020-01-14 11:41:35 +01:00
Vinoth Kannan
3b7f5db5ba
FIX: parallel spec system needs a dedicated upload folder for each worker. (#8547) 2019-12-18 11:21:57 +05:30
David Taylor
481efebe76
DEV: Update backup/restore pipeline to avoid cd (#8347) 2019-11-13 15:52:28 +00:00
Justin DiRose
c3f06943c7
FIX: Account for empty uploads directory upon backup restore (#8262)
This commit fixes a case where backup restores would fail if the uploads/default directory is empty.
2019-10-30 09:33:07 -05:00
Krzysztof Kotlarek
f530378df3 FIX: Restore for non-multisite is not raising an error on reconnect step (#8237)
That commit introduced a bug to the system: f69dacf979

Restore works fine for multisite, however, stopped working for non-multisite.

Reason for that was that `establish_connection` method got a check if the multisite instance is available:
```
    def self.instance
      @instance
    end

    def self.establish_connection(opts)
      @instance.establish_connection(opts) if @instance
    end
```
However, the reload method don't have that check
```
    def self.reload
      @instance = new(instance.config_filename)
    end
```

To solve it, let's ensure we are in a multisite environment before call reload
2019-10-24 11:46:22 +11:00
Krzysztof Kotlarek
f69dacf979 FIX: Reconnect in restore process connects to correct DB (#8218)
Simplified flow of restore is like that
```
migrate_database
reconnect
extract_uploads
```

Problem with incorrect current database started with this fix https://github.com/discourse/discourse/commit/025d4ee91f4

Dump task is reconnecting to default database https://github.com/rails/rails/blob/master/activerecord/lib/active_record/railties/databases.rake#L429

And then, we are trying to reconnect to the original database with that code:
```
def reconnect_database
  log "Reconnecting to the database..."
  RailsMultisite::ConnectionManagement::establish_connection(db: @current_db)
end
```

This reconnect is not switching us back to correct database because of that check
https://github.com/discourse/rails_multisite/blob/master/lib/rails_multisite/connection_management.rb#L181
Basically, it finds existing handler and it thinks that we are connected to correct DB and this step can be skipped.

To solve it, we can reload RailsMultisite::ConnectionManagement which creates a new instance of that class
https://github.com/discourse/rails_multisite/blob/master/lib/rails_multisite/connection_management.rb#L38
2019-10-23 17:23:50 +11:00
Roman Rizzi
01bc465db8
DEV: Split max decompressed setting for themes and backups (#8179) 2019-10-11 14:38:10 -03:00
Roman Rizzi
5357ab3324
SECURITY: Safely decompress backups when restoring. (#8166)
* SECURITY: Safely decompress backups when restoring.

* Fix tests and update theme_controller_spec to work with zip files instead of .tar.gz
2019-10-09 11:41:16 -03:00
Gerhard Schlager
1f118b1309 FEATURE: Allow plugins to manipulate site settings during backup restore 2019-08-22 22:41:26 +02:00
Gerhard Schlager
d686318133 FIX: Prevent failed remaps during restores
Additional changes:
* Verbose logging of remaps during restores
* Exclude the backup_metadata table from restores
2019-08-12 17:15:01 +02:00
Gerhard Schlager
7cb51d0e40 FIX: Create readonly functions during backup
Temporarily recreate already dropped functions in the discourse_functions schema in order to allow restoring of backups which still reference dropped functions.
2019-08-09 11:39:46 +02:00
Michael Brown
31f583855a DEV: pull static check out of loop
* followup to 08b28680
* as per https://review.discourse.org/t/4713/2
2019-07-23 17:18:16 -04:00
Gerhard Schlager
68b082e1a4 FIX: Ensure that jobs don't run immediately after migrate_to_s3 2019-07-23 17:42:12 +02:00
Michael Brown
08b286808a
FIX: backups taken by pg_dump >= 11 are nonportable (#7893) 2019-07-15 18:07:44 -04:00
Gerhard Schlager
5f0d38341e FIX: Remapping during restore was wrong for CDN URLs 2019-07-09 17:34:41 +02:00
Gerhard Schlager
4c1b8c7559 FIX: Remap differently when backup comes from multisite 2019-07-09 16:11:32 +02:00
Gerhard Schlager
a65a9a85d5 FEATURE: Remap uploads during restore when S3 or CDN changes
In order for this to work the Backuper stores a couple of site settings
in the new backup_metadata table, because the old setting values might
not be available on restore anymore.
2019-07-09 14:04:16 +02:00
Gerhard Schlager
f7a2648694 FEATURE: Migrate uploads to S3 during restore 2019-06-04 15:47:36 +02:00
Sam Saffron
30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Gerhard Schlager
1ddd4a44d5 FIX: Wrong color palette after backup restore 2019-05-07 17:02:57 +02:00
Gerhard Schlager
2487e01c73 FIX: Optimized site icons were missing after backup restore 2019-05-07 17:02:57 +02:00
Guo Xiang Tan
ebca588fd0 DEV: Remove unused line of code. 2019-05-02 16:54:10 +08:00
Gerhard Schlager
3aca070311 FIX: Restoring backup shouldn't change disable_emails from "yes" to "non-staff" 2019-04-16 11:48:07 +02:00
Gerhard Schlager
78f8114989 FEATURE: Allow discourse script to skip disabling of emails after restore 2019-03-07 21:49:33 +01:00
Joffrey JAFFEUX
42df20e4f0
typo (#7065) 2019-02-25 16:36:22 +01:00
Gerhard Schlager
dc961fecb9 FIX: Outgoing emails were not disabled after restoring backup 2019-02-25 16:07:24 +01:00
Gerhard Schlager
99ad61afb7 FEATURE: Trigger an event after a backup restore 2019-02-18 11:48:03 +01:00
Gerhard Schlager
b087719340 FEATURE: Setting for excluding optimized images from backups 2019-02-13 11:10:51 +01:00
Gerhard Schlager
5bb955dcb7 FIX: Allow restore when latest migration is a post_migration 2019-02-08 17:37:05 +01:00
Joffrey JAFFEUX
f9648de897
DEV: upgrades from Ember 2.13 to Ember 3.5.1 (#6808)
Co-Authored-By: Bianca Nenciu <nbianca@users.noreply.github.com>
Co-Authored-By: David Taylor <david@taylorhq.com>
2019-01-10 11:06:01 +01:00
Sam
1b4f2029d7 FIX: clear theme cache when restoring
Previously old themes may be cached incorrectly, this also forces
a rebake of old themes to ensure version can compile cleanly
2018-11-20 13:37:58 +11:00
Guo Xiang Tan
84d4c81a26 FEATURE: Support backup uploads/downloads directly to/from S3.
This reverts commit 3c59106bac.
2018-10-15 09:43:31 +08:00
Guo Xiang Tan
3c59106bac Revert "FEATURE: Support backup uploads/downloads directly to/from S3."
This reverts commit c29a4dddc1.

We're doing a beta bump soon so un-revert this after that is done.
2018-10-11 11:08:23 +08:00
Gerhard Schlager
c29a4dddc1 FEATURE: Support backup uploads/downloads directly to/from S3. 2018-10-11 10:38:43 +08:00
Gerhard Schlager
469a2c36ed FIX: Always unpause Sidekiq after backup and restore
* Logs exceptions during the cleanup phase, but doesn't stop executing subsequent cleanup tasks.
* Notifies the user at the end of the cleanup phase, so that the log contains possible errors during that phase.
2018-09-19 20:35:43 +02:00
Guo Xiang Tan
212ee15804 FIX: Create BaseDropper functions in a different schema.
https://meta.discourse.org/t/error-when-restore-db-backup/93145/25?u=tgxworld
2018-08-23 12:52:21 +08:00
Régis Hanol
8f1db615db FIX: don't break restore if function does not exist 2018-07-30 22:11:38 +02:00