Commit Graph

76 Commits

Author SHA1 Message Date
Justin DiRose
c1517e428e
DEV: Add vBulletin5 bulk importer (#12904)
This is a pretty straightforward bulk importer, just tailored to the vBulletin 5 database structure.

Also made a few minor improvements to the base importer -- should be self explanatory in the code.
2021-04-30 11:03:33 -05:00
Michael Maroszek
144584aacb
fix vbulletin importer to hide soft-deleted posts (#12057)
equal to theads posts can be soft-deleted which results in a visibile = 2 state. at the moment those posts will be imported fully visible.
2021-02-12 14:29:05 +01:00
Justin DiRose
f6e87e1e5e
DEV: Improvements to Discourse Merger script (#11660)
After running the Discourse merge script, it was pretty evident it held up well after all these years ;)

Made a few fixes:

    Included an environment variable for DB_PASS as likely the password will need to be changed if running the import in an official Docker container (recommended)
    Set a hard order for imported categories, otherwise sometimes they'd be imported in a weird order making things unpredictable for parent/child category imports
    Fixed a couple of instances where we added unique indexes (such as on category slugs)
    Set up upload regex to handle AWS URLs better
    Fixed the script to work with frozen string literals
2021-01-08 09:31:39 -06:00
Régis Hanol
a85d5edbf1
DEV: set digest_attempted_at during migrations (#11369) 2020-12-14 10:58:14 +11:00
David Taylor
cf21de0e7a
DEV: Migrate Github authentication to ManagedAuthenticator (#11170)
This commit adds an additional find_user_by_email hook to ManagedAuthenticator so that GitHub login can continue to support secondary email addresses

The github_user_infos table will be dropped in a follow-up commit.

This is the last core authenticator to be migrated to ManagedAuthenticator 🎉
2020-11-10 10:09:15 +00:00
Krzysztof Kotlarek
93ff54e184
FIX: improvements for vanilla bulk import (#10212)
Adjustments to the base:
1. PG connection doesn't require host - it was broken on import droplet
2. Drop `topic_reply_count` - it was removed here - https://github.com/discourse/discourse/blob/master/db/post_migrate/20200513185052_drop_topic_reply_count.rb
3. Error with `backtrace.join("\n")` -> `e.backtrace.join("\n")`
4. Correctly link the user and avatar to quote block

Adjustments to vanilla:
1. Top-level Vanilla categories are valid categories
2. Posts have `format` column which should be used to decide if the format is HTML or Markdown
3. Remove no UTF8 characters
4. Remove not supported HTML elements like `font` `span` `sub` `u`
2020-07-14 15:58:27 +10:00
Régis Hanol
47a1157458 DEV: various bugfixes in bulk importer 2020-06-19 17:53:06 +02:00
Régis Hanol
5143309014 DEV: ensure values are converted to integers in bulk importer 2020-06-18 17:42:14 +02:00
Régis Hanol
823b940b9d PERF: improve loading of indexes in bulk import
Similar strategy as for c52191d in which we stream the results from the database into
an automatically growing array instead of using a hash.
2020-06-18 16:32:27 +02:00
Régis Hanol
c52191d49e PERF: improve loading a imported_ids in bulk imports
- Stream the queries that load the imported_ids
- Use an array instead of a hash for keeping the mapping between imported_ids and new ids
- Ensure we always treat the imported_ids as integers instead of strings
2020-06-16 19:55:08 +02:00
Sam Saffron
d0d5a138c3
DEV: stop freezing frozen strings
We have the `# frozen_string_literal: true` comment on all our
files. This means all string literals are frozen. There is no need
to call #freeze on any literals.

For files with `# frozen_string_literal: true`

```
puts %w{a b}[0].frozen?
=> true

puts "hi".frozen?
=> true

puts "a #{1} b".frozen?
=> true

puts ("a " + "b").frozen?
=> false

puts (-("a " + "b")).frozen?
=> true
```

For more details see: https://samsaffron.com/archive/2018/02/16/reducing-string-duplication-in-ruby
2020-04-30 16:48:53 +10:00
David Taylor
5919618a87
DEV: Drop legacy OpenID 2.0 support (#8894)
This is not used in core or official plugins, and has been printing a deprecation notice since v2.3.0beta4. All OpenID 2.0 code and dependencies have been dropped. The user_open_ids table remains for now, in case anyone has missed the deprecation notice, and needs to migrate their data.

Context at https://meta.discourse.org/t/-/113249
2020-02-07 17:32:35 +00:00
Gerhard Schlager
ab07b945c2
Merge pull request #8736 from gschlager/rename_reply_id_column
REFACTOR: Rename `post_replies.reply_id` column to `post_replies.reply_post_id`
2020-01-17 17:24:49 +01:00
Sam Saffron
0c52537f10 DEV: update rubocop to version 0.77
We like to stay as close as possible to latest with rubocop cause the cops
get better.

This update required some code changes, specifically the default is to avoid
explicit returns where implicit is done

Also this renames a few rules
2019-12-10 11:48:39 +11:00
Penar Musaraj
067696df8f DEV: Apply Rubocop redundant return style 2019-11-14 15:10:51 -05:00
Gerhard Schlager
b788948985 FEATURE: English locale with international date formats
Makes en_US the new default locale
2019-05-20 13:47:20 +02:00
Sam Saffron
30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Arpit Jalan
110512d4d0 Improvements to vBulletin bulk import script
- import attachments
- import avatars
- import user signatures
- create permalink file
- reconnect to MySQL db in case of failure
2019-04-11 12:35:19 +05:30
Arpit Jalan
a20f58554b IMPORT: create category definitions in import:ensure_consistency task 2019-04-11 12:06:37 +05:30
David Taylor
fc7938f7e0
REFACTOR: Migrate GoogleOAuth2Authenticator to use ManagedAuthenticator (#7120)
https://meta.discourse.org/t/future-social-authentication-improvements/94691/3
2019-03-07 11:31:04 +00:00
Joffrey JAFFEUX
703c724cf3
REFACTOR: Migrate InstagramAuthenticator to use ManagedAuthenticator (#7081) 2019-03-04 14:54:28 +01:00
Bianca Nenciu
714f6cde79 FIX: Remove duplicate definition of create_categories. 2019-03-04 10:32:09 +02:00
Arpit Jalan
71a5369fef FIX: do not convert quote tags to markdown 2018-12-11 20:09:46 +05:30
Arpit Jalan
735a48415d FEATURE: option to use ruby-bbcode-to-md in bulk import script
ruby-bbcode-to-md provides better bbcode to markdown conversion
2018-12-10 10:28:07 +05:30
Arpit Jalan
0365d50797 Improve vBulletin bulk import script to support table prefix.
Improve base bulk import script to convert list tags to ul/li.
2018-12-10 10:10:44 +05:30
David Taylor
160d29b18a
REFACTOR: Migrate TwitterAuthenticator to use ManagedAuthenticator (#6739)
No changes to functionality. TwitterAuthenticator goes from 136 lines to 24, and all twitter-specific logic elsewhere has been deleted 🎉
2018-12-07 15:39:06 +00:00
Régis Hanol
3c9c95ac83 Update Rubocop to 0.60 2018-12-04 10:48:16 +01:00
David Taylor
9248ad1905 DEV: Enable Style/SingleLineMethods and Style/Semicolon in Rubocop (#6717) 2018-12-04 11:48:13 +08:00
David Taylor
208005f9c9 REFACTOR: Migrate FacebookAuthenticator to use ManagedAuthenticator
Changes to functionality
  - Removed syncing of user metadata including gender, location etc.
    These are no longer available to standard Facebook applications.
  - Removed the remote 'revoke' functionality. No other providers have
    it, and it does not appear to be standard practice in other apps.
  - The 'facebook_no_email' event is no longer logged. The system can
    cope fine with a missing email address.

Data is migrated to the new user_associated_accounts table.
facebook_user_infos can be dropped once we are confident the data has
been migrated successfully.
2018-11-30 11:18:11 +00:00
Régis Hanol
a0f0bac752
Add a comment to run the 'import:ensure_consistency' rake task after a bulk import 2018-11-21 16:28:35 +01:00
David Taylor
9bf522f227
FEATURE: Mixed case tagging (#6454)
- By default, behaviour is not changed: tags are made lowercase upon creation and edit.

- If force_lowercase_tags is disabled, then mixed case tags are allowed.

- Tags must remain case-insensitively unique. This is enforced by ActiveRecord and Postgres.

- A migration is added to provide a `UNIQUE` index on `lower(name)`. Migration includes a safety to correct any current tags that do not meet the criteria.

- A `where_name` scope is added to `models/tag.rb`, to allow easy case-insensitive lookups. This is used instead of `Tag.where(name: "blah")`.

- URLs remain lowercase. Mixed case URLs are functional, but have the lowercase equivalent as the canonical.
2018-10-05 10:23:52 +01:00
Neil Lalonde
8af6d81891 FIX: improved category merging in discourse_merger. Use full paths to look for uniqueness instead of category names. 2018-09-20 12:33:58 -04:00
Neil Lalonde
b9891c2641 FIX: error because last_id is nil in discourse_merger script 2018-09-17 09:57:11 -04:00
Arpit Jalan
7af0da9498 Fix Vanilla bulk import script 2018-08-16 22:12:26 +05:30
Arpit Jalan
0e04e3990e Improve Vanilla bulk import script 2018-08-16 22:00:26 +05:30
Neil Lalonde
f7f24a5399 FIX: discourse_merger: skip collisions on join models when both objects were merged 2018-08-02 16:05:55 -04:00
Mohammad AlTawil
64f533db99 Add display name to user (#6198) 2018-07-31 14:43:16 +10:00
Neil Lalonde
bf7ebecb76 FIX: discourse_merger: many foreign keys were not being updated 2018-07-22 22:05:07 -04:00
Neil Lalonde
4e09206061 FIX: set uploads sequence after copying uplaods in discourse_merger 2018-07-19 11:07:15 -04:00
Neil Lalonde
def2653fc8 FIX: discourse_merger: copied topic_link records had wrong url, and update all internal links to use new topic URLs in copied posts 2018-07-18 16:45:48 -04:00
Neil Lalonde
24da2940a7 FIX: copy uploads quickly in discourse_merger.rb, and fix user avatar upload id for copied users 2018-07-18 16:42:59 -04:00
Neil Lalonde
dbfa491ee2 FIX: avatars in discourse_merger.rb 2018-07-17 21:40:24 -04:00
Neil Lalonde
f146f94ef6 FIX: errors when copying post_uploads in discourse_merger.rb 2018-07-17 16:47:23 -04:00
Neil Lalonde
04077a7df6 WIP: a fast method of copying uploads in discourse_merger.rb. not working yet. 2018-07-17 16:46:32 -04:00
Neil Lalonde
2786c79354 another check to avoid unique index error in discourse_merger.rb 2018-07-16 13:34:41 -04:00
Neil Lalonde
8d11df6971 FIX: support amazon S3 upload urls in discourse_merger.rb 2018-07-13 16:10:31 -04:00
Neil Lalonde
71814009bd FIX: badges for merged users don't get merged by discourse_merger.rb 2018-07-12 17:43:21 -04:00
Neil Lalonde
cba292cb56 FIX: personal messages not being copied by discourse_merger.rb 2018-07-12 17:41:16 -04:00
Neil Lalonde
c33ee13c4c FIX: discourse_merger halts when topic has nil category 2018-06-29 12:21:25 -04:00
Neil Lalonde
a1c0d0e6e5 fixes to discourse_merger: failures for Uploads, UserBadges, PostUploads hack 2018-06-21 12:16:05 -04:00