Commit Graph

13 Commits

Author SHA1 Message Date
Selase Krakani
d896f5cb70
DEV: Include post and topic attributes in imported quotes (#27851)
Currently, quotes imported via generic bulk import script do not include
references to the quoted post. This change includes both topic and post attributes
in a quote if the placeholder metadata includes a `post_id`
2024-07-11 16:47:21 +00:00
Gerhard Schlager
7c26d5d084
FIX: Import script was broken after upgrade of sqlite3 gem (#27648) 2024-07-02 17:38:15 +10:00
Selase Krakani
f2c4474c1e
DEV: Improve user generic bulk importer anonymization (#27307)
* DEV: Improve user generic bulk importer anonymization

Add support for properly anonymizing:
 - email
 - date_of_birth
 - location
 - website
 - bio

* DEV: Remove uneeded anon username check in `import_user_emails`
2024-06-05 11:25:17 +00:00
Loïc Guitaut
2a28cda15c DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
Selase Krakani
949c70372c
DEV: Add support for various fields in generic bulk importer (#27114)
* user_profiles - `location`
* users - `date_of_birth`
* topics - `pinned_at`, `pinned_until`, `pinned_globally`

This also include changes to correctly import PMs. Currently PM topics
are skipped because of a check in `import_users` step which requires `category_id`
to be present.
2024-05-24 13:46:06 +02:00
Selase Krakani
61e12aaebe
FEATURE: Extend PM recipient bulk imports (#27063)
* FIX: Support multiple topic allowed user imports

* FEATURE: Add topic allowed groups import support
2024-05-17 13:45:20 +02:00
Ítalo Alves
73481e8f45
FIX: Add check for existing provider_uids to generic import (#26914)
Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
2024-05-17 11:36:31 +02:00
Gerhard Schlager
e3882a0c48
DEV: Add support for user_associated_accounts to import script (#26779) 2024-04-29 19:48:32 +02:00
Gerhard Schlager
a538e2f153
DEV: Import script should use case-insensitive check for tag names (#26699) 2024-04-29 19:27:28 +02:00
Gerhard Schlager
4d045bfc61
DEV: Import script should insert more data into user_stats table (#26551)
This SQL tries to insert as much data as possible into the `user_stats` table by either calculating or by approximating stats based on existing. It also fixes an error in the calculation of `reply_count`which mistakenly contained all posts, not just replies.

This change also disables some steps in the `import:ensure_consistency` rake task by setting the `SKIP_USER_STATS` env variable. Otherwise, the rake task will overwrite the calculated data in the `user_stats` table with inaccurate data. I'm not changing or removing the logic from the rake task yet because other bulk import scripts seem to depend on it.
2024-04-11 14:05:21 +02:00
Gerhard Schlager
bc98740205
DEV: Improve generic import script (#25972)
* FEATURE: Import into `category_users` table
* FIX: Failed to import `user_options` unless `timezone` was set
* FIX: Prevent reusing original `id` from intermediate DB in `user_fields`
* FEATURE: Order posts by `post_nuber` if available
* FEATURE: Allow `[mention]` placeholder to reference users by"id" or "name" (username)
* FEATURE: Support `[quote]` placeholders in posts
* FEATURE: Support `[link]` placeholders in posts
* FEATURE: Support all kinds of permalinks and remove support for `old_relative_url`
* PERF: Speed up pre-cooking by removing DB lookups
2024-03-05 22:23:36 +01:00
Gerhard Schlager
dc8c6b8958 DEV: Lots of improvements to the generic_bulk import script
Notable changes:
* Imports a lot more tables from core and plugins
  * site settings
  * uploads with necessary upload references
  * groups and group members
  * user profiles
  * user options
  * user fields & values
  * muted users
  * user notes (plugin)
  * user followers (plugin)
  * user avatars
  * tag groups and tags
  * tag users (notification settings for tags / user)
  * category permissions
  * polls with options and votes
  * post votes (plugin)
  * solutions (plugin)
  * gamification scores (plugin)
  * events (plugin)
  * badges and badge groupings
  * user badges
  * optimized images
  * topic users (notification settings for topics)
  * post custom fields
  * permalinks and permalink normalizations

* It creates the `migration_mappings` table which is used to store the mapping for a handful of imported tables

* Detects duplicate group names and renames them

* Pre-cooking for attachments, images and mentions

* Outputs instructions when gems are missing

* Supports importing uploads from a DB generated by `uploads_importer.rb`

* Checks that all required plugins exists and enables them if needed

* A couple of optimizations and additions in `import.rake`
2023-12-11 16:23:07 +01:00
Gerhard Schlager
0b29dc5d38 DEV: Add experimental generic bulk import script 2023-08-09 20:56:14 +02:00