19 Commits

Author SHA1 Message Date
Ruben Oussoren
20a16ea231
Adding Import Support for Discourse Reactions Plugin ()
* Adding Import Support for Discourse Reactions Plugin

---------

Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
2025-01-07 08:58:26 -05:00
Ruben Oussoren
f81bf31993
Fix the import votes for posts table to the new table () 2024-11-19 10:53:33 -05:00
Ruben Oussoren
4b8b7b2fb5
DEV: Add import support for Topic Voting plugin ()
DEV: Add import support for Topic Voting plugin
- Implemented bulk import functionality specifically for topic_voting_votes.
- Updated the method name from import_votes to import_post_voting_votes for improved clarity and maintainability.
- Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
2024-10-31 08:30:28 -04:00
Selase Krakani
dd34f1927b
FIX: Imports of upload-only chat messages ()
The current implementation adds a "note" for chat messages with empty
messages, however chat messages with only uploads  are allowed. This change
allows such messages to be imported.
2024-10-10 15:18:10 +00:00
Selase Krakani
9825bde811
DEV: Generic bulk chat import support ()
* DEV: WIP generic bulk chat import support

This first iteration implements bulk import for:

* direct_messages
* chat_channels
* user_chat_channel_memberships
* chat_threads
* user_chat_thread_memberships
* chat_messages
* chat_reactions
* chat_mentions

* DEV: Refactor raw placeholder interpolation to support chat messages

This change adds support for chat message placeholder interpolation
and switches to using `Chat::Message.cook` for cooking in the interim
instead of hand-cooking chat messages like we currently do for posts

* DEV: Extend upload references import to support chat message uploads

* DEV: Explicity set chat retention

- Set both channel and dm chats to 0
- Add temporary workaround for testing only chat imports

* DEV: Compute channel and thread membership metadata

Compute and set various user channel/thread membership stats and
remove hardcoded test index seed data

* FIX: Fix chat reactions import

Allow multiple reactions on a message by a user
2024-10-08 11:55:30 +00:00
Gerhard Schlager
d4379af7f2
FIX: Import script didn't set public attribute of polls () 2024-10-02 20:02:13 +02:00
Selase Krakani
d896f5cb70
DEV: Include post and topic attributes in imported quotes ()
Currently, quotes imported via generic bulk import script do not include
references to the quoted post. This change includes both topic and post attributes
in a quote if the placeholder metadata includes a `post_id`
2024-07-11 16:47:21 +00:00
Gerhard Schlager
7c26d5d084
FIX: Import script was broken after upgrade of sqlite3 gem () 2024-07-02 17:38:15 +10:00
Selase Krakani
f2c4474c1e
DEV: Improve user generic bulk importer anonymization ()
* DEV: Improve user generic bulk importer anonymization

Add support for properly anonymizing:
 - email
 - date_of_birth
 - location
 - website
 - bio

* DEV: Remove uneeded anon username check in `import_user_emails`
2024-06-05 11:25:17 +00:00
Loïc Guitaut
2a28cda15c DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
Selase Krakani
949c70372c
DEV: Add support for various fields in generic bulk importer ()
* user_profiles - `location`
* users - `date_of_birth`
* topics - `pinned_at`, `pinned_until`, `pinned_globally`

This also include changes to correctly import PMs. Currently PM topics
are skipped because of a check in `import_users` step which requires `category_id`
to be present.
2024-05-24 13:46:06 +02:00
Selase Krakani
61e12aaebe
FEATURE: Extend PM recipient bulk imports ()
* FIX: Support multiple topic allowed user imports

* FEATURE: Add topic allowed groups import support
2024-05-17 13:45:20 +02:00
Ítalo Alves
73481e8f45
FIX: Add check for existing provider_uids to generic import ()
Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
2024-05-17 11:36:31 +02:00
Gerhard Schlager
e3882a0c48
DEV: Add support for user_associated_accounts to import script () 2024-04-29 19:48:32 +02:00
Gerhard Schlager
a538e2f153
DEV: Import script should use case-insensitive check for tag names () 2024-04-29 19:27:28 +02:00
Gerhard Schlager
4d045bfc61
DEV: Import script should insert more data into user_stats table ()
This SQL tries to insert as much data as possible into the `user_stats` table by either calculating or by approximating stats based on existing. It also fixes an error in the calculation of `reply_count`which mistakenly contained all posts, not just replies.

This change also disables some steps in the `import:ensure_consistency` rake task by setting the `SKIP_USER_STATS` env variable. Otherwise, the rake task will overwrite the calculated data in the `user_stats` table with inaccurate data. I'm not changing or removing the logic from the rake task yet because other bulk import scripts seem to depend on it.
2024-04-11 14:05:21 +02:00
Gerhard Schlager
bc98740205
DEV: Improve generic import script ()
* FEATURE: Import into `category_users` table
* FIX: Failed to import `user_options` unless `timezone` was set
* FIX: Prevent reusing original `id` from intermediate DB in `user_fields`
* FEATURE: Order posts by `post_nuber` if available
* FEATURE: Allow `[mention]` placeholder to reference users by"id" or "name" (username)
* FEATURE: Support `[quote]` placeholders in posts
* FEATURE: Support `[link]` placeholders in posts
* FEATURE: Support all kinds of permalinks and remove support for `old_relative_url`
* PERF: Speed up pre-cooking by removing DB lookups
2024-03-05 22:23:36 +01:00
Gerhard Schlager
dc8c6b8958 DEV: Lots of improvements to the generic_bulk import script
Notable changes:
* Imports a lot more tables from core and plugins
  * site settings
  * uploads with necessary upload references
  * groups and group members
  * user profiles
  * user options
  * user fields & values
  * muted users
  * user notes (plugin)
  * user followers (plugin)
  * user avatars
  * tag groups and tags
  * tag users (notification settings for tags / user)
  * category permissions
  * polls with options and votes
  * post votes (plugin)
  * solutions (plugin)
  * gamification scores (plugin)
  * events (plugin)
  * badges and badge groupings
  * user badges
  * optimized images
  * topic users (notification settings for topics)
  * post custom fields
  * permalinks and permalink normalizations

* It creates the `migration_mappings` table which is used to store the mapping for a handful of imported tables

* Detects duplicate group names and renames them

* Pre-cooking for attachments, images and mentions

* Outputs instructions when gems are missing

* Supports importing uploads from a DB generated by `uploads_importer.rb`

* Checks that all required plugins exists and enables them if needed

* A couple of optimizations and additions in `import.rake`
2023-12-11 16:23:07 +01:00
Gerhard Schlager
0b29dc5d38 DEV: Add experimental generic bulk import script 2023-08-09 20:56:14 +02:00