Commit Graph

18 Commits

Author SHA1 Message Date
Ruben Oussoren
f81bf31993
Fix the import votes for posts table to the new table (#29810) 2024-11-19 10:53:33 -05:00
Ruben Oussoren
4b8b7b2fb5
DEV: Add import support for Topic Voting plugin (#29442)
Some checks are pending
Licenses / run (push) Waiting to run
Linting / run (push) Waiting to run
Tests / core frontend (${{ matrix.browser }}) (Chrome) (push) Waiting to run
Tests / ${{ matrix.target }} ${{ matrix.build_type }} (annotations, core) (push) Waiting to run
Tests / ${{ matrix.target }} ${{ matrix.build_type }} (backend, core) (push) Waiting to run
Tests / ${{ matrix.target }} ${{ matrix.build_type }} (backend, plugins) (push) Waiting to run
Tests / ${{ matrix.target }} ${{ matrix.build_type }} (frontend, plugins) (push) Waiting to run
Tests / ${{ matrix.target }} ${{ matrix.build_type }} (frontend, themes) (push) Waiting to run
Tests / ${{ matrix.target }} ${{ matrix.build_type }} (system, chat) (push) Waiting to run
Tests / ${{ matrix.target }} ${{ matrix.build_type }} (system, core) (push) Waiting to run
Tests / ${{ matrix.target }} ${{ matrix.build_type }} (system, plugins) (push) Waiting to run
Tests / ${{ matrix.target }} ${{ matrix.build_type }} (system, themes) (push) Waiting to run
Tests / core frontend (${{ matrix.browser }}) (Firefox ESR) (push) Waiting to run
Tests / core frontend (${{ matrix.browser }}) (Firefox Evergreen) (push) Waiting to run
DEV: Add import support for Topic Voting plugin
- Implemented bulk import functionality specifically for topic_voting_votes.
- Updated the method name from import_votes to import_post_voting_votes for improved clarity and maintainability.
- Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
2024-10-31 08:30:28 -04:00
Selase Krakani
dd34f1927b
FIX: Imports of upload-only chat messages (#29162)
The current implementation adds a "note" for chat messages with empty
messages, however chat messages with only uploads  are allowed. This change
allows such messages to be imported.
2024-10-10 15:18:10 +00:00
Selase Krakani
9825bde811
DEV: Generic bulk chat import support (#28697)
* DEV: WIP generic bulk chat import support

This first iteration implements bulk import for:

* direct_messages
* chat_channels
* user_chat_channel_memberships
* chat_threads
* user_chat_thread_memberships
* chat_messages
* chat_reactions
* chat_mentions

* DEV: Refactor raw placeholder interpolation to support chat messages

This change adds support for chat message placeholder interpolation
and switches to using `Chat::Message.cook` for cooking in the interim
instead of hand-cooking chat messages like we currently do for posts

* DEV: Extend upload references import to support chat message uploads

* DEV: Explicity set chat retention

- Set both channel and dm chats to 0
- Add temporary workaround for testing only chat imports

* DEV: Compute channel and thread membership metadata

Compute and set various user channel/thread membership stats and
remove hardcoded test index seed data

* FIX: Fix chat reactions import

Allow multiple reactions on a message by a user
2024-10-08 11:55:30 +00:00
Gerhard Schlager
d4379af7f2
FIX: Import script didn't set public attribute of polls (#28864) 2024-10-02 20:02:13 +02:00
Selase Krakani
d896f5cb70
DEV: Include post and topic attributes in imported quotes (#27851)
Currently, quotes imported via generic bulk import script do not include
references to the quoted post. This change includes both topic and post attributes
in a quote if the placeholder metadata includes a `post_id`
2024-07-11 16:47:21 +00:00
Gerhard Schlager
7c26d5d084
FIX: Import script was broken after upgrade of sqlite3 gem (#27648) 2024-07-02 17:38:15 +10:00
Selase Krakani
f2c4474c1e
DEV: Improve user generic bulk importer anonymization (#27307)
* DEV: Improve user generic bulk importer anonymization

Add support for properly anonymizing:
 - email
 - date_of_birth
 - location
 - website
 - bio

* DEV: Remove uneeded anon username check in `import_user_emails`
2024-06-05 11:25:17 +00:00
Loïc Guitaut
2a28cda15c DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
Selase Krakani
949c70372c
DEV: Add support for various fields in generic bulk importer (#27114)
* user_profiles - `location`
* users - `date_of_birth`
* topics - `pinned_at`, `pinned_until`, `pinned_globally`

This also include changes to correctly import PMs. Currently PM topics
are skipped because of a check in `import_users` step which requires `category_id`
to be present.
2024-05-24 13:46:06 +02:00
Selase Krakani
61e12aaebe
FEATURE: Extend PM recipient bulk imports (#27063)
* FIX: Support multiple topic allowed user imports

* FEATURE: Add topic allowed groups import support
2024-05-17 13:45:20 +02:00
Ítalo Alves
73481e8f45
FIX: Add check for existing provider_uids to generic import (#26914)
Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
2024-05-17 11:36:31 +02:00
Gerhard Schlager
e3882a0c48
DEV: Add support for user_associated_accounts to import script (#26779) 2024-04-29 19:48:32 +02:00
Gerhard Schlager
a538e2f153
DEV: Import script should use case-insensitive check for tag names (#26699) 2024-04-29 19:27:28 +02:00
Gerhard Schlager
4d045bfc61
DEV: Import script should insert more data into user_stats table (#26551)
This SQL tries to insert as much data as possible into the `user_stats` table by either calculating or by approximating stats based on existing. It also fixes an error in the calculation of `reply_count`which mistakenly contained all posts, not just replies.

This change also disables some steps in the `import:ensure_consistency` rake task by setting the `SKIP_USER_STATS` env variable. Otherwise, the rake task will overwrite the calculated data in the `user_stats` table with inaccurate data. I'm not changing or removing the logic from the rake task yet because other bulk import scripts seem to depend on it.
2024-04-11 14:05:21 +02:00
Gerhard Schlager
bc98740205
DEV: Improve generic import script (#25972)
* FEATURE: Import into `category_users` table
* FIX: Failed to import `user_options` unless `timezone` was set
* FIX: Prevent reusing original `id` from intermediate DB in `user_fields`
* FEATURE: Order posts by `post_nuber` if available
* FEATURE: Allow `[mention]` placeholder to reference users by"id" or "name" (username)
* FEATURE: Support `[quote]` placeholders in posts
* FEATURE: Support `[link]` placeholders in posts
* FEATURE: Support all kinds of permalinks and remove support for `old_relative_url`
* PERF: Speed up pre-cooking by removing DB lookups
2024-03-05 22:23:36 +01:00
Gerhard Schlager
dc8c6b8958 DEV: Lots of improvements to the generic_bulk import script
Notable changes:
* Imports a lot more tables from core and plugins
  * site settings
  * uploads with necessary upload references
  * groups and group members
  * user profiles
  * user options
  * user fields & values
  * muted users
  * user notes (plugin)
  * user followers (plugin)
  * user avatars
  * tag groups and tags
  * tag users (notification settings for tags / user)
  * category permissions
  * polls with options and votes
  * post votes (plugin)
  * solutions (plugin)
  * gamification scores (plugin)
  * events (plugin)
  * badges and badge groupings
  * user badges
  * optimized images
  * topic users (notification settings for topics)
  * post custom fields
  * permalinks and permalink normalizations

* It creates the `migration_mappings` table which is used to store the mapping for a handful of imported tables

* Detects duplicate group names and renames them

* Pre-cooking for attachments, images and mentions

* Outputs instructions when gems are missing

* Supports importing uploads from a DB generated by `uploads_importer.rb`

* Checks that all required plugins exists and enables them if needed

* A couple of optimizations and additions in `import.rake`
2023-12-11 16:23:07 +01:00
Gerhard Schlager
0b29dc5d38 DEV: Add experimental generic bulk import script 2023-08-09 20:56:14 +02:00