Commit Graph

80 Commits

Author SHA1 Message Date
Sam
5c49009c6c
PERF: properly preload emails to speed up user exports (#12778)
scopes are incredibly annoying to preload, simply adding :user_emails is not
enough.

Instead of relying on scopes simply iterate through user_emails which is
properly preloaded.

This removes 2 * N+1 when generating user reports.
2021-04-21 10:42:07 +10:00
David Taylor
821bb1e8cb
FEATURE: Rename 'Discourse SSO' to DiscourseConnect (#11978)
The 'Discourse SSO' protocol is being rebranded to DiscourseConnect. This should help to reduce confusion when 'SSO' is used in the generic sense.

This commit aims to:
- Rename `sso_` site settings. DiscourseConnect specific ones are prefixed `discourse_connect_`. Generic settings are prefixed `auth_`
- Add (server-side-only) backwards compatibility for the old setting names, with deprecation notices
- Copy `site_settings` database records to the new names
- Rename relevant translation keys
- Update relevant translations

This commit does **not** aim to:
- Rename any Ruby classes or methods. This might be done in a future commit
- Change any URLs. This would break existing integrations
- Make any changes to the protocol. This would break existing integrations
- Change any functionality. Further normalization across DiscourseConnect and other auth methods will be done separately

The risks are:
- There is no backwards compatibility for site settings on the client-side. Accessing auth-related site settings in Javascript is fairly rare, and an error on the client side would not be security-critical.
- If a plugin is monkey-patching parts of the auth process, changes to locale keys could cause broken error messages. This should also be unlikely. The old site setting names remain functional, so security-related overrides will remain working.

A follow-up commit will be made with a post-deploy migration to delete the old `site_settings` rows.
2021-02-08 10:04:33 +00:00
Kane York
7bf199b0c4 DEV: Switch to new ExportUserArchive job
We now use the newly created job class from the previous commit.
2020-08-28 11:46:53 -07:00
Dan Ungureanu
3a7ca97c36
FIX: Use include-subcategories filter in report export (#10007)
Some filters were renamed and the conversion of the filter names and arguments
was removed.
2020-06-10 18:57:39 +03:00
David Taylor
3106f85983
FIX: Support exporting reports which reference topics (#9957) 2020-06-01 18:23:58 +01:00
Dan Ungureanu
7a946b9f5c
DEV: Fix typo
Follow-up to 0a85a7aef9.
2020-04-29 12:38:57 +03:00
Dan Ungureanu
0a85a7aef9
FEATURE: Add user_profile to user_archive CSV export (#9571) 2020-04-29 12:09:50 +03:00
Robin Ward
ce663d67e1 FIX: CSV Exports were throwing errors with invalid dates
This fix will consider any invalid dates to be non-existant.
2020-04-07 14:05:27 -04:00
Joffrey JAFFEUX
aeaea3c154
FIX: correctly take category/group filters into csv export (#9300) 2020-03-30 19:08:47 +02:00
Dan Ungureanu
5060811464
FIX: Export all category names in user archives (#8790) 2020-01-28 12:52:07 +02:00
Bianca Nenciu
0a27086764
FEATURE: Export all types of reports (#8748)
There is a single stacked_chart which was not exportable
2020-01-21 18:43:19 +02:00
Roman Rizzi
10565e4623
SECURITY: Safely decompress files. (#8124)
* FEATURE: Adds an extra protection layer when decompressing files.

* Rename exporter/importer to zip importer. Update old locale

* Added a new composite class to decompress a file with multiple strategies

* Set max file size inside a site setting

* Ensure that file is deleted after compression

* Sanitize path and files before compressing/decompressing
2019-10-03 10:19:35 -03:00
Krzysztof Kotlarek
427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Gerhard Schlager
fd12c414e7 DEV: Refactor helper methods for upload markdown
Follow-up to a61ff167
2019-07-25 16:36:35 +02:00
Gerhard Schlager
a61ff16740 DEV: Make attachment markdown reusable 2019-07-25 14:04:18 +02:00
Roman Rizzi
f5c707c97a
FEATURE: Gz to zip for exports (#7889)
* Revert "Revert "FEATURE: admin/user exports are compressed using the zip format (#7784)""

This reverts commit f89bd55576.

* Replace .tar.zip with .zip
2019-07-18 09:34:48 -03:00
romanrizzi
f89bd55576 Revert "FEATURE: admin/user exports are compressed using the zip format (#7784)"
This reverts commit 8b2580e20f.
2019-07-10 11:38:51 -03:00
Roman Rizzi
8b2580e20f
FEATURE: admin/user exports are compressed using the zip format (#7784)
* FEATURE: admin/user exports are compressed using the zip format

* Update translations. Theme exporter now exports .zip file. Theme importer supports .zip and .gz files

* Fix controller test, updated locale and skip saving the csv export to disk
2019-07-10 11:13:03 -03:00
Bianca Nenciu
b2eb0f4ad6 FEATURE: Export any type of report supporting table mode. (#7662) 2019-06-28 08:50:31 +02:00
Guo Xiang Tan
9d0fba64c0 FIX: Use attachment format in user export system post take 2. 2019-06-11 12:15:11 +08:00
Arpit Jalan
028121b95b
FIX: delete system generated message when user_export record is deleted (#7595)
FIX: system generated message for user export should be closed by default
2019-05-28 16:38:41 +05:30
Sam Saffron
954293655f FEATURE: rake posts:destroy_old_user_data_exports
Historically we would keep the user data export posts around but delete
the uploads.

This leaves a lot of broken uploads in the system.

This rake task allows us to clean up old mess.
2019-05-23 11:11:37 +10:00
Sam Saffron
30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Guo Xiang Tan
451f7842ff DEV: More send -> public_send. 2019-05-07 10:05:58 +08:00
Régis Hanol
326d892f5e Aadd 'secondary_emails' field in users export
FIX: escape_comma wasn't working in CSV exports
FIX: group_names field wasn't properly serialized
2019-02-27 10:12:20 +01:00
Régis Hanol
3c9c95ac83 Update Rubocop to 0.60 2018-12-04 10:48:16 +01:00
Guo Xiang Tan
ad5082d969 Make rubocop happy again. 2018-06-07 13:28:18 +08:00
Arpit Jalan
cafd1241b9 FIX: make report data export more resilient to inconsistent date format
The DAU/MAU report export was failing because of date being a string
and not a valid Date value
2018-05-24 17:14:08 +05:30
Arpit Jalan
4f55fbfefa FEATURE: include report title in PM subject and filename 2018-04-24 22:25:54 +05:30
Arpit Jalan
a1f0c58f23 FIX: export download link was broken 2018-04-20 01:12:41 +05:30
Arpit Jalan
91bf10bd12 FIX: create upload record for exported csv files 2018-04-20 00:27:49 +05:30
Arpit Jalan
f5febe5843 FIX: date range was inconsistent for admin dashboard reports 2018-04-08 16:49:41 +05:30
Arpit Jalan
33df2d6a02 FIX: data export should fill missing dates with zero value 2018-02-18 23:52:09 +05:30
Neil Lalonde
b35960fc35 FIX: export admin reports to csv would have 0 rows in the csv file 2017-12-01 15:50:36 -05:00
Robin Ward
971e302ff2 FEATURE: Support an end date for user silencing 2017-11-14 13:20:19 -05:00
Robin Ward
1f14350220 Rename "Blocked" to "Silenced" 2017-11-10 14:10:27 -05:00
Régis Hanol
d9465bac13 add 'staged' column to user export 2017-09-13 18:09:11 +02:00
Arpit Jalan
e993d53260 FIX: handle missing parent category when exporting user archive
https://meta.discourse.org/t/download-my-posts-failed/67613?u=techapj
2017-08-09 14:37:21 +05:30
Guo Xiang Tan
5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Arpit Jalan
df246c79b6 FEATURE: add user profile fields in user list export 2017-03-21 15:51:52 +05:30
Robin Ward
1957d12670 SECURITY: Don't use backticks for exporting your archive 2017-03-16 16:24:59 -04:00
Arpit Jalan
f1566ef80b FIX: escape comma in exported CSV file 2016-07-19 13:19:46 +05:30
Régis Hanol
86d50d0017 PERF: stream CSV exports so they don't fail when they are too big 2016-05-25 22:20:35 +02:00
shakti katare
cd4c34204d Fix: Suspensions and Blocks do not give enough details in logs 2016-04-20 10:45:19 +05:30
Arpit Jalan
05288144b5 FIX: export user list based on trust level filter 2016-03-07 18:49:31 +05:30
Devon Estes
0aa59956fa Extract method refactoring in Jobs::ExportCsvFile
I was combing through some of the files with worse grades on Code Climate as a guide for places where I could jump in and help, and I saw this as one of the ones in need of some love. I reduced duplication in the #user_list_export method by extracting several methods that were common to both branches of the logic in that method.
2016-02-05 16:47:15 +01:00
Arpit Jalan
eec8436cfe FEATURE: filter admin reports via user group 2016-02-04 11:23:49 +05:30
Arpit Jalan
2eba00a326 FIX: user archive url field should have absolute URLs 2015-11-25 15:13:15 +05:30
Arpit Jalan
700c005c26 PERF: optimize export user list CSV queries 2015-11-02 13:15:58 +05:30
Neil Lalonde
b4a724e80a FEATURE: export dashboard reports to csv file 2015-09-15 16:45:10 -04:00