Commit Graph

776 Commits

Author SHA1 Message Date
Arpit Jalan
3684337e4a FIX: email always settings were not being respected
https://meta.discourse.org/t/email-notification-for-messages/163937/10?u=techapj
2020-09-23 22:00:15 +05:30
Martin Brennan
5268568d23
FEATURE: Remove user topic timers and migrate to bookmarks with reminders (#10474)
This PR removes the user reminder topic timers, because that system has been supplanted and improved by bookmark reminders. The option is removed from the UI and all existing user reminder topic timers are migrated to bookmark reminders.

Migration does this:

* Get all topic_timers with status_type 5 (reminders)
* Gets all bookmarks where the user ID and topic ID match
* Loops through the found topic timers
  * If there is no bookmark for the OP of the topic, then we just create a bookmark with a reminder
  * If there is a bookmark for the OP of the topic and it does **not** have a reminder set, then just 
update it with the topic timer reminder
  * If there is a bookmark for the OP of the topic with a reminder then just discard the topic timer
* Cancels all outstanding user reminder topic timers
* **Trashes (not deletes) all user reminder topic timers**

Notes:

* For now I have left the user reminder topic timer job class in place; this is so the jobs can be cancelled in the migration. It and the specs will be deleted in the next PR.
* At a later date I will write a migration to delete all trashed user topic timers. They are not deleted here in case there are data issues and they need to be recovered.
* A future PR will change the UI of the topic timer modal to make it look more like the bookmark modal.
2020-09-14 11:11:55 +10:00
Kane York
e0a0928420
FEATURE: Add bookmarks to the user export (#10591) 2020-09-11 11:03:22 +10:00
Gerhard Schlager
ac70c48be4 FIX: Prevent "uploads are missing in S3" alerts after restoring a backup
After restoring a backup it takes up to 48 hours for uploads stored on S3 to appear in the S3 inventory. This change prevents alerts about missing uploads by preventing the EnsureS3UploadsExistence job from running in the first 48 hours after a restore. During the restore it  deletes the count of missing uploads from the PluginStore, so that an alert isn't triggered by an old number.
2020-09-10 21:37:48 +02:00
Martin Brennan
7f2f87bf59
DEV: Review fixes (#10641)
See comments in https://review.discourse.org/t/dev-imap-log-to-database-10435/14337/6 for context.
2020-09-10 13:41:46 +10:00
Blake Erickson
67dec38f31 FIX: Gravatar download attempt if user is missing their email
It is possible that a user could exist without an email, if so we should
not enqueue a job to download their gravatar.

This commit resolves this error that can occur:

```
Job exception: undefined method `email' for nil:NilClass
/var/www/discourse/app/models/user.rb:1204:in `email'
/var/www/discourse/app/jobs/regular/update_gravatar.rb:12:in `execute'
```

This commit also fixes the original spec which actually was wrong. The
job never enqueued in the original spec and so the gravatar was never
actually updated and the test was checking if the two values were the
same, but they were both null and never updated, so of course they were
the same!

A new test has also been added to make sure the gravatar job isn't
enqueued when a user's email is missing.
2020-09-02 20:19:46 -06:00
Krzysztof Kotlarek
9954a677ab
FIX: don't send mailing list for post with empty content (#10577)
discourse-assign is creating posts with empty content to show that a specific user was assign/unassigned for a specific topic.

It is causing confusing emails with empty content

The bug was mentioned here: https://meta.discourse.org/t/again-on-empty-emails-and-notifications-generated-on-topic-assignment/162213
2020-09-03 08:58:25 +10:00
Guo Xiang Tan
01600492de
FIX: Don't raise error in update username job if user has been deleted. 2020-09-02 11:17:17 +08:00
Kane York
26ec4fd25b
FIX: User export category preferences on a deleted category. (#10573)
Tests from a1dd761bd9 were incomplete and did not test a deleted category's category_users record.
2020-09-01 13:22:59 -07:00
Kane York
a1dd761bd9
FIX: Handle deleted categories in post export (#10567)
Fixes a crash when exporting my own archive on Meta.
2020-08-31 17:33:28 -07:00
Kane York
5ec5fbd7ba
User export improvements 2 (#10560)
* FEATURE: Use predictable filenames inside the user archive export

* FEATURE: Include badges in user archive export

* FEATURE: Add user_visits table to the user archive export
2020-08-31 15:26:51 -07:00
Krzysztof Kotlarek
94152e4640
FIX: dont error when bookmark topic is nil (#10555) 2020-08-31 09:15:36 +10:00
Kane York
c5dc729e77
FEATURE: Add category tracking state to user archive export (#10557)
Tackling a simple table for the first actual new file in the user archive export.
2020-08-28 13:16:31 -07:00
Kane York
225cdba676 DEV: drop the explicit .each in UserArchive CSV writing
I think this is mostly stylistic, but this helps prevent explosive typos in the enum_for() line.
2020-08-28 11:46:53 -07:00
Kane York
4aed861336 DEV: minor refactors to ExportUserArchive(Spec) 2020-08-28 11:46:53 -07:00
Kane York
7bf199b0c4 DEV: Switch to new ExportUserArchive job
We now use the newly created job class from the previous commit.
2020-08-28 11:46:53 -07:00
Kane York
a8560d741f DEV: Create ExportUserArchive as clone of ExportCsvFile
This is in preparation for improvements to the user archive export data.
Some refactors happened along the way, including calling the different _export methods 'components' of the zip file.

Additionally, make the test for post export much more comprehensive.

Copy sources:
  app/jobs/regular/export_csv_file.rb
  spec/jobs/export_csv_file_spec.rb
2020-08-28 11:46:53 -07:00
Guo Xiang Tan
40c6d90df3 PERF: Create a partial regular post_search_data index on large sites.
With the addition of `PostSearchData#private_message`, a partial
index consisting of only search data from regular posts can be created.
The partial index helps to speed up searches on large sites since PG
will not have to do an index scan on the entire search data index which
has shown to be a bottle neck.
2020-08-27 13:42:00 +08:00
Guo Xiang Tan
3cc761ac44
FIX: Clean up toggle closed topic timer when user is not authorized. 2020-08-26 12:59:05 +08:00
Guo Xiang Tan
5bca1aec48
FIX: Clean up topic_timers when no longer valid.
This was causing a bug where the jobs for the topic_timers will keep
getting enqueued over and over again.
2020-08-26 12:18:51 +08:00
Martin Brennan
4670b62969
DEV: IMAP log to database (#10435)
Convert all IMAP logging to write to a database table for easier inspection. These logs are cleaned up daily if they are > 5 days old.

Logs can easily be watched in dev by setting DISCOURSE_DEV_LOG_LEVEL=\"debug\" and running tail -f development.log | grep IMAP
2020-08-14 12:01:31 +10:00
Régis Hanol
bc63232d2e
FIX: sync reviewable count when opening the hamburger menu (#10368)
When a tab is open but left unattended for a while, the red, green, and blue
pills tend to go out of sync.

So whevener we open the notifications menu, we sync up the notification count
(eg. blue and green pills) with the server.

However, the reviewable count (eg. the red pill) is not a notification and
is located in the hamburger menu. This commit adds a new route on the server
side to retrieve the reviewable count for the current user and a ping
(refreshReviewableCount) from the client side to sync the reviewable count
whenever they open the hamburger menu.

REFACTOR: I also refactored the hamburger-menu widget code to prevent repetitive uses
of "this.".

PERF: I improved the performance of the 'notify_reviewable' job by doing only 1 query
to the database to retrieve all the pending reviewables and then tallying based on the
various rights.
2020-08-07 18:13:02 +02:00
David Taylor
df39e372d7
DEV: Add spec for removing and re-adding hotlinked images
Before the recent refactor, this would fail. https://meta.discourse.org/t/154184
2020-08-06 10:01:53 +01:00
David Taylor
ceb858c70a
PERF: Release post_upload records when downloaded image is removed (#10379)
Previously we would unconditionally keep all images downloaded via pull_hotlinked_images, even if they are later removed from the post. This commit removes that logic, and relies on the existing link_post_uploads process to pick up the downloaded images in `cooked`. Specs are added to ensure this is working correctly for regular hotlinked images, and for oneboxes.
2020-08-06 10:06:34 +10:00
David Taylor
cb12a721c4
REFACTOR: Refactor pull_hotlinked_images job
This commit should cause no functional change
- Split into functions to avoid deep nesting
- Register custom field type, and remove manual json parse/serialize
- Recover from deleted upload records

Also adds a test to ensure pull_hotlinked_images redownloads secure images only once
2020-08-05 12:14:59 +01:00
Guo Xiang Tan
162958380a
DEV: Remove stray code that has been commented out. 2020-07-29 09:58:29 +08:00
Guo Xiang Tan
c6202af005
Update rubocop to 2.3.1. 2020-07-24 17:19:21 +08:00
Guo Xiang Tan
0cbf86f1e7
DEV: Refactor reindex_search_spec.
Follow-up 20dc845418
2020-07-24 09:29:54 +08:00
Mark VanLandingham
20dc845418
FIX: tests for reindex_search_spec pass regardless of seed (#10297) 2020-07-23 13:51:45 -05:00
Guo Xiang Tan
3766122a82
DEV: Allow developmental post search index versions. 2020-07-23 15:19:46 +08:00
Guo Xiang Tan
609ba50fe8
DEV: Add more granularity to SearchIndexer versions.
Sometimes, we just want to reindex a specific model and not all the
things.
2020-07-23 14:24:06 +08:00
jbrw
06073fe8c6
FEATURE: Allow group moderators to close/archive topics
* FEATURE: Allow group moderators to close/archive topics
2020-07-14 12:36:19 -04:00
Robin Ward
a73da42691 FIX: Don't award new user of the month in the wrong month
see: https://meta.discourse.org/t/new-user-of-the-month-badge-awarded-before-registraton-date/157347/2?u=eviltrout
2020-07-13 15:05:30 -04:00
Dan Ungureanu
c72bc27888
FEATURE: Implement support for IMAP and SMTP email protocols. (#8301)
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2020-07-10 12:05:55 +03:00
Penar Musaraj
67582e7d27
FIX: Do not send system emails to suspended users (#10192) 2020-07-08 13:30:32 -04:00
David Taylor
c5078e5dc1
DEV: Remove accidentally committed puts statements 2020-06-23 12:41:47 +01:00
Guo Xiang Tan
3370ef188e
FEATURE: Remove deprecated uploads url site settings.
The site settings have been replaced with direct image upload since
Discourse 2.3.
2020-06-22 14:32:29 +08:00
David Taylor
17c4f76eac
FIX: Do not attempt to pull_hotlinked on emoji images when CDN enabled (#10091) 2020-06-19 20:21:05 +01:00
David Taylor
a99bb0ded4
Revert "FIX: Do not attempt to pull_hotlinked on emoji images when CDN enabled"
This changed cause plugin spec failures and needs further investigation

This reverts commit 78626d2832.
2020-06-19 14:39:16 +01:00
David Taylor
9f2e7e4651
FIX: Handle invalid URLs gracefully when pulling hotlinked images 2020-06-19 12:52:51 +01:00
David Taylor
78626d2832
FIX: Do not attempt to pull_hotlinked on emoji images when CDN enabled 2020-06-19 12:45:06 +01:00
Dan Ungureanu
3a7ca97c36
FIX: Use include-subcategories filter in report export (#10007)
Some filters were renamed and the conversion of the filter names and arguments
was removed.
2020-06-10 18:57:39 +03:00
Vinoth Kannan
0f20a6f0aa FIX: use short_path of flair upload to get signed url for secure media.
If we use `upload.url` for secure urls then the images won't render.
2020-06-05 07:43:15 +05:30
David Taylor
3106f85983
FIX: Support exporting reports which reference topics (#9957) 2020-06-01 18:23:58 +01:00
Krzysztof Kotlarek
9a6ef80739
FEATURE: notify admins about old credentials (#9918)
* FEATURE: notify admins about old credentials

Security and API keys should be renewed periodically.
This additional notification should help admins keep their Discourse safe and secure.
2020-06-01 13:49:27 +10:00
David Taylor
ecfce93f28
FIX: Support IRIs (unicode URIs) when pulling hotlinked images (#9928) 2020-05-29 17:47:05 +01:00
Andrew Schleifer
74d28a43d1
new S3 backup layout (#9830)
* DEV: new S3 backup layout

Currently, with $S3_BACKUP_BUCKET of "bucket/backups", multisite backups
end up in "bucket/backups/backups/dbname/" and single-site will be in
"bucket/backups/".

Both _should_ be in "bucket/backups/dbname/"

- remove MULTISITE_PREFIX,
- always include dbname,
- method to move to the new prefix
- job to call the method

* SPEC: add tests for `VacateLegacyPrefixBackups` onceoff job.

Co-authored-by: Vinoth Kannan <vinothkannan@vinkas.com>
2020-05-29 00:28:23 +05:30
David Taylor
8a3d9d7036
DEV: Run jobs sequentially in test mode (#9897)
When running jobs in tests, we use `Jobs.run_immediately!`. This means that jobs are run synchronously when they are enqueued. Jobs sometimes enqueue other jobs, which are also executed synchronously. This means that the outermost job will block until the inner jobs have finished executing. In some cases (e.g. process_post with hotlinked images) this can lead to a deadlock.

This commit changes the behavior slightly. Now we will never run jobs inside other jobs. Instead, we will queue them up and run them sequentially in the order they were enqueued. As a whole, they are still executed synchronously. Consider the example

```ruby
class Jobs::InnerJob < Jobs::Base
  def execute(args)
    puts "Running inner job"
  end
end

class Jobs::OuterJob < Jobs::Base
  def execute(args)
    puts "Starting outer job"
    Jobs.enqueue(:inner_job)
    puts "Finished outer job"
  end
end

Jobs.enqueue(:outer_job)
puts "All jobs complete"
```

The old behavior would result in:

```
Starting outer job
Running inner job
Finished outer job
All jobs complete
```

The new behavior will result in:
```
Starting outer job
Finished outer job
Running inner job
All jobs complete
```
2020-05-28 12:52:27 +01:00
Régis Hanol
2a4db15544 FIX: don't send digests to users with no primary email
It might happen that some User records have no associated primary emails.
In which case we don't ever want to send them a digest.

Also added a new "user_email_no_email" skipped email log to ensure these cases
are properly handled and surfaced.
2020-05-27 17:09:40 +02:00
Krzysztof Kotlarek
34e5f0a9a3
Revert "FEATURE: notify admins about old credentials (#9854)" (#9886)
This reverts commit 349a67bee6.
2020-05-27 09:52:53 +10:00
Krzysztof Kotlarek
349a67bee6
FEATURE: notify admins about old credentials (#9854)
* FEATURE: notify admins about old credentials

Security and API keys should be renewed periodically.
This additional notification should help admins keep their Discourse safe and secure.
2020-05-27 08:13:47 +10:00
Vinoth Kannan
5fb9271878
DEV: ignore flair_url column in group model. (#9873) 2020-05-26 00:43:50 +05:30
Vinoth Kannan
505122bb45 FIX: skip onceoff job for groups with invalid flair URL. 2020-05-25 13:11:00 +05:30
Vinoth Kannan
8e56197728
UX: use "icon-picker" & "image-uploader" fields to set group flair. (#9779) 2020-05-25 11:08:47 +05:30
Michael Brown
d9a02d1336
Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eee.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
Jeff Atwood
20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6f, reversing
changes made to 2660c2e21d.
2020-05-22 20:25:56 -07:00
Guo Xiang Tan
96c02caba7
DEV: Change use of Redis flushall to flushdb.
FLUSHALL removes all keys from all databases. Instead we only want to
remove keys from the current Redis database.
2020-05-19 10:20:00 +08:00
Kane York
869f9b20a2
PERF: Dematerialize topic_reply_count (#9769)
* PERF: Dematerialize topic_reply_count

It's only ever used for trust level promotions that run daily, or compared to 0. We don't need to track it on every post creation.

* UX: Add symbol in TL3 report if topic reply count is capped

* DEV: Drop user_stats.topic_reply_count column
2020-05-14 15:42:00 -07:00
Martin Brennan
fa572d3a7a
DEV: Remove all code referencing at_desktop bookmark reminders (#9650)
We have found no need for these reminder types, so we are removing the code for them.
2020-05-06 15:22:43 +10:00
Dan Ungureanu
0a85a7aef9
FEATURE: Add user_profile to user_archive CSV export (#9571) 2020-04-29 12:09:50 +03:00
Roman Rizzi
394babcae3
FIX: Only show the review page to users that can see it. Do not publish the reviewable count update message to everyone. (#9556) 2020-04-27 14:51:25 -03:00
Sam Saffron
6a18c9aa0b
Revert "FEATURE: enforce_canonical_emails site setting"
This reverts commit 6f9177e2ed.

We decided on a completely different approach to the problem.

Instead we will let blocked emails be treated as canonical.
2020-04-24 13:52:06 +10:00
Vinoth Kannan
df0c386f8a
UX: drop the automatic_membership_retroactive column from groups model. (#9430) 2020-04-22 22:07:39 +05:30
Martin Brennan
628ba9d1e2
FEATURE: Promote bookmarks with reminders to core functionality (#9369)
The main thrust of this PR is to take all the conditional checks based on the `enable_bookmarks_with_reminders` away and only keep the code from the `true` path, making bookmarks with reminders the core bookmarks feature. There is also a migration to create `Bookmark` records out of `PostAction` bookmarks for a site.

### Summary

* Remove logic based on whether enable_bookmarks_with_reminders is true. This site setting is now obsolete, the old bookmark functionality is being removed. Retain the setting and set the value to `true` in a migration.
* Use the code from the rake task to create a database migration that creates bookmarks from post actions.
* Change the bookmark report to read from the new table.
* Get rid of old endpoints for bookmarks
* Link to the new bookmarks list from the user summary page
2020-04-22 13:44:19 +10:00
David Taylor
b6c19cba20
FIX: Abort emit_web_hook_event job cleanly if web hook was deleted (#9445)
Raising an error causes the job to be retried, and causes a lot of noise in the logs
2020-04-16 21:24:09 +01:00
Neil Lalonde
074509fd95
FIX: don't demote users to TL2 when default trust level is 3
Within 24 hours of signing up, new users were losing their
default trust level of 3. With this fix, demotions from
trust level 3 won't happen when the "default trust level"
setting is 3 or 4.
2020-04-16 12:28:16 -04:00
Sam Saffron
6f9177e2ed
FEATURE: enforce_canonical_emails site setting
The new `enforce_canonical_emails` site setting ensures that emails in the
canonical form are unique.

This mean that if `s.a.m+1@gmail.com` is registered `sam@gmail.com` will
not be allowed.

The commit contains a blanket "tag strip" (stripping everything after +)
it also contains special handling of a "dot strip" for googlemail and gmail.

The setting only impacts new registrations after `enforce_canonical_emails`

The setting is default false so it will not impact any existing installs.
2020-04-14 14:16:30 +10:00
Robin Ward
ce663d67e1 FIX: CSV Exports were throwing errors with invalid dates
This fix will consider any invalid dates to be non-existant.
2020-04-07 14:05:27 -04:00
Vinoth Kannan
82201fa466 FIX: jobs/delete_replies: Add Time+Duration, not Time+Time #9314
Co-authored-by: Kane York <kanepyork@gmail.com>
2020-04-03 09:23:40 +05:30
Joffrey JAFFEUX
aeaea3c154
FIX: correctly take category/group filters into csv export (#9300) 2020-03-30 19:08:47 +02:00
Martin Brennan
097851c135
FIX: Change secure media to encompass attachments as well (#9271)
If the “secure media” site setting is enabled then ALL files uploaded to Discourse (images, video, audio, pdf, txt, zip etc. etc.) will follow the secure media rules. The “prevent anons from downloading files” setting will no longer have any bearing on upload security. Basically, the feature will more appropriately be called “secure uploads” instead of “secure media”.

This is being done because there are communities out there that would like all attachments and media to be secure based on category rules but still allow anonymous users to download attachments in public places, which is not possible in the current arrangement.
2020-03-26 07:16:02 +10:00
Roman Rizzi
27bc4f51c7
FIX: Ignore suspect users that were migrated or users who were created more than six months ago (#9205) 2020-03-14 08:47:53 -03:00
Martin Brennan
793f39139a
FEATURE: Send notifications for time-based and At Desktop bookmark reminders (#9071)
* This PR implements the scheduling and notification system for bookmark reminders. Every 5 minutes a schedule runs to check any reminders that need to be sent before now, limited to **300** reminders at a time. Any leftover reminders will be sent in the next run. This is to avoid having to deal with fickle sidekiq and reminders in the far-flung future, which would necessitate having a background job anyway to clean up any missing `enqueue_at` reminders.

* If a reminder is sent its `reminder_at` time is cleared and the `reminder_last_sent_at` time is filled in. Notifications are only user-level notifications for now.

* All JavaScript and frontend code related to displaying the bookmark reminder notification is contained here. The reminder functionality is now re-enabled in the bookmark modal as well.

* This PR also implements the "Remind me next time I am at my desktop" bookmark reminder functionality. When the user is on a mobile device they are able to select this option. When they choose this option we set a key in Redis saying they have a pending at desktop reminder. The next time they change devices we check if the new device is desktop, and if it is we send reminders using a DistributedMutex. There is also a job to ensure consistency of these reminders in Redis (in case Redis drops the ball) and the at desktop reminders expire after 20 days.

* Also in this PR is a fix to delete all Bookmarks for a user via `UserDestroyer`
2020-03-12 10:16:00 +10:00
Roman Rizzi
b9aaa9718d
FIX: When must_approve_users is enabled, we don't want to send suspect users to the review queue. Only non-approved users should be sent. Provide a migration to auto-approve every problematic review item (#9179) 2020-03-11 17:05:44 -03:00
Jarek Radosz
29b35aa64c
DEV: Improve flaky time-sensitive specs (#9141) 2020-03-10 22:13:17 +01:00
Jarek Radosz
85e03a7f68
DEV: Replace Time.new with Time.now (#9142)
(or `Time.zone.now`)
2020-03-09 17:37:49 +01:00
Gerhard Schlager
edc8d58ac3 FEATURE: Add site setting to disable staged user cleanup
... and disabled the cleanup during imports, otherwise a running Sidekiq might delete users before posts are created
2020-03-09 10:26:41 +01:00
Blake Erickson
43b54c631d
DEV: Reserve webhook event types to be used in plugins (#9110)
* DEV: Reserve webhook event types to be used in plugins

Based on feedback on the following PR's:

https://github.com/discourse/discourse-solved/pull/85

https://github.com/discourse/discourse-assign/pull/61

This commit reserves ID's to be used for webhook event types to ensure
that some other webhook or plugin doesn't end up using the same ID.

* Fix broken test

I don't think this test has to test ALL event types to verify that this
feature is working. Now that we added some event types that plugins are
using this test was failing for missing fabricators that exist in the
respective plugins.

* remove loop and just test first record
2020-03-06 10:16:19 -07:00
Martin Brennan
0388653a4d
DEV: Upload and secure media retroactive rake task improvements (#9027)
* Add uploads:sync_s3_acls rake task to ensure the ACLs in S3 are the correct (public-read or private) setting based on upload security

* Improved uploads:disable_secure_media to be more efficient and provide better messages to the user.

* Rename uploads:ensure_correct_acl task to uploads:secure_upload_analyse_and_update as it does more than check the ACL

* Many improvements to uploads:secure_upload_analyse_and_update

* Make sure that upload.access_control_post is unscoped so deleted posts are still fetched, because they still affect the security of the upload.

* Add escape hatch for capture_stdout in the form of RAILS_ENABLE_TEST_STDOUT. If provided the capture_stdout code will be ignored, so you can see the output if you need.
2020-03-03 10:03:58 +11:00
tshenry
a09e5d12c2
FIX: Topics should honor auto-close when published to category (#8963)
* FIX: Topics should honor auto-close when published to category

* Add test
2020-03-02 14:21:35 -05:00
Vinoth Kannan
5774107a2d
FIX: downloaded image URLs incorrectly replaced in post raw. (#9014)
Previously, while replacing the downloaded image URL `http://wiki.mozilla.org/images/2/2e/Longcat1.png` similar non-image URL `http://wiki.mozilla.org/images/2` was replaced wrongly.
2020-02-27 10:22:55 +05:30
Dan Ungureanu
cf0c6d5761
FIX: Ensure web hooks are retried at most 5 times 2020-02-21 17:02:40 +02:00
Martin Brennan
6a2bde4d48 Fix broken secure media specs 2020-02-21 10:01:32 +10:00
Robin Ward
345764565f FIX: Respect muted tags for mailing list mode
If a user has a tag muted, don't send them emails about that tag.
We've done this forever for categories so it makes sense to do it
for tags too.
2020-02-19 15:14:42 -05:00
Roman Rizzi
9441362c72
FEATURE: Support uploading a csv with either user emails or usernames (#8971) 2020-02-18 10:53:12 -03:00
Roman Rizzi
2ee6a615b7
FEATURE: Send suspect users to the review queue (#8811) 2020-01-29 15:38:27 -03:00
Martin Brennan
ab3bda6cd0
FIX: Mitigate issue where legacy pre-secure hotlinked media would not be redownloaded (#8802)
Basically, say you had already downloaded a certain image from a certain URL
using pull_hotlinked_images and the onebox. The upload would be stored
by its sha as an upload record. Whenever you linked to the same URL again
in a post (e.g. in our case an og:image on review.discourse) we would
would reuse the original upload record because of the sha1.

However when you turned on secure media this could cause problems as
the first post that uses that upload after secure media is enabled
will set the access control post for the upload to the new post.
Then if the post is deleted every single onebox/link to that same image
URL will fail forever with 403 as the secure-media-uploads URL fails
if the access control post has been deleted.

To fix this when cooking posts and pulling hotlinked images, we only
allow using an original upload by URL if its access control post
matches the current post, and if the original_sha1 is filled in,
meaning it was uploaded AFTER secure media was enabled. otherwise
we just redownload the media again to be safe, as the URL will always
be new then.
2020-01-29 10:11:38 +10:00
Krzysztof Kotlarek
20e7fb1c95
FIX: correct notification when tag or category is added (#8801)
Regression was created here:
https://github.com/discourse/discourse/pull/8750

When tag or category is added and the user is watching that category/tag
we changed notification type to `edited` instead of `new post`.

However, the logic here should be a little bit more sophisticated.

If the user has already seen the post, notification should be `edited`.

However, when user hasn't yet seen post, notification should be "new
reply". The case for that is when for example topic is under private
category and set for publishing later. In that case, we modify an
existing topic, however, for a user, it is like a new post.

Discussion on meta:
https://meta.discourse.org/t/publication-of-timed-topics-dont-trigger-new-topic-notifications/139335/13
2020-01-29 11:03:47 +11:00
Gerhard Schlager
9621af2214 FIX: Award "User of the month" badge at the beginning of month
Previously the badge was granted one month after the last time the badge was granted. The exact date shifted by one day each month. The new logic tries to grant the badge always at the beginning of a new month by looking at new users of the previous month. The "granted at" date is set to the end of the previous month.
2020-01-28 22:59:13 +01:00
David Taylor
a9d0d55817 FEATURE: Add message to log when admins are automatically deactivated 2020-01-28 12:16:24 +00:00
Dan Ungureanu
5060811464
FIX: Export all category names in user archives (#8790) 2020-01-28 12:52:07 +02:00
Martin Brennan
45b37a8bd1
FIX: Resolve pull hotlinked image and broken link issues for secure media URLs (#8777)
When pull_hotlinked_images tried to run on posts with secure media (which had already been downloaded from external sources) we were getting a 404 when trying to download the image because the secure endpoint doesn't allow anon downloads.

Also, we were getting into an infinite loop of pull_hotlinked_images because the job didn't consider the secure media URLs as "downloaded" already so it kept trying to download them over and over.

In this PR I have also refactored secure-media-upload URL checks and mutations into single source of truth in Upload, adding a SECURE_MEDIA_ROUTE constant to check URLs against too.
2020-01-24 11:59:30 +10:00
Bianca Nenciu
0a27086764
FEATURE: Export all types of reports (#8748)
There is a single stacked_chart which was not exportable
2020-01-21 18:43:19 +02:00
Krzysztof Kotlarek
0420be88a6
FIX: when tag or category is added notify users that topic was modified (#8750)
There is a feature, that when tag or category is added to the topic,
customers who are watching that category or tag are notified.

The problem is that it is using default notification type "new post"

It would be better to use "new post" only when there really is a new
post and "edited" when categories or tags were modified.
2020-01-21 08:41:13 +11:00
Gerhard Schlager
a2fd8ac990 DEV: Add spec for disabled UpdateS3Inventory job
Follow-up to c351ffe580
2020-01-16 13:57:57 +01:00
Martin Brennan
7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
David Taylor
4fdfc2665d FIX: Update featured badge ranking when mass-awarding badges
Follow-up to cff6e941de
2020-01-14 14:39:20 +00:00
Roman Rizzi
d69c5eebcf
Feature: Mass award badge (#8694)
* UI: Mass grant a badge from the admin ui

* Send the uploaded CSV and badge ID to the backend

* Read the CSV and grant badge in batches

* UX: Communicate the result to the user

* Don't award if badge is disabled

* Create a 'send_notification' method to remove duplicated code, slightly shrink badge image. Replace router transition with href.

* Dynamically discover current route
2020-01-13 11:20:26 -03:00
Vinoth Kannan
3b7f5db5ba
FIX: parallel spec system needs a dedicated upload folder for each worker. (#8547) 2019-12-18 11:21:57 +05:30
Joffrey JAFFEUX
0d3d2c43a0
DEV: s/\$redis/Discourse\.redis (#8431)
This commit also adds a rubocop rule to prevent global variables.
2019-12-03 10:05:53 +01:00
Martin Brennan
02cb01406e
FIX: Allow secure uploads if global s3 setting active and enable_s3_uploads validations (#8373)
The secure media functionality relied on `SiteSetting.enable_s3_uploads?` which, as we found in dev, did not take into account global S3 settings via `GlobalSetting.use_s3?`. We now use `SiteSetting.Upload.enable_s3_uploads` instead to be more consistent.

Also, we now validate `enable_s3_uploads` changes, because if `GlobalSetting.use_s3?` is true users should NOT be enabling S3 uploads manually.
2019-11-20 07:46:44 +10:00
David Taylor
836b3f4d82
FIX: Do not deactivate admin accounts with recent posts or api keys (#8342)
This prevents 'bot' users being deactivated
2019-11-12 16:56:01 +00:00
Mark VanLandingham
3dd2f2f701
DEV: Remove RSS feed polling in favor of plugin (#8233) 2019-11-12 09:49:02 -06:00
Krzysztof Kotlarek
69266f60ed FIX: tag and category watchers regression (#8336)
I made a regression here 17366d3bcc (diff-ddeebb36d131f89ca91be9d04c2baefaR10)

When the tag is added, people watching specific tag are notified but also people watching specific category.

Therefore, `notify_post_users` should accept options who should be notified.

So when `category` is added to the topic, users watching topic and users watching category are notified.

When `tag` is added to the topic, users watching topic and users watching tag are notified

Finally, when a new post is created, everybody is notified, topic watchers, category watchers, tag watchers.
2019-11-12 16:44:46 +11:00
Martin Brennan
56d3e29a69
FIX: Badge and user title interaction fixes (#8282)
* Fix user title logic when badge name customized
* Fix an issue where a user's title was not considered a badge granted title when the user used a badge for their title and the badge name was customized. this affected the effectiveness of revoke_ungranted_titles! which only operates on badge_granted_titles.
* When a user's title is set now it is considered a badge_granted_title if the badge name OR the badge custom name from TranslationOverride is the same as the title
* When a user's badge is revoked we now also revoke their title if the user's title matches the badge name OR the badge custom name from TranslationOverride
* Add a user history log when the title is revoked to remove confusion about why titles are revoked
* Add granted_title_badge_id to user_profile, now when we set badge_granted_title on a user profile when updating a user's title based on a badge, we also remember which badge matched the title
* When badge name (or custom text) changes update titles of users in a background job
* When the name of a badge changes, or in the case of system badges when their custom translation text changes, then we need to update the title of all corresponding users who have a badge_granted_title and matching granted_title_badge_id. In the case of system badges we need to first get the proper badge ID based on the translation key e.g. badges.regular.name
* Add migration to backfill all granted_title_badge_ids for both normal badge name titles and titles using custom badge text.
2019-11-08 15:34:24 +10:00
Krzysztof Kotlarek
17366d3bcc
FEATURE: notify tag watchers when tag was added to post (#8299)
Issue was mentioned in this [meta topic](https://meta.discourse.org/t/send-a-notification-to-watching-users-when-adding-tag/125314)

It is working well when category is changed because NotifyCategoryChange job already got that code:
```
if post&.topic&.visible?
  post_alerter = PostAlerter.new
  post_alerter.notify_post_users(post, User.where(id: args[:notified_user_ids]))
  post_alerter.notify_first_post_watchers(post, post_alerter.category_watchers(post.topic))
end
```

For NotifyTagChange job notify post users were missing so it worked only when your notification was set to `watching first post`
2019-11-07 08:20:15 +11:00
Arpit Jalan
2ae71b7a87 FEATURE: allow sending bulk invites to staged users 2019-10-30 11:40:03 +05:30
Sam Saffron
3d85cc1e69 PERF: run expensive clean up uploads less frequently
Previously every hour we would run a full scan of the entire DB searching
for expired uploads that need to be moved to the tombstone folder.

This commit amends it so we only run the job 2 times per clean_orpha_uploads_grace_period_hours

There is a upper bound of 7 days so even if the grace period is set really
high it will still run at least once a week.

By default we have a 48 grace period so this amends it to run this cleanup
daily instead of hourly. This eliminates 23 times we run this ultra expensive
query.
2019-10-28 11:14:52 +11:00
Dan Ungureanu
96b8710b39
DEV: Fix heisentest (ensure that user ID really does not exist). 2019-10-14 12:25:43 +03:00
Sam Saffron
78c5adc093 DEV: stop hardcoding ids in specs
hardcoded ids are unreliable for running specs and lead to erratic failures
2019-10-08 09:37:07 +11:00
Krzysztof Kotlarek
427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Robin Ward
d251f12c9c Tweak calculation for reviewable sensitivities/priorities
Previously, calculating thresholds for reviewables was done based on the
50th and 85th percentile across all reviewables. However, many forum
owners provided feedback that these thresholds were too easy to hit, in
particular when it came to auto hiding content.

The calculation has been adjusted to base the priorities on reviewables
that have a minimum of 2 scores (flags). This should push the amount of
flags required to hide something higher then before.
2019-09-19 14:07:56 -04:00
Robin Ward
d5b52abf2f FIX: Require a min amount of reviewables before calculating thresholds
On forums with very few flags you don't want to calculate averages
because they won't be very useful. Stick with the defaults until we hit
15 reviewables at least.
2019-09-19 13:42:50 -04:00
Régis Hanol
aa511c5b59 FIX: support <img> in code blocks when inlining uploads
Simpler code is better :)

Also added moar specs to ensure <img> tag inside code blocks are properly ignored.
2019-09-12 21:25:14 +02:00
David Taylor
67a98946b8 FIX: Do not log 'pull_hotlinked_images' edits in the staff action log 2019-09-12 15:55:45 +01:00
Joffrey JAFFEUX
a25869969a
DEV: adds event hook when add/remove user to group (#8038) 2019-09-10 11:58:08 -05:00
David Taylor
e2449f9f23 Revert "Revert "Revert "FIX: Heartbeat check per sidekiq process (#7873)"""
This reverts commit c3497559be.
2019-08-30 11:26:16 +01:00
Sam Saffron
c3497559be Revert "Revert "FIX: Heartbeat check per sidekiq process (#7873)""
This reverts commit e805d44965.
We now have mechanisms in place to ensure heartbeat will always
be scheduled even if the scheduler is overloaded per: 098f938b
2019-08-30 10:12:10 +10:00
OsamaSayegh
e805d44965 Revert "FIX: Heartbeat check per sidekiq process (#7873)"
This reverts commit 340855da55.
2019-08-27 11:56:23 +00:00
Osama Sayegh
340855da55
FIX: Heartbeat check per sidekiq process (#7873)
* FIX: Heartbeat check per sidekiq process

* Rename method

* Remove heartbeat queues of previous bootups

* Regis feedback

* Refactor before_start

* Update lib/demon/sidekiq.rb

Co-Authored-By: Régis Hanol <regis@hanol.fr>

* Update lib/demon/sidekiq.rb

Co-Authored-By: Régis Hanol <regis@hanol.fr>

* Expire redis keys after 3600 seconds

* Don't use redis to store the list of queues
2019-08-26 09:33:49 +03:00
Gerhard Schlager
00b75b4f4e FIX: Don't try to delete staged, unused admins and mods 2019-08-21 15:29:51 +02:00
Arpit Jalan
0603636cea FIX: include default label when exporting reports 2019-07-26 12:57:13 +05:30
Arpit Jalan
eb9155f3fe
FEATURE: send max 200 emails every minute for bulk invites (#7875)
DEV: deprecate `invite.via_email` in favor of `invite.emailed_status`

This commit adds a new column `emailed_status` in `invites` table for
 tracking email sending status.
 0 - not required
 1 - pending
 2 - bulk pending
 3 - sending
 4 - sent

For normal email invites, invite record is created with emailed_status
 set to 'pending'.

When bulk invites are sent invite record is created with emailed_status
 set to 'bulk pending'.

For invites that generates link, invite record is created with
 emailed_status set to 'not required'.

When invite email is in queue emailed_status is updated to 'sending'

Once the email is sent via `InviteEmail` job the invite emailed_status
 is updated to 'sent'.
2019-07-19 11:29:12 +05:30
Gerhard Schlager
26b843e5e8 Correct typo in spec name 2019-07-04 11:37:37 +02:00
Gerhard Schlager
d513c28e3b FIX: Don't send notification email when user isn't allowed to see topic 2019-07-01 14:03:03 +02:00
Bianca Nenciu
b2eb0f4ad6 FEATURE: Export any type of report supporting table mode. (#7662) 2019-06-28 08:50:31 +02:00
Guo Xiang Tan
8deaef3872 FIX: Don't replace img tags within anchor tags with markdown format.
Follow up to 9a25b0d614.
2019-06-21 12:32:02 +08:00
Guo Xiang Tan
9a25b0d614 FIX: Edge case with anchor tag in InlineUploads. 2019-06-21 09:55:56 +08:00
Vinoth Kannan
9f0574dcfd SPEC: Update webhook event attributes even when an error raised
1e3cb7575d
2019-06-21 00:45:35 +05:30
Guo Xiang Tan
c9db897777 FIX: Remove onebox src from Jobs::PullHotlinkedImages.
The test that was added is incorrect because the post was not cooked.
2019-06-14 09:21:25 +08:00
Guo Xiang Tan
40e67971f9 DEV: Add spec for Email::Sender for upload links in plain text emails. 2019-06-11 16:02:24 +08:00
Guo Xiang Tan
fb0a655e8a FEATURE: Update pull hotlinked images to use Upload#short_url. 2019-06-11 15:17:29 +08:00
Guo Xiang Tan
9d0fba64c0 FIX: Use attachment format in user export system post take 2. 2019-06-11 12:15:11 +08:00
David Taylor
54afa314fb FIX: Do not download emojis in pull_hotlinked_images 2019-06-07 13:00:52 +01:00
Guo Xiang Tan
3b4432f258 DEV: Refactor specs for Jobs::PullHotlinkedImages. 2019-06-06 14:56:21 +08:00
Arpit Jalan
e7fe7010b8
FIX: use hijack for processing bulk invites (#7679)
FIX: do not store bulk invite CSV file on server
2019-06-04 20:19:46 +05:30
Sam Saffron
74c4f926fc FIX: drop deleted posts from search index
This does two things

1. Our "index grace period" has been wound down to 1 day, there is no point
keeping a bloated index for a week, usually when people delete stuff they
mean for it to be removed

2. We were never dropping deleted posts from the index, only posts from
deleted topics

These changes speed up search a tiny bit and reduce background work.
2019-06-04 17:19:59 +10:00
Arpit Jalan
028121b95b
FIX: delete system generated message when user_export record is deleted (#7595)
FIX: system generated message for user export should be closed by default
2019-05-28 16:38:41 +05:30
Robin Ward
b85fa62882 FIX: Don't send replies to flags that are ignored by expiry
The `AutoQueueHandler` will ignore really old flags. In that case, don't
notify the user that the moderator is looking into it. They probably
never saw it because it didn't meet the reviewable minimum priority.
2019-05-27 17:24:36 -04:00
Robin Ward
e74cd54fc6 REFACTOR: Replace score bonuses with low/med/high priorities
We removed score from the UX so it makes more sense to have sites set
priorities instead of score bonuses.
2019-05-23 11:54:45 -04:00
Guo Xiang Tan
df1e6eed5a FIX: Pull hotlinked images for lightbox links as well. 2019-05-23 15:44:37 +08:00
Robin Ward
225593d445 FIX: Small tweak to percentiles based on feedback 2019-05-16 13:45:36 -04:00
Gerhard Schlager
bbab60fa9b FIX: Don’t try to delete inactive moderators
follow-up to 6d77156a
2019-05-13 21:44:03 +02:00
Guo Xiang Tan
8165ceb320 Make rubocop happy. 2019-05-13 09:55:44 +08:00
Daniel Waterworth
54cfc48975 Use prefabricated users in bulk_invite_spec.rb 2019-05-10 08:34:04 -04:00
Robin Ward
5af7c90bab FEATURE: Hide Reviewable scores, change score filter to Priority
We found score hard to understand. It is still there behind the scenes
for sorting purposes, but it is no longer shown.

You can now filter by minimum priority (low, med, high) instead of
score.
2019-05-07 14:05:23 -04:00
Daniel Waterworth
e219588142 DEV: Prefabrication (test optimization) (#7414)
* Introduced fab!, a helper that creates database state for a group

It's almost identical to let_it_be, except:

 1. It creates a new object for each test by default,
 2. You can disable it using PREFABRICATION=0
2019-05-07 13:12:20 +10:00
Guo Xiang Tan
152238b4cf DEV: Prefer public_send over send. 2019-05-07 09:33:21 +08:00
Sam Saffron
9be70a22cd DEV: introduce new API to look up dynamic site setting
This removes all uses of both `send` and `public_send` from consumers of
SiteSetting and instead introduces a `get` helper for dynamic lookup

This leads to much cleaner and safer code long term as we are always explicit
to test that a site setting is really there before sending an arbitrary
string to the class

It also removes a couple of risky stubs from the auth provider test
2019-05-07 11:00:30 +10:00
Guo Xiang Tan
24347ace10 FIX: Properly associate user_profiles background urls via upload id.
`Upload#url` is more likely and can change from time to time. When it
does changes, we don't want to have to look through multiple tables to
ensure that the URLs are all up to date. Instead, we simply associate
uploads properly to `UserProfile` so that it does not have to replicate
the URLs in the table.
2019-05-02 14:58:24 +08:00
Robin Ward
404b35bd04 FEATURE: Category Reviewable by Group
Allow a group to review content in a particular category.
2019-04-30 15:23:06 -04:00
Sam Saffron
4ea21fa2d0 DEV: use #frozen_string_literal: true on all spec
This change both speeds up specs (less strings to allocate) and helps catch
cases where methods in Discourse are mutating inputs.

Overall we will be migrating everything to use #frozen_string_literal: true
it will take a while, but this is the first and safest move in this direction
2019-04-30 10:27:42 +10:00
Sam Saffron
45285f1477 DEV: remove update_attributes which is deprecated in Rails 6
See: https://github.com/rails/rails/pull/31998

update_attributes is a relic of the past, it should no longer be used.
2019-04-29 17:32:25 +10:00
Daniel Waterworth
ad44243a57 Removed unused let blocks (#7446)
The bodies of these blocks were never evaluated.
2019-04-29 15:08:56 +08:00
Gerhard Schlager
6d77156a94 FIX: Don't try to delete inactive admins 2019-04-25 11:59:50 +02:00
Robin Ward
70097966ed FIX: Don't add the reviewable score twice 2019-04-23 15:46:18 -04:00
Guo Xiang Tan
d07605d885 DEV: Fix failing spec. 2019-04-23 18:25:28 +08:00
Guo Xiang Tan
97dcd50e51 DEV: Fix incorrect spec file names.
Co-authored-by: Daniel Waterworth <me@danielwaterworth.com>
2019-04-23 18:03:55 +08:00
Tarek Khalil
6e46197bc8
FIX: Disable webhooks on 410 and 404 HTTP responses (#7392)
FIX: Disable webhooks on 410 and 404 HTTP responses (#7392)
2019-04-18 12:36:37 +01:00
Tarek Khalil
02a9429c38
REFACTOR: Quick refactor of the webhook event emitter job (#7385)
* REFACTOR: Quick refactor of the webhook event emitter job
2019-04-17 10:03:23 +01:00
Robin Ward
ba6d4b2a8d FIX: Better handling for toggling must_approve_users
If you turn it on now, default all users to approved since they were
previously. Also support approving a user that doesn't have a reviewable
record (it will be created first.)

This also includes a refactor to move class method calls to
`DiscourseEvent` into an initializer. Otherwise the load order of
classes makes a difference in the test environment and some settings
might be triggered and others not, randomly.
2019-04-16 15:56:35 -04:00
Guo Xiang Tan
784940bea0 Fix the build. 2019-04-16 15:47:16 +08:00
Guo Xiang Tan
d6e45864ce Fix failing spec. 2019-04-16 09:59:35 +08:00
Guo Xiang Tan
25514550f0 FIX: Bulk invite should skip invite and add existing users to groups. 2019-04-16 09:42:25 +08:00
Guo Xiang Tan
6e6cdfaf80 FIX: Bulk invite should not add users to automatic groups.
* Also checks that the user creating the bulk invite has permission.
2019-04-16 09:42:25 +08:00
Arpit Jalan
594674703c
FIX: properly log webhook errors in UI on rescue (#7376) 2019-04-15 12:19:48 +05:30
Guo Xiang Tan
afbf852e0e DEV: Add specs for cleaning up uploads in non pending reviable posts.
Follow up to f7ebfb1acc
2019-04-15 09:20:34 +08:00
David Taylor
268d4d4c82
FEATURE: Multiple SCSS file support for themes (#7351)
Theme developers can include any number of scss files within the /scss/ directory of a theme. These can then be imported from the main common/desktop/mobile scss.
2019-04-12 11:36:08 +01:00
Robin Ward
d21dd521d2 FIX: Don't try to create a reviewable user twice 2019-04-11 12:30:54 -04:00
Robin Ward
331a809738 FEATURE: Display the reason for many reviewable items
Queued Posts and Users will now display a reason why they are in the
review queue.
2019-04-11 12:04:45 -04:00
Arpit Jalan
318aba1f53 fix flaky spec 2019-04-11 12:25:41 +05:30
Régis Hanol
f665862ee8 SPEC: ensure we don't send activation reminder to staged users 2019-04-10 17:40:11 +02:00
Régis Hanol
4fe562ece5 fix build 2019-04-10 17:37:38 +02:00
Bianca Nenciu
3d545d66df FEATURE: Send user activation reminders. (#7280) 2019-04-10 16:53:52 +02:00
Tarek Khalil
442fb2facb FEATURE: Remove ignore feature SiteSetting and enable ignore by default (#7349) 2019-04-10 12:54:59 +02:00
Vinoth Kannan
914ada1c74 DEV: convert scheduled job EnsurePostUploadsExistence into a rake task 2019-04-09 02:07:35 +05:30
David Taylor
f524f8f811
Remove Yahoo login support from core and deprecate OpenID2.0 (#7310)
- Plugin developers using OpenID2.0 should migrate to OAuth2 or OIDC. OpenID2.0 APIs will be removed in v2.4.0

- For sites requiring Yahoo login, it can be implemented using the OpenID Connect plugin: https://meta.discourse.org/t/103632

For more information, see https://meta.discourse.org/t/113249
2019-04-08 10:38:25 +01:00
Guo Xiang Tan
108c231d1c FIX: Clean up topic_search_data of trashed topics.
This keeps the index and table smaller.
2019-04-08 16:53:39 +08:00
Guo Xiang Tan
c4997ce85f FIX: Don't try and reindex posts that have been trashed. 2019-04-08 16:38:43 +08:00
Robin Ward
c9ddc6ce30 FIX: Make sure the site setting works and fix build 2019-04-04 17:44:10 -04:00
Vinoth Kannan
64ec4a6a1b fix the EnsurePostUploadsExistence job spec 2019-04-04 21:53:58 +05:30
Vinoth Kannan
b3fb0a7039 FEATURE: ensure consistency of post uploads in cooked content 2019-04-04 02:23:28 +05:30
Robin Ward
111a502231 FIX: Deleting Users should work nicely with Reviewable Users
"Rejecting" a user in the queue is equivalent to deleting them, which
would then making it impossible to review rejected users. Now we store
information about the user in the payload so if they are deleted things
still display in the Rejected view.

Secondly, if a user is destroyed outside of the review queue, it will
now automatically "Reject" that queue item.
2019-04-03 16:42:39 -04:00
Guo Xiang Tan
d151425353
PERF: Delete search data of posts from trashed topics periodically. (#7302)
This keeps both the index and table smaller.
2019-04-03 10:10:41 +08:00
Guo Xiang Tan
d8704c11ca PERF: Better use of index when queueing a topci for search reindex.
Also move `Search::INDEX_VERSION` to `SearchIndexer` which is where the
version is actually being used.
2019-04-02 09:53:37 +08:00
Guo Xiang Tan
aa2311a7b0 FIX: Don't reindex posts belonging to a deleted topic for search.
Posts belonging to a deleted topic can't be index for search so we need
to avoid loading those post ids.
2019-04-02 07:36:53 +08:00
Guo Xiang Tan
3fc5dbb045 FIX: Don't attempt to reindex posts that have an empty raw.
If the post ids keep loading, we might end up in a situations where
we're always loading the same post ids over and over again without
indexing anything new.

Follow up to daeda80ada.
2019-04-02 07:13:33 +08:00
Robin Ward
76669bb5a6 FIX: Don't refer to pending review items as flags
They could be queued posts or users, and the notice should reflect that
properly.
2019-04-01 14:46:56 -04:00
Guo Xiang Tan
daeda80ada
FIX: Don't index posts with empty Post#raw for search. (#7263)
* DEV: Remove unnecessary join in `Jobs::ReindexSearch`.

* FIX: Don't index posts with empty `Post#raw` for search.
2019-04-01 10:06:27 +08:00
Guo Xiang Tan
730ebdfcba DEV: Refactor Jobs::EmitWebHookEvent specs. 2019-04-01 09:46:39 +08:00
Vinoth Kannan
904ba266cf SPEC: Add test case in emit_web_hook_event_spec for commit 4c6bfb9 2019-03-31 16:28:40 +05:30
Tarek Khalil
b1cb95fc23
FEATURE: Introduce ignore duration selection (#7266)
* FEATURE: Introducing new UI for tracking User's ignored or muted states
2019-03-29 10:14:53 +00:00
Robin Ward
118f98b6ed Linting error 2019-03-28 12:52:46 -04:00
Robin Ward
b58867b6e9 FEATURE: New 'Reviewable' model to make reviewable items generic
Includes support for flags, reviewable users and queued posts, with REST API
backwards compatibility.

Co-Authored-By: romanrizzi <romanalejandro@gmail.com>
Co-Authored-By: jjaffeux <j.jaffeux@gmail.com>
2019-03-28 12:45:10 -04:00
Sam Saffron
9ebabc1de8 FEATURE: unconditionally update Topic updated_at when posts change in topic
Previously we would bypass touching `Topic.updated_at` for whispers and post
recovery / deletions.

This meant that certain types of caching can not be done where we rely on
this information for cache accuracy.

For example if we know we have zero unread topics as of yesterday and whisper
is made I need to bump this date so the cache remains accurate

This is only half of a larger change but provides the groundwork.

Confirmed none of our serializers leak out Topic.updated_at so this is safe
spot for this info

At the moment edits still do not change this but it is not relevant for the
unread cache.

This commit also cleans up some specs to use the new `eq_time` matcher for
millisecond fidelity comparison of times

Previously `freeze_time` would fudge this which is not that clean.
2019-03-28 17:28:01 +11:00
David Taylor
95d5819218 FIX: Re-download hotlinked optimized images (#7249)
* FIX: Download local images, even if download remote is disabled
2019-03-27 21:31:12 +01:00
David Taylor
bb3f8e32e5 DEV: Run pull_hotlinked_images onebox specs without synchronous jobs
This was changed in fa5a1586, and caused a mutex to lock up, adding 60 seconds to the test suite.
2019-03-22 19:27:37 +00:00
Neil Lalonde
399e937a38 FIX: prevent sending multiple summary emails due to Sidekiq delays 2019-03-22 12:34:34 -04:00
David Taylor
a9d5ffbe3d FIX: Prevent critical emails bypassing disable, and improve email test logic
- The test_email job is removed, because it was always being run synchronously (not in sidekiq)
- 34b29f62 added a bypass for critical emails, to match the spec. This removes the bypass, and removes the spec.
- This adapts the specs for 72ffabf6, so that they check for emails being sent
- This reimplements c2797921, allowing test emails to be sent even when emails are disabled
2019-03-22 17:28:43 +08:00
Tarek Khalil
a31a35b334 FEATURE: Ignored user notification behaviour should be as a muted user (#7227) 2019-03-21 12:15:34 +01:00
Tarek Khalil
3b59ff0d02 [FEATURE] Disallow ignoring self, admins or moderators users (#7202) 2019-03-20 11:18:46 +01:00
Tarek Khalil
fed2dd9148 FEATURE: Add scheduled job to purge expired ignored users (#7211) 2019-03-20 11:01:43 +01:00
Guo Xiang Tan
4020c87680 DEV: Refactor tests for Jobs::CleanUpInactiveUsers.
* Remove use of 0 in favor of `TrustLevel.levels[:newuser]`.
* Consolidate two tests into a single one.
* Test that disabling the feature works.
* Avoid loading full ActiveRecord object in test when we only need to
know the existence of the record.
2019-03-19 09:57:21 +08:00
Bianca Nenciu
2347661a74 FEATURE: Clean up inactive users. (#7172) 2019-03-18 16:25:15 +01:00
Penar Musaraj
9334d2f4f7
FEATURE: add more granular user option levels for email notifications (#7143)
Migrates email user options to a new data structure, where `email_always`, `email_direct` and `email_private_messages` are replace by

* `email_messages_level`, with options: `always`, `only_when_away` and `never` (defaults to `always`)
* `email_level`, with options: `always`, `only_when_away` and `never` (defaults to `only_when_away`)
2019-03-15 10:55:11 -04:00
Tarek Khalil
bd6d31c9ec
FEATURE: Add IgnoredUsersSummary daily job (#7144)
* FEATURE: Add `IgnoredUsersSummary` daily job

## Why?

This is part of the [Ability to ignore a user feature](https://meta.discourse.org/t/ability-to-ignore-a-user/110254/8).

We want to:

1. Send an automatic group PM that goes out to moderators
2. When {x} users have Ignored the same user, threshold defined by a site setting, default of 5
3. Only send this message every X days which is defined by another site setting
2019-03-14 22:51:43 +00:00
Robin Ward
fa5a158683 REFACTOR: Move queue_jobs out of SiteSetting
It is not a setting, and only relevant in specs. The new API is:

```
Jobs.run_later!        # jobs will be thrown on the queue
Jobs.run_immediately!  # jobs will run right away, avoid the queue
```
2019-03-14 10:47:38 -04:00
Guo Xiang Tan
b0c8fdd7da FIX: Properly support defaults for upload site settings. 2019-03-13 16:36:57 +08:00
Penar Musaraj
95532814df DEV: Make Rubocop happy 2019-03-11 22:33:24 -04:00
Guo Xiang Tan
34b29f62db DEV: Remove the use of stubs and mocks in Jobs::UserEmail tests.
We can only be sure that an email is sent when we get a mailer in
`ActionMailer::Deliveries`. A couple of tests were actually incorrect
because it didn't flow through our email sender where there are more
conditions in determining whether an email is sent or not.
2019-03-12 09:39:16 +08:00
Robin Ward
d1d9a4f128 Add new run_jobs_synchronously! helper for tests
Previously if you wanted to have jobs execute in test mode, you'd have
to do `SiteSetting.queue_jobs = false`, because the opposite of queue
is to execute.

I found this very confusing, so I created a test helper called
`run_jobs_synchronously!` which is much more clear about what it does.
2019-03-11 16:58:35 -04:00
David Taylor
fc7938f7e0
REFACTOR: Migrate GoogleOAuth2Authenticator to use ManagedAuthenticator (#7120)
https://meta.discourse.org/t/future-social-authentication-improvements/94691/3
2019-03-07 11:31:04 +00:00
Joffrey JAFFEUX
703c724cf3
REFACTOR: Migrate InstagramAuthenticator to use ManagedAuthenticator (#7081) 2019-03-04 14:54:28 +01:00
Régis Hanol
326d892f5e Aadd 'secondary_emails' field in users export
FIX: escape_comma wasn't working in CSV exports
FIX: group_names field wasn't properly serialized
2019-02-27 10:12:20 +01:00
Vinoth Kannan
484bd82278 FIX: Add onceoff job to remove double quotes from s3 etags 2019-02-14 05:19:41 +05:30
Vinoth Kannan
ba9cc83d4c FIX: Destination prefix in S3 inventory configuration is incorrect 2019-02-06 20:51:28 +05:30
Vinoth Kannan
ff12c4b2d4 FIX: Bucket name is missing in S3 inventory data path 2019-02-06 19:16:08 +05:30
Vinoth Kannan
b4f713ca52
FEATURE: Use amazon s3 inventory to manage upload stats (#6867) 2019-02-01 10:10:48 +05:30
Robin Ward
78ddc82952 FIX: Respect min_flags_staff_visibility for new flags too
There was a situation where if:

* There were new flags to review that met the visibility threshold

AND

* There were old flags that *didn't* meet the threshold

THEN

a pending flags notification would be sent out. This fixes that case.
Staff should not be notified of flags if they do not meet the threshold
and are old.
2019-01-25 11:27:43 -05:00
Robin Ward
96b2585a91 REFACTOR: Remove unncessary stubs from pending flags reminder
They seem to be calculated fine by the application, and stubbing
makes the tests more brittle and prone to regression.
2019-01-24 13:45:58 -05:00
Robin Ward
f32de88dfc FIX: Don't notify of pending flags if min_flags_staff_visibility not met 2019-01-22 11:01:18 -05:00
Sam
384135845b FEATURE: introduce ultra_low priority queue
This commit introduces an ultra low priority queue for post rebakes. This
way rebakes can never interfere with regular sidekiq processing for cases
where we perform a large scale rebake.

Additionally it allows Post.rebake_old to be run with rate_limiter: false
to avoid triggering the limiter when rebaking. This is handy for cases
where you want to just force the full rebake and not wait for it to trickle
2019-01-17 14:53:19 +11:00
Sam
824c3420e9 DEV: make Jobs.enqueue tests less fragile
Previously we depended on non Sidekiq specific mocking which is not the
official way of testing Sidekiq, this made these tests very fragile

New testing is more robust and complete
2019-01-09 09:51:11 +11:00
Sam
e08a3f719c FEATURE: push post rebake regular task to low priority queue
This allows us to run regular rebakes without starving the normal queue.

It additionally adds the ability to specify queue with `Jobs.enqueue` so
we can specifically queue a job with lower priority using the `queue` arg.
2019-01-09 08:57:20 +11:00
David Taylor
5bf16d7d10 FEATURE: Topic timer for bumping a topic in the future 2019-01-04 13:08:04 +00:00
David Taylor
1960236822
FIX: Suspicious login detection (#6772) 2018-12-14 16:30:34 +00:00
Neil Lalonde
3b76f19668 FIX: invalidating inactive admin emails should mark them as not active 2018-12-12 17:07:49 -05:00
Neil Lalonde
a1db15fead FEATURE: require admins to re-validate their email addresses if they haven't been seen for a number of days, configurable with the invalidate_inactive_admin_email_after_days site setting. Social logins are also revoked. Default is 365 days. 2018-12-12 15:32:38 -05:00
David Taylor
3fedb2ad20 DEV: Style and performance improvements
Follow-up from 9db8291
2018-12-11 09:58:20 +00:00
David Taylor
9db829134c
FIX: Use database to persist metadata during social registration (#6750)
Previously was using the cookie_store, which is limited to 4kb. This caused issues for providers sending large volumes of metadata about a user.
2018-12-10 15:10:06 +00:00
Rishabh
da41a515cd Add test to check if posts are auto rebaked by PeriodicalUpdates job when gravatar download is disabled
follow-up for 8e307e633e
2018-12-10 11:10:57 +11:00
Vinoth Kannan
d33d031742
FEATURE: Filter topic and post web hook events by tags (#6726)
* FEATURE: Filter topic and post web hook events by tags

* Add a spec test with unmatched tags
2018-12-05 14:44:06 +05:30
Guo Xiang Tan
eecd1a7d8c FIX: Jobs::CleanUpUploads fails when value of upload data_type is an empty string. 2018-11-30 10:46:39 +08:00
Penar Musaraj
03deda2147
Upgrade to FontAwesome 5 (take two) (#6673)
* Add missing icons to set

* Revert FA5 revert

 This reverts commit 42572ff

* use new SVG syntax in locales

* Noscript page changes (remove login button, center "powered by" footer text)

* Cast wider net for SVG icons in settings

- include any _icon setting for SVG registry (offers better support for plugin settings)

- let themes store multiple pipe-delimited icons in a setting

- also replaces broken onebox image icon with SVG reference in cooked post processor

* interpolate icons in locales

* Fix composer whisper icon alignment

* Add support for stacked icons

* SECURITY: enforce hostname to match discourse hostname

This ensures that the hostname rails uses for various helpers always matches
the Discourse hostname

* load SVG sprite with pre-initializers

* FIX: enable caching on SVG sprites

* PERF: use JSONP for SVG sprites so they are served from CDN

This avoids needing to deal with CORS for loading of the SVG

Note, added the svg- prefix to the filename so we can quickly tell in
dev tools what the file is

* Add missing SVG sprite JSONP script to CSP

* Upgrade to FA 5.5.0

* Add support for all FA4.7 icons

- adds complete frontend and backend for renamed FA4.7 icons

- improves performance of SvgSprite.bundle and SvgSprite.all_icons

* Fix group avatar flair preview

- adds an endpoint at /svg-sprites/search/:keyword

- adds frontend ajax call that pulls icon in avatar flair preview even when it is not in subset

* Remove FA 4.7 font files
2018-11-26 16:49:57 -05:00
Guo Xiang Tan
81b3bdaabd FIX: Remove site settings override for deprecated url site settings. 2018-11-20 11:42:39 +08:00
Guo Xiang Tan
9e86b425bc FIX: Job to clean up old URL settings when new setting has been set.
Related to 44391ee8ab
2018-11-16 09:33:31 +08:00
Guo Xiang Tan
df111259fe More URL site settings into a onceoff job.
* Doing it in a post migration was a bad idea
  because the migration will fail if the site
  is down while trying to download uploads
  which points to the instance. This mainly
  affects self-hosters using `discourse_docker`
  where `./launcher rebuild` will take the
  existing container down.
2018-11-14 20:29:20 +08:00
Bianca Nenciu
fce0a0ccc8 FEATURE: Compute distance between logins to generate login alerts. (#6562) 2018-11-14 13:26:47 +01:00
Guo Xiang Tan
44391ee8ab
FEATURE: Upload Site Settings. (#6573) 2018-11-14 15:03:02 +08:00
Guo Xiang Tan
7b44339529 FIX: Prevent uploads used in site settings from being deleted. 2018-11-13 09:15:16 +08:00
Gerhard Schlager
24e5be3f0c FIX: Relative links in translations should work with subfolder 2018-11-08 23:31:05 +00:00
Bianca Nenciu
2070edf889 FIX: Clarify User.group_locked_trust_level.
* Rename User.group_locked_trust_level to User.group_granted_trust_level.

* Remove the column from users table.
2018-11-07 10:27:44 +08:00
Guo Xiang Tan
bdb8e9efdb DEV: Remove mocks from specs. 2018-11-07 09:55:58 +08:00
Bianca Nenciu
e0ccd36dbe FEATURE: Suspicious logins report. (#6544) 2018-10-30 22:51:58 +00:00
Régis Hanol
d17c8df926 Only check for suspicious login for staff members 2018-10-26 00:29:28 +02:00
Bianca Nenciu
6a3767cde7 FEATURE: Warn users via email about suspicious logins. (#6520)
* FEATURE: Warn users via email about suspicious logins.

* DEV: Move suspicious login check to a job.
2018-10-25 09:45:31 +00:00
Kyle Zhao
0f1afad6da FIX: extracted theme JavaScripts for multisite (#6502)
* FIX: extracted theme javascripts for multisite

* onceoff to rebake all theme fields
2018-10-18 17:05:34 +11:00
Neil Lalonde
12f132736b FIX: error looking at users in admin when tl3_promotion_min_duration is set to a very high value 2018-10-11 15:11:48 -04:00
Guo Xiang Tan
1b1ef21481 DEV: Improve Jobs::CleanUpEmailLogs specs. 2018-10-08 11:11:20 +08:00
Vinoth Kannan
8430ea927e FIX: Generate webhook payloads before destroy events (#6325) 2018-10-05 16:53:59 +08:00