Commit Graph

1128 Commits

Author SHA1 Message Date
Mark VanLandingham
0cba4d73c1
FEATURE: Add user custom fields to user directory (#13238) 2021-06-07 12:34:01 -05:00
Sam
435c4817cb
FEATURE: enable tagging by default (#13175)
Over the years we have found that a few communities never discovered tags.

Instead of having them default off we now have them default on, ensuring
that everyone finds out about them.

Co-authored-by: Dan Ungureanu <dan@ungureanu.me>
2021-06-07 18:07:46 +03:00
Bianca Nenciu
d9484db718
FIX: Split link watched words from replace (#13196)
It was not clear that replace watched words can be used to replace text
with URLs. This introduces a new watched word type that makes it easier
to understand.
2021-06-02 15:36:49 +10:00
Gerhard Schlager
41ee5b7c86
FIX: Don't store translated trust level names in anonymous cache (#13224)
Refactors `TrustLevel` and moves translations from server to client

Additional changes:
  * "staff" and "admin" wasn't translatable in site settings
  * it replaces a concatenated string with a translation
  * uses translation for trust levels in users_by_trust_level report
  * adds a DB migration to rename keys of translation overrides affected by this commit
2021-06-01 22:11:48 +02:00
Joffrey JAFFEUX
1cd0424ccd
FEATURE: lets users favorite 2 badges to show on user-card (#13151) 2021-06-01 10:33:40 +02:00
Martin Brennan
964da21817
FEATURE: Improve group email settings UI (#13083)
This overhauls the user interface for the group email settings management, aiming to make it a lot easier to test the settings entered and confirm they are correct before proceeding. We do this by forcing the user to test the settings before they can be saved to the database. It also includes some quality of life improvements around setting up IMAP and SMTP for our first supported provider, GMail. This PR does not remove the old group email config, that will come in a subsequent PR. This is related to https://meta.discourse.org/t/imap-support-for-group-inboxes/160588 so read that if you would like more backstory.

### UI

Both site settings of `enable_imap` and `enable_smtp` must be true to test this. You must enable SMTP first to enable IMAP.

You can prefill the SMTP settings with GMail configuration. To proceed with saving these settings you must test them, which is handled by the EmailSettingsValidator.

If there is an issue with the configuration or credentials a meaningful error message should be shown.

IMAP settings must also be validated when IMAP is enabled, before saving.

When saving IMAP, we fetch the mailboxes for that account and populate them. This mailbox must be selected and saved for IMAP to work (the feature acts as though it is disabled until the mailbox is selected and saved):

### Database & Backend

This adds several columns to the Groups table. The purpose of this change is to make it much more explicit that SMTP/IMAP is enabled for a group, rather than relying on settings not being null. Also included is an UPDATE query to backfill these columns. These columns are automatically filled when updating the group.

For GMail, we now filter the mailboxes returned. This is so users cannot use a mailbox like Sent or Trash for syncing, which would generally be disastrous.

There is a new group endpoint for testing email settings. This may be useful in the future for other places in our UI, at which point it can be extracted to a more generic endpoint or module to be included.
2021-05-28 09:28:18 +10:00
Sam
d45682716b
FIX: automatically expire bad push channels (#13156)
Previously we would retry push notifications indefinitely for all errors
except for ExpiredSubscription

Under certain conditions other persistent errors may arise such as a persistent
rate limit.

If we track more than 3 errors in a period of time longer than a day we will
delete the subscription

Also performs a bit of internal cleanup to ensure protected methods really
are private.
2021-05-27 06:49:20 +10:00
Roman Rizzi
83dd47f0b4
FEATURE: Navigate to an approved queued topic from the review queue. (#12841)
Admins can visit an approved queued topic from the review queue by clicking their title. We no longer store the created post and topic ids in the reviewable's payload object. Instead, we set the `topic_id` and `target_id` attributes.
2021-05-26 15:43:18 -03:00
Krzysztof Kotlarek
e29605b79f
FEATURE: the ability to search users by custom fields (#12762)
When the admin creates a new custom field they can specify if that field should be searchable or not.

That setting is taken into consideration for quick search results.
2021-04-27 15:52:45 +10:00
Josh Soref
308c032293
DEV: Fix spelling mistakes in DB migrations (#12811)
* spelling: initial
* spelling: integer
* spelling: irreversible
* spelling: mislabeled
* spelling: notification
* spelling: nullable
* spelling: targeted
2021-04-23 18:25:10 +02:00
Dan Ungureanu
528cfea079
FEATURE: Auto-activate users invited by email (#12675)
When invited by email, users will receive an invite URL which contains
a token. If that token is present when the invite is redeemed, their
account will be automatically activated.
2021-04-14 12:15:56 +03:00
Martin Brennan
eeaecd4fd2
FEATURE: Category setting to allow unlimited first post edits by the owner of the topic (#12690)
This PR adds a new category setting which is a column in the `categories` table, `allow_unlimited_owner_edits_on_first_post`.

What this does is:

* Inside the `can_edit_post?` method of `PostGuardian`, if the current user editing a post is the owner of the post, it is the first post, and the topic's category has `allow_unlimited_owner_edits_on_first_post`, then we bypass the check for `LimitedEdit#edit_time_limit_expired?` on that post.
* Also, similar to wiki topics, in `PostActionNotifier#after_create_post_revision` we send a notification to all users watching a topic when the OP is edited in a topic with the category setting `allow_unlimited_owner_edits_on_first_post` enabled.

This is useful for forums where there is a Marketplace or similar category, where topics are created and then updated indefinitely by the OP rather than the OP making new topics or additional replies. In a way this acts similar to a wiki that only one person can edit.
2021-04-14 15:54:09 +10:00
Martin Brennan
66d17fdd6b
FIX: Topic user bookmarked column is out of sync after post moves (#12612)
When posts are moved from one topic to another, the `topic_user.bookmarked` column for all users in the new and the old topic needs to be resynced, for example because a user bookmarks post 12 in topic 1, then it is moved to topic 2, the topic_user record for topic 1 should no longer be bookmarked. A background job has been added to sync the column for a specified topic, or for no topic at all, which does it for all topics like the migration.

Also includes a migration that we have run in the past to fix bad data.

----

This has been addressed in other places in the past:

https://github.com/discourse/discourse/pull/10211
https://github.com/discourse/discourse/pull/10188
2021-04-14 09:10:53 +10:00
Sam
c47f403dd9
FIX: correct mailing list migration (#12501)
Migration mistakenly enabled mailing list mode after it was disabled explicitly
2021-03-24 08:49:27 +11:00
Dan Ungureanu
2a4ddc621d
FIX: Add migration to set correct redemption_count (#12491)
Redeeming email invites did not increase the redemption_count which let
those invites in a weird state were they were both pending and redeemed.
2021-03-23 18:57:39 +02:00
Sam
e45bca7298
PERF: avoid regex on uploads table (#12485)
In extreme circumstances when the uploads table is huge, the old version of
this migration could take a very long time.

The rewrite extracts the sha1 directly from the badges table and does an index
based match on the uploads table
2021-03-23 09:19:02 +11:00
Martin Brennan
49f4c548ef
FEATURE: Bookmark pinning (#12431)
Users can now pin bookmarks from their bookmark list. This will anchor the bookmark to the top of the list, and show a pin icon next to it. This also applies in the nav bookmarks panel. If there are multiple pinned bookmarks they sort by last updated order.
2021-03-22 09:50:22 +10:00
Krzysztof Kotlarek
c5a116859d
FIX: delete post action from permanently deleted posts (#12309)
When Post is permanently deleted, we should delete correlated PostAction as well.
2021-03-18 15:22:41 +11:00
Bianca Nenciu
16b5fa030b
DEV: Set disable_mailing_list_mode automatically (#12402)
The user mailing list mode continued to be silently enabled and
UserEmail job checked just that ignoring site setting
disable_mailing_list_mode.

An additional migrate was added to set disable_mailing_list_mode
to false if any users enabled the mailing list mode already.
2021-03-17 17:39:10 +02:00
Osama Sayegh
a23d0f9961
UX: Add image uploader widget for uploading badge images (#12377)
Currently the process of adding a custom image to badge is quite clunky; you have to upload your image to a topic, and then copy the image URL and pasting it in a text field. Besides being clucky, if the topic or post that contains the image is deleted, the image will be garbage-collected in a few days and the badge will lose the image because the application is not that the image is referenced by a badge.

This commit improves that by adding a proper image uploader widget for badge images.
2021-03-17 08:55:23 +03:00
Alan Guo Xiang Tan
839caa274a DEV: Drop show_filter_by_tag site settings.
Follow-up to 83519e7f3a
2021-03-11 10:48:18 +08:00
Alan Guo Xiang Tan
ebe4896e48 FEATURE: Change very high/low search priority to rank at absolute ends.
Prior to this change, we had weights for very_high, high, low and
very_low. This means there were 4 weights to tweak and what weights to
use for `very_high/high` and `very_low/low` pair was hard to explain.
This change makes it such that `very_high` search priority will always
ensure that the posts are ranked at the top while `very_low` search
priority will ensure that the posts are ranked at the very bottom.
2021-03-09 09:20:37 +08:00
Dan Ungureanu
fecf3e20d9
FEATURE: Various improvements to invite system (#12314)
* FEATURE: Do not delete invite if link was copied

* FIX: Show error to user if invite redeeming fails

The error was only displayed to console.

* UX: Better placement of bulk buttons

Destroy all expired invites should be on the expired tab, not pending.

* FIX: Ensure invited_groups is unique per invite and group

* FIX: Do not refresh topic list if title unchanged

* FIX: Do not close modal on enter

This intereferes with the group and topic chooser.
Wrapping everything in a form disables this behavior.

* FIX: Move link and email options outside advanced section

* FIX: Do not close modal if saving a link invite

User may still want to copy the link.
2021-03-09 00:15:14 +02:00
Bianca Nenciu
4b0496b2fc
FIX: Reintroduce auto_silence_first_post_regex (#12223)
Follow-up to 533800a87b.
2021-02-26 15:07:04 +02:00
Penar Musaraj
45c5fd2172
DEV: Remove JoyPixels emoji option (#12197)
- removes the option from site settings
- deletes the site setting on existing sites that have it
- marks posts using emojis as requiring a rebake

Note that the actual image files are not removed here, the plan is to
remove them in a few weeks/months (when presumably the rebaking of old
posts has been completed).
2021-02-26 07:44:52 -05:00
Bianca Nenciu
533800a87b
Add watched words of type "replace" (#12020)
This commit includes other various improvements to watched words.

auto_silence_first_post_regex site setting was removed because it overlapped
with 'require approval' watched words.
2021-02-25 14:00:58 +02:00
David Taylor
b22ea7911c
DEV: Drop old SSO site setting rows from the database (#12148)
These were copied to their new names in 821bb1e8cb
2021-02-19 19:05:49 +00:00
Krzysztof Kotlarek
a696cc07d2
Revert "FEATURE: Ability to dismiss all new topics (#12018)" (#12058)
This reverts commits 7426764af4 and f5b18e2a31
2021-02-12 08:50:25 +11:00
Krzysztof Kotlarek
7426764af4
FIX: Optimize move to dismiss_new_topics migration (#12041)
This migration is quite heavy because of join to all potential topics which should be `dismissed` for each user. To make it a little bit more efficient I did two things:
- move conditions to join so it should use fewer rows
- do that in batches - 1000 users at the time
2021-02-11 16:05:38 -05:00
Krzysztof Kotlarek
f5b18e2a31
FEATURE: Ability to dismiss all new topics (#12018)
Follow up https://github.com/discourse/discourse/pull/11968

Dismiss all new topics using the same DismissTopicService. In addition, MessageBus receives exact topic ids which should be marked as `seen`.
2021-02-11 13:35:09 +11:00
David Taylor
821bb1e8cb
FEATURE: Rename 'Discourse SSO' to DiscourseConnect (#11978)
The 'Discourse SSO' protocol is being rebranded to DiscourseConnect. This should help to reduce confusion when 'SSO' is used in the generic sense.

This commit aims to:
- Rename `sso_` site settings. DiscourseConnect specific ones are prefixed `discourse_connect_`. Generic settings are prefixed `auth_`
- Add (server-side-only) backwards compatibility for the old setting names, with deprecation notices
- Copy `site_settings` database records to the new names
- Rename relevant translation keys
- Update relevant translations

This commit does **not** aim to:
- Rename any Ruby classes or methods. This might be done in a future commit
- Change any URLs. This would break existing integrations
- Make any changes to the protocol. This would break existing integrations
- Change any functionality. Further normalization across DiscourseConnect and other auth methods will be done separately

The risks are:
- There is no backwards compatibility for site settings on the client-side. Accessing auth-related site settings in Javascript is fairly rare, and an error on the client side would not be security-critical.
- If a plugin is monkey-patching parts of the auth process, changes to locale keys could cause broken error messages. This should also be unlikely. The old site setting names remain functional, so security-related overrides will remain working.

A follow-up commit will be made with a post-deploy migration to delete the old `site_settings` rows.
2021-02-08 10:04:33 +00:00
Martin Brennan
4af77f1e38
FEATURE: Allow durations < 1 hour and < 1 day for topic timers where duration is specified (auto delete replies, close based on last post) (#11961)
This PR allows entering a float value for topic timers e.g. 0.5 for 30 minutes when entering hours, 0.5 for 12 hours when entering days. This is achieved by adding a new column to store the duration of a topic timer in minutes instead of the ambiguous both hours and days that it could be before.

This PR has ommitted the post migration to delete the duration column in topic timers; it will be done in a subsequent PR to ensure that no data is lost if the UPDATE query to set duration_mintues fails.

I have to keep the old keyword of duration in set_or_create_topic_timer for backwards compat, will remove at a later date after plugins are updated.
2021-02-05 10:12:56 +10:00
Krzysztof Kotlarek
f39e7fe81d
FEATURE: New way to dismiss new topics (#11927)
This is a try to simplify logic around dismiss new topics to have one solution to work in all places - dismiss all-new, dismiss new in a specific category or even in a specific tag.
2021-02-04 11:27:34 +11:00
Martin Brennan
4af4d36175
FIX: IMAP allow unknown senders to reply to group topics via email (#11877)
Adds a new column/setting to groups, allow_unknown_sender_topic_replies, which is default false. When enabled, this scenario is allowed via IMAP:

* OP sends an email to the support email address which is synced to a group inbox via IMAP, creating a group topic
* Group user replies to the group topic
* An email notification is sent to the OP of the topic via GroupSMTPMailer
* The OP has several email accounts and the reply is sent to all of them, or they forward their reply to another email account
* The OP replies from a different email address than the OP (gloria@gmail.com instead of gloria@hey.com for example)
* The a new staged user is created, the new reply is accepted and added to the topic, and the staged user is added to the topic allowed users

Without allow_unknown_sender_topic_replies enabled the new reply creates an entirely new topic (because the email address it is sent from is not previously part of the topic email chain).
2021-01-29 09:59:10 +10:00
Martin Brennan
f49e3e5731
DEV: Add security_last_changed_at and security_last_changed_reason to uploads (#11860)
This PR adds security_last_changed_at and security_last_changed_reason to uploads. This has been done to make it easier to track down why an upload's secure column has changed and when. This necessitated a refactor of the UploadSecurity class to provide reasons why the upload security would have changed.

As well as this, a source is now provided from the location which called for the upload's security status to be updated as they are several (e.g. post creator, topic security updater, rake tasks, manual change).
2021-01-29 09:03:44 +10:00
Mark VanLandingham
809274fe0d
DEV: Replace 'processed' column on notifications with new table (#11864) 2021-01-27 10:29:24 -06:00
Martin Brennan
9ee8a01c3a
FIX: Change default for IncomingEmail#created_via to 0 (unknown) and make NOT NULL (#11782)
Follow up to https://review.discourse.org/t/dev-add-created-via-column-to-incomingemail-pr-11751/18366/6
2021-01-21 12:59:50 +10:00
Gerhard Schlager
3b2f6e129a
FEATURE: Add English (UK) as locale (#11768)
* "English" gets renamed into "English (US)"
* "English (UK)" replaces "English"

@discourse-translator-bot keep_translations_and_approvals
2021-01-20 21:32:22 +01:00
Mark VanLandingham
1a7922bea2
FEATURE: Create notification schedule to automatically set do not disturb time (#11665)
This adds a new table UserNotificationSchedules which stores monday-friday start and ends times that each user would like to receive notifications (with a Boolean enabled to remove the use of the schedule). There is then a background job that runs every day and creates do_not_disturb_timings for each user with an enabled notification schedule. The job schedules timings 2 days in advance. The job is designed so that it can be run at any point in time, and it will not create duplicate records.

When a users saves their notification schedule, the schedule processing service will run and schedule do_not_disturb_timings. If the user should be in DND due to their schedule, the user will immediately be put in DND (message bus publishes this state).

The UI for a user's notification schedule is in user -> preferences -> notifications. By default every day is 8am - 5pm when first enabled.
2021-01-20 10:31:52 -06:00
Martin Brennan
fb184fed06
DEV: Add created_via column to IncomingEmail (#11751)
This should make it easier to track down how the incoming email was created, which is one of four locations:

The POP3 poller (which picks up reply via email replies)
The admin email controller #handle_mail (which is where hosted mail is sent)
The IMAP sync tool
The group SMTP mailer, which sends emails when replying to IMAP topics, pre-emptively creating IncomingEmail records to avoid double syncing
2021-01-20 13:22:41 +10:00
Krzysztof Kotlarek
06b7c44593
FEATURE: reason to reject user signup (#11700)
Feature for `Must Approve Users` setup. When a user is rejected, a staff member can optionally set a reason for audit purposes. In addition, feedback email can be sent to the user.

Meta: https://meta.discourse.org/t/account-rejection-email/103112/8
2021-01-15 09:43:26 +11:00
Martin Brennan
87961534ea
FEATURE: IMAP detect spammed email and delete associated Discourse topic (#11654)
This PR adds functionality for the IMAP sync code to detect if a UID that is missing from the mail group mailbox is in the Spam/Junk folder for the mail account, and if so delete the associated Discourse topic. This is identical to what we do for emails that are moved for Trash.

If an email is missing but not in Spam or Trash, then we mark the incoming email record with imap_missing: true. This may be used in future to further filter or identify these emails, and perhaps go hunting for them in the email account in bulk.

Note: This adds some code duplication because the trash and spam email detection and handling is very similar. I intend to do more refactors/improvements to the IMAP sync code in time because there is a lot of room for improvement.
2021-01-14 09:54:18 +10:00
Bianca Nenciu
ec0212e56b
FIX: Make category slugs lowercase (#11277)
Admins could specify category slug with upper case characters and same slug,
but with different cases could be used simultaneously.
2021-01-12 17:28:33 +02:00
Mark VanLandingham
4601f3be7e
FEATURE: Send notification emails when users leave do not disturb mode (#11643) 2021-01-07 10:49:49 -06:00
Sam
53f4d54f23
PERF: add indexes to speed up profile pages (#11598)
These 2 indexes optimise performance on profile pages.

The summary page displays:

1. A list of "Top Link" - links sorted by number of clicks posted by user
2. A list of "Top Replies" - replies made by a user that go the most hearts

These two areas could devolve into full index or table scans, new indexes are there to avoid this cost on large dbs

One minor downside is that storage requirements go a tiny bit up to maintain the new indexes
2020-12-29 15:54:05 +11:00
Alan Guo Xiang Tan
d2a0462186 FIX: Re-run remove flag and queued post web hook migrations.
There have been production instances where the migrations has run but the records
that were meant to be deleted have not been deleted. The root cause is
unknown but the migration is safe and simple to re-run. The problem is
not reproducible locally so we're not spending too much time on digging
up the root. Time vs business cost tradeoff.

Follow-up fb15da43da
2020-12-21 10:17:08 +08:00
Mark VanLandingham
649ed24bb4
FEATURE: Do not disturb (#11484) 2020-12-18 09:03:51 -06:00
David Taylor
02278109d6
DEV: Maintain github_user_info primary key values during migration (#11286)
This will only happen if the user_associated_accounts table is currently empty. It's useful for people that may be depending on primary key values in data explorer queries. This change will only have an effect on sites which have not already run this migration.
2020-11-19 10:30:18 +00:00
Dan Ungureanu
aee5e80038
FIX: Disable auto_update for existent themes (#11244) 2020-11-16 15:35:07 +02:00
Dan Ungureanu
bc8423a1bf
FEATURE: Add auto update field to themes (#11102)
Themes marked for auto update will be automatically updated when
Discourse is updated. This is triggered by discourse_docker or
docker_manager running Rake task 'themes:update'.
2020-11-16 14:44:09 +02:00
Roman Rizzi
f2bef7ea8f
FIX: Store Reviewable's force_review as a boolean. (#11219)
* FIX: Store Reviewable's force_review as a boolean.

Using the `force_review` flag raises the score to hit the minimum visibility threshold. This strategy turned out to be ineffective on sites with a high number of flags, where these values could rapidly fluctuate.

This change adds a `force_review` column on the reviewables table and modifies the `Reviewable#list_for` method to show these items when passing the `status: :pending` option, even if the score is not high enough. ReviewableQueuedPosts and ReviewableUsers are always created using this option.
2020-11-13 08:19:01 -03:00
Dan Ungureanu
ab314218d3
FEATURE: Implement edit functionality for post notices (#11140)
All post notice related custom fields were moved to a single one.
2020-11-11 14:49:53 +02:00
David Taylor
cf21de0e7a
DEV: Migrate Github authentication to ManagedAuthenticator (#11170)
This commit adds an additional find_user_by_email hook to ManagedAuthenticator so that GitHub login can continue to support secondary email addresses

The github_user_infos table will be dropped in a follow-up commit.

This is the last core authenticator to be migrated to ManagedAuthenticator 🎉
2020-11-10 10:09:15 +00:00
David Taylor
5140ec9acf
DEV: Cleanup ignored user logic (#11107)
- IgnoredUser records should all now have an expiring_at value. This commit enforces that in the DB, and fixes any corrupt rows
- Changes to the ignored user list are now handled by the `/u/{username}/notification_level` endpoint. This allows setting expiration dates on the ignore. This commit removes the old logic for saving a list of usernames in the user preferences.
- Many specs were calling `IgnoredUser.create`. This commit changes them to use `Fabricate(:ignored_user)` for consistency
2020-11-03 12:38:54 +00:00
Arpit Jalan
1476e17c35
FEATURE: new setting to create a linked topic on autoclosing mega topics (#11001)
This commit adds a site setting `auto_close_topics_create_linked_topic`
which when enabled works in conjunction with `auto_close_topics_post_count`
setting and creates a new linked topic for the topic just closed.

The auto-created new topic contains a link for all the previous topics
and the topic titles are appended with `(Part {n})`.

The setting is enabled by default.
2020-11-02 12:18:48 +05:30
Jarek Radosz
2f4a1ff61b
DEV: Update rubocop-discourse from 2.3.2 to 2.4.0 (#11079)
Also fixes whitespace related issues raised by rubocop.
2020-10-30 15:04:29 +01:00
David Taylor
23e5c605f6 DEV: Add support for allowed parameters in user api key scopes
Initially, this feature is only intended for use in core/plugins, so there is no API for requesting a parameter-scoped key. That may change in future.
2020-10-19 10:40:55 +01:00
Roman Rizzi
21c53ed249
FEATURE: Topic slow mode. (#10904)
Adds a new slow mode for topics that are heating up. Users will have to wait for a period of time before being able to post again.

We store this interval inside the topics table and track the last time a user posted using the last_posted_at datetime in the TopicUser relation.
2020-10-16 16:24:38 -03:00
Bianca Nenciu
43e52a7dc1
DEV: Remove gifsicle dependency (#10357)
Dependency on gifsicle, allow_animated_avatars and allow_animated_thumbnails
site settings were all removed. Animated GIF images are still allowed, but
the generated optimized images are no longer animated for those (which were
used for avatars and thumbnails).

The added 'animated' is populated by extracting information using FastImage.
This field was used to selectively reoptimize old animations. This process
happens in the background.
2020-10-16 13:41:27 +03:00
Bianca Nenciu
25b8ed740b
DEV: Make site setting type uploaded_image_list use upload IDs (#10401)
It used to be a list of concatenated upload URLs which was prone to
break.
2020-10-13 16:17:06 +03:00
Sam Saffron
cd3254e170 DEV: remove legacy code
This code is a no-op on all sites, even though it looks rather dangerous
this migration has long run prior to people trying exploit it.

That said ... hygiene here ... is not good.

Remove this legacy, we do not want it, even in historical migrations.
2020-10-08 10:53:07 +08:00
David Taylor
c0293339b8
PERF: Do not enqueue digest emails when attempted recently (#10849)
Previously, Jobs::EnqueueDigestEmails would enqueue a digest job for every user, even if there are no topics to send. The digest job would exit, no email would send, and last_emailed_at would not change. 30 minutes later, Jobs::EnqueueDigestEmails would run again and re-enqueue jobs for the same users.

120fa8ad introduced a temporary mitigation for this issue, by randomly selecting a subset of those users each time.

This commit adds a new `digest_attempted_at` column to the `user_stats` table. This column is updated every time a digest job completes for a user. Using this, we can avoid scheduling digest jobs for the same user every 30 minutes. This also removes the random user selection in 120fa8ad, and instead prioritizes users who had digests attempted the longest time ago.
2020-10-07 15:30:38 +01:00
Martin Brennan
6e2be3e60b
FIX: When admin changes an email for the user the user must confirm the change (#10830)
See https://meta.discourse.org/t/changing-a-users-email/164512 for additional context.

Previously when an admin user changed a user's email we assumed that they would need a password reset too because they likely did not have access to their account. This proved to be incorrect, as there are other reasons a user needs admin to change their email. This PR:

* Changes the admin change email for user flow so the user is sent an email to confirm the change
* We now record who the email change request was requested by
* If the requested by user is admin and not the user we note this in the email sent to the user
* We also make the confirm change email route open to anonymous users, so it can be clicked by the user even if they do not have access to their account. If there is a logged in user we make sure the confirmation matches the current user.
2020-10-07 13:02:24 +10:00
Roman Rizzi
7d5b18b7cf
FIX: Migration should only update ReviewableUsers where the user is not approved. (#10790)
The reviewable was updated despite the user not being approved because a u.id = r.target_id condition is missing. It only affected user reviewables that were pending when the migration ran. Users were not auto-approved.
2020-10-01 08:49:10 -03:00
David Taylor
1ba9b34b03
DEV: Move UserApiKey scopes to dedicated table (#10704)
This has no functional impact yet, but it is the first step in adding more granular scopes to UserApiKeys
2020-09-29 10:57:48 +01:00
Guo Xiang Tan
b47b640598
FEATURE: Hidden SiteSetting.keep_old_ip_address_count to track IP history. 2020-09-17 12:50:39 +08:00
Martin Brennan
c2a660ead3
FIX: Fix verification_status migration for uploads
* Missing ELSE 1 even though there is a default on the
  column.
2020-09-17 14:23:37 +10:00
Martin Brennan
80268357e7
DEV: Change upload verified column to be integer (#10643)
Per review https://review.discourse.org/t/dev-add-verified-to-uploads-and-fill-in-s3-inventory-10406/14180

Change the verified column for Upload to a verified_status integer column, to avoid having NULL as a weird implicit status.
2020-09-17 13:35:29 +10:00
Martin Brennan
5268568d23
FEATURE: Remove user topic timers and migrate to bookmarks with reminders (#10474)
This PR removes the user reminder topic timers, because that system has been supplanted and improved by bookmark reminders. The option is removed from the UI and all existing user reminder topic timers are migrated to bookmark reminders.

Migration does this:

* Get all topic_timers with status_type 5 (reminders)
* Gets all bookmarks where the user ID and topic ID match
* Loops through the found topic timers
  * If there is no bookmark for the OP of the topic, then we just create a bookmark with a reminder
  * If there is a bookmark for the OP of the topic and it does **not** have a reminder set, then just 
update it with the topic timer reminder
  * If there is a bookmark for the OP of the topic with a reminder then just discard the topic timer
* Cancels all outstanding user reminder topic timers
* **Trashes (not deletes) all user reminder topic timers**

Notes:

* For now I have left the user reminder topic timer job class in place; this is so the jobs can be cancelled in the migration. It and the specs will be deleted in the next PR.
* At a later date I will write a migration to delete all trashed user topic timers. They are not deleted here in case there are data issues and they need to be recovered.
* A future PR will change the UI of the topic timer modal to make it look more like the bookmark modal.
2020-09-14 11:11:55 +10:00
Martin Brennan
7f2f87bf59
DEV: Review fixes (#10641)
See comments in https://review.discourse.org/t/dev-imap-log-to-database-10435/14337/6 for context.
2020-09-10 13:41:46 +10:00
Guo Xiang Tan
9b75d95fc6 PERF: Keep track of first unread PM and first unread group PM for user.
This optimization helps to filter away topics so that the joins on
related tables when querying for unread messages is not expensive.
2020-09-09 14:05:41 +08:00
Rafael dos Santos Silva
58cdf87674
PERF: Add partial index on reviewables for topic view (#10492)
On the topic view route we query for reviewables of each post in the stream,
using a query that filters on two unindexed columns. This results in a Parallel Seq Scan
over all rows, which can take quite some time (~20ms was seen) on forums with lots of flags

After index is added PostgreSQL planner opts for a simple Index Scan and runs in sub 1ms.

Before:

```
                                     QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Finalize GroupAggregate  (cost=11401.08..11404.87 rows=20 width=28) (actual time=19.209..19.209 rows=1 loops=1)
   Group Key: r.target_id
   ->  Gather Merge  (cost=11401.08..11404.41 rows=26 width=28) (actual time=19.202..20.419 rows=1 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         ->  Partial GroupAggregate  (cost=10401.06..10401.38 rows=13 width=28) (actual time=16.958..16.958 rows=0 loops=3)
               Group Key: r.target_id
               ->  Sort  (cost=10401.06..10401.09 rows=13 width=16) (actual time=16.956..16.956 rows=0 loops=3)
                     Sort Key: r.target_id
                     Sort Method: quicksort  Memory: 25kB
                     Worker 0:  Sort Method: quicksort  Memory: 25kB
                     Worker 1:  Sort Method: quicksort  Memory: 25kB
                     ->  Nested Loop  (cost=0.42..10400.82 rows=13 width=16) (actual time=15.894..16.938 rows=0 loops=3)
                           ->  Parallel Seq Scan on reviewables r  (cost=0.00..10302.47 rows=8 width=12) (actual time=15.882..16.927 rows=0 loops=3)
                                 Filter: (((target_type)::text = 'Post'::text) AND (target_id = ANY ('{7565483,7565563,7565566,7565567,7565568,7565569,7565579,7565580,7565583,7565586,7565588,7565589,7565601,7565602,7565603,7565613,7565620,7565623,7565624,7565626}'::integer[])))
                                 Rows Removed by Filter: 49183
                           ->  Index Scan using index_reviewable_scores_on_reviewable_id on reviewable_scores s  (cost=0.42..12.27 rows=2 width=8) (actual time=0.029..0.030 rows=1 loops=1)
                                 Index Cond: (reviewable_id = r.id)
 Planning Time: 0.318 ms
 Execution Time: 20.470 ms
```

After:
```
                                                                                                          QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 GroupAggregate  (cost=0.84..342.54 rows=20 width=28) (actual time=0.038..0.038 rows=1 loops=1)
   Group Key: r.target_id
   ->  Nested Loop  (cost=0.84..341.95 rows=31 width=16) (actual time=0.020..0.033 rows=1 loops=1)
         ->  Index Scan using index_reviewables_on_target_id on reviewables r  (cost=0.42..96.07 rows=20 width=12) (actual time=0.013..0.026 rows=1 loops=1)
               Index Cond: (target_id = ANY ('{7565483,7565563,7565566,7565567,7565568,7565569,7565579,7565580,7565583,7565586,7565588,7565589,7565601,7565602,7565603,7565613,7565620,7565623,7565624,7565626}'::integer[]))
         ->  Index Scan using index_reviewable_scores_on_reviewable_id on reviewable_scores s  (cost=0.42..12.27 rows=2 width=8) (actual time=0.005..0.005 rows=1 loops=1)
               Index Cond: (reviewable_id = r.id)
 Planning Time: 0.253 ms
 Execution Time: 0.067 ms
```
2020-08-28 11:47:11 -03:00
Penar Musaraj
b7cfc9e861
FEATURE: User selectable color schemes (#10544) 2020-08-28 10:36:52 -04:00
Guo Xiang Tan
f4a0d200ee
DEV: Drop search index on non-pm posts take 2.
Follow-up to 87de8948c0
2020-08-21 07:27:37 +08:00
Guo Xiang Tan
87de8948c0
DEV: Drop search index on non-pm posts.
The problem with this index is that on sites with a high non-pm to pm
posts ratio, the index is esstentially duplicating the existing index on
`PostSearchData#search_data`. If the site is huge, the index ends up
taking up more diskspace.
2020-08-21 07:21:34 +08:00
Guo Xiang Tan
0967ce478d
DEV: Remove migration that is no longer required.
Avoid creating the index twice.

Follow-up to 628319aad3
2020-08-20 14:25:20 +08:00
jbrw
aa1fc01307
FEATURE - Moderators can create and manage groups (#10432)
Enabling the moderators_manage_categories_and_groups site setting will allow moderator users to create/manage groups.

* show New Group form to moderators

* Allow moderators to update groups and read logs, where appropriate

* Rename site setting from create -> manage

* improved tests

* Migration should rename old log entries

* Log group changes, even if those changes mean you can no longer see the group

* Slight reshuffle

* RouteTo /g if they no longer have permissions to view group
2020-08-19 10:41:40 -04:00
Guo Xiang Tan
2161abfabd
DEV: Move data migration of PostSearchData#private_message into post_migration.
Follow-up to 92b7fe4c62
2020-08-18 16:46:14 +08:00
Guo Xiang Tan
92b7fe4c62
PERF: Add partial index for non-pm search. 2020-08-18 15:55:08 +08:00
Vinoth Kannan
476d26159a
FEATURE: add new user option skip_new_user_tips. (#10437)
And add new site setting `default_other_skip_new_user_tips` in user preferences category.
2020-08-14 19:10:56 +05:30
Martin Brennan
4670b62969
DEV: IMAP log to database (#10435)
Convert all IMAP logging to write to a database table for easier inspection. These logs are cleaned up daily if they are > 5 days old.

Logs can easily be watched in dev by setting DISCOURSE_DEV_LOG_LEVEL=\"debug\" and running tail -f development.log | grep IMAP
2020-08-14 12:01:31 +10:00
Martin Brennan
b950b3fb3f
DEV: Add verified to uploads and fill in S3 inventory (#10406)
When we run the S3 inventory, mark uploads that exist as verified true, those that don't as verified false, and uploads not included in the check / not yet checked as verified nil.
2020-08-11 14:43:51 +10:00
jbrw
3593e582a3
FIX - limit number of embedded media items in a post (#10391)
* FIX - limit number of embedded media items in a post

* Add renamed settings to DeprecatedSettings
2020-08-07 12:08:59 -04:00
Neil Lalonde
1ca81fbb95
FEATURE: set notification levels when added to a group (#10378)
* FEATURE: set notification levels when added to a group

This feature allows admins and group owners to define default
category and tag tracking levels that will be applied to user
preferences automatically at the time when users are added to the
group. Users are free to change those preferences afterwards.
When removed from a group, the user's notification preferences aren't
changed.
2020-08-06 12:27:27 -04:00
Penar Musaraj
6fdc711b4a
FEATURE: Allow users to opt out of automatic dark mode (#10377) 2020-08-06 09:45:37 -04:00
Martin Brennan
2920988b3a
FIX: IMAP sync email update uniqueness across groups and minor improvements (#10332)
Adds a imap_group_id column to IncomingEmail to deal with an issue where we were trying to update emails in the mailbox, calling IncomingEmail.where(imap_sync: true). However UID and UIDVALIDITY could be the same across accounts. So if group A used IMAP details for Gmail account A, and group B used IMAP details for Gmail account B, and both tried to sync changes to an email with UID of 3 (e.g. changing Labels), one account could affect the other. This even applied to Archiving!

Also in this PR:

* Fix error occurring if we do a uid_fetch and no emails are returned
* Allow for creating labels within the target mailbox (previously we would not do this, only use existing labels)
* Improve consistency for log messages
* Add specs for generic IMAP provider (Gmail specs still to come)
* Add custom archiving support for Gmail
* Only use Message-ID for uniqueness of IncomingEmail if it was generated by us
* Various refactors and improvements
2020-08-03 13:10:17 +10:00
Martin Brennan
9e5b213089
FIX: Ensure topic user bookmarked synced on bookmark auto-delete (#10323)
For the following conditions, the TopicUser.bookmarked column was not updated correctly:

* When a bookmark was auto-deleted because the reminder was sent
* When a bookmark was auto-deleted because the owner of the bookmark replied to the topic

This adds another migration to fix the out-of-sync column and also some refactors to BookmarkManager to allow for more of these delete cases. BookmarkManager is used instead of directly destroying the bookmark in PostCreator and BookmarkReminderNotificationHandler.
2020-07-29 09:43:32 +10:00
Krzysztof Kotlarek
14003abc37
FIX: Improve allowed_path column migration (#10321)
Because previous migration was already deployed and some databases were already migrated, I needed to add some conditions to the migration.

Previous migration - https://github.com/discourse/discourse/blob/master/db/post_migrate/20200629232159_rename_path_whitelist_to_allowed_paths.rb

What will happen in a scenario when previous migration was not run.
1. column allowed_paths will be created
2. allowed_path will be populated with data from path_whitelist
3. path_whitelist column will be dropped

What will happen in a scenario when previous migration was already run.
1. column allowed_paths will not be created because already exists - `unless column_exists?(:embeddable_hosts, :allowed_paths)`
2. Data will not be copied because path_whitelist is missing - `if column_exists?(:embeddable_hosts, :path_whitelist) && column_exists?(:embeddable_hosts, :allowed_paths)`
3. path_whitelist column deletion will be skipped - `if column_exists?(:embeddable_hosts, :path_whitelist)`
2020-07-28 13:31:51 +10:00
Krzysztof Kotlarek
e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Martin Brennan
41b43a2a25
FEATURE: Add "delete on owner reply" bookmark functionality (#10231)
This adds an option to "delete on owner reply" to bookmarks. If you select this option in the modal, then reply to the topic the bookmark is in, the bookmark will be deleted on reply.

This PR also changes the checkboxes for these additional bookmark options to an Integer column in the DB with a combobox to select the option you want.

The use cases are:

* Sometimes I will bookmark the topics to read it later. In this case we definitely don’t need to keep the bookmark after I replied to it.
* Sometimes I will read the topic in mobile and I will prefer to reply in PC later. Or I may have to do some research before reply. So I will bookmark it for reply later.
2020-07-21 10:00:39 +10:00
Blake Erickson
690f17bcbe
FEATURE: Allow List for PMs (#10270)
* FEATURE: Allow List for PMs

This feature adds a new user setting that is disabled by default that
allows them to specify a list of users that are allowed to send them
private messages. This way they don't have to maintain a large list of
users they don't want to here from and instead just list the people they
know they do want. Staff will still always be able to send messages to
the user.

* Update PR based on feedback
2020-07-20 15:23:49 -06:00
Martin Brennan
716ccf7fe4
FIX: Bookmark search fixes (#10239)
* Remove unneeded bookmark name index.
* Change bookmark search query to use post_search_data. This allows searching on topic title and post content
* Tweak the style/layout of the bookmark list so the search looks better and the whole page fits better on mobile.
2020-07-17 15:55:07 +10:00
Roman Rizzi
f13ec11c64
FEATURE: Add scopes to API keys (#9844)
* Added scopes UI

* Create scopes when creating a new API key

* Show scopes on the API key show route

* Apply scopes on API requests

* Extend scopes from plugins

* Add missing scopes. A mapping can be associated with multiple controller actions

* Only send scopes if the use global key option is disabled. Use the discourse plugin registry to add new scopes

* Add not null validations and index for api_key_id

* Annotate model

* DEV: Move default mappings to ApiKeyScope

* Remove unused attribute and improve UI for existing keys

* Support multiple parameters separated by a comma
2020-07-16 15:51:24 -03:00
jbrw
06073fe8c6
FEATURE: Allow group moderators to close/archive topics
* FEATURE: Allow group moderators to close/archive topics
2020-07-14 12:36:19 -04:00
David Taylor
3d65678a13
DEV: Add timestamp columns to optimized_images table (#10199)
This allows us to filter by created/updated date when comparing to an S3 inventory.
2020-07-14 11:50:33 +01:00
Martin Brennan
bcc80e0ea8
FEATURE: Add search to user bookmark list (#10230)
User bookmarks can now be searched by name or post raw content. The q querystring param is hooked up from the Ember router as well.
2020-07-14 14:43:41 +10:00
Bianca Nenciu
4492718864
FIX: Skip whisper posts when updating topic like count (#10157) 2020-07-13 16:30:00 +10:00
Dan Ungureanu
c72bc27888
FEATURE: Implement support for IMAP and SMTP email protocols. (#8301)
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2020-07-10 12:05:55 +03:00
Martin Brennan
9a2955d471
FIX: Migrate topic_users.bookmarked to false when it is incorrectly true (#10211)
Follow up to https://github.com/discourse/discourse/pull/10188/files

There are still TopicUser records where bookmarked is true even though there are no Bookmark or PostAction records with the type of bookmark for the associated topic and user. This migration corrects this issue by setting bookmarked to false for these cases.
2020-07-10 12:22:15 +10:00
Martin Brennan
6be7a66ba7
FIX: Cap bookmark name at 100 chars and truncate existing names (#10189)
We have a couple of examples of enormous amounts of text being entered in the name column of bookmarks. This is not desirable...it is just meant to be a short note / reminder of why you bookmarked this.

This PR caps the column at 100 characters and truncates existing names in the database to 100 characters.
2020-07-08 17:19:01 +10:00