Commit Graph

1492 Commits

Author SHA1 Message Date
David Taylor
31db83527b DEV: Introduce PresenceChannel API for core and plugin use
PresenceChannel aims to be a generic system for allow the server, and end-users, to track the number and identity of users performing a specific task on the site. For example, it might be used to track who is currently 'replying' to a specific topic, editing a specific wiki post, etc.

A few key pieces of information about the system:
- PresenceChannels are identified by a name of the format `/prefix/blah`, where `prefix` has been configured by some core/plugin implementation, and `blah` can be any string the implementation wants to use.
- Presence is a boolean thing - each user is either present, or not present. If a user has multiple clients 'present' in a channel, they will be deduplicated so that the user is only counted once
- Developers can configure the existence and configuration of channels 'just in time' using a callback. The result of this is cached for 2 minutes.
- Configuration of a channel can specify permissions in a similar way to MessageBus (public boolean, a list of allowed_user_ids, and a list of allowed_group_ids). A channel can also be placed in 'count_only' mode, where the identity of present users is not revealed to end-users.
- The backend implementation uses redis lua scripts, and is designed to scale well. In the future, hard limits may be introduced on the maximum number of users that can be present in a channel.
- Clients can enter/leave at will. If a client has not marked itself 'present' in the last 60 seconds, they will automatically 'leave' the channel. The JS implementation takes care of this regular check-in.
- On the client-side, PresenceChannel instances can be fetched from the `presence` ember service. Each PresenceChannel can be used entered/left/subscribed/unsubscribed, and the service will automatically deduplicate information before interacting with the server.
- When a client joins a PresenceChannel, the JS implementation will automatically make a GET request for the current channel state. To avoid this, the channel state can be serialized into one of your existing endpoints, and then passed to the `subscribe` method on the channel.
- The PresenceChannel JS object is an ember object. The `users` and `count` property can be used directly in ember templates, and in computed properties.
- It is important to make sure that you `unsubscribe()` and `leave()` any PresenceChannel objects after use

An example implementation may look something like this. On the server:

```ruby
register_presence_channel_prefix("site") do |channel|
  next nil unless channel == "/site/online"
  PresenceChannel::Config.new(public: true)
end
```

And on the client, a component could be implemented like this:

```javascript
import Component from "@ember/component";
import { inject as service } from "@ember/service";

export default Component.extend({
  presence: service(),
  init() {
    this._super(...arguments);
    this.set("presenceChannel", this.presence.getChannel("/site/online"));
  },
  didInsertElement() {
    this.presenceChannel.enter();
    this.presenceChannel.subscribe();
  },
  willDestroyElement() {
    this.presenceChannel.leave();
    this.presenceChannel.unsubscribe();
  },
});
```

With this template:

```handlebars
Online: {{presenceChannel.count}}
<ul>
  {{#each presenceChannel.users as |user|}} 
    <li>{{avatar user imageSize="tiny"}} {{user.username}}</li>
  {{/each}}
</ul>
```
2021-08-27 16:26:06 +01:00
Martin Brennan
e43a8af3bd
FIX: Do not send emails to mailing_list_mode subscribers for PMs (#14159)
This bug was introduced by f66007ec83.

In PostJobsEnqueuer we previously did not fire the after_post_create
event and after_topic_create event for private message topics. This was
changed in the above commit in order to publish message bus messages
for topic tracking state updates. Unfortunately this caused the
NotifyMailingListSubscribers job to be enqueued for all posts including
private messages, and admins and the users involved in the PMs got
emailed the contents of the PMs if they had mailing list mode enabled.

Luckily the impact of this was mitigated by a Guardian#can_see? check
for each mailing list mode user in the NotifyMailingListSubscribers job.
We never want to notify mailing list mode subscribers for private messages
so an early return has been added there, plus the logic in PostJobsEnqueuer
has been fixed, and tests have been added to that class where there were
none before.
2021-08-26 15:16:35 +10:00
Alan Guo Xiang Tan
f66007ec83
FEATURE: Display unread and new counts for messages. (#14059)
There are certain design decisions that were made in this commit.

Private messages implements its own version of topic tracking state because there are significant differences between regular and private_message topics. Regular topics have to track categories and tags while private messages do not. It is much easier to design the new topic tracking state if we maintain two different classes, instead of trying to mash this two worlds together.

One MessageBus channel per user and one MessageBus channel per group. This allows each user and each group to have their own channel backlog instead of having one global channel which requires the client to filter away unrelated messages.
2021-08-25 11:17:56 +08:00
Vinoth Kannan
c75500b871
DEV: decrease webhook timeout and move to 'low' priority queue. (#14038)
Previously, webhooks has 60 seconds default timeout of Excon and were added in the 'default' queue.
2021-08-13 10:41:02 +05:30
Alan Guo Xiang Tan
849827841f Revert "DEV: Move private message topic tracking state publish into sidekiq."
This reverts commit c51c80466c.

There is a conditional in post jobs enqueuer that only enqueues the job
for non-PM posts.
2021-07-30 11:36:01 +08:00
Alan Guo Xiang Tan
c51c80466c DEV: Move private message topic tracking state publish into sidekiq.
Same intend as b1f32f2f57.
2021-07-29 14:57:19 +08:00
Martin Brennan
b500949ef6
FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787)
This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads`  settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader.

A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used. 

### Starting a direct S3 upload

When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded.

Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage.

### Completing a direct S3 upload

Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`.

1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this.

2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues.

    We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large.

3. Finally we return the serialized upload record back to the client

There are several errors that could happen that are handled by `UploadsController` as well.

Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
2021-07-28 08:42:25 +10:00
Alan Guo Xiang Tan
0d8144b62b DEV: Improve logging of errors in Jobs::ProcessUserNotificationSchedules
Gives us the actual error and backtrace to work with. Otherwise, the
logging of the error is not useful at all.
2021-07-21 12:20:44 +08:00
Dan Ungureanu
079d2af55f
FIX: Clear stale status of reloaded reviewables (#13750)
* FIX: Clear stale status of reloaded reviewables

Navigating away from and back to the reviewables reloaded Reviewable
records, but did not clear the "stale" attribute.

* FEATURE: Show user who last acted on reviewable

When a user acts on a reviewable, all other clients are notified and a
generic "reviewable was resolved by someone" notice was shown instead of
the buttons. There is no need to keep secret the username of the acting
user.
2021-07-16 19:57:12 +03:00
Penar Musaraj
361c8be547
PERF: Add scheduled job to delete old stylesheet cache rows (#13747) 2021-07-16 10:58:01 -04:00
Osama Sayegh
31aa701518
FEATURE: Add option to grant badge multiple times to users using Bulk Award (#13571)
Currently when bulk-awarding a badge that can be granted multiple times, users in the CSV file are granted the badge once no matter how many times they're listed in the file and only if they don't have the badge already.

This PR adds a new option to the Badge Bulk Award feature so that it's possible to grant users a badge even if they already have the badge and as many times as they appear in the CSV file.
2021-07-15 05:53:26 +03:00
Martin Brennan
068889cb5f
FIX: Email threads sometimes not grouping for group SMTP (#13727)
This PR fixes a couple of issues related to group SMTP:

1. When running the group SMTP job, we were exiting early if the email was for the OP because of an IMAP race condition. However this causes issues when replying as a new topic for an existing SMTP topic, as the recipient does not get the OP email which can cause threading problems.
2. When sending emails for a new topic spun out like the issue in 1., we are not maintaining the original subject/topic title because that is based on the incoming email record, which we were not doing because the group SMTP email was never sent because of issue 1.
2021-07-14 14:23:14 +10:00
Martin Brennan
c6f2459cc4
FIX: Do not prevent other topic timers running on error (#13665)
There was an issue with the TopicTimerEnqueuer where any timer
that failed to enqueue_typed_job with an error would prevent
all other pending timers after the one that errored from running.

To mitigate this we just capture the error and log it (so we can
still fix it if needed for bug crushing) and proceed with the
rest of the timer enqueues.

The commit https://github.com/discourse/discourse/pull/13544 highlighted
this issue originally in hosted sites.

<!-- NOTE: All pull requests should have tests (rspec in Ruby, qunit in JavaScript). If your code does not include test coverage, please include an explanation of why it was omitted. -->
2021-07-08 12:49:58 +10:00
Martin Brennan
b3d3ad250b
FIX: Handle SMTPServerBusy for group smtp email (#13632)
Use the `sidekiq_retry_in` code from Jobs::UserEmail in group SMTP. Also we don't need to keep `seconds_to_delay` -- sidekiq uses the default delay calculation if you return 0 or nil from the block. See 3330df0ee3/lib/sidekiq/job_retry.rb (L216-L234) for sidekiq default retry delay logic.

I experimented with extracting this into a concern or a module, but `sidekiq_retry_in` is quite magic and it would not allow me to abstract away into a module that calls some method specificall in the child job class.

I would love to write tests for this, but it does not seem possible (not sure if its because of our test
setup) to write tests that test sidekiq's retry capability, and I am not sure if we should be anyway. Initial addition
to UserEmail did not test this functionality 
d224966a0e
2021-07-06 13:37:52 +10:00
Martin Brennan
0f688f45bd
FIX: Handle edge cases for group SMTP email job (#13631)
Skip group SMTP email (and add log) if:

* topic is deleted
* post is deleted
* smtp has been disabled for the group

Skip without log if:

* enable_smtp site setting is false
* disable_emails site setting is yes

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2021-07-05 14:56:32 +10:00
Martin Brennan
03338f9086
FIX: Remove legacy topic timer code (#13544)
The new topic timer backend code introduced six months ago
in 0034cbd is now used instead of this legacy code. It can be safely removed
now.
2021-06-29 09:16:25 +10:00
Martin Brennan
87684f7c5e
FEATURE: Use group SMTP job and mailer instead of UserNotifications change (#13489)
This PR backtracks a fair bit on this one https://github.com/discourse/discourse/pull/13220/files.

Instead of sending the group SMTP email for each user via `UserNotifications`, we are changing to send only one email with the existing `Jobs::GroupSmtpEmail` job and `GroupSmtpMailer`. We are changing this job and mailer along with `PostAlerter` to make the first topic allowed user the `to_address` for the email and any other `topic_allowed_users` to be the CC address on the email. This is to cut down on emails sent via SMTP, which is subject to daily limits from providers such as Gmail. We log these details in the `EmailLog` table now.

In addition to this, we have changed `PostAlerter` to no longer rely on incoming email email addresses for sending the `GroupSmtpEmail` job. This was unreliable as a user's email could have changed in the meantime. Also it was a little overcomplicated to use the incoming email records -- it is far simpler to reason about to just use topic allowed users.

This also adds a fix to include cc_addresses in the EmailLog.addressed_to_user scope.
2021-06-28 08:55:13 +10:00
Roman Rizzi
2c918a3161
FEATURE: Staff can receive pending user reminders more frequently. (#13422)
* FEATURE: Staff can receive pending user reminders more frequently.

We now express the "pending_users_reminder_delay"  in minutes instead of hours so staff can have finer control over the delay.

We need to keep in mind that the reminders could still take up to 20 minutes, even when using a lower value. We send them from a scheduled job.

* Migrate to a new site setting for the reminders delay
2021-06-24 10:02:56 -03:00
Joffrey JAFFEUX
2654a6685c
DEV: adds support for bannered until (#13417)
ATM it only implements server side of it, as my need is for automation purposes. However it should probably be added in the UI too as it's unexpected to have pinned_until and no bannered_until.
2021-06-24 11:35:36 +02:00
Jarek Radosz
046a875222
DEV: Improve script/downsize_uploads.rb (#13508)
* Only shrink images that are used in Posts and no other models
* Don't save the upload if the size is the same
2021-06-24 00:09:40 +02:00
Bianca Nenciu
d1b2e9db3b
PERF: Update post uploads secure status in a job (#13459)
When secure uploads are enabled, editing a post with many uploads can
cause a timeout because the store has to be contacted for each upload.
2021-06-21 19:15:24 +03:00
Dan Ungureanu
c893b20298
FIX: Destroy invites of anonymized emails (#13404)
Anonymizing a user changed their email address, destroyed all
associated InvitedUser records, but did not destroy the invites
associated to user's email.
2021-06-17 10:45:40 +03:00
Martin Brennan
c659e3e95b
FIX: Make sure topic_user.bookmarked is synced in more places (#13383)
When we call Bookmark.cleanup! we want to make sure that
topic_user.bookmarked is updated for topics linked to the
bookmarks that were deleted. Also when PostDestroyer calls
destroy and recover. We have a job for this already --
SyncTopicUserBookmarked -- so we just utilize that.
2021-06-16 08:30:40 +10:00
Roman Rizzi
4dc8c3c409
FEATURE: Blocking is optional when deleting a user from the review queue. (#13375)
Subclasses must call #delete_user_actions inside build_actions to support user deletion. The method adds a delete user bundle, which has a delete and a delete + block option. Every subclass is responsible for implementing these actions.
2021-06-15 12:35:45 -03:00
Dan Ungureanu
fa02775095
PERF: Perform user filtering in SQL (#13358)
Notifying about a tag change sometimes resulted in loading a large
number of users in memory just to perform an exclusion. This commit
prefers to do inclusion (i.e. instead of exclude users X, do include
users in groups Y) and does it in SQL to avoid fetching unnecessary
data that is later discarded.
2021-06-11 10:55:50 +10:00
Sam
023ff9a282
DEV: ensure user export ordering is predictable (#13340)
Flaky spec due to random ordering for the post_actions table.

Introduces consistent ordering.
2021-06-09 15:55:52 +10:00
Dan Ungureanu
da2889a7a8
DEV: Add more verbose logging for image uploads (#13270)
Image optimization fails randomly (very rare) without a trace and it is
near impossible to find culprit image, reproduce the issue and attempt
to fix.
2021-06-04 15:13:58 +03:00
Dan Ungureanu
197e3f24ce
FEATURE: Show stale reviewable to other clients (#13114)
The previous commits removed reviewables leading to a bad user
experience. This commit updates the status, replaces actions with a
message and greys out the reviewable.
2021-05-26 09:47:35 +10:00
Josh Soref
59097b207f
DEV: Correct typos and spelling mistakes (#12812)
Over the years we accrued many spelling mistakes in the code base. 

This PR attempts to fix spelling mistakes and typos in all areas of the code that are extremely safe to change 

- comments
- test descriptions
- other low risk areas
2021-05-21 11:43:47 +10:00
Dan Ungureanu
d903d4dc5a
DEV: Periodically delete old email change requests (#13054)
Email change requests are never deleted no matter if they completed
successfully or not. The abandoned requests have the disadvantage of
showing up as unconfirmed emails in user's preferences page.
2021-05-14 10:34:56 +03:00
Roman Rizzi
d4b5a81b05
FIX: Recalculate scores only when approving or transitioning to pending. (#13009)
Recalculating a ReviewableFlaggedPost's score after rejecting or ignoring it sets the score as 0, which means that we can't find them after reviewing. They don't surpass the minimum priority threshold and are hidden.

Additionally, we only want to use agreed flags when calculating the different priority thresholds.
2021-05-10 14:09:04 -03:00
Jeff Wong
75e159f0ed
FEATURE: add support for like webhooks (#12917)
* FEATURE: add support for like webhooks

Add support for like webhooks. Webhook events only send on user membership
in the defined webhook group filters.

This also fixes group webhook events, as before this was never used, and
the logic was not correct.
2021-04-30 17:08:38 -07:00
Roman Rizzi
60059a7190
FEATURE: A low priority filter for the review queue. (#12822)
This filter hides reviewables with a score lower than the "reviewable_low_priority_threshold" setting. We only use reviewables that already met this threshold to calculate the Medium and High priority filters.
2021-04-23 15:34:24 -03:00
Bianca Nenciu
8c4a11c006
DEV: Move autotag to topic creator (#12790)
This move was necessary to automatically tag the topic with the right
tags from creation time. The process post job may be delayed for a
short time.
2021-04-23 16:55:34 +03:00
Dan Ungureanu
31d3990986
FIX: Send a different message if export fails (#12799)
It used to check if an upload record exists, which is wrong because an
invalid upload record exists even if the upload was not created.

The other improvement is a better log message.
2021-04-22 20:21:31 +03:00
Sam
e4f1760bab
FEATURE: watch title for automatic tagging (#12782)
Previously watched words ignored topic titles when applying auto tagging rules.

Also copy has been improved to reflect how the system behaves.

The text hints that we are only watching first post now
2021-04-21 18:16:25 +03:00
Sam
5c49009c6c
PERF: properly preload emails to speed up user exports (#12778)
scopes are incredibly annoying to preload, simply adding :user_emails is not
enough.

Instead of relying on scopes simply iterate through user_emails which is
properly preloaded.

This removes 2 * N+1 when generating user reports.
2021-04-21 10:42:07 +10:00
Gerhard Schlager
92e222d246
FIX: POP3 polling shouldn't stop after exception or old email (#12742) 2021-04-19 10:27:29 +02:00
Dan Ungureanu
99dadb2129
PERF: Async notify users after inviting group (#12697)
Inviting a group generates a notification for each member. If this
happens synchronously it may take a while, leading to a poor user
experience.
2021-04-14 19:30:51 +03:00
Martin Brennan
66d17fdd6b
FIX: Topic user bookmarked column is out of sync after post moves (#12612)
When posts are moved from one topic to another, the `topic_user.bookmarked` column for all users in the new and the old topic needs to be resynced, for example because a user bookmarks post 12 in topic 1, then it is moved to topic 2, the topic_user record for topic 1 should no longer be bookmarked. A background job has been added to sync the column for a specified topic, or for no topic at all, which does it for all topics like the migration.

Also includes a migration that we have run in the past to fix bad data.

----

This has been addressed in other places in the past:

https://github.com/discourse/discourse/pull/10211
https://github.com/discourse/discourse/pull/10188
2021-04-14 09:10:53 +10:00
Dan Ungureanu
e704f0a541
FIX: Autocorrect values for dropdown imported user fields (#12572)
When bulk inviting, the uploaded CSV file may contain wrong values for
the user fields. This tries to automatically correct them by finding
the most similar option (by ignoring the case).
2021-03-31 19:19:57 +03:00
Dan Ungureanu
e8c576cca9
FIX: User fields are case insensitive in bulk CSV (#12559)
The CSV column title had to be case sensitive match with the name of
the user field which was unnecessary complex.
2021-03-31 13:42:53 +03:00
Dan Ungureanu
8335c8dc1a
FEATURE: Allow admins to pre-populate user fields (#12361)
Admins can use bulk invites to pre-populate user fields. The imported
CSV file must have a header with "email" column (first position) and
names of the user fields (exact match).

Under the hood, the bulk invite will create staged users and populate
the user fields of those.
2021-03-29 14:03:19 +03:00
Bianca Nenciu
97623f5351
FIX: Do not raise if post no longer exists (#12428) 2021-03-17 19:22:05 +02:00
Bianca Nenciu
16b5fa030b
DEV: Set disable_mailing_list_mode automatically (#12402)
The user mailing list mode continued to be silently enabled and
UserEmail job checked just that ignoring site setting
disable_mailing_list_mode.

An additional migrate was added to set disable_mailing_list_mode
to false if any users enabled the mailing list mode already.
2021-03-17 17:39:10 +02:00
Osama Sayegh
a23d0f9961
UX: Add image uploader widget for uploading badge images (#12377)
Currently the process of adding a custom image to badge is quite clunky; you have to upload your image to a topic, and then copy the image URL and pasting it in a text field. Besides being clucky, if the topic or post that contains the image is deleted, the image will be garbage-collected in a few days and the badge will lose the image because the application is not that the image is referenced by a badge.

This commit improves that by adding a proper image uploader widget for badge images.
2021-03-17 08:55:23 +03:00
Dan Ungureanu
fb19ee9eee
FIX: Correctly use invite to topic email templates (#12411)
It was used both when inviting from a topic page and when creating
invites with "Send to topic on first login", while it should be used
only in the former case.
2021-03-16 17:08:54 +02:00
David Taylor
bce837db0c
FIX: Update excerpt after process_post (#12340)
Onebox content may only be resolved during the process_post job. Onebox content could change the content of the excerpt, so we need to make sure the excerpt is updated accordingly.
2021-03-10 17:07:13 +00:00
David Taylor
4430bc153d
FIX: Do not clean up uploads when they're used by theme settings (#12326)
We intend to move ThemeSetting to use an upload_id column, rather than storing the URL. So this is a short-term solution.
2021-03-09 19:16:45 +00:00
Blake Erickson
2123561125
FIX: Mobile app notification urls w/ subfolder (#12282)
The urls that we generate for mobile post notifications don't take into
account the subfolder url if a site happens to have one configured. When
this happens when you tap on a new mobile notification it takes you to
a url that doesn't work because it is missing the subfolder portion.

I honestly think this should be handled in the Post model like we do
with the Topic model. `Post.url` should know how to handle subfolder
installs, but that seemed like a very risky change because there are
lots of other places in the codebase where we tack on the base_path and
I didn't want to risk duplicating it.

I also found a small typo in the topics controller spec.
2021-03-04 07:07:37 -07:00
Dan Ungureanu
c047640ad4
FEATURE: Various improvements to invite system (#12023)
The user interface has been reorganized to show email and link invites
in the same screen. Staff has more control over creating and updating
invites. Bulk invite has also been improved with better explanations.

On the server side, many code paths for email and link invites have
been merged to avoid duplicated logic. The API returns better responses
with more appropriate HTTP status codes.
2021-03-03 11:45:29 +02:00
Bianca Nenciu
b49b455e47
FEATURE: Autotag watched words (#12244)
New topics with be matched against a set of watched words and be
tagged accordingly.
2021-03-03 10:53:38 +02:00
Rafael dos Santos Silva
83f332b5a5
FEATURE: Add a site setting to allow emojis to come from an external URL (#12180) 2021-03-02 16:04:16 -03:00
Gerhard Schlager
a96a5db0fb
DEV: Add option to send system message to groups (#12256) 2021-03-02 18:51:50 +01:00
Roman Rizzi
4e716e9ce5
FIX: Reduce the time_read threshold to one minute. (#12159)
* FIX: Reduce the time_read threshold to one minute.

Five minutes is too much and could fill the queue with false positives.

* Update spec/jobs/enqueue_suspect_users_spec.rb

Co-authored-by: Arpit Jalan <arpit@techapj.com>

Co-authored-by: Arpit Jalan <arpit@techapj.com>
2021-02-20 08:25:32 -03:00
Roman Rizzi
95fb363c2a
FEATURE: Use the "time_read" stat to flag users as suspicious. (#12145)
Completing the discobot tutorial gives you ~3m of reading time, so we set the limit at 5m. Additionally, we use an "OR" clause to cover the case when you just scroll through a single topic.
2021-02-19 13:10:19 -03:00
Krzysztof Kotlarek
ad3ec5809f
FIX: Dismiss new with better migration (#12062)
Original PR was reverted because of broken migration https://github.com/discourse/discourse/pull/12058

I fixed it by adding this line
```
          AND topics.id IN(SELECT id FROM topics ORDER BY created_at DESC LIMIT :max_new_topics)
```

This time it is left joining a limited amount of topics. I tested it on few databases and it worked quite smooth
2021-02-15 08:50:33 +11:00
Penar Musaraj
900d4187ef
DEV: Prevents rate limits for new feature checks on multisite (#12053) 2021-02-12 08:52:59 -05:00
Krzysztof Kotlarek
a696cc07d2
Revert "FEATURE: Ability to dismiss all new topics (#12018)" (#12058)
This reverts commits 7426764af4 and f5b18e2a31
2021-02-12 08:50:25 +11:00
Krzysztof Kotlarek
f5b18e2a31
FEATURE: Ability to dismiss all new topics (#12018)
Follow up https://github.com/discourse/discourse/pull/11968

Dismiss all new topics using the same DismissTopicService. In addition, MessageBus receives exact topic ids which should be marked as `seen`.
2021-02-11 13:35:09 +11:00
Krzysztof Kotlarek
a1aa37758c
FIX: do not notify when the hidden tag is added or removed (#12025)
The bug was mentioned on meta https://meta.discourse.org/t/users-are-seeing-handling-of-unhandled-tag-again/155367

It was related to users who are watching a specific topic. In that case, when the hidden tag was added or removed to the topic they were notified by `NotifyTagChangeJob`.

That job should take hidden tags into consideration. If all changed tags are in a hidden group, it should exclude user not belong to that group.

At the same time, if visible to anyone tag is added or removed users watching topic should be notified.
2021-02-11 10:03:45 +11:00
Bianca Nenciu
f4db1675f3
FIX: Check if post.topic exists before publishing topic updates (#11900) 2021-02-09 16:41:22 +11:00
David Taylor
821bb1e8cb
FEATURE: Rename 'Discourse SSO' to DiscourseConnect (#11978)
The 'Discourse SSO' protocol is being rebranded to DiscourseConnect. This should help to reduce confusion when 'SSO' is used in the generic sense.

This commit aims to:
- Rename `sso_` site settings. DiscourseConnect specific ones are prefixed `discourse_connect_`. Generic settings are prefixed `auth_`
- Add (server-side-only) backwards compatibility for the old setting names, with deprecation notices
- Copy `site_settings` database records to the new names
- Rename relevant translation keys
- Update relevant translations

This commit does **not** aim to:
- Rename any Ruby classes or methods. This might be done in a future commit
- Change any URLs. This would break existing integrations
- Make any changes to the protocol. This would break existing integrations
- Change any functionality. Further normalization across DiscourseConnect and other auth methods will be done separately

The risks are:
- There is no backwards compatibility for site settings on the client-side. Accessing auth-related site settings in Javascript is fairly rare, and an error on the client side would not be security-critical.
- If a plugin is monkey-patching parts of the auth process, changes to locale keys could cause broken error messages. This should also be unlikely. The old site setting names remain functional, so security-related overrides will remain working.

A follow-up commit will be made with a post-deploy migration to delete the old `site_settings` rows.
2021-02-08 10:04:33 +00:00
Martin Brennan
4af77f1e38
FEATURE: Allow durations < 1 hour and < 1 day for topic timers where duration is specified (auto delete replies, close based on last post) (#11961)
This PR allows entering a float value for topic timers e.g. 0.5 for 30 minutes when entering hours, 0.5 for 12 hours when entering days. This is achieved by adding a new column to store the duration of a topic timer in minutes instead of the ambiguous both hours and days that it could be before.

This PR has ommitted the post migration to delete the duration column in topic timers; it will be done in a subsequent PR to ensure that no data is lost if the UPDATE query to set duration_mintues fails.

I have to keep the old keyword of duration in set_or_create_topic_timer for backwards compat, will remove at a later date after plugins are updated.
2021-02-05 10:12:56 +10:00
Krzysztof Kotlarek
f39e7fe81d
FEATURE: New way to dismiss new topics (#11927)
This is a try to simplify logic around dismiss new topics to have one solution to work in all places - dismiss all-new, dismiss new in a specific category or even in a specific tag.
2021-02-04 11:27:34 +11:00
Robin Ward
93c25070fa
FIX: In FastImage 2.2.2 an error is raised with a nil path (#11954)
* FIX: In FastImage 2.2.2 an error is raised with a `nil` path

Sometimes Discourse.store.path_for would return `nil`, which the job
handled gracefully before, but raises an error with the new version of
the gem.

Note the logic of this job is a bit awkward since it depends on `nil`
being a string, but at least now it's no longer filling logs with
errors.

* Update app/jobs/onceoff/fix_invalid_gravatar_uploads.rb

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>
2021-02-03 11:45:12 -05:00
Alan Guo Xiang Tan
0cc178d58b
FIX: Avoid pulling hotlinked images of post that have been deleted. (#11913) 2021-02-03 16:45:07 +11:00
Martin Brennan
6d72c8ab19
FEATURE: Topic timer UI revamp (#11912)
This PR revamps the topic timer UI, using the time shortcut selector from the bookmark modal.

* Fixes an issue where the duration of hours/days after last reply or auto delete replies was not enforced to be > 0
* Fixed an issue where the timer dropdown options were not reloaded correctly if the topic status changes in the background (use `MessageBus` to publish topic state in the open/close timer jobs)
* Moved the duration input and the "based on last post" option from the `future-date-input` component, as it was only used for topic timers. Also moved out the notice that is displayed which was also only relevant for topic timers.
2021-02-03 10:13:32 +10:00
Penar Musaraj
49e97279c7 FEATURE: Add daily job to check for new features 2021-02-01 10:31:44 +08:00
Mark VanLandingham
809274fe0d
DEV: Replace 'processed' column on notifications with new table (#11864) 2021-01-27 10:29:24 -06:00
Régis Hanol
cd3d24ed8c
FIX: move post_search_data migration into onceoff job (#11851)
And reduce the size of the batches to 100k.

That should hopefully make the migrations run smoother...
2021-01-26 16:29:00 +01:00
Régis Hanol
aa1138ff71
FIX: reindex_search job should work on model with no search data (#11819)
Lots of changes but it's mostly a refactoring.

The interesting part that was fix are the 'load_problem_<model>_ids' methods.
They will now return records with no search data associated so they can be properly indexed for the search.
This "bad" state usually happens after a migration.
2021-01-25 11:23:36 +01:00
Mark VanLandingham
1a7922bea2
FEATURE: Create notification schedule to automatically set do not disturb time (#11665)
This adds a new table UserNotificationSchedules which stores monday-friday start and ends times that each user would like to receive notifications (with a Boolean enabled to remove the use of the schedule). There is then a background job that runs every day and creates do_not_disturb_timings for each user with an enabled notification schedule. The job schedules timings 2 days in advance. The job is designed so that it can be run at any point in time, and it will not create duplicate records.

When a users saves their notification schedule, the schedule processing service will run and schedule do_not_disturb_timings. If the user should be in DND due to their schedule, the user will immediately be put in DND (message bus publishes this state).

The UI for a user's notification schedule is in user -> preferences -> notifications. By default every day is 8am - 5pm when first enabled.
2021-01-20 10:31:52 -06:00
Martin Brennan
fb184fed06
DEV: Add created_via column to IncomingEmail (#11751)
This should make it easier to track down how the incoming email was created, which is one of four locations:

The POP3 poller (which picks up reply via email replies)
The admin email controller #handle_mail (which is where hosted mail is sent)
The IMAP sync tool
The group SMTP mailer, which sends emails when replying to IMAP topics, pre-emptively creating IncomingEmail records to avoid double syncing
2021-01-20 13:22:41 +10:00
Martin Brennan
0034cbda8a
DEV: Change Topic Timer from enqueue_at scheduled jobs to incrementally executed jobs (#11698)
Moves the topic timer jobs from being scheduled ahead of time with enqueue_at to a 5 minute scheduled run like bookmark reminders, in a new job called Jobs::EnqueueTopicTimers. Backwards compatibility is maintained by checking if an existing topic timer job is enqueued in sidekiq for the timer, and if it is not running it inside the new job.

The functionality to close/open a topic if it is in the opposite state still remains in the after_save block of TopicTimer, with further commentary, which is used for Open/Close Temporarily.

This also removes the ensure_consistency! functionality of topic timers as it is no longer needed; the new job will always pick up the timers because they are not stored in a fragile state of sidekiq.
2021-01-19 13:30:58 +10:00
Martin Brennan
5710d5d771
FIX: Do not process pop3 mails > 1 week old (#11740)
This adds a safe default to not process pop3 emails when the pop3 polling option is set up that are > 1 week old. This is to avoid the situation where an older mailbox is used, which causes us to go and process all emails in that mailbox, sending out error emails to the senders of emails which cannot be parsed successfully.
2021-01-19 09:49:50 +10:00
Krzysztof Kotlarek
06b7c44593
FEATURE: reason to reject user signup (#11700)
Feature for `Must Approve Users` setup. When a user is rejected, a staff member can optionally set a reason for audit purposes. In addition, feedback email can be sent to the user.

Meta: https://meta.discourse.org/t/account-rejection-email/103112/8
2021-01-15 09:43:26 +11:00
Martin Brennan
2404fa7a23
DEV: Split toggle topic close job (#11679)
Splits the `ToggleTopicClosed` job into two distinct `OpenTopic` and `CloseTopic` jobs to make the code clearer. The old job cannot be deleted yet because of outstanding sidekiq schedules, so a todo has been added to do so later this year.

Also replaced mentions of `topic_status_update` with `topic_timer` in some files, because the `topic_status_update` model is obsolete and replaced by topic timer.

Added some shortcut methods for checking if a topic is open/whether a user can change an open topic.
2021-01-13 08:49:29 +10:00
jbrw
b1f32f2f57
DEV: TopicTrackingState calls should happen in the background (#11682)
* DEV: TopicTrackingState calls should happen in the background

It was observed that calling TopicTrackingState on popular topics could result in a large number of calls to redis, resulting in slow response times when posting replies.

These calls should be moved to a background job.

* DEV: PostUpdateTopicTrackingState should execute on default queue
2021-01-11 15:58:27 -05:00
Mark VanLandingham
4601f3be7e
FEATURE: Send notification emails when users leave do not disturb mode (#11643) 2021-01-07 10:49:49 -06:00
Martin Brennan
64ba5b1d21
FIX: Group SMTP email improvements (#11633)
Fixes a rare race condition causing the `Imap::Sync` class to create an incoming email and associated post/topic, which then kicks off the PostAlerter to notify others in the PM about a reply in the topic, but for the OP which is not necessary (because the person emailing the IMAP inbox already knows about the OP). Basically, we should never be sending the group SMTP email for the first post in a topic.

Also in this PR:

* Custom attribute accessors for the to/from/cc addresses on `IncomingEmail`, to parse them from an array to a joined string so the logic for this is only in one place.
* Store extra detail against the `IncomingEmail` created in `GroupSmtpMailer`
* regex test Mail header Reply-To as string instead of Field, which fixes `warning: deprecated Object#=~ is called on Mail::Field; it always returns nil`
* Add DEBUG_IMAP to log all IMAP logs as warnings for easier debugging
* Changed the Rails logging to `ImapSyncLog` in the `GroupSmtpMailer`
2021-01-05 15:32:04 +10:00
Bianca Nenciu
755627caa5
FEATURE: Introduce skip_auto_delete_reply_likes site setting (#11562)
osts from topics with 'auto delete replies timer' with more than
skip_auto_delete_reply_likes likes will no longer be deleted. If 0,
all posts will be deleted.
2020-12-23 16:30:10 +11:00
Roman Rizzi
8a7fe3b276
FIX: Don't enqueue imported users when there're multiple custom fields. (#11559)
My initial implementation didn't consider this case. We should skip imported users if the "imported_id" field is present, even if there're other custom fields.
2020-12-22 14:28:07 -03:00
Krzysztof Kotlarek
6dd3f986b2
FIX: no email when the category is muted and mailing list is on (#11490)
When `mute_all_categories_by_default` setting is enabled we should not send mailing list until category, tag or topic is explicitly watched.
2020-12-16 09:30:21 +11:00
Roman Rizzi
c7b9f044a4
FEATURE: Automatically disable slow mode. (#11461)
Staff and TL4 users can decide the slow mode duration. We'll internally set a topic timer to disable it.
2020-12-14 14:06:50 -03:00
Vinoth Kannan
eb60fc86dc
PERF: run user merging task in a background job. (#10961)
* PERF: run user merging task in a background job.

Currently, admin page is timing out while merging the users with lots of posts.
2020-12-10 15:52:08 +11:00
Kane York
901a45eeb3
FEATURE: Add likes, flags to user data export (#11439)
This commit is dedicated to https://twitter.com/FiloSottile/status/1335666583126073354 for reminding me that like timestamps are valuable data.

Likes additionally include the topic_id and post_number of the acted post, to aid in analysis. Flag export does not include the disposition by staff.
2020-12-09 15:34:13 -08:00
David Taylor
c69bb5d5be
DEV: Always enqueue sidekiq jobs after database transaction commit (#11293)
When jobs are enqueued inside a transaction, it's possible that they will be executed before the necessary data is available in the database. This commit ensures all jobs are enqueued in an ActiveRecord after_commit hook.

One potential downside here is if the job fails to enqueue, the transaction will no longer be aborted. However, the chance of that happening is reasonably low, and the impact is significantly lower than the current issue where jobs are scheduled before their data is ready.
2020-12-08 11:05:01 +11:00
Krzysztof Kotlarek
9c5ee4923b
FEATURE: silently close topic (#11392)
New TopicTimer to silently close topic. It will be used by discourse-solved plugin

Meta: https://meta.discourse.org/t/allow-auto-close-for-solved-to-do-so-silently/169300
2020-12-03 10:43:19 +11:00
Krzysztof Kotlarek
b7490e894f FIX: don't error-index category job when missing category
When the category is missing we should not throw an error but just finish the job.
2020-12-01 09:30:53 +08:00
Gerhard Schlager
333f0af0ec
FIX: Cached badge_count isn't updated after backfilling badges (#11281) 2020-11-18 22:01:56 +01:00
Joffrey JAFFEUX
de174ef0c4
DEV: cap notifications per run at 300 as stated in comment (#11252) 2020-11-17 09:08:12 +10:00
Arpit Jalan
fec9d6e578
FIX: when creating linked topics make sure they belong to same category (#11188)
This PR fixes a bug where the newly created linked topic was being
assigned "uncategorized" category. Now we'll assign linked topics the
same category as that of parent topic.

Meta reference: https://meta.discourse.org/t/linked-topics-splitting-and-managing-megatopics/168992/10?u=techapj
2020-11-11 00:44:27 +05:30
Guo Xiang Tan
650da7b626 PERF: Update index for category in a background job.
Search indexing can get expensive and there is no need for us to block
the entire request just to wait for index to finish.
2020-11-09 13:51:26 +08:00
Arpit Jalan
436bd48512
UX: update topic small action post to add link for new linked topic URL (#11132)
https://meta.discourse.org/t/linked-topics-splitting-and-managing-megatopics/168992/4?u=techapj
2020-11-05 22:39:21 +05:30
Arpit Jalan
d237ada12f
FIX: enqueue PostCreator jobs outside transaction block (#11120)
The `enqueue_jobs` is not correctly post-processing the post since the
post is being created inside a transaction block. This commit explicitly
enqueues the job outside transaction block.
2020-11-04 21:41:05 +05:30
Arpit Jalan
24d1b1f159
UX: when creating linked topic use URL and let onebox work its magic (#11118)
When the linked topic is created we'll not hardcode the topic title and
let onebox work its magic instead so that the title can be updated
automatically.
2020-11-04 17:19:09 +05:30
David Taylor
5140ec9acf
DEV: Cleanup ignored user logic (#11107)
- IgnoredUser records should all now have an expiring_at value. This commit enforces that in the DB, and fixes any corrupt rows
- Changes to the ignored user list are now handled by the `/u/{username}/notification_level` endpoint. This allows setting expiration dates on the ignore. This commit removes the old logic for saving a list of usernames in the user preferences.
- Many specs were calling `IgnoredUser.create`. This commit changes them to use `Fabricate(:ignored_user)` for consistency
2020-11-03 12:38:54 +00:00
David Taylor
1b7d39fa85
FIX: Remove 4 month limit on IgnoredUser records (#11105)
b8c676e7 added the 'forever' option to the UI, and this is correctly stored in the database. However, we had a hard-coded limit of 4 months in the cleanup job. This commit removes the limit, so ignores can last forever.
2020-11-03 12:12:43 +00:00
Arpit Jalan
1476e17c35
FEATURE: new setting to create a linked topic on autoclosing mega topics (#11001)
This commit adds a site setting `auto_close_topics_create_linked_topic`
which when enabled works in conjunction with `auto_close_topics_post_count`
setting and creates a new linked topic for the topic just closed.

The auto-created new topic contains a link for all the previous topics
and the topic titles are appended with `(Part {n})`.

The setting is enabled by default.
2020-11-02 12:18:48 +05:30
Martin Brennan
632942e697
FIX: Ensure group SMTP and message builder always uses from address for Reply-To when IMAP is enabled (#11037)
There is a site setting reply_by_email_enabled which when combined with reply_by_email_address creates a Reply-To header in emails in the format "test+%{reply_key}@test.com" along with a PostReplyKey record, so when replying Discourse knows where to route the reply.

However this conflicts with the IMAP implementation. Since we are sending the email for a group via SMTP and from their actual email account, we want all replys to go to that email account as well so the IMAP sync job can pick them up and put them in the correct place. So if the group has IMAP enabled and configured, then the reply-to header will be correct.

This PR also makes a further fix to 64b0b50 by using the correct recipient user for the PostReplyKey record. If the post user is used we encounter this error:

if destination.user_id != user.id && !forwarded_reply_key?(destination, user)
  raise ReplyUserNotMatchingError, "post_reply_key.user_id => #{destination.user_id.inspect}, user.id => #{user.id.inspect}"
end
This is because the user above is found from the from_address, but the destination which is the PostReplyKey is made by the post.user, which will be different people.
2020-10-28 07:01:58 +10:00
Kane York
e35fcd3340
FEATURE: Include rejected queued posts in the user archive export (#10859)
Requested at https://meta.discourse.org/t/where-can-a-user-find-the-post-that-was-rejected-by-the-moderator/165671?u=riking

Field whitelisting is applied to the json field using Hash#slice, which was activesupport until Ruby 2.5.
2020-10-27 07:48:48 -07:00
Martin Brennan
64b0b50ac0
FIX: Pass user to Email::Sender to avoid broken reply key for group_smtp email (#10978)
Our Email::Sender class accepts an optional user argument, which is used to create a PostReplyKey record when present. This record is used to sub out the %{reply_key} placeholder in the Reply-To mail header, so if we do not pass in the user we get a broken Reply-To header.

This is especially problematic in the IMAP group SMTP situation, because these emails go to customers that we are replying to, and when they reply to us the email bounces! This fixes the issue by passing user to the Email::Sender when sending a group_smtp email but there is still more to do in another PR.

This Email::Sender optional user is a bit of a footgun IMO, especially because most of the time we use it there is a user we can source. I would like to do another PR for this after this one to make the parameter not optional, so we don't end up with these reply issues down the line again.
2020-10-22 10:49:08 +10:00
Bianca Nenciu
be5efc9410
FIX: Ensure old uploads can have animated field updated (#10963)
If admins decreased the maximum filesize limit the ActiveRecord
validations would fail.
2020-10-20 19:11:43 +03:00
Bianca Nenciu
43e52a7dc1
DEV: Remove gifsicle dependency (#10357)
Dependency on gifsicle, allow_animated_avatars and allow_animated_thumbnails
site settings were all removed. Animated GIF images are still allowed, but
the generated optimized images are no longer animated for those (which were
used for avatars and thumbnails).

The added 'animated' is populated by extracting information using FastImage.
This field was used to selectively reoptimize old animations. This process
happens in the background.
2020-10-16 13:41:27 +03:00
Martin Brennan
c3cede697d
FEATURE: Add weekly bookmark cleanup code (#10899)
When posts or topics are deleted we don't want to immediately delete associated bookmarks, so we have a grace period to recover them and their reminders if the post or topic is un-deleted. This PR adds a task to the Weekly scheduled job to go and delete bookmarks attached to posts or topics deleted > 3 days ago.
2020-10-14 09:38:57 +10:00
Bianca Nenciu
25b8ed740b
DEV: Make site setting type uploaded_image_list use upload IDs (#10401)
It used to be a list of concatenated upload URLs which was prone to
break.
2020-10-13 16:17:06 +03:00
Gerhard Schlager
bdbee36961 DEV: Fix typo 2020-10-07 23:43:11 +02:00
David Taylor
16daf3c986
FIX: Compare Jobs::UserEmail type argument as string (#10851)
In specs, symbols are passed through correctly. But in production, all arguments are provided to jobs as strings.

Followup to c0293339b8
2020-10-07 16:55:32 +01:00
David Taylor
c0293339b8
PERF: Do not enqueue digest emails when attempted recently (#10849)
Previously, Jobs::EnqueueDigestEmails would enqueue a digest job for every user, even if there are no topics to send. The digest job would exit, no email would send, and last_emailed_at would not change. 30 minutes later, Jobs::EnqueueDigestEmails would run again and re-enqueue jobs for the same users.

120fa8ad introduced a temporary mitigation for this issue, by randomly selecting a subset of those users each time.

This commit adds a new `digest_attempted_at` column to the `user_stats` table. This column is updated every time a digest job completes for a user. Using this, we can avoid scheduling digest jobs for the same user every 30 minutes. This also removes the random user selection in 120fa8ad, and instead prioritizes users who had digests attempted the longest time ago.
2020-10-07 15:30:38 +01:00
Sam
120fa8ad2f
PERF: Introduce absolute limit of digests per 30 minutes (#10845)
To avoid blocking the sidekiq queue a limit of 10,000 digests per 30 minutes
is introduced.

This acts as a safety measure that makes sure we don't keep pouring oil on
a fire.

On multisites it is recommended to set the number way lower so sites do not
dominate the backlog. A reasonable default for multisites may be 100-500.

This can be controlled with the environment var

DISCOURSE_MAX_DIGESTS_ENQUEUED_PER_30_MINS_PER_SITE
2020-10-07 17:30:15 +11:00
Martin Brennan
6e2be3e60b
FIX: When admin changes an email for the user the user must confirm the change (#10830)
See https://meta.discourse.org/t/changing-a-users-email/164512 for additional context.

Previously when an admin user changed a user's email we assumed that they would need a password reset too because they likely did not have access to their account. This proved to be incorrect, as there are other reasons a user needs admin to change their email. This PR:

* Changes the admin change email for user flow so the user is sent an email to confirm the change
* We now record who the email change request was requested by
* If the requested by user is admin and not the user we note this in the email sent to the user
* We also make the confirm change email route open to anonymous users, so it can be clicked by the user even if they do not have access to their account. If there is a logged in user we make sure the confirmation matches the current user.
2020-10-07 13:02:24 +10:00
Kane York
68e87bb58e
User export: profile as json, export auth token logs (#10819)
* FEATURE: Export the entire user profile as json, not just bio/website

* FEATURE: Add session log information to user export

Even though the columns are named 'auth_token' etc, the content is not actually usable to log into the forum with. Despite all that, it is still truncated for export, to avoid any 'token hash cracking' situations.
2020-10-06 15:51:53 -07:00
Roman Rizzi
a8c47e7c7f
SECURITY: Ensure users can see the topic before setting a topic timer. (#10841) 2020-10-06 16:49:06 -03:00
Arpit Jalan
3684337e4a FIX: email always settings were not being respected
https://meta.discourse.org/t/email-notification-for-messages/163937/10?u=techapj
2020-09-23 22:00:15 +05:30
Martin Brennan
5268568d23
FEATURE: Remove user topic timers and migrate to bookmarks with reminders (#10474)
This PR removes the user reminder topic timers, because that system has been supplanted and improved by bookmark reminders. The option is removed from the UI and all existing user reminder topic timers are migrated to bookmark reminders.

Migration does this:

* Get all topic_timers with status_type 5 (reminders)
* Gets all bookmarks where the user ID and topic ID match
* Loops through the found topic timers
  * If there is no bookmark for the OP of the topic, then we just create a bookmark with a reminder
  * If there is a bookmark for the OP of the topic and it does **not** have a reminder set, then just 
update it with the topic timer reminder
  * If there is a bookmark for the OP of the topic with a reminder then just discard the topic timer
* Cancels all outstanding user reminder topic timers
* **Trashes (not deletes) all user reminder topic timers**

Notes:

* For now I have left the user reminder topic timer job class in place; this is so the jobs can be cancelled in the migration. It and the specs will be deleted in the next PR.
* At a later date I will write a migration to delete all trashed user topic timers. They are not deleted here in case there are data issues and they need to be recovered.
* A future PR will change the UI of the topic timer modal to make it look more like the bookmark modal.
2020-09-14 11:11:55 +10:00
Kane York
e0a0928420
FEATURE: Add bookmarks to the user export (#10591) 2020-09-11 11:03:22 +10:00
Gerhard Schlager
ac70c48be4 FIX: Prevent "uploads are missing in S3" alerts after restoring a backup
After restoring a backup it takes up to 48 hours for uploads stored on S3 to appear in the S3 inventory. This change prevents alerts about missing uploads by preventing the EnsureS3UploadsExistence job from running in the first 48 hours after a restore. During the restore it  deletes the count of missing uploads from the PluginStore, so that an alert isn't triggered by an old number.
2020-09-10 21:37:48 +02:00
Guo Xiang Tan
9b75d95fc6 PERF: Keep track of first unread PM and first unread group PM for user.
This optimization helps to filter away topics so that the joins on
related tables when querying for unread messages is not expensive.
2020-09-09 14:05:41 +08:00
Blake Erickson
67dec38f31 FIX: Gravatar download attempt if user is missing their email
It is possible that a user could exist without an email, if so we should
not enqueue a job to download their gravatar.

This commit resolves this error that can occur:

```
Job exception: undefined method `email' for nil:NilClass
/var/www/discourse/app/models/user.rb:1204:in `email'
/var/www/discourse/app/jobs/regular/update_gravatar.rb:12:in `execute'
```

This commit also fixes the original spec which actually was wrong. The
job never enqueued in the original spec and so the gravatar was never
actually updated and the test was checking if the two values were the
same, but they were both null and never updated, so of course they were
the same!

A new test has also been added to make sure the gravatar job isn't
enqueued when a user's email is missing.
2020-09-02 20:19:46 -06:00
Krzysztof Kotlarek
9954a677ab
FIX: don't send mailing list for post with empty content (#10577)
discourse-assign is creating posts with empty content to show that a specific user was assign/unassigned for a specific topic.

It is causing confusing emails with empty content

The bug was mentioned here: https://meta.discourse.org/t/again-on-empty-emails-and-notifications-generated-on-topic-assignment/162213
2020-09-03 08:58:25 +10:00
Guo Xiang Tan
01600492de
FIX: Don't raise error in update username job if user has been deleted. 2020-09-02 11:17:17 +08:00
Kane York
26ec4fd25b
FIX: User export category preferences on a deleted category. (#10573)
Tests from a1dd761bd9 were incomplete and did not test a deleted category's category_users record.
2020-09-01 13:22:59 -07:00
Guo Xiang Tan
069a109cbb
DEV: Require scheduled job in development to avoid loading file twice.
This removes the need to memoize constant in order to avoid the "warning: already initialized constant".
2020-09-01 10:14:40 +08:00
Kane York
a1dd761bd9
FIX: Handle deleted categories in post export (#10567)
Fixes a crash when exporting my own archive on Meta.
2020-08-31 17:33:28 -07:00
Kane York
5ec5fbd7ba
User export improvements 2 (#10560)
* FEATURE: Use predictable filenames inside the user archive export

* FEATURE: Include badges in user archive export

* FEATURE: Add user_visits table to the user archive export
2020-08-31 15:26:51 -07:00
Kane York
c5dc729e77
FEATURE: Add category tracking state to user archive export (#10557)
Tackling a simple table for the first actual new file in the user archive export.
2020-08-28 13:16:31 -07:00
Kane York
225cdba676 DEV: drop the explicit .each in UserArchive CSV writing
I think this is mostly stylistic, but this helps prevent explosive typos in the enum_for() line.
2020-08-28 11:46:53 -07:00
Kane York
4aed861336 DEV: minor refactors to ExportUserArchive(Spec) 2020-08-28 11:46:53 -07:00
Kane York
7bf199b0c4 DEV: Switch to new ExportUserArchive job
We now use the newly created job class from the previous commit.
2020-08-28 11:46:53 -07:00
Kane York
a8560d741f DEV: Create ExportUserArchive as clone of ExportCsvFile
This is in preparation for improvements to the user archive export data.
Some refactors happened along the way, including calling the different _export methods 'components' of the zip file.

Additionally, make the test for post export much more comprehensive.

Copy sources:
  app/jobs/regular/export_csv_file.rb
  spec/jobs/export_csv_file_spec.rb
2020-08-28 11:46:53 -07:00
Guo Xiang Tan
a4ec4c06e3
Fix the build. 2020-08-27 13:50:07 +08:00
Guo Xiang Tan
40c6d90df3 PERF: Create a partial regular post_search_data index on large sites.
With the addition of `PostSearchData#private_message`, a partial
index consisting of only search data from regular posts can be created.
The partial index helps to speed up searches on large sites since PG
will not have to do an index scan on the entire search data index which
has shown to be a bottle neck.
2020-08-27 13:42:00 +08:00
Guo Xiang Tan
3cc761ac44
FIX: Clean up toggle closed topic timer when user is not authorized. 2020-08-26 12:59:05 +08:00
Guo Xiang Tan
5bca1aec48
FIX: Clean up topic_timers when no longer valid.
This was causing a bug where the jobs for the topic_timers will keep
getting enqueued over and over again.
2020-08-26 12:18:51 +08:00
Martin Brennan
4670b62969
DEV: IMAP log to database (#10435)
Convert all IMAP logging to write to a database table for easier inspection. These logs are cleaned up daily if they are > 5 days old.

Logs can easily be watched in dev by setting DISCOURSE_DEV_LOG_LEVEL=\"debug\" and running tail -f development.log | grep IMAP
2020-08-14 12:01:31 +10:00
Krzysztof Kotlarek
7194b31443
FEATURE: don't notify about changed tags for a private message (#10408)
* FEATURE: don't notify about changed tags for a private message

Only staff members observing specific tag should receive a notification

* FIX: remove other category which is not used

* FIX: improved specs to ensure that revise was succesful
2020-08-13 17:22:34 +10:00
Régis Hanol
bc63232d2e
FIX: sync reviewable count when opening the hamburger menu (#10368)
When a tab is open but left unattended for a while, the red, green, and blue
pills tend to go out of sync.

So whevener we open the notifications menu, we sync up the notification count
(eg. blue and green pills) with the server.

However, the reviewable count (eg. the red pill) is not a notification and
is located in the hamburger menu. This commit adds a new route on the server
side to retrieve the reviewable count for the current user and a ping
(refreshReviewableCount) from the client side to sync the reviewable count
whenever they open the hamburger menu.

REFACTOR: I also refactored the hamburger-menu widget code to prevent repetitive uses
of "this.".

PERF: I improved the performance of the 'notify_reviewable' job by doing only 1 query
to the database to retrieve all the pending reviewables and then tallying based on the
various rights.
2020-08-07 18:13:02 +02:00
David Taylor
cb12a721c4
REFACTOR: Refactor pull_hotlinked_images job
This commit should cause no functional change
- Split into functions to avoid deep nesting
- Register custom field type, and remove manual json parse/serialize
- Recover from deleted upload records

Also adds a test to ensure pull_hotlinked_images redownloads secure images only once
2020-08-05 12:14:59 +01:00
Gerhard Schlager
957e851ffe Revert "FIX: Regularly reset unknown extension of uploads"
This reverts commit cc7b24b88b as it shouldn't be needed anymore for new uploads.
2020-08-03 13:37:32 +02:00
Régis Hanol
da7e6b1f7e DEV: make rubocop 🤖 happy 2020-07-31 18:16:54 +02:00
Régis Hanol
3f59d0cffe
DEV: prevent initialized constant warning
Was tired of seeing the following warnings in the logs

```
/discourse/app/jobs/scheduled/old_keys_reminder.rb:7: warning: already initialized constant Jobs::OldKeysReminder::OLD_CREDENTIALS_PERIOD
/discourse/app/jobs/scheduled/old_keys_reminder.rb:7: warning: previous definition of OLD_CREDENTIALS_PERIOD was here
```
2020-07-31 18:05:12 +02:00
David Taylor
4ef389bc04
FIX: Define s3_helper in EnsureS3UploadsExistence job
Followup to 16c65a94
2020-07-29 12:23:04 +01:00
David Taylor
16c65a94f7
PERF: Preload S3 inventory data for multisite clusters 2020-07-29 10:31:55 +01:00
Krzysztof Kotlarek
e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Guo Xiang Tan
c6202af005
Update rubocop to 2.3.1. 2020-07-24 17:19:21 +08:00
Guo Xiang Tan
3766122a82
DEV: Allow developmental post search index versions. 2020-07-23 15:19:46 +08:00
Guo Xiang Tan
609ba50fe8
DEV: Add more granularity to SearchIndexer versions.
Sometimes, we just want to reindex a specific model and not all the
things.
2020-07-23 14:24:06 +08:00
jbrw
06073fe8c6
FEATURE: Allow group moderators to close/archive topics
* FEATURE: Allow group moderators to close/archive topics
2020-07-14 12:36:19 -04:00
Robin Ward
a73da42691 FIX: Don't award new user of the month in the wrong month
see: https://meta.discourse.org/t/new-user-of-the-month-badge-awarded-before-registraton-date/157347/2?u=eviltrout
2020-07-13 15:05:30 -04:00
Dan Ungureanu
c72bc27888
FEATURE: Implement support for IMAP and SMTP email protocols. (#8301)
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2020-07-10 12:05:55 +03:00
Penar Musaraj
67582e7d27
FIX: Do not send system emails to suspended users (#10192) 2020-07-08 13:30:32 -04:00
Vinoth Kannan
9426d12c1d FEATURE: trigger user_updated event if username is changed. 2020-07-03 00:14:19 +05:30
Sam Saffron
9ffc022cf4
DEV: improve verbose mode for reindexer
This makes the verbose mode provide a bit of progress notification
while reindexing as it can take many hours to do a giant site
2020-06-24 17:29:45 +10:00
Sam Saffron
dcad720a4c
DEV: add optional verbose logging to re-index job
This verbose logging can be useful when executing the job by hand
for debugging purposes

In general people will not use this
2020-06-24 15:37:08 +10:00
David Taylor
d473ddd02c
FIX: Use raw content when checking for watched words to flag (#10104)
This makes flagged watched words match the behavior of censor, requires_approval and blocked watched words.
2020-06-22 19:48:06 +01:00
Guo Xiang Tan
3370ef188e
FEATURE: Remove deprecated uploads url site settings.
The site settings have been replaced with direct image upload since
Discourse 2.3.
2020-06-22 14:32:29 +08:00
David Taylor
17c4f76eac
FIX: Do not attempt to pull_hotlinked on emoji images when CDN enabled (#10091) 2020-06-19 20:21:05 +01:00
David Taylor
a99bb0ded4
Revert "FIX: Do not attempt to pull_hotlinked on emoji images when CDN enabled"
This changed cause plugin spec failures and needs further investigation

This reverts commit 78626d2832.
2020-06-19 14:39:16 +01:00
David Taylor
9f2e7e4651
FIX: Handle invalid URLs gracefully when pulling hotlinked images 2020-06-19 12:52:51 +01:00
David Taylor
78626d2832
FIX: Do not attempt to pull_hotlinked on emoji images when CDN enabled 2020-06-19 12:45:06 +01:00
Dan Ungureanu
5bfe1ee4f1
FEATURE: Improve UX support for multiple email addresses (#9691) 2020-06-10 19:11:49 +03:00
Dan Ungureanu
3a7ca97c36
FIX: Use include-subcategories filter in report export (#10007)
Some filters were renamed and the conversion of the filter names and arguments
was removed.
2020-06-10 18:57:39 +03:00
Arpit Jalan
3094459cd9
FEATURE: multiple use invite links (#9813) 2020-06-09 20:49:32 +05:30
Robin Ward
4e3a84c687 FIX: If creating a flag for a watched word, include the reason 2020-06-02 11:49:02 -04:00
David Taylor
3106f85983
FIX: Support exporting reports which reference topics (#9957) 2020-06-01 18:23:58 +01:00
Krzysztof Kotlarek
9a6ef80739
FEATURE: notify admins about old credentials (#9918)
* FEATURE: notify admins about old credentials

Security and API keys should be renewed periodically.
This additional notification should help admins keep their Discourse safe and secure.
2020-06-01 13:49:27 +10:00
David Taylor
ecfce93f28
FIX: Support IRIs (unicode URIs) when pulling hotlinked images (#9928) 2020-05-29 17:47:05 +01:00
David Taylor
28f46c171c
FIX: Pull hotlinked images even when edited by system users (#9890)
Previously the pull hotlinked images job was skipped after system edits. This ensured that we never had an infinite loop of system-edit/pull-hotlinked/system-edit/pull-hotlinked etc.

A side effect was that edits made by system for any other reason (e.g. API, removing full quotes) would prevent pulling hotlinked images. This commit removes the system edit check, and replaces it with another method to avoid an infinite job scheduling loop.
2020-05-29 13:07:47 +01:00
Andrew Schleifer
74d28a43d1
new S3 backup layout (#9830)
* DEV: new S3 backup layout

Currently, with $S3_BACKUP_BUCKET of "bucket/backups", multisite backups
end up in "bucket/backups/backups/dbname/" and single-site will be in
"bucket/backups/".

Both _should_ be in "bucket/backups/dbname/"

- remove MULTISITE_PREFIX,
- always include dbname,
- method to move to the new prefix
- job to call the method

* SPEC: add tests for `VacateLegacyPrefixBackups` onceoff job.

Co-authored-by: Vinoth Kannan <vinothkannan@vinkas.com>
2020-05-29 00:28:23 +05:30
David Taylor
8a3d9d7036
DEV: Run jobs sequentially in test mode (#9897)
When running jobs in tests, we use `Jobs.run_immediately!`. This means that jobs are run synchronously when they are enqueued. Jobs sometimes enqueue other jobs, which are also executed synchronously. This means that the outermost job will block until the inner jobs have finished executing. In some cases (e.g. process_post with hotlinked images) this can lead to a deadlock.

This commit changes the behavior slightly. Now we will never run jobs inside other jobs. Instead, we will queue them up and run them sequentially in the order they were enqueued. As a whole, they are still executed synchronously. Consider the example

```ruby
class Jobs::InnerJob < Jobs::Base
  def execute(args)
    puts "Running inner job"
  end
end

class Jobs::OuterJob < Jobs::Base
  def execute(args)
    puts "Starting outer job"
    Jobs.enqueue(:inner_job)
    puts "Finished outer job"
  end
end

Jobs.enqueue(:outer_job)
puts "All jobs complete"
```

The old behavior would result in:

```
Starting outer job
Running inner job
Finished outer job
All jobs complete
```

The new behavior will result in:
```
Starting outer job
Finished outer job
Running inner job
All jobs complete
```
2020-05-28 12:52:27 +01:00
Blake Erickson
a788427b46
FIX: broken query if upload id is missing (#9900)
Log warnings if there are any errors trying to upload group flair image
and ensure upload.id exists before trying to use it in a sql query.
2020-05-27 12:39:30 -06:00
Régis Hanol
2a4db15544 FIX: don't send digests to users with no primary email
It might happen that some User records have no associated primary emails.
In which case we don't ever want to send them a digest.

Also added a new "user_email_no_email" skipped email log to ensure these cases
are properly handled and surfaced.
2020-05-27 17:09:40 +02:00
Krzysztof Kotlarek
34e5f0a9a3
Revert "FEATURE: notify admins about old credentials (#9854)" (#9886)
This reverts commit 349a67bee6.
2020-05-27 09:52:53 +10:00
Krzysztof Kotlarek
349a67bee6
FEATURE: notify admins about old credentials (#9854)
* FEATURE: notify admins about old credentials

Security and API keys should be renewed periodically.
This additional notification should help admins keep their Discourse safe and secure.
2020-05-27 08:13:47 +10:00
Vinoth Kannan
5fb9271878
DEV: ignore flair_url column in group model. (#9873) 2020-05-26 00:43:50 +05:30
Vinoth Kannan
505122bb45 FIX: skip onceoff job for groups with invalid flair URL. 2020-05-25 13:11:00 +05:30
Vinoth Kannan
8e56197728
UX: use "icon-picker" & "image-uploader" fields to set group flair. (#9779) 2020-05-25 11:08:47 +05:30
Kane York
869f9b20a2
PERF: Dematerialize topic_reply_count (#9769)
* PERF: Dematerialize topic_reply_count

It's only ever used for trust level promotions that run daily, or compared to 0. We don't need to track it on every post creation.

* UX: Add symbol in TL3 report if topic reply count is capped

* DEV: Drop user_stats.topic_reply_count column
2020-05-14 15:42:00 -07:00
Sam Saffron
451e9c4bb9
DEV: minor SQL formatting change
Moved join prior to left join to make query less confusing.

Has no material impact on performance.
2020-05-12 16:55:42 +10:00
Rafael dos Santos Silva
4d137b4f89
FIX: Handle a deleted topic in thumbnail generation
Follow up to 03818e642a

A topic can be deleted between job enqueue and job execution
2020-05-11 13:32:37 -03:00
Krzysztof Kotlarek
a3e1e9ef16
FIX: remove deprecated URI.escape (#9697)
During Nokogumbo changes I introduced back URI.escape which is deprecated.
2020-05-08 11:14:59 +10:00
Martin Brennan
fa572d3a7a
DEV: Remove all code referencing at_desktop bookmark reminders (#9650)
We have found no need for these reminder types, so we are removing the code for them.
2020-05-06 15:22:43 +10:00
David Taylor
03818e642a
FEATURE: Include optimized thumbnails for topics (#9215)
This introduces new APIs for obtaining optimized thumbnails for topics. There are a few building blocks required for this:

- Introduces new `image_upload_id` columns on the `posts` and `topics` table. This replaces the old `image_url` column, which means that thumbnails are now restricted to uploads. Hotlinked thumbnails are no longer possible. In normal use (with pull_hotlinked_images enabled), this has no noticeable impact

- A migration attempts to match existing urls to upload records. If a match cannot be found then the posts will be queued for rebake

- Optimized thumbnails are generated during post_process_cooked. If thumbnails are missing when serializing a topic list, then a sidekiq job is queued

- Topic lists and topics now include a `thumbnails` key, which includes all the available images:
   ```
   "thumbnails": [
   {
     "max_width": null,
     "max_height": null,
     "url": "//example.com/original-image.png",
     "width": 1380,
     "height": 1840
   },
   {
     "max_width": 1024,
     "max_height": 1024,
     "url": "//example.com/optimized-image.png",
     "width": 768,
     "height": 1024
   }
   ]
  ```

- Themes can request additional thumbnail sizes by using a modifier in their `about.json` file:
   ```
    "modifiers": {
      "topic_thumbnail_sizes": [
        [200, 200],
        [800, 800]
      ],
      ...
  ```
  Remember that these are generated asynchronously, so your theme should include logic to fallback to other available thumbnails if your requested size has not yet been generated

- Two new raw plugin outlets are introduced, to improve the customisability of the topic list. `topic-list-before-columns` and `topic-list-before-link`
2020-05-05 09:07:50 +01:00
Krzysztof Kotlarek
9bff0882c3
FEATURE: Nokogumbo (#9577)
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
Kane York
784bf2a173
FIX: Unassign user titles when a badge is deleted (#9573) 2020-05-02 18:02:28 -07:00
Sam Saffron
d0d5a138c3
DEV: stop freezing frozen strings
We have the `# frozen_string_literal: true` comment on all our
files. This means all string literals are frozen. There is no need
to call #freeze on any literals.

For files with `# frozen_string_literal: true`

```
puts %w{a b}[0].frozen?
=> true

puts "hi".frozen?
=> true

puts "a #{1} b".frozen?
=> true

puts ("a " + "b").frozen?
=> false

puts (-("a " + "b")).frozen?
=> true
```

For more details see: https://samsaffron.com/archive/2018/02/16/reducing-string-duplication-in-ruby
2020-04-30 16:48:53 +10:00
Dan Ungureanu
7a946b9f5c
DEV: Fix typo
Follow-up to 0a85a7aef9.
2020-04-29 12:38:57 +03:00
Dan Ungureanu
0a85a7aef9
FEATURE: Add user_profile to user_archive CSV export (#9571) 2020-04-29 12:09:50 +03:00
Roman Rizzi
394babcae3
FIX: Only show the review page to users that can see it. Do not publish the reviewable count update message to everyone. (#9556) 2020-04-27 14:51:25 -03:00
Guo Xiang Tan
16f0d1c503
PERF: Remove database query when publishing to staff users. 2020-04-27 11:50:21 +08:00
Jarek Radosz
e0aec23844 DEV: Don't warn about valid data 2020-04-26 22:18:44 +02:00
Vinoth Kannan
df0c386f8a
UX: drop the automatic_membership_retroactive column from groups model. (#9430) 2020-04-22 22:07:39 +05:30
Martin Brennan
628ba9d1e2
FEATURE: Promote bookmarks with reminders to core functionality (#9369)
The main thrust of this PR is to take all the conditional checks based on the `enable_bookmarks_with_reminders` away and only keep the code from the `true` path, making bookmarks with reminders the core bookmarks feature. There is also a migration to create `Bookmark` records out of `PostAction` bookmarks for a site.

### Summary

* Remove logic based on whether enable_bookmarks_with_reminders is true. This site setting is now obsolete, the old bookmark functionality is being removed. Retain the setting and set the value to `true` in a migration.
* Use the code from the rake task to create a database migration that creates bookmarks from post actions.
* Change the bookmark report to read from the new table.
* Get rid of old endpoints for bookmarks
* Link to the new bookmarks list from the user summary page
2020-04-22 13:44:19 +10:00
Jarek Radosz
5a81e3999c
DEV: Remove bypass_bump from CookedPostProcessor (#9468)
It was only passing it along to `PullHotlinkedImages` and that class have not used that arg since April 2014 (c52ee665b4)
2020-04-21 03:48:19 +02:00
Robin Ward
25bed4f643 FIX: Concurrency issues with making topic embedded posts visible 2020-04-20 15:11:59 -04:00
David Taylor
b6c19cba20
FIX: Abort emit_web_hook_event job cleanly if web hook was deleted (#9445)
Raising an error causes the job to be retried, and causes a lot of noise in the logs
2020-04-16 21:24:09 +01:00
Neil Lalonde
074509fd95
FIX: don't demote users to TL2 when default trust level is 3
Within 24 hours of signing up, new users were losing their
default trust level of 3. With this fix, demotions from
trust level 3 won't happen when the "default trust level"
setting is 3 or 4.
2020-04-16 12:28:16 -04:00
Robin Ward
ce663d67e1 FIX: CSV Exports were throwing errors with invalid dates
This fix will consider any invalid dates to be non-existant.
2020-04-07 14:05:27 -04:00
David Taylor
3cfe086a94
FIX: Limit personal message participants when converting from topic (#9343)
Previously all topic posters would be added which could lead to major performance issues. Now if there are too many posters, only the acting user will be added as a participant.
2020-04-03 16:42:01 +01:00
Daniel Waterworth
76bb01c31c DEV: Increase process_post lock timeout
The process_post job uses CookedPostProcessor which also uses a
DistributedMutex. There's no good reason for the timeout of the outer
lock to be smaller than the timeout of the inner lock.
2020-04-03 13:34:44 +01:00
Vinoth Kannan
82201fa466 FIX: jobs/delete_replies: Add Time+Duration, not Time+Time #9314
Co-authored-by: Kane York <kanepyork@gmail.com>
2020-04-03 09:23:40 +05:30
Joffrey JAFFEUX
aeaea3c154
FIX: correctly take category/group filters into csv export (#9300) 2020-03-30 19:08:47 +02:00
Joffrey JAFFEUX
96e841a635
DEV: trigger :username_changed when a user is renamed (#9301) 2020-03-30 18:02:34 +02:00
Martin Brennan
097851c135
FIX: Change secure media to encompass attachments as well (#9271)
If the “secure media” site setting is enabled then ALL files uploaded to Discourse (images, video, audio, pdf, txt, zip etc. etc.) will follow the secure media rules. The “prevent anons from downloading files” setting will no longer have any bearing on upload security. Basically, the feature will more appropriately be called “secure uploads” instead of “secure media”.

This is being done because there are communities out there that would like all attachments and media to be secure based on category rules but still allow anonymous users to download attachments in public places, which is not possible in the current arrangement.
2020-03-26 07:16:02 +10:00
Vinoth Kannan
aad12822b7
FEATURE: automatically delete replies on a topic after N days. (#9209) 2020-03-19 21:06:31 +05:30
Roman Rizzi
27bc4f51c7
FIX: Ignore suspect users that were migrated or users who were created more than six months ago (#9205) 2020-03-14 08:47:53 -03:00
Martin Brennan
793f39139a
FEATURE: Send notifications for time-based and At Desktop bookmark reminders (#9071)
* This PR implements the scheduling and notification system for bookmark reminders. Every 5 minutes a schedule runs to check any reminders that need to be sent before now, limited to **300** reminders at a time. Any leftover reminders will be sent in the next run. This is to avoid having to deal with fickle sidekiq and reminders in the far-flung future, which would necessitate having a background job anyway to clean up any missing `enqueue_at` reminders.

* If a reminder is sent its `reminder_at` time is cleared and the `reminder_last_sent_at` time is filled in. Notifications are only user-level notifications for now.

* All JavaScript and frontend code related to displaying the bookmark reminder notification is contained here. The reminder functionality is now re-enabled in the bookmark modal as well.

* This PR also implements the "Remind me next time I am at my desktop" bookmark reminder functionality. When the user is on a mobile device they are able to select this option. When they choose this option we set a key in Redis saying they have a pending at desktop reminder. The next time they change devices we check if the new device is desktop, and if it is we send reminders using a DistributedMutex. There is also a job to ensure consistency of these reminders in Redis (in case Redis drops the ball) and the at desktop reminders expire after 20 days.

* Also in this PR is a fix to delete all Bookmarks for a user via `UserDestroyer`
2020-03-12 10:16:00 +10:00
Roman Rizzi
b9aaa9718d
FIX: When must_approve_users is enabled, we don't want to send suspect users to the review queue. Only non-approved users should be sent. Provide a migration to auto-approve every problematic review item (#9179) 2020-03-11 17:05:44 -03:00
Roman Rizzi
826b4793c0
FEATURE: Approve suspect users is now true by default. The suspect users list was removed (#9151) 2020-03-10 08:56:42 -03:00
Jarek Radosz
85e03a7f68
DEV: Replace Time.new with Time.now (#9142)
(or `Time.zone.now`)
2020-03-09 17:37:49 +01:00
Gerhard Schlager
edc8d58ac3 FEATURE: Add site setting to disable staged user cleanup
... and disabled the cleanup during imports, otherwise a running Sidekiq might delete users before posts are created
2020-03-09 10:26:41 +01:00
Bianca Nenciu
f14dd1f82d
FIX: Prevent race condition when post processing post (#8819)
If a post is being cooked twice (for example after an edit), there is a
chance the 'raw' and 'cooked' column to be inconsistent. This reduces
the chances of that happening.
2020-03-07 14:36:54 +02:00
Kane York
10ddb8a9c4 FIX: Use destroy_all instead of delete_all for shared drafts
Rails has an odd behavior for calling .delete_all on a has_many relation - the
default behavior is to nullify the foreign key fields instead of actually
'DELETE'ing the records.

Additionally, publishing a shared draft topic creates a PostRevision that the
NotifyPostRevision job picks up which is then promptly deleted.

Use destroy_all when cleaning up the revisions and have the NotifyPostRevision
job tolerate deleted PostRevision records.

This takes a small performance hit (several SQL DELETEs instead of just one)
but shouldn't be too much of an issue (high cardinalities range from 30-100).
2020-03-05 11:13:43 -08:00
tshenry
a09e5d12c2
FIX: Topics should honor auto-close when published to category (#8963)
* FIX: Topics should honor auto-close when published to category

* Add test
2020-03-02 14:21:35 -05:00
Vinoth Kannan
5774107a2d
FIX: downloaded image URLs incorrectly replaced in post raw. (#9014)
Previously, while replacing the downloaded image URL `http://wiki.mozilla.org/images/2/2e/Longcat1.png` similar non-image URL `http://wiki.mozilla.org/images/2` was replaced wrongly.
2020-02-27 10:22:55 +05:30
Sam Saffron
372f6f4f22
FEATURE: limit number of notifications per user to 10,000
Introduces a new site setting `max_notifications_per_user`.

Out-of-the-box this is set to 10,000. If a user exceeds this number of
notifications, we will delete the oldest notifications keeping only 10,000.

To disable this safeguard set the setting to 0.

Enforcement happens weekly.

This is in place to protect the system from pathological states where a
single user has enormous amounts of notifications causing various queries
to time out. In practice nobody looks back more than a few hundred notifications.
2020-02-24 11:42:50 +11:00
Dan Ungureanu
cf0c6d5761
FIX: Ensure web hooks are retried at most 5 times 2020-02-21 17:02:40 +02:00
Robin Ward
345764565f FIX: Respect muted tags for mailing list mode
If a user has a tag muted, don't send them emails about that tag.
We've done this forever for categories so it makes sense to do it
for tags too.
2020-02-19 15:14:42 -05:00
Robin Ward
c954d083df Link website when reviewing users 2020-02-19 10:18:05 -05:00
Roman Rizzi
9441362c72
FEATURE: Support uploading a csv with either user emails or usernames (#8971) 2020-02-18 10:53:12 -03:00
Sam Saffron
28292d2759
PERF: avoid shelling to get hostname aggressively
Previously we had many places in the app that called `hostname` to get
hostname of a server. This commit replaces the pattern in 2 ways

1. We cache the result in `Discourse.os_hostname` so it is only ever called once

2. We prefer to use Socket.gethostname which avoids making a shell command

This improves performance as we are not spawning hostname processes throughout
the app lifetime
2020-02-18 15:13:19 +11:00
Roman Rizzi
2ee6a615b7
FEATURE: Send suspect users to the review queue (#8811) 2020-01-29 15:38:27 -03:00
Martin Brennan
ab3bda6cd0
FIX: Mitigate issue where legacy pre-secure hotlinked media would not be redownloaded (#8802)
Basically, say you had already downloaded a certain image from a certain URL
using pull_hotlinked_images and the onebox. The upload would be stored
by its sha as an upload record. Whenever you linked to the same URL again
in a post (e.g. in our case an og:image on review.discourse) we would
would reuse the original upload record because of the sha1.

However when you turned on secure media this could cause problems as
the first post that uses that upload after secure media is enabled
will set the access control post for the upload to the new post.
Then if the post is deleted every single onebox/link to that same image
URL will fail forever with 403 as the secure-media-uploads URL fails
if the access control post has been deleted.

To fix this when cooking posts and pulling hotlinked images, we only
allow using an original upload by URL if its access control post
matches the current post, and if the original_sha1 is filled in,
meaning it was uploaded AFTER secure media was enabled. otherwise
we just redownload the media again to be safe, as the URL will always
be new then.
2020-01-29 10:11:38 +10:00
Krzysztof Kotlarek
20e7fb1c95
FIX: correct notification when tag or category is added (#8801)
Regression was created here:
https://github.com/discourse/discourse/pull/8750

When tag or category is added and the user is watching that category/tag
we changed notification type to `edited` instead of `new post`.

However, the logic here should be a little bit more sophisticated.

If the user has already seen the post, notification should be `edited`.

However, when user hasn't yet seen post, notification should be "new
reply". The case for that is when for example topic is under private
category and set for publishing later. In that case, we modify an
existing topic, however, for a user, it is like a new post.

Discussion on meta:
https://meta.discourse.org/t/publication-of-timed-topics-dont-trigger-new-topic-notifications/139335/13
2020-01-29 11:03:47 +11:00
Gerhard Schlager
9621af2214 FIX: Award "User of the month" badge at the beginning of month
Previously the badge was granted one month after the last time the badge was granted. The exact date shifted by one day each month. The new logic tries to grant the badge always at the beginning of a new month by looking at new users of the previous month. The "granted at" date is set to the end of the previous month.
2020-01-28 22:59:13 +01:00
David Taylor
a9d0d55817 FEATURE: Add message to log when admins are automatically deactivated 2020-01-28 12:16:24 +00:00
Dan Ungureanu
5060811464
FIX: Export all category names in user archives (#8790) 2020-01-28 12:52:07 +02:00
Gerhard Schlager
ea11ad4d99 DEV: Drop unused columns 2020-01-27 15:28:56 +01:00
Jarek Radosz
8a82ceb3bc
FIX: Improve downsize_uploads (#8409)
With this change the script:
* Actually removes original large-sized images
* Doesn't save processed files if their size has increased
* Prevents inconsistent state
2020-01-27 03:31:11 +01:00
Martin Brennan
45b37a8bd1
FIX: Resolve pull hotlinked image and broken link issues for secure media URLs (#8777)
When pull_hotlinked_images tried to run on posts with secure media (which had already been downloaded from external sources) we were getting a 404 when trying to download the image because the secure endpoint doesn't allow anon downloads.

Also, we were getting into an infinite loop of pull_hotlinked_images because the job didn't consider the secure media URLs as "downloaded" already so it kept trying to download them over and over.

In this PR I have also refactored secure-media-upload URL checks and mutations into single source of truth in Upload, adding a SECURE_MEDIA_ROUTE constant to check URLs against too.
2020-01-24 11:59:30 +10:00
Martin Brennan
1b3b0708c0
FEATURE: Update upload security status on post move, topic conversion, category change (#8731)
Add TopicUploadSecurityManager to handle post moves. When a post moves around or a topic changes between categories and public/private message status the uploads connected to posts in the topic need to have their secure status updated, depending on the security context the topic now lives in.
2020-01-23 12:01:10 +10:00
Bianca Nenciu
7b7e1717f2
FIX: Quoting a quote preserves the original post information (#8746)
Let's say post #2 quotes post number #1. If a user decides to quote the
quote in post #2, it should keep the information of post #1
("user_1, post: 1, topic: X"), instead of replacing with current post
info ("user_2, post: 2, topic: X").
2020-01-22 16:10:23 +02:00
Bianca Nenciu
0a27086764
FEATURE: Export all types of reports (#8748)
There is a single stacked_chart which was not exportable
2020-01-21 18:43:19 +02:00
Krzysztof Kotlarek
0420be88a6
FIX: when tag or category is added notify users that topic was modified (#8750)
There is a feature, that when tag or category is added to the topic,
customers who are watching that category or tag are notified.

The problem is that it is using default notification type "new post"

It would be better to use "new post" only when there really is a new
post and "edited" when categories or tags were modified.
2020-01-21 08:41:13 +11:00
Gerhard Schlager
f216c6d60b FEATURE: Drop "backup" schema 7 days after restore
The "backup" schema is used to rollback a failed restore. It isn't useful after a longer period of time and turns into a waste of disk space.
2020-01-16 17:48:47 +01:00
Martin Brennan
7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
Gerhard Schlager
c351ffe580 FEATURE: Add hidden setting to disable configuration of inventory bucket 2020-01-14 17:23:12 +01:00
David Taylor
cff6e941de
PERF: Cache ranks for featured badges, to simplify user serialization (#8698) 2020-01-14 14:26:49 +00:00
Roman Rizzi
d69c5eebcf
Feature: Mass award badge (#8694)
* UI: Mass grant a badge from the admin ui

* Send the uploaded CSV and badge ID to the backend

* Read the CSV and grant badge in batches

* UX: Communicate the result to the user

* Don't award if badge is disabled

* Create a 'send_notification' method to remove duplicated code, slightly shrink badge image. Replace router transition with href.

* Dynamically discover current route
2020-01-13 11:20:26 -03:00
Martin Brennan
edbc356593
FIX: Replace deprecated URI.encode, URI.escape, URI.unescape and URI.unencode (#8528)
The following methods have long been deprecated in ruby due to flaws in their implementation per http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-core/29293?29179-31097:

URI.escape
URI.unescape
URI.encode
URI.unencode
escape/encode are just aliases for one another. This PR uses the Addressable gem to replace these methods with its own encode, unencode, and encode_component methods where appropriate.

I have put all references to Addressable::URI here into the UrlHelper to keep them corralled in one place to make changes to this implementation easier.

Addressable is now also an explicit gem dependency.
2019-12-12 12:49:21 +10:00
Sam Saffron
0c52537f10 DEV: update rubocop to version 0.77
We like to stay as close as possible to latest with rubocop cause the cops
get better.

This update required some code changes, specifically the default is to avoid
explicit returns where implicit is done

Also this renames a few rules
2019-12-10 11:48:39 +11:00
Joffrey JAFFEUX
0d3d2c43a0
DEV: s/\$redis/Discourse\.redis (#8431)
This commit also adds a rubocop rule to prevent global variables.
2019-12-03 10:05:53 +01:00
Martin Brennan
e7c7a05097
FIX: Mark secure media upload insecure automatically if used for theme component (#8413)
When uploading a file to a theme component, and that file is existing and has already been marked as secure, we now automatically mark the file as secure: false, change the ACL, and log the action as the user (also rebake the posts for the upload)
2019-11-28 07:32:17 +10:00
David Taylor
e1fcbf4aef DEV: Remove new_version_emails global setting reference
All site settings are now shadowed by global settings, so there is no need to lookup the global setting explicitly
2019-11-20 15:54:09 +00:00
Martin Brennan
02cb01406e
FIX: Allow secure uploads if global s3 setting active and enable_s3_uploads validations (#8373)
The secure media functionality relied on `SiteSetting.enable_s3_uploads?` which, as we found in dev, did not take into account global S3 settings via `GlobalSetting.use_s3?`. We now use `SiteSetting.Upload.enable_s3_uploads` instead to be more consistent.

Also, we now validate `enable_s3_uploads` changes, because if `GlobalSetting.use_s3?` is true users should NOT be enabling S3 uploads manually.
2019-11-20 07:46:44 +10:00
Krzysztof Kotlarek
975165f25f
FIX: Don't error CleanUpInactiveUserJob when user is missing (#8362) 2019-11-18 16:14:15 +11:00
Penar Musaraj
102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Penar Musaraj
067696df8f DEV: Apply Rubocop redundant return style 2019-11-14 15:10:51 -05:00
David Taylor
836b3f4d82
FIX: Do not deactivate admin accounts with recent posts or api keys (#8342)
This prevents 'bot' users being deactivated
2019-11-12 16:56:01 +00:00