Commit Graph

1445 Commits

Author SHA1 Message Date
David Taylor
cd88af8876
FIX: Ensure reviewable counts are updated correctly for new user menu (#21222)
On the client-side, message-bus subscriptions and reviewable count UI is based on the 'redesigned_user_menu_enabled' boolean. We need to use the same logic on the server-side to ensure things work correctly when legacy navigation is used alongside the new user menu.
2023-04-24 16:59:32 +01:00
David Taylor
93c33e02f0
PERF: Avoid full posts table scans during anonymisation (#21081)
2e78045a fixed the anonymization job so that it correctly updated self-mentions, which are not logged in the post_actions table. The solution was to scan the entire `posts` table with an `raw ILIKE` query. On sites with many posts, this can take a very long time.

This commit updates the job to take a two-pass approach:

First, we update posts based on the post_actions table. This is much more efficient than a full table scan, and takes care of all 'non-self' mentions.

Then, we make a second pass using the `raw ILIKE` approach. Since we already took care of most posts, we can scope this down to self-mentions only. By filtering the query to a specific posts.user_id, it is significantly more performant than a full table scan.
2023-04-12 18:39:10 +01:00
Loïc Guitaut
d151f4ee9d
FIX: Don’t assume post is available in UserEmail job (#21054)
Currently, we’re performing a check when a user is suspended in the
`UserEmail` job and we’re assuming a `post` is always available, which
is not the case. The code indeed breaks when the job is called with the
`account_suspended` type option.

This patch fixes this issue by making the check use the safe navigation
operator, thus making it working when `post` is not provided.
2023-04-12 12:34:22 +10:00
Bianca Nenciu
142d2ab65e
FEATURE: Move bootstrap mode indicator to header (#20663)
The information about bootstrap mode has been moved from the notice to
the README topic for admins.
2023-03-24 15:59:03 +02:00
Loïc Guitaut
8a8f2758d5 DEV: Refactor Jobs::UserEmail a little 2023-03-14 09:23:06 +01:00
Loïc Guitaut
27f7cf18b1 FIX: Don’t email suspended users from group PM
Currently, when a suspended user belongs to a group PM (private message
with more than two people in it) and a staff member sends a message to
this group PM, then the suspended user will receive an email.
This happens because a suspended user can only receive emails from staff
members. But in this case, this can be seen as a bug as the expected
behavior would be instead to not send any email to the suspended user. A
staff member can participate in active discussions like any other
member and so their messages in this context shouldn’t be treated
differently than the ones from regular users.

This patch addresses this issue by checking if a suspended user receives
a message from a group PM or not. If that’s the case then an email won’t
be sent no matter if the post originated from a staff member or not.
2023-03-08 15:53:53 +01:00
chapoi
e52bbc1230
UX/DEV: Review queue redesign fixes (#20239)
* UX: add type tag and design update

* UX: clarify status copy in reviewQ

* DEV: switch to selectKit

* UX: color approve/reject buttons in RQ

* DEV: regroup actions

* UX: add type tag and design update

* UX: clarify status copy in reviewQ

* Join questions for flagged post with "or" with new I18n function
* Move ReviewableScores component out of context
* Add CSS classes to reviewable-item based on human type

* UX: add table header for scoring

* UX: don't display % score

* UX: prefix modifier class with dash

* UX: reviewQ flag table styling

* UX: consistent use of ignore icon

* DEV: only show context question on pending status

* UX: only show table headers on pending status

* DEV: reviewQ regroup actions for hidden posts

* UX: reviewQ > approve/reject buttons

* UX: reviewQ add fadeout

* UX: reviewQ styling

* DEV: move scores back into component

* UX: reviewQ mobile styling

* UX: score table on mobile

* UX: reviewQ > move meta info outside table

* UX: reviewQ > score layout fixes

* DEV: readd `agree_and_keep` and fix the spec tests.

* Fix the spec tests

* fix the quint test

* DEV: readd deleting replies

* UX: reviewQ copy tweaks

* DEV: readd test for ignore + delete replies

* Remove old

* FIX: Add perform_ignore back in for backwards compat

* DEV: add an action alias `ignore` for `ignore_and_do_nothing`.

---------

Co-authored-by: Martin Brennan <martin@discourse.org>
Co-authored-by: Vinoth Kannan <svkn.87@gmail.com>
2023-03-02 16:40:53 +01:00
Sam
654959faa4
DEV: reduce amount of errors logged when notifying on flags (#20472)
Forcing distributed muted to raise when a notify reviewable job is running
leads to excessive errors in the logs under many conditions.

The new pattern

1. Optimises the counting of reviewables so it is a lot faster
2. Holds the distributed lock for 2 minutes (max)


The downside is the job queue can get blocked up when tons of notify
reviewables are running at the same time. However this should be very
rare in the real world, as we only notify when stuff is flagged which
is fairly infrequent.

This also give a fair bit more time for the notifications which may be
a little slow on large sites with tons of mods.
2023-03-01 08:58:32 +11:00
Loïc Guitaut
f7c57fbc19 DEV: Enable unless cops
We discussed the use of `unless` internally and decided to enforce
available rules from rubocop to restrict its most problematic uses.
2023-02-21 10:30:48 +01:00
Ted Johansson
25a226279a
DEV: Replace #pluck_first freedom patch with AR #pick in core (#19893)
The #pluck_first freedom patch, first introduced by @danielwaterworth has served us well, and is used widely throughout both core and plugins. It seems to have been a common enough use case that Rails 6 introduced it's own method #pick with the exact same implementation. This allows us to retire the freedom patch and switch over to the built-in ActiveRecord method.

There is no replacement for #pluck_first!, but a quick search shows we are using this in a very limited capacity, and in some cases incorrectly (by assuming a nil return rather than an exception), which can quite easily be replaced with #pick plus some extra handling.
2023-02-13 12:39:45 +08:00
Selase Krakani
2e78045af1
FIX: Extend username updates to self-mentions (#20071)
Posts with self-mentions aren't updated with username updates. This happens
because mention `UserAction` entries aren't logged for self-mentions.

This change updates the lookup of `Post` and `PostRevision` with mentions to bypass
`UserAction` entries.
2023-02-02 12:33:42 +00:00
jbrw
f8863b0f98
DEV: Limit concurrency of NotifyReviewables job (#19968)
Under scenarios of extremely high load where large numbers of `Reviewable*` items are being created, it has been observed that multiple instances of the `NotifyReviewable` job may run simultaneously.

These jobs will work satisfactorily if the concurrency is limited to 1, and the different types of jobs (items reviewable by admins, vs moderators, vs particular groups, etc.) are run eventually.

This change introduces a new option to `DistributedMutex` which allows the `max_get_lock_attempts` to be specified. If the number is exceeded an error will be raised, which will cause Sidekiq to requeue the job. Sidekiq has existing logic to back-off on retry times for jobs that have failed multiple times.
2023-01-25 15:19:11 -05:00
Daniel Waterworth
666536cbd1
DEV: Prefer \A and \z over ^ and $ in regexes (#19936) 2023-01-20 12:52:49 -06:00
Régis Hanol
b704e338ef
DEV: extract anniversary badge query (#19716)
So it can easily be overwritten in a plugin for example.

### Added more tests to provide better coverage

We previously only had `u.silenced_till IS NULL` but I made it consistent with pretty much every other places where we check for "active" users.

These two new lines do change the query a tiny bit though. 

**Before** 

- You could not get the badge if you were currently silenced (no matter what period is being checked)
- You could get the badge if you were suspended 😬 

**After**

- You can't get the badge if you were silenced during the past year
- You can't get the badge if you were suspended during the past year


### Improved the performance of the query by using `NOT EXISTS` instead of `LEFT JOIN / COUNT() = 0`

There is no difference in behaviour between 

```sql
LEFT JOIN user_badges AS ub ON ub.user_id = u.id AND ...
[...]
HAVING COUNT(ub.*) = 0
```

and

```sql
NOT EXISTS (SELECT 1 FROM user_badges AS ub WHERE ub.user_id = u.id AND ...)
```

The only difference is performance-wise. The `NOT EXISTS` is 10-30% faster on very large databases (aka. posts and users in X millions). I checked on 3 of the largest datasets I could find.
2023-01-16 11:55:00 +01:00
Alan Guo Xiang Tan
8a7b62b126
DEV: Fix threading error when running jobs immediately in system tests (#19811)
```
class Jobs::DummyDelayedJob < Jobs::Base
  def execute(args = {})
  end
end

RSpec.describe "Jobs.run_immediately!" do
  before { Jobs.run_immediately! }

  it "explodes" do
    current_user = Fabricate(:user)
    Jobs.enqueue_in(1.seconds, :dummy_delayed_job)
    sign_in(current_user)
  end
end
```

The test above will fail with the following error if `ActiveRecord::Base.connection_handler.clear_active_connections!` is called before the configured Capybara server checks out a connection from the connection pool.

```
     ActiveRecord::ActiveRecordError:
       Cannot expire connection, it is owned by a different thread: #<Thread:0x00007f437391df58@puma srv tp 001 /home/tgxworld/.asdf/installs/ruby/3.1.3/lib/ruby/gems/3.1.0/gems/puma-6.0.2/lib/puma/thread_pool.rb:106 sleep_forever>. Current thread: #<Thread:0x00007f437d6cfc60 run>.
```

We're not exactly sure if this is an ActiveRecord bug or not but we've
invested too much time into investigating this problem. Fundamentally,
we also no longer understand why `ActiveRecord::Base.connection_handler.clear_active_connections!` is being called in an ensure block
within `Jobs::Base#perform` which was added in
ceddb6e0da 10 years ago. This
commit moves the logic for running jobs immediately out of the
`Jobs::Base#perform` method into another `Jobs::Base#perform_immediately` method such that
`ActiveRecord::Base.connection_handler.clear_active_connections!` is not
called. This change will only impact the test environment.
2023-01-10 13:41:25 +08:00
Daniel Waterworth
dfc449a32e
FIX: Continue if checking themes if updating one fails (#19665) 2023-01-09 11:56:43 -06:00
David Taylor
5a003715d3
DEV: Apply syntax_tree formatting to app/* 2023-01-09 14:14:59 +00:00
Alan Guo Xiang Tan
ab3a032b4b
SECURITY: BCC active user emails from group SMTP (#19725)
When sending emails out via group SMTP, if we
are sending them to non-staged users we want
to mask those emails with BCC, just so we don't
expose them to anyone we shouldn't. Staged users
are ones that have likely only interacted with
support via email, and will likely include other
people who were CC'd on the original email to the
group.

Co-authored-by: Martin Brennan <martin@discourse.org>
2023-01-05 06:07:50 +08:00
Selase Krakani
7ba115769a
DEV: Skip push notifications for active online users (#19502)
* DEV: Skip push notifications for active online users

Currently, users with active push subscriptions get push notifications
regardless of their "presence" on the site.

This change introduces a `push_notification_time_window_mins`
site setting which is used in conjunction with a user's `last_seen_at` to
determine if push notifications should be sent. A user is considered to
be actively online if their `last_seen_at` is within `push_notification_time_window_mins`
minutes. `push_notification_time_window_mins` is set to 10 by default.

* DEV: Remove client param for push_notification_time_window_mins site setting

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>
2022-12-19 20:17:40 +00:00
Osama Sayegh
1c03d6f9b9
FEATURE: Send notifications to admins when new features are released (#19460)
This commit adds a new notification that gets sent to admins when the site gets new features after an upgrade/deploy. Clicking on the notification takes the admin to the admin dashboard at `/admin` where they can see the new features under the "New Features" section.

Internal topic: t/87166.
2022-12-15 20:12:53 +03:00
Alan Guo Xiang Tan
fde9e6bc25
DEV: Migrate sidebar site settings (#19336)
This new site setting replaces the
`enable_experimental_sidebar_hamburger` and `enable_sidebar` site
settings as the sidebar feature exits the experimental phase.

Note that we're replacing this without depreciation since the previous
site setting was considered experimental.

Internal Ref: /t/86563
2022-12-08 09:44:29 +08:00
Osama Sayegh
3ff6f6a5e1
FIX: Exclude claimed reviewables from user menu (#19179)
Users who can access the review queue can claim a pending reviewable(s) which means that the claimed reviewable(s) can only be handled by the user who claimed it. Currently, we show claimed reviewables in the user menu, but this can be annoying for other reviewers because they can't do anything about a reviewable claimed by someone. So this PR makes sure that we only show in the user menu reviewables that are claimed by nobody or claimed by the current user.

Internal topic: t/77235.
2022-12-01 07:09:57 +08:00
Alan Guo Xiang Tan
7c321d3aad
PERF: Update Group#user_count counter cache outside DB transaction (#19256)
While load testing our user creation code path in production, we
identified that executing the DB statement to update the `Group#user_count` column within a
transaction is creating a bottleneck for us. This is because the
creation of a user and addition of the user to the relevant groups are
done in a transaction. When we execute the DB statement to update
`Group#user_count` for the relevant group, a row level lock is held
until the transaction completes. This row level lock acts like a global
lock when the server is creating users that will be added to the same
group in quick succession.

Instead of updating the counter cache within a transaction which the
default ActiveRecord `counter_cache` option does, we simply update the
counter cache outside of the committing transaction.

Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>

Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>
2022-11-30 11:52:08 -03:00
Selase Krakani
c7ccb17433
FEATURE: Add cooked post to user archive exports (#18979)
This change allows easily accessible secure media URLs to be available
in the exported data.
2022-11-11 11:07:32 +00:00
Daniel Waterworth
167181f4b7
DEV: Quote values when constructing SQL (#18827)
All of these cases should already be safe, but still good to quote for
"defense in depth".
2022-11-01 14:05:13 -05:00
David Taylor
68b4fe4cf8
SECURITY: Expand and improve SSRF Protections (#18815)
See https://github.com/discourse/discourse/security/advisories/GHSA-rcc5-28r3-23rr

Co-authored-by: OsamaSayegh <asooomaasoooma90@gmail.com>
Co-authored-by: Daniel Waterworth <me@danielwaterworth.com>
2022-11-01 16:33:17 +00:00
Alan Guo Xiang Tan
1b56a55f50
DEV: Sidebar default tags and categories are determined at user creation (#18620)
The previous sidebar default tags and categories implementation did not
allow for a user to configure their sidebar to have no categories or
tags. This commit changes how the defaults are applied. When a user is being created,
we create the SidebarSectionLink records based on the `default_sidebar_categories` and
`default_sidebar_tags` site settings. SidebarSectionLink records are
only created for categories and tags which the user has visibility on at
the point of user creation.

With this change, we're also adding the ability for admins to apply
changes to the `default_sidebar_categories` and `default_sidebar_tags`
site settings historically when changing their site setting. When a new
category/tag has been added to the default, the new category/tag will be
added to the sidebar for all users if the admin elects to apply the changes historically.
Like wise when a tag/category is removed, the tag/category will be
removed from the sidebar for all users if the admin elects to apply the
changes historically.

Internal Ref: /t/73500
2022-10-27 06:38:50 +08:00
David Taylor
3115f38de2
PERF: Move dominant color calculation to separate job (#18501)
This will ensure that any potential problems with this process do not affect the performance or reliability of the PeriodicalUpdates job.
2022-10-06 13:26:08 +01:00
Krzysztof Kotlarek
d5f6262c4f
FIX: watched topic overcome muted category (#18480)
Previously, when categories were not muted by default, we were sending message about unmuted topics (topics which user explicitly set notification level to watching)

The same mechanism can be used to fix a bug. When the user was explicitly watching topic, but category was muted, then the user was not informed about new reply.
2022-10-06 11:10:43 +11:00
Bianca Nenciu
cf646b2061
FIX: Count resulting bulk invites correctly (#18461)
Skipped invites were not counted at all and some invites could generate
more than one error and resulted in a grand total that was not equal to
the count of bulk invites.
2022-10-04 18:41:06 +03:00
Martin Brennan
8ebd5edd1e
DEV: Rename secure_media to secure_uploads (#18376)
This commit renames all secure_media related settings to secure_uploads_* along with the associated functionality.

This is being done because "media" does not really cover it, we aren't just doing this for images and videos etc. but for all uploads in the site.

Additionally, in future we want to secure more types of uploads, and enable a kind of "mixed mode" where some uploads are secure and some are not, so keeping media in the name is just confusing.

This also keeps compatibility with the `secure-media-uploads` path, and changes new
secure URLs to be `secure-uploads`.

Deprecated settings:

* secure_media -> secure_uploads
* secure_media_allow_embed_images_in_emails -> secure_uploads_allow_embed_images_in_emails
* secure_media_max_email_embed_image_size_kb -> secure_uploads_max_email_embed_image_size_kb
2022-09-29 09:24:33 +10:00
Loïc Guitaut
26fe047724 DEV: Use AR enums in reviewables related code
This is the first patch of many to replace our custom enums in Ruby by
the ones provided by `ActiveRecord`.
2022-09-22 14:44:27 +02:00
David Taylor
d0243f741e
UX: Use dominant color as image loading placeholder (#18248)
We previously had a system which would generate a 10x10px preview of images and add their URLs in a data-small-upload attribute. The client would then use that as the background-image of the `<img>` element. This works reasonably well on fast connections, but on slower connections it can take a few seconds for the placeholders to appear. The act of loading the placeholders can also break or delay the loading of the 'real' images.

This commit replaces the placeholder logic with a new approach. Instead of a 10x10px preview, we use imagemagick to calculate the average color of an image and store it in the database. The hex color value then added as a `data-dominant-color` attribute on the `<img>` element, and the client can use this as a `background-color` on the element while the real image is loading. That means no extra HTTP request is required, and so the placeholder color can appear instantly.

Dominant color will be calculated:
1. When a new upload is created
2. During a post rebake, if the dominant color is missing from an upload, it will be calculated and stored
3. Every 15 minutes, 25 old upload records are fetched and their dominant color calculated and stored. (part of the existing PeriodicalUpdates job)

Existing posts will continue to use the old 10x10px placeholder system until they are next rebaked
2022-09-20 10:28:17 +01:00
Alan Guo Xiang Tan
0f0048e8e3
DEV: Enable new user menu when experimental sidebar hamburger is enabled (#18133)
When `enable_experimental_sidebar_hamburger` site setting is enabled, we
will switch to rendering the new user menu.
2022-08-31 21:15:01 +03:00
jbrw
ea5d303a07
DEV: Increase timeout when pulling hotlinked image (#18036)
We observed that some sites seemingly put us in a tarpit when we attempt to pull hotlinked images. Increasing the timeout will help in these situations.
2022-08-23 09:38:10 -04:00
Jarek Radosz
2361833844
FIX: Don't raise on deleted topic in UpdateHotlinkedRaw (#17901)
Fixes:

```
Job exception: undefined method `acting_user=' for nil:NilClass
```

in
```
/var/www/discourse/lib/post_revisor.rb:181:in `revise!'
/var/www/discourse/app/models/post.rb:646:in `revise'
/var/www/discourse/app/jobs/regular/update_hotlinked_raw.rb:24:in `execute'
```
2022-08-13 22:40:26 +02:00
Roman Rizzi
58b135d6d3
DEV:Using symbols is deprecated, use strings instead (#17874) 2022-08-11 11:33:41 -03:00
Osama Sayegh
ce9eec8606
DEV: Combine all header notification bubbles into one in the new user menu (#17718)
Extracted from https://github.com/discourse/discourse/pull/17379.
2022-08-03 08:57:59 +03:00
Roman Rizzi
f1c3670d74
FIX: Publish membership update events when refreshing automatic groups. (#17668)
Adding or removing users from automatic groups is now consistent with `Group#add` and `Group#remove`.
2022-07-27 11:34:08 -03:00
Bianca Nenciu
f75a99e932
FIX: Notify tag watchers when publishing topic (#17576)
When a topic was published from a shared draft and it had tags, the
users watching the tags were not notified. The problem was that the
topics are usually created in a secret category and publishing it just
moves an existent topic to the target category, without making any
changes to the tags.
2022-07-20 19:07:18 +03:00
Bianca Nenciu
4a996825fd
FIX: Skip job if tag edit notification is disabled (#17508)
Previous commit did not schedule any more new jobs, but the jobs in
queue could still create notifications.

Follow up to commit e8d802eb86.
2022-07-15 15:36:27 +03:00
Alan Guo Xiang Tan
7d41e980c9
FIX: Uninitialized class variable error in sidekiq (#17227)
Follow-up to 4199ada1ce
2022-06-24 14:17:39 +10:00
Martin Brennan
3f5e19c62a
FIX: Typo in log_thread (#17226)
Follow up to 4199ada1ce
2022-06-24 12:12:30 +08:00
Alan Guo Xiang Tan
4199ada1ce
DEV: Ensure Sidekiq logging thread is always running (#17211) 2022-06-24 10:28:18 +08:00
Vinoth Kannan
deee3c6f02
DEV: drop unused column flair_url from groups table. (#17179)
It's already included in the `ignored_columns` list in the group model. 03ffb0bf27/app/models/group.rb (L9)

Also, removed the `MigrateGroupFlairImages` onceoff job and spec.
2022-06-22 00:15:05 +05:30
Jarek Radosz
fcb4e5a1a1
DEV: Make wizard an ember addon (#17027)
Co-authored-by: David Taylor <david@taylorhq.com>
2022-06-17 14:50:21 +02:00
Bianca Nenciu
9db8f00b3d
FEATURE: Create upload_references table (#16146)
This table holds associations between uploads and other models. This can be used to prevent removing uploads that are still in use.

* DEV: Create upload_references
* DEV: Use UploadReference instead of PostUpload
* DEV: Use UploadReference for SiteSetting
* DEV: Use UploadReference for Badge
* DEV: Use UploadReference for Category
* DEV: Use UploadReference for CustomEmoji
* DEV: Use UploadReference for Group
* DEV: Use UploadReference for ThemeField
* DEV: Use UploadReference for ThemeSetting
* DEV: Use UploadReference for User
* DEV: Use UploadReference for UserAvatar
* DEV: Use UploadReference for UserExport
* DEV: Use UploadReference for UserProfile
* DEV: Add method to extract uploads from raw text
* DEV: Use UploadReference for Draft
* DEV: Use UploadReference for ReviewableQueuedPost
* DEV: Use UploadReference for UserProfile's bio_raw
* DEV: Do not copy user uploads to upload references
* DEV: Copy post uploads again after deploy
* DEV: Use created_at and updated_at from uploads table
* FIX: Check if upload site setting is empty
* DEV: Copy user uploads to upload references
* DEV: Make upload extraction less strict
2022-06-09 09:24:30 +10:00
David Taylor
5238f6788c
FEATURE: Allow hotlinked media to be blocked (#16940)
This commit introduces a new site setting: `block_hotlinked_media`. When enabled, all attempts to hotlink media (images, videos, and audio) will fail, and be replaced with a linked placeholder. Exceptions to the rule can be added via `block_hotlinked_media_exceptions`.

`download_remote_image_to_local` can be used alongside this feature. In that case, hotlinked images will be blocked immediately when the post is created, but will then be replaced with the downloaded version a few seconds later.

This implementation is purely server-side, and does not impact the composer preview.

Technically, there are two stages to this feature:

1. `PrettyText.sanitize_hotlinked_media` is called during `PrettyText.cook`, and whenever new images are introduced by Onebox. It will iterate over all src/srcset attributes in the post HTML and check if they're allowed. If not, the attributes will be removed and replaced with a `data-blocked-hotlinked-src(set)` attribute

2. In the `CookedPostProcessor`, we iterate over all `data-blocked-hotlinked-src(set)` attributes and check whether we have a downloaded version of the media. If yes, we update the src to use the downloaded version. If not, the entire media element is replaced with a placeholder. The placeholder is labelled 'external media', and is a link to the offsite media.
2022-06-07 15:23:04 +01:00
David Taylor
19f583c449
FIX: Skip pulling hotlinked images for nil user bio (#16901) 2022-05-24 11:52:13 +01:00
David Taylor
bf6f8299a7 FEATURE: Pull hotlinked images immediately after posting
Previously, with the default `editing_grace_period`, hotlinked images were pulled 5 minutes after a post is created. This delay was added to reduce the chance of automated edits clashing with user edits.

This commit refactors things so that we can pull hotlinked images immediately. URLs are immediately updated in the post's `cooked` HTML. The post's raw markdown is updated later, after the `editing_grace_period`.

This involves a number of behind-the-scenes changes including:

- Schedule Jobs::PullHotlinkedImages immediately after Jobs::ProcessPost. Move scheduling to after the `update_column` call to avoid race conditions

- Move raw changes into a separate job, which is delayed until after the ninja-edit window

- Move disable_if_low_on_disk_space logic into the `pull_hotlinked_images` job

- Move raw-parsing/replacing logic into `InlineUpload` so it can be easily be shared between `UpdateHotlinkedRaw` and `PullUserProfileHotlinkedImages`
2022-05-23 14:28:02 +01:00
David Taylor
0baabafa9d DEV: Map already-downloaded hotlinked images in post_process_cooked
Previously this mapping of **cooked** images was only being run for oneboxes. Now it runs for all images, so we can transform hotlinked images without needing to immediately update `raw`
2022-05-23 14:28:02 +01:00
Martin Brennan
faf5b4d3e9
PERF: Speed up secure media and ACL sync rake tasks (#16849)
Incorporates learnings from /t/64227:

* Changes the code to set access control posts in the rake
  task to be an efficient UPDATE SQL query.
  The original version was timing out with 312017 post uploads,
  the new query took ~3s to run.
* Changes the code to mark uploads as secure/not secure in
  the rake task to be an efficient UPDATE SQL query rather than
  using UploadSecurity. This took a very long time previously,
  and now takes only a few seconds.
* Spread out ACL syncing for uploads into jobs with batches of
  100 uploads at a time, so they can be parallelized instead
  of having to wait ~1.25 seconds for each ACL to be changed
  in S3 serially.

One issue that still remains is post rebaking. Doing this serially
is painfully slow. We have a way to do this in sidekiq via PeriodicalUpdates
but this is limited by max_old_rebakes_per_15_minutes. It would
be better to fan this rebaking out into jobs like we did for the
ACL sync, but that should be done in another PR.
2022-05-23 13:14:11 +10:00
Martin Brennan
fcc2e7ebbf
FEATURE: Promote polymorphic bookmarks to default and migrate (#16729)
This commit migrates all bookmarks to be polymorphic (using the
bookmarkable_id and bookmarkable_type) columns. It also deletes
all the old code guarded behind the use_polymorphic_bookmarks setting
and changes that setting to true for all sites and by default for
the sake of plugins.

No data is deleted in the migrations, the old post_id and for_topic
columns for bookmarks will be dropped later on.
2022-05-23 10:07:15 +10:00
Martin Brennan
fbcc35b417
DEV: Remove PostAction/UserAction bookmark refs (#16681)
We have not used anything related to bookmarks for PostAction
or UserAction records since 2020, bookmarks are their own thing
now. Deleting all this is just cleaning up old cruft.
2022-05-10 10:42:18 +10:00
Martin Brennan
222c8d9b6a
FEATURE: Polymorphic bookmarks pt. 3 (reminders, imports, exports, refactors) (#16591)
A bit of a mixed bag, this addresses several edge areas of bookmarks and makes them compatible with polymorphic bookmarks (hidden behind the `use_polymorphic_bookmarks` site setting). The main ones are:

* ExportUserArchive compatibility
* SyncTopicUserBookmarked job compatibility
* Sending different notifications for the bookmark reminders based on the bookmarkable type
* Import scripts compatibility
* BookmarkReminderNotificationHandler compatibility

This PR also refactors the `register_bookmarkable` API so it accepts a class descended from a `BaseBookmarkable` class instead. This was done because we kept having to add more and more lambdas/properties inline and it was very messy, so a factory pattern is cleaner. The classes can be tested independently as well.

Some later PRs will address some other areas like the discourse narrative bot, advanced search, reports, and the .ics endpoint for bookmarks.
2022-05-09 09:37:23 +10:00
Isaac Janzen
2381f18eba
DEV: Convert notify_about_queued_posts_after to accept a float (#16637)
Add support for `notify_about_queued_posts_after` to be set to a float to allow for 15 min increments
2022-05-04 11:33:43 -05:00
Isaac Janzen
dcc7f2a55e
DEV: Convert notify_about_flags_after to float (#16633)
Add support for `notify_about_flags_after` to be set to a float.
2022-05-04 11:19:43 -05:00
David Taylor
c1db968740
DEV: Move hotlinked image information into a dedicated table (#16585)
This will make future changes to the 'pull hotlinked images' system easier. This commit should not introduce any functional change.

For now, the old post_custom_field data is kept in the database. This will be dropped in a future commit.
2022-05-03 13:53:32 +01:00
Loïc Guitaut
008b700a3f DEV: Upgrade to Rails 7
This patch upgrades Rails to version 7.0.2.4.
2022-04-28 11:51:03 +02:00
Mark VanLandingham
1e8a666003
DEV: Accept force_respect_seen_recently argument in UserEmail job (#16460) 2022-04-18 13:32:11 -05:00
Roman Rizzi
6f76a12e0a
FEATURE: Let sites add a sitemap.xml file. (#16357)
* FEATURE: Let sites add a sitemap.xml file.

This PR adds the same features discourse-sitemap provides to core. Sitemaps are only added to the robots.txt file if the `enable_sitemap` setting is enabled and `login_required` disabled.

After merging discourse/discourse-sitemap#34, this change will take priority over the sitemap plugin because it will disable itself. We're also using the same sitemaps table, so our migration won't try to create it
again using `if_not_exists: true`.
2022-04-12 10:33:59 -03:00
Martin Brennan
9f2138dc92
FEATURE: Add a sidekiq job for syncing S3 ACLs (#16449)
Sometimes we need to update a _lot_ of ACLs on S3 (such as when secure media
is enabled), and since it takes ~1s per upload to update the ACL, this is
best spread out over many jobs instead of having to do the whole thing serially.

In future, it will be better to have a job that can be run based on
a column on uploads (e.g. acl_stale) so we can track progress, similar
to how we can set the baked_version to nil to rebake posts.
2022-04-12 14:26:42 +10:00
David Taylor
39ac476db6 FIX: Do not attempt to pull_hotlinked_image for raw_html
raw_html posts (i.e. those which are pulled as part of our comments integration) don't go through our markdown pipeline, so `upload://` URLs are not supported. Running pull_hotlinked_images will break any images in the post.

In future we may add support for pulling hotlinked images in these posts. But for now, disabling it will stop it breaking images.
2022-04-05 16:39:38 +08:00
Martin Brennan
b8828d4a2d
FEATURE: Polymorphic bookmarks pt. 1 (CRUD) (#16308)
This commit introduces a new use_polymorphic_bookmarks site setting
that is default false and hidden, that will be used to help continuous
development of polymorphic bookmarks. This setting **should not** be
enabled anywhere in production yet, it is purely for local development.

This commit uses the setting to enable create/update/delete actions
for polymorphic bookmarks on the server and client side. The bookmark
interactions on topics/posts are all usable. Listing, searching,
sending bookmark reminders, and other edge cases will be handled
in subsequent PRs.

Comprehensive UI tests will be added in the final PR -- we already
have them for regular bookmarks, so it will just be a matter of
changing them to be for polymorphic bookmarks.
2022-03-30 12:43:11 +10:00
Alan Guo Xiang Tan
5743a6ec1e DEV: Remove Zeitwerk inflection monkey patch.
There isn't a good reason we need to patch the inflector.

Co-authored-by: Loïc Guitaut <loic@discourse.org>
2022-03-29 16:04:49 +02:00
Martin Brennan
9d5737fd28
SECURITY: Hide private categories in user activity export (#16273)
In some of the user's own activity export data,
we sometimes showed a secure category's name or
exposed the existence of a secure category.
2022-03-24 15:38:44 +10:00
Bianca Nenciu
30f3e78834
FIX: Reset last sent for existent bookmarks (#16202)
The meaning of reminder_at and reminder_last_sent_at changed after
commit 6d422a8033. A bookmark reminder
will fire only if reminder_last_sent_at is null, but before that it
fired everytime reminder_at was set. This is no longer true because
sometimes reminder_at continues to exist even after a reminder fired.
2022-03-18 16:31:35 +02:00
Bianca Nenciu
6a295ea9e9
DEV: Log more when verbose_upload_logging is enabled (#16177)
A message was logged when download started, but it was not known if a error
during the download.
2022-03-15 23:55:05 +02:00
Penar Musaraj
94750c81fa
FIX: Update group inbox notifications on archive/unarchive (#16152) 2022-03-11 11:57:47 +01:00
Bianca Nenciu
6d422a8033
FEATURE: Highlight expired bookmark reminders (#15317)
The user can select what happens with a bookamrk after it expires. New
option allow bookmark's reminder to be kept even after it has expired.
After a bookmark's reminder notification is created, the reminder date
will be highlighted in red until the user resets the reminder date.
User can do that using the new Clear Reminder button from the dropdown.
2022-03-08 19:44:18 +02:00
Jarek Radosz
2fc70c5572
DEV: Correctly tag heredocs (#16061)
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Michael Brown
3bf3b9a4a5 DEV: pull email address validation out to a new EmailAddressValidator
We validate the *format* of email addresses in many places with a match against
a regex, often with very slightly different syntax.

Adding a separate EmailAddressValidator simplifies the code in a few spots and
feels cleaner.

Deprecated the old location in case someone is using it in a plugin.

No functionality change is in this commit.

Note: the regex used at the moment does not support using address literals, e.g.:
* localpart@[192.168.0.1]
* localpart@[2001:db8::1]
2022-02-17 21:49:22 -05:00
Bianca Nenciu
5eaf214594
FEATURE: New plugin API to check if upload is used (#15545)
This commit introduces two new APIs for handling unused uploads, one
can be used to exclude uploads in bulk when the data model allow and
the other one excludes uploads one by one.
2022-02-16 09:00:30 +02:00
Martin Brennan
4086ee551e
DEV: Add bounce_error_code to EmailLog (#15948)
Whenever we got a bounced email in the Email::Receiver we
previously would just set bounced: true on the EmailLog and
discard the status/diagnostic code. This commit changes this
flow to store the bounce error code (defined in the RFC at
https://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-enhanced-status-codes.xhtml)
not just in the Email::Receiver, but also via webhook events
from other mail services and from SNS.

This commit does not surface the bounce error in the UI,
we can do that later if necessary.
2022-02-15 14:17:26 +10:00
Alan Guo Xiang Tan
6f03b2694d
DEV: Fix typo. (#15857) 2022-02-08 09:04:53 +08:00
David Taylor
15cff27bfe
DEV: Stringify keys of nested hashes in job arguments (#15850)
This provides symmetry with the `.with_indifferent_access` usage in `Jobs#perform`, which is also recursive.
2022-02-07 20:28:45 +00:00
David Taylor
c8c23ba557
DEV: Introduce deprecation warning for non-json Job arguments (#15842)
This commit introduces our own handling and warning for Sidekiq's new 'non-json-serializable' warning. This decouples us from Sidekiq's own deprecation cycle, and allows us to use our own deprecation system. It also means that the dump/parse happens in test mode, which will help us to catch occurrences before they reach production.
2022-02-07 17:59:55 +00:00
David Taylor
9ddd1f739e
DEV: Update :critical_user_email calls to use strings (#15827)
Symbols are converted to strings anyway, so there is no change in behaviour. The latest version of sidekiq introduced a warning for this.
2022-02-04 23:43:53 +00:00
David Taylor
f53d70ac63
DEV: Ensure delay_for and queue are not passed as job arguments (#15824)
This regressed in 3a85c4d680 because deep_stringify_keys makes a copy of the `opts` hash
2022-02-04 20:11:03 +00:00
David Taylor
c3a54eb30f DEV: Use strings for :user_email job type argument
Job arguments go via JSON, and so symbols will appear as strings in the Job's `#execute` method. The latest version of Sidekiq has started warning about this to reduce developer confusion.
2022-02-04 18:28:18 +00:00
David Taylor
3a85c4d680 DEV: Ensure Sidekiq job arguments have stringified keys
The latest version of Sidekiq introduced a warning when jobs are queued with arguments which 'do not stringify to JSON safely'. In the vast majority of cases, this is because a hash is passed with symbols as keys. When those args are passed to the job, the keys will be stringified.

Our job wrapper already takes care of this issue by calling '.with_indifferent_access' on the args before passing them to `#execute`, so we don't need to change anything about our use. All we need to do is satisfy Sidekiq's warning system by 'stringifying' all the keys before enqueuing the job.
2022-02-04 18:28:18 +00:00
Peter Zhu
c5fd8c42db
DEV: Fix methods removed in Ruby 3.2 (#15459)
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
2022-01-05 18:45:08 +01:00
Martin Brennan
2d68e5d942
FEATURE: Scheduled problem checks for admin dashboard (#15327)
This commit introduces scheduled problem checks for the admin dashboard, which are long running or otherwise cumbersome problem checks that will be run every 10 minutes rather than every time the dashboard is loaded. If these scheduled checks add a problem, the problem will remain until it is cleared or until the scheduled job runs again.

An example of a check that should be scheduled is validating credentials against an external provider.

This commit also introduces the concept of a `priority` to the problems generated by `AdminDashboardData` and the scheduled checks. This is `low` by default, and can be set to `high`, but this commit does not change any part of the UI with this information, only adds a CSS class.

I will be making a follow up PR to check group SMTP credentials.
2021-12-20 09:59:11 +10:00
Roman Rizzi
3602f83cf4
FEATURE: Delete previous reviewable reminders. (#15250)
We send the reminder using the GroupMessage class, which supports removing previous messages. We can't match them by raw because they could mention different moderators. Also, I had to change the subject to remove dynamically generated values, which is necessary for finding them.
2021-12-10 10:17:39 -03:00
Angus McLeod
df3886d6e5
FEATURE: Experimental support for group membership via google auth (#14835)
This commit introduces a new site setting "google_oauth2_hd_groups". If enabled, group information will be fetched from Google during authentication, and stored in the Discourse database. These 'associated groups' can be connected to a Discourse group via the "Membership" tab of the group preferences UI. 

The majority of the implementation is generic, so we will be able to add support to more authentication methods in the near future.

https://meta.discourse.org/t/managing-group-membership-via-authentication/175950
2021-12-09 12:30:27 +00:00
Jarek Radosz
28bf9599f5
FEATURE: Pre-setting user locale via bulk invite (#15195) 2021-12-06 02:08:21 +01:00
Sam
3a73028a70
PERF: migrate normalized_emails in a migrations (#15166)
Old OnceOff job could perform pretty slowly on sites with millions of emails

New implementation operates in batches in a migration, minimizing locking.
2021-12-02 15:08:22 +11:00
Mark VanLandingham
bd140948e3
DEV: Changes to support chat uploads (#15153) 2021-12-01 13:24:16 -06:00
Dan Ungureanu
6e2d4a14ac
FIX: Delete unconfirmed AND expired email tokens only (#15089) 2021-11-25 10:34:30 +02:00
Dan Ungureanu
fa8cd629f1
DEV: Hash tokens stored from email_tokens (#14493)
This commit adds token_hash and scopes columns to email_tokens table.
token_hash is a replacement for the token column to avoid storing email
tokens in plaintext as it can pose a security risk. The new scope column
ensures that email tokens cannot be used to perform a different action
than the one intended.

To sum up, this commit:

* Adds token_hash and scope to email_tokens

* Reuses code that schedules critical_user_email

* Refactors EmailToken.confirm and EmailToken.atomic_confirm methods

* Periodically cleans old, unconfirmed or expired email tokens
2021-11-25 09:34:39 +02:00
Bianca Nenciu
3ea8937157
FEATURE: Add email normalization rules setting (#14593)
When this setting is turned on, it will check that normalized emails
are unique. Normalized emails are emails without any dots or plus
aliases.

This setting can be used to block use of aliases of the same email
address.
2021-11-24 11:30:06 +02:00
Martin Brennan
eabe2df8d2
FIX: Drop malformed CC addresses in GroupSmtpEmail job (#14934)
Sometimes, a user may have a malformed email such as
`test@test.com<mailto:test@test.com` their email address,
and as a topic participant will be included as a CC email
when sending a GroupSmtpEmail. This causes the CC parsing to
fail and further down the line in Email::Sender the code
to check the CC addresses expects an array but gets a string
instead because of the parse failure.

Instead, we can just check if the CC addresses are valid
and drop them if they are not in the GroupSmtpEmail job.
2021-11-16 08:15:11 +10:00
Arpit Jalan
aeaf27b7dc
FIX: insert commas even when field value is null (#14791) 2021-11-02 14:59:24 +05:30
Bianca Nenciu
1c3c0f04d9
FEATURE: Pull hotlinked images in user bios (#14726) 2021-10-29 17:58:05 +03:00
Martin Brennan
b659e94a8e
DEV: Delete vacate_legacy_prefix_backups code (#14735)
Introduced in 3037617327, we no
longer need this code, as all of the backups have been
migrated.
2021-10-28 07:53:21 +10:00
David Taylor
010309d108
SECURITY: Improve validation of SNS subscription confirm (#14671)
An upstream validation bug in the aws-sdk-sns library could enable RCE under certain circumstances. This commit updates the upstream gem, and adds additional validation to provide defense-in-depth.
2021-10-20 22:20:52 +01:00
Osama Sayegh
228264d17c
Revert "DEV: add routes_lazy_route to boost boot-up time (#14545)" (#14581)
This reverts commit f5cf647e57.

The gem breaks usage of Rails URL helpers when used outside views and
controllers, for example in
88ecb83382/app/models/upload.rb (L239-L242)
the `upload_short_path` method call fails with an undefined method
exception when this gem is enabled.
2021-10-12 17:30:38 +03:00
Sam
f5cf647e57
DEV: add routes_lazy_route to boost boot-up time (#14545)
The lazy route initialization cuts down boot time of rails.

On my local system it cuts out 200ms of boot time taking me from 3.2 to 3 seconds.

This is not a radically enormous amount of time, but paper cuts add up, and a faster boot in dev will make everyone happy.

TBD if we want to also include this in production.

Gem is heavily maintained by @amatsuda, last commit 3 days ago.
2021-10-11 13:22:13 +11:00
Vinoth Kannan
a6de4a5ce9
DEV: use upload id to save in theme setting instead of URL. (#14341)
When we use URL instead it creates the problem while changing the CDN hostname.
2021-09-16 07:58:53 +05:30
Martin Brennan
41e19adb0d
DEV: Ignore reminder_type for bookmarks (#14349)
We don't actually use the reminder_type for bookmarks anywhere;
we are just storing it. It has no bearing on the UI. It used
to be relevant with the at_desktop bookmark reminders (see
fa572d3a7a)

This commit marks the column as readonly, ignores it, and removes
the index, and it will be dropped in a later PR. Some plugins
are relying on reminder_type partially so some stubs have been
left in place to avoid errors.
2021-09-16 09:56:54 +10:00