Commit Graph

9421 Commits

Author SHA1 Message Date
Arpit Jalan
d1fc759ac4
FIX: remove 'crawl_images' site setting (#14646) 2021-10-19 17:12:29 +05:30
Andrei Prigorshnev
b9164d22ee
FEATURE: add a hidden setting that enables using email as a source for username suggestions (#14623)
This is a follow-up to https://github.com/discourse/discourse/pull/14541. This adds a hidden setting for restoring the old behavior for those users who rely on it. We'll likely deprecate this setting at some point in the future.
2021-10-19 14:37:39 +04:00
Martin Brennan
d3678f6930
FIX: Do not show recipient user in email participants list (#14642)
This commit removes the recipient's username from the
respond to / participants list that is shown at the bottom
of user notification emails. For example if the recipient's
username was jsmith, and there were participants ljones and
bmiller, we currently show this:

> "reply to this email to respond to jsmith, ljones, bmiller"

or

> "Participants: jsmith, ljones, bmiller"

However this is a bit redundant, as you are not replying to
yourself here if you are the recipient user. So we omit the
recipient user's username from this list, which is only used
in the text of the email and not elsewhere.
2021-10-19 15:26:22 +10:00
Jean
6275658e3d
FEATURE: Add setting to disable notifications for topic category edits (#14632) 2021-10-18 09:04:01 -04:00
Alan Guo Xiang Tan
fb5a062b1f
DEV: Remove TopicView#first_post_id. (#14631)
The method was only used for mega topics but it was redundant as the
first post can be determined from using the condition where
`Post#post_number` equal to one.
2021-10-18 14:47:47 +08:00
Rafael dos Santos Silva
b301a6b3db
FEATURE: Cache CORS preflight requests for 2h (#14614)
* FEATURE: Cache CORS preflight requests for 2h

Browsers will cache this for 5 seconds by default. If using MessageBus
in a different domain, Discourse will issue a new long polling, by
default, every 30s or so. This means we would be issuing a new preflight
request **every time**. This can be incredibly wasteful, so let's cache
the authorization in the client for 2h, which is the maximum Chromium
allows us as of today.

* fix tests
2021-10-14 22:37:53 -03:00
David Taylor
567c470361
FIX: Allow staff to view pending/expired invites of other users (#14602)
`/u/username/invited.json?filter=expired` and `/u/username/invited.json?filter=pending` APIs are already returning data to admins. However, the `can_see_invite_details?` boolean was false, which prevented the Ember frontend from showing the tabs correctly. This commit updates the guardian method to match reality.
2021-10-14 15:57:01 +01:00
Krzysztof Kotlarek
9062fd9b7a
FIX: improvements for download local dates (#14588)
* FIX: do not display add to calendar for past dates

There is no value in saving past dates into calendar

* FIX: remove postId and move ICS to frontend

PostId is not necessary and will make the solution more generic for dates which doesn't belong to a specific post.

Also, ICS file can be generated in JavaScript to avoid calling backend.
2021-10-14 09:22:44 +11:00
Bianca Nenciu
c4843fc1c1
FEATURE: Allow admins to permanently delete posts and topics (#14406)
Sometimes administrators want to permanently delete posts and topics
from the database. To make sure that this is done for a good reasons,
administrators can do this only after one minute has passed since the
post was deleted or immediately if another administrator does it.
2021-10-13 12:53:23 +03:00
Yasuo Honda
9a083a550c FIX: BackupRestore::DatabaseRestorer failures with Ruby 3
Implemented workaround suggested at
https://github.com/freerange/mocha/issues/445#issuecomment-644944003
2021-10-12 17:25:51 -04:00
Andrei Prigorshnev
88ecb83382
FEATURE: stop using email as source for username and name suggestions for Single Sign On (#14541)
We don't want to be using emails as source for username and name suggestions in cases when it's possible that a user have no chance to intervene and correct a suggested username. It risks exposing email addresses.
2021-10-12 17:25:54 +04:00
Vinoth Kannan
eeecb2460d
FIX: nil the baked version after moving the posts. (#14483)
Previosuly, quotes from original topics are rendered incorrectly since the moved posts are not rebaked.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2021-10-12 17:31:18 +11:00
Vinoth Kannan
fd9a5bc023
FIX: use category's default sort order in latest & unseen filters only. (#14571)
Previously, even the top topics filter rendered all the topics in default sort order.
2021-10-12 10:25:03 +05:30
Alan Guo Xiang Tan
d0595127cc
FIX: Missing excerpt for post small actions in topic timeline. (#14547) 2021-10-12 09:20:35 +08:00
Bianca Nenciu
79e55ec3f0
FIX: Remove List-Post email header (#14554)
* FIX: Remove List-Post email header

This header is used for mailing lists and can confuse some email clients
such as Thunderbird to display wrong replying options.

* FIX: Replace reply_key in email custom headers

Admins can add custom email headers from site settings. Email sender
will try to replace the reply key if %{reply_key} exists or remove the
header if a reply key does not exist.
2021-10-11 20:57:42 +03:00
Bianca Nenciu
8b99a7f73d
FIX: Move check if user is suspended later (#14566)
Calling create_notification_alert could still send a notification to a
suspended user. This just moves the check if user is suspended right
before sending the notification.
2021-10-11 20:55:18 +03:00
Roman Rizzi
7bc7e1aceb
FIX: update translation key to match flag reason. (#14573)
The refactor introduced in https://github.com/discourse/discourse/pull/14475 made this visible. Adds a safeguard to make sure translation is present before trying to linkify backticks.
2021-10-11 10:24:41 -03:00
Dan Ungureanu
d0bd96e19c
FIX: Handle separately invite to topic and forum (#14562)
Invite is used in two contexts, when inviting a new user to the forum
and when inviting an existent user to a topic. The first case is more
complex and it involves permission checks to ensure that new users can
be created. In the second case, it is enough to ensure that the topic
is visible for both users and that all preconditions are met.

One edge case is the invite to topic via email functionality which
checks for both conditions because first the user must be invited to
create an account first and then to the topic.

A side effect of these changes is that all site settings related to
invites refer to inviting new users only now.
2021-10-11 12:19:31 +03:00
Krzysztof Kotlarek
0301b775a8
FIX: topic_tracking_state not erroring when missing user_stat (#14559)
publish_unread should not error when UserStat is missing
2021-10-11 13:20:55 +11:00
Andrei Prigorshnev
e68dc0c598
DEV: add test cases that prove that sso can override username from "bob1" to "bob" (#14555)
The problem was reported a long time ago (https://meta.discourse.org/t/cannot-change-username-with-sso-enabled/31307). It's hard to say if this bag didn't exist or was fixed later, but test cases in this PR prove that this works now.
2021-10-07 22:29:24 +04:00
Roman Rizzi
9f3b82eeb2
DEV: Move settings to linkify to the serializer code. (#14553)
We aren't translating these settings, so it makes more sense to move them into the code. I added an instance method so plugins can add mappings for custom reasons.
2021-10-07 12:41:57 -03:00
David Taylor
a55642a30a
DEV: Various behind-the-scenes improvements to PresenceChannel (#14518)
- Allow the `/presence/get` endpoint to return multiple channels in a single request (limited to 50)
- When multiple presence channels are initialized in a single Ember runloop, batch them into a single GET request
- Introduce the `presence-pretender` to allow easy testing of PresenceChannel-related features
- Introduce a `use_cache` boolean (default true) on the the server-side PresenceChannel initializer. Useful during testing.
2021-10-07 15:50:14 +01:00
David Taylor
ba380c5f52
DEV: Update invite API docs expires_at default (#14550) 2021-10-07 12:41:04 +01:00
David Taylor
7a52ce0d6d
FIX: Strip discourse-logged-in header during force_anonymous! (#14533)
When the anonymous cache forces users into anonymous mode, it strips the cookies from their request. However, the discourse-logged-in header from the JS client remained.

When the discourse-logged-in header is present without any valid auth_token, the current_user_provider [marks the request as ['logged out'](dbbfad7ed0/lib/auth/default_current_user_provider.rb (L125-L125)), and a [discourse-logged-out header is returned to the client](dbbfad7ed0/lib/middleware/request_tracker.rb (L209-L211)). This causes the JS app to [popup a "you were logged out" modal](dbbfad7ed0/app/assets/javascripts/discourse/app/components/d-document.js (L29-L29)), which is very disruptive.

This commit strips the discourse-logged-in header from the request at the same time as the auth cookie.
2021-10-07 12:31:42 +01:00
Robin Ward
ae13839f98 FEATURE: Adds an API to exclude a tag from a TopicQuery
To exclude a tag from a topic list, add the `exclude_tag` query
parameter. For example: `latest?exclude_tag=music`
2021-10-06 16:07:08 -04:00
Blake Erickson
2fb9834821
DEV: Fix api docs for default calendar (#14539)
Change the type for default_calendar to a string.

The type specified for the default calendar in the api docs wasn't a
valid type. The linting in the api docs repo reports:

```
`type` can be one of the following only: "object", "array", "string", "number", "integer", "boolean", "null".
```

This linting currently is only in the `discourse_api_docs` repo.
2021-10-06 13:36:11 -06:00
Blake Erickson
aaf7ac8936
DEV: Add include_subcategories param to api docs (#14534)
* DEV: Add include_subcategories param to api docs

Adding the `include_subcategories=true` query param to the
`/categories.json` api docs.

Follow up to: fe676f334a

* fix spec
2021-10-06 12:34:03 -06:00
Penar Musaraj
e9b1d29d8b
UX: Revamp quick search (#14499)
Co-authored-by: Robin Ward <robin.ward@gmail.com>
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2021-10-06 11:42:52 -04:00
Osama Sayegh
d9d877fee7
DEV: Pass kwargs to the redis gem when calling methods/commands that we don't wrap (#14530)
This commit fixes the `eval` and `evalsha` commands/methods and any other methods that don't have a wrapper in `DiscourseRedis` and expect keyword arguments. I noticed this problem in Logster when I was trying to fetch some log messages in JSON format using the rails console and saw the messages were missing the `env` field. Logster uses the `eval` command to fetch messages `env`s:

dc351fd00f/lib/logster/redis_store.rb (L250-L253)

and that code was not fetching anything because `DiscourseRedis` didn't pass the `keys` keyword arg to the redis gem.
2021-10-06 17:42:04 +03:00
Bianca Nenciu
f58ab2283d
FIX: Parse address lists in embedded emails (#14514)
Same fix is applied to emails immediately after being parsed because
long headers are sometimes in an invalid format.
2021-10-06 15:07:29 +03:00
Krzysztof Kotlarek
cb5b0cb9d8
FEATURE: save local date to calendar (#14486)
It allows saving local date to calendar.
Modal is giving option to pick between ics and google. User choice can be remembered as a default for the next actions.
2021-10-06 14:11:52 +11:00
Blake Erickson
fe676f334a
FEATURE: Return subcategories on categories endpoint (#14492)
* FEATURE: Return subcategories on categories endpoint

When using the API subcategories will now be returned nested inside of
each category response under the `subcategory_list` param. We already
return all the subcategory ids under the `subcategory_ids` param, but
you then would have to make multiple separate API calls to fetch each of
those subcategories. This way you can get **ALL** of the categories
along with their subcategories in a single API response.

The UI will not be affected by this change because you need to pass in
the `include_subcategories=true` param in order for subcategories to be
returned.

In a follow up PR I'll add the API scoping for fetching categories so
that a readonly API key can be used for the `/categories.json` endpoint. This
endpoint should be used instead of the `/site.json` endpoint for
fetching a sites categories and subcategories.

* Update PR based on feedback

- Have spec check for specific subcategory
- Move comparison check out of loop
- Only populate subcategory list if option present
- Remove empty array initialization
- Update api spec to allow null response

* More PR updates based on feedback

- Use a category serializer for the subcategory_list
- Don't include the subcategory_list param if empty
- For the spec check for the subcategory by id
- Fix spec to account for param not present when empty
2021-10-05 12:12:31 -06:00
Yasuo Honda
dbbfad7ed0 FIX: Support Ruby 3 keyword arguments 2021-10-05 11:25:00 -04:00
Dan Ungureanu
74a9c0509b
FIX: Use addresses to compare email header (#14509)
Usually, when an email is received a user lookup is performed using the
email address found in the `From` header. When an email has an
`X-Original-From` header, if it is equal to `Reply-To` then it uses that
one instead. The comparison was sensitive to whitespaces and other
insignificant characters such as quotes because it reconstructed the
`From` header.

For the fixture added in this commit, it compared the reconstructed
`From` header `John Doe <johndoe@example.com>` with the `Reply-To`
header `"John Doe"    <johndoe@example.com>`.
2021-10-05 12:42:19 +03:00
Roman Rizzi
a9d20610d4
FIX: Make score's reason link building more explicit (#14475)
We relied on backticks to identify and replace site setting names with links. Unfortunately, some translations don't follow this convention, breaking this feature.

Additionally, this lets us linkify `category settings` and `watched words` without using HTML in the translations.

You may notice that I split the texts we want to linkify into two groups. I did this on purpose to emphasize those that should be translated (regular_links) from those who don't (site_settings_link). If you can think of a better solution, I'm open to suggestions.
2021-10-04 16:55:09 -03:00
Roman Rizzi
90a3fbc07b
DEV: Remove HTML setting type and sanitization logic. (#14440)
* DEV: Remove HTML setting type and sanitization logic.

We concluded that we don't want settings to contain HTML, so I'm removing the setting type and sanitization logic. Additionally, we no longer allow the global-notice text to contain HTML.

I searched for usages of this setting type in the `all-the-plugins` repo and found none, so I haven't added a migration for existing settings.

* Mark Global notices containing links as HTML Safe.
2021-10-04 15:40:35 -03:00
Andrei Prigorshnev
149e869c22
FEATURE: make username suggester suggest user1, user2 etc. for input that contains invalid characters only (#14179)
We were suggesting 111, 1111, 1112 before.

See the discussion on Meta – https://meta.discourse.org/t/curious-account-creation-behaviour/199970/14.
2021-10-04 16:47:55 +04:00
Alan Guo Xiang Tan
34cebfd867
FIX: Exclude PMs that user sent to themselves. (#14496)
Regression from 016efeadf6

Follow-up to 016efeadf6
2021-10-04 11:55:35 +08:00
Alan Guo Xiang Tan
4cade9d7f6
FIX: Topic.similar_to results in invalid query for certain locales. (#14497)
For `zh_CN`, we use the `cppjieba_rb` gem to remove stop words so
calling `Search.prepare_data` may result in an empty string.
2021-10-04 11:40:22 +08:00
Roman Rizzi
4c2d5158c5
FIX: Follow the canonical URL when importing a remote topic. (#14489)
FinalDestination now supports the `follow_canonical` option, which will perform an initial GET request, parse the canonical link if present, and perform a HEAD request to it.

We use this mode during embeds to avoid treating URLs with different query parameters as different topics.
2021-10-01 12:48:21 -03:00
Vinoth Kannan
c8d5c049eb
DEV: skip S3 CDN urls with different path in prefix. (#14488)
Previously, while retrieving each upload urls in a post S3 CDN urls with different path in prefix (external urls technically) are considered as uploaded url. It created issue while checking missing uploads.
2021-10-01 12:25:17 +05:30
Dan Ungureanu
76a7b75d8a
DEV: Reuse can_invite_to_forum? in can_invite_to? (#14392)
This commit resolves refactors can_invite_to? to use
can_invite_to_forum? for checking the site-wide permissions and then
perform topic specific checkups.

Similarly, can_invite_to? is always used with a topic object and this is
now enforced.

There was another problem before when `must_approve_users` site setting
was not checked when inviting users to forum, but was checked when
inviting to a topic.

Another minor security issue was that group owners could invite to
group topics even if they did not have the minimum trust level to do
it.
2021-09-29 17:40:16 +03:00
Andrei Prigorshnev
b609f6c11c
FIX: restrict other user's notification routes (#14442)
It was possible to see notifications of other users using routes:
- notifications/responses
- notifications/likes-received
- notifications/mentions
- notifications/edits

We weren't showing anything private (like notifications about private messages), only things that're publicly available in other places. But anyway, it feels strange that it's possible to look at notifications of someone else. Additionally, there is a risk that we can unintentionally leak something on these pages in the future.

This commit restricts these routes.
2021-09-29 16:24:28 +04:00
Alan Guo Xiang Tan
a1745e05ae
FIX: Do not publish post for PM topic tracking if not new for user. (#14469) 2021-09-29 13:54:24 +08:00
Alan Guo Xiang Tan
9d5da2b383
PERF: Revert all inboxes from messages route. (#14445)
The all inboxes was introduced in
016efeadf6 but we decided to roll it back
for performance reasons. The main performance challenge here is that PG
has to basically loop through all the PMs that a user is allowed to view
before being able to order by `Topic#bumped_at`. The all inboxes was not
planned as part of the new/unread filter so we've decided not to tackle
the performance issue for the upcoming release.

Follow-up to 016efeadf6
2021-09-28 11:58:04 +08:00
Alan Guo Xiang Tan
cd64e88711
PERF: Improve database query perf when loading topics for a category. (#14416)
* PERF: Improve database query perf when loading topics for a category.

Instead of left joining the `topics` table against `categories` by filtering with `categories.id`,
we can improve the query plan by filtering against `topics.category_id`
first before joining which helps to reduce the number of rows in the
topics table that has to be joined against the other tables and also
make better use of our existing index.

The following is a before and after of the query plan for a category
with many subcategories.

Before:

```
                                                                                                       QUERY PLAN

-------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------
 Limit  (cost=1.28..747.09 rows=30 width=12) (actual time=85.502..2453.727 rows=30 loops=1)
   ->  Nested Loop Left Join  (cost=1.28..566518.36 rows=22788 width=12) (actual time=85.501..2453.722 rows=30 loops=1)
         Join Filter: (category_users.category_id = topics.category_id)
         Filter: ((topics.category_id = 11) OR (COALESCE(category_users.notification_level, 1) <> 0) OR (tu.notification_level > 1))
         ->  Nested Loop Left Join  (cost=1.00..566001.58 rows=22866 width=20) (actual time=85.494..2453.702 rows=30 loops=1)
               Filter: ((COALESCE(tu.notification_level, 1) > 0) AND ((topics.category_id <> 11) OR (topics.pinned_at IS NULL) OR ((t
opics.pinned_at <= tu.cleared_pinned_at) AND (tu.cleared_pinned_at IS NOT NULL))))
               Rows Removed by Filter: 1
               ->  Nested Loop  (cost=0.57..528561.75 rows=68606 width=24) (actual time=85.472..2453.562 rows=31 loops=1)
                     Join Filter: ((topics.category_id = categories.id) AND ((categories.topic_id <> topics.id) OR (categories.id = 1
1)))
                     Rows Removed by Join Filter: 13938306
                     ->  Index Scan using index_topics_on_bumped_at on topics  (cost=0.42..100480.05 rows=715549 width=24) (actual ti
me=0.010..633.015 rows=464623 loops=1)
                           Filter: ((deleted_at IS NULL) AND ((archetype)::text <> 'private_message'::text))
                           Rows Removed by Filter: 105321
                     ->  Materialize  (cost=0.14..36.04 rows=30 width=8) (actual time=0.000..0.002 rows=30 loops=464623)
                           ->  Index Scan using categories_pkey on categories  (cost=0.14..35.89 rows=30 width=8) (actual time=0.006.
.0.040 rows=30 loops=1)
                                 Index Cond: (id = ANY ('{11,53,57,55,54,56,112,94,107,115,116,117,97,95,102,103,101,105,99,114,106,1
13,104,98,100,96,108,109,110,111}'::integer[]))
               ->  Index Scan using index_topic_users_on_topic_id_and_user_id on topic_users tu  (cost=0.43..0.53 rows=1 width=16) (a
ctual time=0.004..0.004 rows=0 loops=31)
                     Index Cond: ((topic_id = topics.id) AND (user_id = 1103877))
         ->  Materialize  (cost=0.28..2.30 rows=1 width=8) (actual time=0.000..0.000 rows=0 loops=30)
               ->  Index Scan using index_category_users_on_user_id_and_last_seen_at on category_users  (cost=0.28..2.29 rows=1 width
=8) (actual time=0.004..0.004 rows=0 loops=1)
                     Index Cond: (user_id = 1103877)
 Planning Time: 1.359 ms
 Execution Time: 2453.765 ms
(23 rows)
```

After:

```
                                                                                                                            QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=1.28..438.55 rows=30 width=12) (actual time=38.297..657.215 rows=30 loops=1)
   ->  Nested Loop Left Join  (cost=1.28..195944.68 rows=13443 width=12) (actual time=38.296..657.211 rows=30 loops=1)
         Filter: ((categories.topic_id <> topics.id) OR (topics.category_id = 11))
         Rows Removed by Filter: 29
         ->  Nested Loop Left Join  (cost=1.13..193462.59 rows=13443 width=16) (actual time=38.289..657.092 rows=59 loops=1)
               Join Filter: (category_users.category_id = topics.category_id)
               Filter: ((topics.category_id = 11) OR (COALESCE(category_users.notification_level, 1) <> 0) OR (tu.notification_level > 1))
               ->  Nested Loop Left Join  (cost=0.85..193156.79 rows=13489 width=20) (actual time=38.282..657.059 rows=59 loops=1)
                     Filter: ((COALESCE(tu.notification_level, 1) > 0) AND ((topics.category_id <> 11) OR (topics.pinned_at IS NULL) OR ((topics.pinned_at <= tu.cleared_pinned_at) AND (tu.cleared_pinned_at IS NOT NULL))))
                     Rows Removed by Filter: 1
                     ->  Index Scan using index_topics_on_bumped_at on topics  (cost=0.42..134521.06 rows=40470 width=24) (actual time=38.267..656.850 rows=60 loops=1)
                           Filter: ((deleted_at IS NULL) AND ((archetype)::text <> 'private_message'::text) AND (category_id = ANY ('{11,53,57,55,54,56,112,94,107,115,116,117,97,95,102,103,101,105,99,114,106,113,104,98,100,96,108,109,110,111}'::integer[])))
                           Rows Removed by Filter: 569895
                     ->  Index Scan using index_topic_users_on_topic_id_and_user_id on topic_users tu  (cost=0.43..1.43 rows=1 width=16) (actual time=0.003..0.003 rows=0 loops=60)
                           Index Cond: ((topic_id = topics.id) AND (user_id = 1103877))
               ->  Materialize  (cost=0.28..2.30 rows=1 width=8) (actual time=0.000..0.000 rows=0 loops=59)
                     ->  Index Scan using index_category_users_on_user_id_and_last_seen_at on category_users  (cost=0.28..2.29 rows=1 width=8) (actual time=0.004..0.004 rows=0 loops=1)
                           Index Cond: (user_id = 1103877)
         ->  Index Scan using categories_pkey on categories  (cost=0.14..0.17 rows=1 width=8) (actual time=0.001..0.001 rows=1 loops=59)
               Index Cond: (id = topics.category_id)
 Planning Time: 1.633 ms
 Execution Time: 657.255 ms
(22 rows)
```

* PERF: Optimize index on topics bumped_at.

Replace `index_topics_on_bumped_at` index with a partial index on `Topic#bumped_at` filtered by archetype since there is already another index that covers private topics.
2021-09-28 10:05:00 +08:00
Bianca Nenciu
1f57b29147
SECURITY: Escape watched word in error message (#14434) 2021-09-24 11:55:15 +03:00
Dan Ungureanu
2e085915cc
FIX: include_ serializer methods must end with ? (#14407)
Otherwise, they are simply dead code and the attribute is visible by
default. These bugs did not expose any sensitive information.
2021-09-22 16:01:25 +03:00
Martin Brennan
a27d2b124c
FIX: Use unread post excerpt for topic-level bookmark excerpt (#14414)
In the user bookmark list, when we show the excerpt of the bookmark
(which is usually just the bookmarked post excerpt), we want to show
the first unread post's excerpt instead for for_topic bookmarks. This
is because when the user clicks on that bookmark link, they are taken
to the first unread post in the topic, not the OP, as per:

27699648ef
2021-09-22 12:47:36 +10:00
Martin Brennan
dba6a5eabf
FEATURE: Humanize file size error messages (#14398)
The file size error messages for max_image_size_kb and
max_attachment_size_kb are shown to the user in the KB
format, regardless of how large the limit is. Since we
are going to support uploading much larger files soon,
this KB-based limit soon becomes unfriendly to the end
user.

For example, if the max attachment size is set to 512000
KB, this is what the user sees:

> Sorry, the file you are trying to upload is too big (maximum
size is 512000KB)

This makes the user do math. In almost all file explorers that
a regular user would be familiar width, the file size is shown
in a format based on the maximum increment (e.g. KB, MB, GB).

This commit changes the behaviour to output a humanized file size
instead of the raw KB. For the above example, it would now say:

> Sorry, the file you are trying to upload is too big (maximum
size is 512 MB)

This humanization also handles decimals, e.g. 1536KB = 1.5 MB
2021-09-22 07:59:45 +10:00
Martin Brennan
27699648ef
FEATURE: Go to last unread for topic-level bookmark links (#14396)
Instead of going to the OP of the topic for topic-level bookmarks
(which are bookmarks where for_topic is true) when clicking on the
bookmark in the quick access menu or on the user bookmark list,
this commit takes the user to the last unread post in
the topic instead. This should be generally more useful than landing
on the unchanging OP.

To make this work nicely, I needed to add the last_read_post_number to
the BookmarkQuery based on the TopicUser association. It should not add
too much extra weight to the query, because it is limited to the user
that we are fetching bookmarks for.

Also fixed an issue where the bookmark serializer highest_post_number was
not taking into account whether the user was staff, which is when we
should use highest_staff_post_number instead.
2021-09-21 13:49:56 +10:00
Alan Guo Xiang Tan
7a8b5cdd5c
DEV: Improve tests coverage when listing private messages. (#14385)
This is in response to the security incident published in
https://github.com/discourse/discourse/security/advisories/GHSA-vm3x-w6jm-j9vv.

The security incident highlighted a gap in our test suite so we're
adding more test cases to ensure that personal and group messages do not
leak between users in the future.
2021-09-21 10:39:59 +08:00
Vinoth Kannan
28be284b27
FIX: use active record update_attribute instead of mini sql. (#14367)
* DEV: use active record `save!` instead of mini sql.

The "save" method will trigger the before_save callback "match_primary_group_changes" for User model. Else `flair_group_id` won't be removed from the user.

* check whether the method `match_primary_group_changes` called or not.
2021-09-21 09:29:12 +08:00
Martin Brennan
0c42a1e5f3
FEATURE: Topic-level bookmarks (#14353)
Allows creating a bookmark with the `for_topic` flag introduced in d1d2298a4c set to true. This happens when clicking on the Bookmark button in the topic footer when no other posts are bookmarked. In a later PR, when clicking on these topic-level bookmarks the user will be taken to the last unread post in the topic, not the OP. Only the OP can have a topic level bookmark, and users can also make a post-level bookmark on the OP of the topic.

I had to do some pretty heavy refactors because most of the bookmark code in the JS topics controller was centred around instances of Post JS models, but the topic level bookmark is not centred around a post. Some refactors were just for readability as well.

Also removes some missed reminderType code from the purge in 41e19adb0d
2021-09-21 08:45:47 +10:00
Blake Erickson
4a4881613b
DEV: Refactor the api docs for the user endpoint (#14377)
Due to the way that rswag expands shared components we were getting this
warning when linting our api docs:

```
Component: "user_response" is never used.
```

This change refactors the `api/users_spec.rb` file so that it uses the
new way of doing things with a separate `user_get_response.json` schema
file rather then the old way of loading a shared response inside of the
swagger_helper.rb file.
2021-09-20 10:04:57 -06:00
Martin Brennan
4fb7d045a0
FIX: Handle forwarded email quotes around Reply-To display name (#14384)
The display name can have quotes around it, which does not work
with our current comparison of a from field (in this case Reply-To)
and another header (X-Original-From), because we are not comparing
the two values in the same way. This causes an issue where the
commit here: b88d8c8 will not
work properly; the forwarded email gets the From address instead
of the Reply-To address as intended.
2021-09-20 16:26:18 +10:00
Vinoth Kannan
a8b2e7e343
DEV: trash category definition topic instead of destroying. (#14356)
After deleting a category, we should soft-delete the category definition topic instead of hard deleting it. Else it causes issues while doing the user merge action if the source user has an orphan post that belongs to the deleted topic.
2021-09-20 11:20:49 +05:30
Bianca Nenciu
c9ad9bff8a
FIX: Update only passed custom fields (#14357)
It used to replace custom fields instead of updating only the custom
fields that were passed. The changes to custom fields will also be
logged.
2021-09-17 13:37:56 +03:00
Gerhard Schlager
962ccf0ab5
FIX: Hoisting linebreaks shouldn't fail for HTML5 elements (#14364) 2021-09-17 10:41:34 +02:00
jbrw
da88cad648
FIX: Offer site_logo_dark_url as an option for dark mode themes (#14361) 2021-09-16 17:47:51 -04:00
Arpit Jalan
d5424161f8
FIX: add locales for group mention PM variants (#14358) 2021-09-16 23:07:45 +05:30
Vinoth Kannan
a6de4a5ce9
DEV: use upload id to save in theme setting instead of URL. (#14341)
When we use URL instead it creates the problem while changing the CDN hostname.
2021-09-16 07:58:53 +05:30
Blake Erickson
91453dd3fc
DEV: Fix flaky site.json api test (#14351)
The color_scheme_id needs to be an integer not a string.

This is one of the failing tests that showed this error:

 https://github.com/discourse/discourse/runs/3598414971

It showed this error

`POSSIBLE ISSUE W/: /user_themes/0/color_scheme_id`

And this is part of the site.json response:

```
...
"user_themes"=>[{"theme_id"=>149, "name"=>"Cool theme 111", "default"=>false, "color_scheme_id"=>37}]
...
```
2021-09-15 18:03:08 -06:00
Martin Brennan
41e19adb0d
DEV: Ignore reminder_type for bookmarks (#14349)
We don't actually use the reminder_type for bookmarks anywhere;
we are just storing it. It has no bearing on the UI. It used
to be relevant with the at_desktop bookmark reminders (see
fa572d3a7a)

This commit marks the column as readonly, ignores it, and removes
the index, and it will be dropped in a later PR. Some plugins
are relying on reminder_type partially so some stubs have been
left in place to avoid errors.
2021-09-16 09:56:54 +10:00
Alan Guo Xiang Tan
ddb458343d
PERF: Improve query performance all inbox private messages. (#14304)
First reported in https://meta.discourse.org/t/-/202482/19

There are two optimizations being applied here:

1. Fetch a user's group ids in a seperate query instead of including it
   as a sub-query. When I tried a subquery, the query plan becomes very
inefficient.

1. Join against the `topic_allowed_users` and `topic_allowed_groups`
   table instead of doing an IN against a subquery where we UNION the
`topic_id`s from the two tables. From my profiling, this enables PG to
do a backwards index scan on the `index_topics_on_timestamps_private`
index.

This commit fixes a bug where listing all messages was incorrectly
excluding topics if a topic has been archived by a group even if the
user did not belong to the group.

This commit also fixes another bug where dismissing private messages
selectively was subjected to the default limit of 30.
2021-09-15 10:29:42 +08:00
Martin Brennan
d1d2298a4c
DEV: Add for_topic column to bookmarks (#14343)
This new column will be used to indicate that a bookmark
is at the topic level. The first post of a topic can be
bookmarked twice after this change -- with for_topic set
to true and with for_topic set to false.

A later PR will use this column for logic to bookmark the
topic, and then topic-level bookmark links will take you
to the last unread post in the topic.

See also 22208836c5
2021-09-15 11:29:22 +10:00
Martin Brennan
22208836c5
DEV: Ignore bookmarks.topic_id column and remove references to it in code (#14289)
We don't need no stinkin' denormalization! This commit ignores
the topic_id column on bookmarks, to be deleted at a later date.
We don't really need this column and it's better to rely on the
post.topic_id as the canonical topic_id for bookmarks, then we
don't need to remember to update both columns if the bookmarked
post moves to another topic.
2021-09-15 10:16:54 +10:00
Krzysztof Kotlarek
d99735e24d
FEATURE: remove duplicated messages about new advices (#14319)
Discourse is sending regularly message to admins when potential problems are persisted. Most of the time they have exactly the same content. In that case, when there are no replies, the old one should be trashed before a new one is created.
2021-09-15 08:59:25 +10:00
Mark VanLandingham
68bb7c5a66
DEV: Support translated title in desktop/notifications (#14325) 2021-09-14 09:57:38 -05:00
Bianca Nenciu
6a7ea66670
FEATURE: Use second factor for admin confirmation (#14293)
Administrators can use second factor to confirm granting admin access
without using email. The old method of confirmation via email is still
used as a fallback when second factor is unavailable.
2021-09-14 15:19:28 +03:00
Dan Ungureanu
f517b6997c
FEATURE: Cook drafts excerpt in user activity (#14315)
The previous excerpt was a simple truncated raw message. Starting with
this commit, the raw content of the draft is cooked and an excerpt is
extracted from it. The logic for extracting the excerpt mimics the the
`ExcerptParser` class, but does not implement all functionality, being
a much simpler implementation.

The two draft controllers have been merged into one and the /draft.json
route has been changed to /drafts.json to be consistent with the other
route names.
2021-09-14 15:18:01 +03:00
Bianca Nenciu
dde66b9e16
FIX: Update only present fields in request (#14310)
Some category fields were always updated, even if they were not present
in the request. When this happened, these field were erased.
2021-09-14 15:04:54 +03:00
Rafael dos Santos Silva
2e0992c757
DEV: Allow TopicEmbed.import to optionally receive a list of tags (#14301)
This will be used by the rss-polling plugin
2021-09-13 17:01:59 -03:00
Martin Brennan
0d809197aa
FIX: Make sure S3 object headers are preserved on copy (#14302)
When copying an existing upload stub temporary object
on S3 to its final destination we were not copying across
its additional headers such as content-disposition and
cache-control, which led to issues like attachments not
downloading with their original filename when clicking
the download links in posts.

This is because the metadata_directive = REPLACE option
was not being passed to object.copy_from(), so only the
source object's headers were being used. Added an option
for apply_metadata_to_destination to apply this option
conditionally, because we may not always want to replace
this metadata, but we definitely do when copying a temporary
upload.
2021-09-10 12:59:51 +10:00
Alan Guo Xiang Tan
bc23dcd30b
FIX: Don't publish PM archive events to acting user. (#14291)
When a user archives a personal message, they are redirected back to the
inbox and will refresh the list of the topics for the given filter.
Publishing an event to the user results in an incorrect incoming message
because the list of topics has already been refreshed.

This does mean that if a user has two tabs opened, the non-active tab
will not receive the incoming message but at this point we do not think
the technical trade-offs are worth it to support this feature. We
basically have to somehow exclude a client from an incoming message
which is not easy to do.

Follow-up to fc1fd1b416
2021-09-10 09:20:50 +08:00
Bianca Nenciu
0532a5a43e
FIX: Do not replace in mentions and hashtags (#14260)
Watched words of type 'replace' or 'link' replaced the text inside
mentions or hashtags too, which broke these. These types of watched
words must skip any match that has an @ or # before it.
2021-09-09 12:03:59 +03:00
Alan Guo Xiang Tan
7b77dd5c05
FIX: Display unread/new PM links only when viewing own user. (#14290)
At this point in time, we do not think supporting unread and new when an
admin is looking at another user's messages is worth supporting.

Follow-up to fc1fd1b416
2021-09-09 14:02:17 +08:00
Alan Guo Xiang Tan
ee8c943326
FIX: Remove dismissed new topics from PM topic tracking state. (#14288)
Follow-up to fc1fd1b416
2021-09-09 12:39:27 +08:00
Alan Guo Xiang Tan
412587f70a
FEATURE: Publish read topic tracking events for private messages. (#14274)
Follow-up to fc1fd1b416
2021-09-09 09:16:53 +08:00
Alan Guo Xiang Tan
1e05175364
PERF: Avoid running query unnecessarily when updating bookmark. (#14276)
* Avoid loading an entire ActiveRecord object when saving and updating.
* Avoid running a DB query when `post_id` or `user_id` is not changed.
2021-09-09 10:50:26 +10:00
Alan Guo Xiang Tan
5de64b3630
FIX: Correct highest post number for read topic tracking state. (#14273) 2021-09-08 11:55:12 +08:00
Krzysztof Kotlarek
e3793e6d7c
FIX: better filter for groups search (#14262)
Follow up of https://github.com/discourse/discourse/pull/14216

Allow plugins to register custom filter with block
2021-09-08 09:38:45 +10:00
Blake Erickson
c6bcf1f06c
DEV: Add site.json to api docs (#14249)
Documenting the site.json api endpoint. This endpoint is often used as a
way to get all of the categories and subcategories in a single api call.
2021-09-07 10:36:05 -06:00
Blake Erickson
d500c436c6
DEV: Add license field to api docs info section (#14248)
One of the fields that should be present for openapi docs is the "license"
field.

https://spec.openapis.org/oas/latest.html#infoObject

Our API docs already had a license, so this commit just specifies that
and provides a link to it.
2021-09-07 10:35:56 -06:00
Alan Guo Xiang Tan
fc1fd1b416
FEATURE: Display new/unread count in browse more messages for PMs. (#14188)
In order to include the new/unread count in the browse more message
under suggested topics, a couple of technical changes have to be made.

1. `PrivateMessageTopicTrackingState` is now auto-injected which is
   similar to how it is done for `TopicTrackingState`. This is done so
we don't have to attempt to pass the `PrivateMessageTopicTrackingState`
object multiple levels down into the suggested-topics component. While
the object is auto-injected, we only fetch the initial state and start
tracking when the relevant private messages routes has been hit and only
when a private message's suggested topics is loaded. This is
done as we do not want to add the extra overhead of fetching the inital
state to all page loads but instead wait till the private messages
routes are hit.

2. Previously, we would stop tracking once the `user-private-messages`
   route has been deactivated. However, that is not ideal since
navigating out of the route and back means we send an API call to the
server each time. Since `PrivateMessageTopicTrackingState` is kept in
sync cheaply via messageBus, we can just continue to track the state
even if the user has navigated away from the relevant stages.
2021-09-07 12:30:40 +08:00
Martin Brennan
9f36d8ad43
FIX: Capture CC addresses for forwarded emails (#14254)
When forwarding emails into the group inbox, we now use the
original sender email as the from_address since
2ac9fd9dff. However, we have not
been saving the original CC addresses of the forwarded email,
which are needed to include those recipients in on the conversation
when replying via the group inbox.

This commit captures the CC addresses on the incoming email, and
makes sure the emails are created as staged users and added to the
list of topic allowed users so they are included on CC's sent by
the GroupSmtpEmail and other jobs.
2021-09-07 08:46:28 +10:00
Jean
34ff7bfeeb
FEATURE: Hide suspended users from site-wide search to regular users (#14245) 2021-09-06 09:59:35 -04:00
Vinoth Kannan
0c777825b3
FIX: perform agree_and_keep action only if possible. (#13967)
While deleting spammers from flag modal it's trying to perform `agree_and_keep` action where it's not possible (or already performed).
2021-09-06 11:41:44 +05:30
Martin Brennan
7b392cee50
FIX: Correct the forwarded by user small post for group inbox (#14252)
When 2ac9fd9dff was done, this
affected the small post that is created when forwarding an email
into the group inbox. Instead of using the name and the email of
the user who forwarded the email, it used the original from email
and name to create the small post. So instead of something like
"Discourse Team forwarded the above email" we ended up with
"John Smith forwarded the above email" which is incorrect.

This fixes the issue by creating a staged user for the forwarding
email address (if such a user does not yet exist) and uses that
for the "forwarded" small post instead.
2021-09-06 15:02:13 +10:00
Martin Brennan
dd4b8c2afa
FIX: Use random file name for temporary uploads (#14250)
Other locale characters in file names (e.g. é, ä) as well
as special characters can cause issues on S3, notably the S3
copy object operation does not support these special characters.
Instead of storing the original file name in the key, which is
unnecessary, we now generate a random file name with the original
extension for the temporary file and use that for all external
upload stub operations.
2021-09-06 10:21:20 +10:00
Krzysztof Kotlarek
f859fd6bde
FEATURE: allow plugins to extend Groups (#14216)
* add_permitted_group_param API for plugins
* add groups-interaction-custom-options outlet
* custom search can use custom group scope
2021-09-06 10:18:51 +10:00
Blake Erickson
ee7809e8a8
DEV: Add missing operationIds to the api docs (#14235)
From the openapi spec:

 https://spec.openapis.org/oas/latest.html#fixed-fields-7

each endpoint needs to have an `operationId`:

> Unique string used to identify the operation. The id MUST be unique
> among all operations described in the API. The operationId value is
> case-sensitive. Tools and libraries MAY use the operationId to uniquely
> identify an operation, therefore, it is RECOMMENDED to follow common
> programming naming conventions.

Running the linter on our openapi.json file with this command:

`npx @redocly/openapi-cli lint openapi.json`

produced the following warning on all of our endpoints:

> Operation object should contain `operationId` field

This commit resolves these warnings by adding an operationId field to
each endpoint.
2021-09-03 07:39:29 -06:00
Jean
85c31c73ba
FIX: allow single string values on custom multiple select fields and not just arrays (#14236) 2021-09-03 09:26:57 -04:00
Arpit Jalan
763f48abc7
FIX: increase chunk size to fetch title tag correctly (#14144) 2021-09-03 13:15:58 +05:30
Penar Musaraj
90a23c6fc8
FEATURE: Enable auto dark mode on new instances (#14208) 2021-09-02 14:55:38 -04:00
Rafael dos Santos Silva
ef0471d7ae
DEV: Allow passing cook_method to TopicEmbed.import to override default (#14209)
DEV: Allow passing cook_method to TopicEmbed.import to override default

This will be used in the rss-polling plugin when we want to have
oneboxes on feed content, like youtube for example.
2021-09-01 15:46:39 -03:00
Vinoth Kannan
49b2bb294e
FEATURE: option to update default notification level of existing users. (#14084)
Previously, a group's `default_notification_level` change will only affect the users added after it.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2021-08-31 16:11:26 +05:30
Alan Guo Xiang Tan
c2f87e0a36
PERF: Make TopicViewSerializer#requested_group_name more efficient. (#14196)
* Avoid executing a query when the custom field doesn't exist
* Avoid generating an ActiveRecord when all we need is the name.
2021-08-31 12:05:32 +08:00
Blake Erickson
70eca1dc4e
DEV: Update api docs for search endpoint (#14181) 2021-08-30 11:25:34 -06:00
David Taylor
ed688bec8c
FIX: Ensure id sequences are not reset during db:migrate (#14184)
The seed-fu gem resets the sequence on all the tables it touches. In some situations, this can cause primary keys to be re-used. This commit introduces a freedom patch which ensures seed-fu only touches the sequence when it is less than the id of one of the seeded records.
2021-08-30 12:31:22 +01:00
Vinoth Kannan
08dce4f477
UX: use existing guardian method to check messageable group. (#14174)
We should display "Message" button only if personal messages are enabled. Currently, it's not respecting that site setting.
2021-08-30 10:38:33 +05:30
Vinoth Kannan
465774cf2c
UX: display correct replies count in embedded comments view. (#14175)
Previosuly, the reply count included the "small_action" posts too. It also caused the broken embed HTML issue.
2021-08-30 10:37:53 +05:30
David Taylor
31db83527b DEV: Introduce PresenceChannel API for core and plugin use
PresenceChannel aims to be a generic system for allow the server, and end-users, to track the number and identity of users performing a specific task on the site. For example, it might be used to track who is currently 'replying' to a specific topic, editing a specific wiki post, etc.

A few key pieces of information about the system:
- PresenceChannels are identified by a name of the format `/prefix/blah`, where `prefix` has been configured by some core/plugin implementation, and `blah` can be any string the implementation wants to use.
- Presence is a boolean thing - each user is either present, or not present. If a user has multiple clients 'present' in a channel, they will be deduplicated so that the user is only counted once
- Developers can configure the existence and configuration of channels 'just in time' using a callback. The result of this is cached for 2 minutes.
- Configuration of a channel can specify permissions in a similar way to MessageBus (public boolean, a list of allowed_user_ids, and a list of allowed_group_ids). A channel can also be placed in 'count_only' mode, where the identity of present users is not revealed to end-users.
- The backend implementation uses redis lua scripts, and is designed to scale well. In the future, hard limits may be introduced on the maximum number of users that can be present in a channel.
- Clients can enter/leave at will. If a client has not marked itself 'present' in the last 60 seconds, they will automatically 'leave' the channel. The JS implementation takes care of this regular check-in.
- On the client-side, PresenceChannel instances can be fetched from the `presence` ember service. Each PresenceChannel can be used entered/left/subscribed/unsubscribed, and the service will automatically deduplicate information before interacting with the server.
- When a client joins a PresenceChannel, the JS implementation will automatically make a GET request for the current channel state. To avoid this, the channel state can be serialized into one of your existing endpoints, and then passed to the `subscribe` method on the channel.
- The PresenceChannel JS object is an ember object. The `users` and `count` property can be used directly in ember templates, and in computed properties.
- It is important to make sure that you `unsubscribe()` and `leave()` any PresenceChannel objects after use

An example implementation may look something like this. On the server:

```ruby
register_presence_channel_prefix("site") do |channel|
  next nil unless channel == "/site/online"
  PresenceChannel::Config.new(public: true)
end
```

And on the client, a component could be implemented like this:

```javascript
import Component from "@ember/component";
import { inject as service } from "@ember/service";

export default Component.extend({
  presence: service(),
  init() {
    this._super(...arguments);
    this.set("presenceChannel", this.presence.getChannel("/site/online"));
  },
  didInsertElement() {
    this.presenceChannel.enter();
    this.presenceChannel.subscribe();
  },
  willDestroyElement() {
    this.presenceChannel.leave();
    this.presenceChannel.unsubscribe();
  },
});
```

With this template:

```handlebars
Online: {{presenceChannel.count}}
<ul>
  {{#each presenceChannel.users as |user|}} 
    <li>{{avatar user imageSize="tiny"}} {{user.username}}</li>
  {{/each}}
</ul>
```
2021-08-27 16:26:06 +01:00
Martin Brennan
99ec8eb6df
FIX: Capture S3 metadata when calling create_multipart (#14161)
The generate_presigned_put endpoint for direct external uploads
(such as the one for the uppy-image-uploader) records allowed
S3 metadata values on the uploaded object. We use this to store
the sha1-checksum generated by the UppyChecksum plugin, for later
comparison in ExternalUploadManager.

However, we were not doing this for the create_multipart endpoint,
so the checksum was never captured and compared correctly.

Also includes a fix to make sure UppyChecksum is the last preprocessor to run.
It is important that the UppyChecksum preprocessor is the last one to
be added; the preprocessors are run in order and since other preprocessors
may modify the file (e.g. the UppyMediaOptimization one), we need to
checksum once we are sure the file data has "settled".
2021-08-27 09:50:23 +10:00
Dan Ungureanu
3406a49e21
FEATURE: Create notification for redeemed invite (#14146)
Users can invite people to topic and they will be automatically
redirected to the topic when logging in after signing up. This commit
ensures a "invited_to_topic" notification is created when the invite is
redeemed.

The same notification is used for the "Notify" sharing method that is
found in share topic modal.
2021-08-26 10:43:56 +03:00
Martin Brennan
e43a8af3bd
FIX: Do not send emails to mailing_list_mode subscribers for PMs (#14159)
This bug was introduced by f66007ec83.

In PostJobsEnqueuer we previously did not fire the after_post_create
event and after_topic_create event for private message topics. This was
changed in the above commit in order to publish message bus messages
for topic tracking state updates. Unfortunately this caused the
NotifyMailingListSubscribers job to be enqueued for all posts including
private messages, and admins and the users involved in the PMs got
emailed the contents of the PMs if they had mailing list mode enabled.

Luckily the impact of this was mitigated by a Guardian#can_see? check
for each mailing list mode user in the NotifyMailingListSubscribers job.
We never want to notify mailing list mode subscribers for private messages
so an early return has been added there, plus the logic in PostJobsEnqueuer
has been fixed, and tests have been added to that class where there were
none before.
2021-08-26 15:16:35 +10:00
Martin Brennan
1646856974
FIX: Topic reset_new unscoped causing huge queries (#14158)
Since ad3ec5809f when a user chooses
the Dismiss New... option in the New topic list, we send a request
to topics/reset-new.json with ?tracked=false as the only parameter.

This then uses Topic as the scope for topics to dismiss, with no
other limitations. When we do topic_scope.pluck(:id), it gets the
ID of every single topic in the database (that is not deleted) to
pass to TopicsBulkAction, causing a huge query with severe performance
issues.

This commit changes the default scope to use
`TopicQuery.new(current_user).new_results(limit: false)`
which should only use the topics in the user's New list, which
will be a much smaller list, depending on the user's "new_topic_duration_minutes"
setting.
2021-08-26 11:25:20 +10:00
Martin Brennan
841e054907
FIX: Do not prefix temp/ S3 keys with s3_bucket_folder_path in S3Helper (#14145)
This is unnecessary, as when the temporary key is created
in S3Store we already include the s3_bucket_folder_path, and
the key will always start with temp/ to assist with lifecycle
rules for multipart uploads.

This was affecting Discourse.store.object_from_path,
Discourse.store.signed_url_for_path, and possibly others.

See also: e0102a5
2021-08-26 08:50:49 +10:00
Penar Musaraj
85b8fea262
UX: Add Styling step to wizard (#14132)
Refactors three wizard steps (colors, fonts, homepage style) into one new step called Styling.
2021-08-25 17:10:12 -04:00
Alan Guo Xiang Tan
d2fe46d476
DEV: Remove the use of stubs. (#14142)
Follow-up to 419d71abcb
2021-08-25 13:25:01 +08:00
Arpit Jalan
419d71abcb
FEATURE: allow admin to delete all posts by a user irrespectively (#14128)
This commit allows admin to delete all posts by a user irrespective of
site settings `delete_user_max_post_age` and `delete_all_posts_max`.
2021-08-25 10:14:22 +05:30
Alan Guo Xiang Tan
f66007ec83
FEATURE: Display unread and new counts for messages. (#14059)
There are certain design decisions that were made in this commit.

Private messages implements its own version of topic tracking state because there are significant differences between regular and private_message topics. Regular topics have to track categories and tags while private messages do not. It is much easier to design the new topic tracking state if we maintain two different classes, instead of trying to mash this two worlds together.

One MessageBus channel per user and one MessageBus channel per group. This allows each user and each group to have their own channel backlog instead of having one global channel which requires the client to filter away unrelated messages.
2021-08-25 11:17:56 +08:00
Penar Musaraj
1167b16913
FIX: Order outputted theme stylesheets (#14133) 2021-08-25 09:37:07 +08:00
Martin Brennan
d66b258b0e
DEV: Do not destroy external upload stub on error in debug mode (#14139)
We do not want to destroy the external upload stub records
in debug mode because they allow for investigation of problems
occuring.
2021-08-25 11:11:19 +10:00
Martin Brennan
e0102a533a
FIX: Restructure temp/ folders for direct S3 uploads (#14137)
Previously we had temp/ in the middle of the S3 key path like so

* /uploads/default/temp/randomstring/test.png (normal site)
* /sitename/uploads/default/temp/randomstring/test.png (s3 folder path site)
* /standard10/uploads/sitename/temp/randomstring/test.png (multisite site)

However this necessitates making a lifecycle rule to clean up incomplete
S3 multipart uploads for every site, something which we cannot do. It makes
much more sense to have a structure with /temp at the start of the key,
which is what this commit does:

* /temp/uploads/default/randomstring/test.png (normal site)
* /temp/sitename/uploads/default/randomstring/test.png (s3 folder path site)
* /temp/standard10/uploads/sitename/randomstring/test.png (multisite site)
2021-08-25 09:22:36 +10:00
Martin Brennan
d295a16dab
FEATURE: Uppy direct S3 multipart uploads in composer (#14051)
This pull request introduces the endpoints required, and the JavaScript functionality in the `ComposerUppyUpload` mixin, for direct S3 multipart uploads. There are four new endpoints in the uploads controller:

* `create-multipart.json` - Creates the multipart upload in S3 along with an `ExternalUploadStub` record, storing information about the file in the same way as `generate-presigned-put.json` does for regular direct S3 uploads
* `batch-presign-multipart-parts.json` - Takes a list of part numbers and the unique identifier for an `ExternalUploadStub` record, and generates the presigned URLs for those parts if the multipart upload still exists and if the user has permission to access that upload
* `complete-multipart.json` - Completes the multipart upload in S3. Needs the full list of part numbers and their associated ETags which are returned when the part is uploaded to the presigned URL above. Only works if the user has permission to access the associated `ExternalUploadStub` record and the multipart upload still exists.

  After we confirm the upload is complete in S3, we go through the regular `UploadCreator` flow, the same as `complete-external-upload.json`, and promote the temporary upload S3 into a full `Upload` record, moving it to its final destination.
* `abort-multipart.json` - Aborts the multipart upload on S3 and destroys the `ExternalUploadStub` record if the user has permission to access that upload.

Also added are a few new columns to `ExternalUploadStub`:

* multipart - Whether or not this is a multipart upload
* external_upload_identifier - The "upload ID" for an S3 multipart upload
* filesize - The size of the file when the `create-multipart.json` or `generate-presigned-put.json` is called. This is used for validation.

When the user completes a direct S3 upload, either regular or multipart, we take the `filesize` that was captured when the `ExternalUploadStub` was first created and compare it with the final `Content-Length` size of the file where it is stored in S3. Then, if the two do not match, we throw an error, delete the file on S3, and ban the user from uploading files for N (default 5) minutes. This would only happen if the user uploads a different file than what they first specified, or in the case of multipart uploads uploaded larger chunks than needed. This is done to prevent abuse of S3 storage by bad actors.

Also included in this PR is an update to vendor/uppy.js. This has been built locally from the latest uppy source at d613b849a6. This must be done so that I can get my multipart upload changes into Discourse. When the Uppy team cuts a proper release, we can bump the package.json versions instead.
2021-08-25 08:46:54 +10:00
Bianca Nenciu
ff367e22fb
FEATURE: Make allow_uploaded_avatars accept TL (#14091)
This gives admins more control over who can upload custom profile
pictures.
2021-08-24 10:46:28 +03:00
Bianca Nenciu
eb6d66fe6f
FIX: Do not allow negative values for LIMIT (#14122)
Negative values generated invalid SQL queries.
2021-08-24 10:45:26 +03:00
Martin Brennan
2ac9fd9dff
FIX: Use original from address when forwarding to group inbox (#14114)
When emails were forwarded to a group inbox by the email address
of the group, for example when an email ends up in spam and must
be manually forwarded to the group+site@discoursemail.com address,
the OP of the topic ended up being the group's email address instead
of the sender who originally sent the email to the group inbox.

This commit detects that an email has been forwarded using existing
tools, and if the from address matches one of the group incoming
email addresses, then we look at the forwarded email's from address
and use that instead for the incoming email from address as well as
the staged/regular user used for the Topic.user.

This will make it much cleaner to forward emails into a group inbox,
and will prevent issues with PostAlerter where the OP is double-notified
for these emails.
2021-08-24 08:57:28 +10:00
Kris
bde6f7e9b0
UX: Update "get a room" composer message (#14104) 2021-08-23 18:34:23 -04:00
Joffrey JAFFEUX
2bbc97fda5
DEV: UserOption.user_tzinfo (#14088)
Provides a safe way to retrieve the timezone of a user.

This is not used in core yet, but used in multiple plugins.
2021-08-19 21:56:14 +02:00
Osama Sayegh
ec352a1969
FEATURE: Order pinned topics by their pinned_at column (#14090)
Currently, pinned topics are ordered by the `bumped_at` column. This behavior is not desired because it gives admins no control over the order of pinned topics. This PR makes pinned topics ordered by the `pinned_at` column. A topic that is pinned last appears first in topic lists. If an admin wants an already pinned topic to appear first in the list of pinned topics, they'll have to unpin that topic and pin it again.

Meta topic: https://meta.discourse.org/t/how-do-i-set-the-order-of-pinned-topics/16935/23?u=osama.
2021-08-19 14:43:58 +03:00
Roman Rizzi
a50cb61dd5
FIX: Deprecated method should still behave the same. (#14067) 2021-08-19 09:58:26 +08:00
Grayden
64ead3c3a1
FIX: Revoking admin or moderator status doesn't require refresh to delete/anonymize/merge user (#14073)
* FIX: Revoking admin or moderator status doesn't require refresh to delete/anonymize/merge user

On the /admin/users/<id>/<username> page, there are action buttons that are either visible or hidden depending on a few fields from the AdminDetailsSerializer: `can_be_deleted`, `can_be_anonymized`, `can_be_merged`, `can_delete_all_posts`.

These fields are updated when granting/revoking admin or moderator status. However, those updates were not being reflected on the page. E.g. if a user is granted moderation privileges, the 'anonymize user' and 'merge' buttons still appear on the page, which is inconsistent with the backend state of the user. It requires refreshing the page to update the state.

This commit fixes that issue, by syncing the client model state with the server state when handling a successful response from the server. Now, when revoking privileges, the buttons automatically appear without refreshing the page. Similarly, when granting moderator privileges, the buttons automatically disappear without refreshing the page.

* Add detailed user response to spec for changed routes.

Add tests to verify that the revoke_moderation, grant_moderation, and revoke_admin routes return a response formatted according to the AdminDetailedUserSerializer.
2021-08-19 09:57:16 +08:00
Bianca Nenciu
5c9abab0cd
FIX: Remove markers from all translated languages (#14081)
Emails can include the marker in a different language, depending on
site and user settings. The email receiver always looked for the marker
in default language.
2021-08-18 19:42:04 +03:00
Arpit Jalan
7db3888f17
FIX: log proper error message when SSO nonce verification fails (#14077) 2021-08-18 18:44:12 +05:30
Alan Guo Xiang Tan
d13716286c
FIX: Unread group PMs should use GroupUser#first_unread_pm_at. (#14075)
This bug was causing unread PMs for groups to appear inaccurate.
2021-08-18 11:23:28 +08:00
Rafael dos Santos Silva
560c13211a
DEV: Allow passing a category parameter when importing a topic (#14069)
This will be used in the rss pooling plugin to address the feature
request at https://meta.discourse.org/t/-/200644?u=falco
2021-08-17 18:17:07 -03:00
Penar Musaraj
08a3aa546b
DEV: Include login_required attribute in basic info endpoint (#14064)
This is useful in the DiscourseHub mobile app, currently the app queries
the `about.json` endpoint, which can raise a CORS issue in some cases,
for example when the site only accepts logins from an external provider.
2021-08-17 14:05:51 -04:00
Rafael dos Santos Silva
b136375582
FEATURE: Rate limit exceptions via ENV (#14033)
Allow admins to configure exceptions to our Rails rate limiter.

Configuration happens in the environment variables, and work with both
IPs and CIDR blocks.

Example:

```
env:
  DISCOURSE_MAX_REQS_PER_IP_EXCEPTIONS: >-
    14.15.16.32/27
    216.148.1.2
```
2021-08-13 12:00:23 -03:00
Blake Erickson
b35695e411
DEV: Fix some openapi spec issues (#14037)
- Remove duplicate paths
- Remove query param listed in the path
2021-08-13 04:22:15 -06:00
Blake Erickson
65f6d46045
DEV: Fix several type issues with the api docs (#14016)
`nullable` is no longer a valid type, and types also can't be an empty
string, so just bringing a number of issues with types in compliance
with the openapi spec.
2021-08-12 12:25:17 -06:00
Roman Rizzi
29bb79de37
FIX: Query the items in the queue to calculate a user's flagged post count. (#14028)
When a staff member clicks on a user's number of flagged posts, we redirect them to the review queue, so it makes sense to count the number of items there to calculate the count.

We used to look at post action items to calculate this number, which doesn't match the number of items in the queue if old flags exist.
2021-08-12 14:20:46 -03:00
Chema Balsas
745b99edbf TEST: Adds test for urls with url-encoded section hash 2021-08-12 10:43:50 -04:00
Chema Balsas
6b8ee4d5ef TEST: Adds test for urls with section hash 2021-08-12 10:43:50 -04:00
Alan Guo Xiang Tan
aed65ec16d SECURITY: User's read state for topic is leaked to unauthorized clients.
A user's read state for a topic such as the last read post number and the notification level is exposed.
2021-08-12 12:16:45 +08:00
Alan Guo Xiang Tan
9a60c83535 FIX: TopicTrackingState.report not including unread for staff posts.
Follow-up to e15c86e8c5
2021-08-12 11:10:49 +08:00
jbrw
20a6bad87e
FIX: Category group moderators can read flagged post meta_topics (#14014)
When a post is flagged with the reason of 'Something Else' a brief message can be added by the user which subsequently creates a `meta_topic` private message. The group `moderators` is automatically added to this topic.

If category group moderation is enabled, and the post belongs to a category with a reviewable group, that group should also be added to the meta_topic.

Note: This extends the `notify_moderators` logic, and will add the reviewable group to the meta_topic, regardless of the settings of that group.
2021-08-11 18:11:22 -04:00
Roman Rizzi
630d485f0f
DEV: Remove unused server-side route. (#14011)
We no longer use this route. When a staff member wants to see a user flagged posts, we redirect them to the review queue.
2021-08-11 17:29:19 -03:00
Blake Erickson
1799944a04
DEV: Specify the latest openapi spec version (#14012)
The latest openapi spec version is v3.1.0

 https://spec.openapis.org/oas/v3.1.0

Specifying the latest version will allow our openapi spec linter to use
this version and allow use to use the new type format that allows for
specifying a type as "null", which we need because sometimes our api
responses include null values instead of a "string", "integer", or
"object" type.

See: https://stackoverflow.com/a/48114322/588458
2021-08-11 12:38:02 -06:00
Blake Erickson
ce015f5b75
DEV: Fix api docs tagging format (#14010)
When specifying multiple tags they should be separate strings, not a
single string.
2021-08-11 11:00:48 -06:00
Krzysztof Kotlarek
195f7346ba
FIX: check if BasicBadge is enabled for TL1 welcome message (#13983)
In 2018 check was added that TL1 welcome message is sent unless user already has BasicBadge granted.

I think we should also check if BasicBadge is even enabled. Otherwise, each time group is assigned to a user and trust level is recalculated, they will receive a welcome message.
2021-08-11 08:39:25 +10:00
Andrei Prigorshnev
622859dbe6
FEATURE: add Unseen view (#13977)
This view is the same as Latest except it hides the topics you have fully read. Based on this plugin of @davidtaylorhq https://meta.discourse.org/t/simple-unread-list-plugin-discourse-simple-unread/70013.
2021-08-10 18:30:34 +04:00
Dmitry Klimensky
d54b339809 Escape values of HTML attributes 2021-08-10 10:25:15 -04:00
David Taylor
7dc8f8b794 FEATURE: Allow linking an existing account during external-auth signup
When a user signs up via an external auth method, a new link is added to the signup modal which allows them to connect an existing Discourse account. This will only happen if:

- There is at least 1 other auth method available

and

- The current auth method permits users to disconnect/reconnect their accounts themselves
2021-08-10 15:07:40 +01:00
David Taylor
46dc189850 DEV: Improve robustness of associate_accounts_controller
This handles a few edge cases which are extremely rare (due to the UI layout), but still technically possible:

- Ensure users are authenticated before attempting association.

- Add a message and logic for when a user already has an association for a given auth provider.
2021-08-10 15:07:40 +01:00
David Taylor
2cae29f644 DEV: Update associate_accounts_controller to use secure_session
This is much cleaner than using redis directly. It also opens the door to more complex association change flows which may happen during login.
2021-08-10 15:07:40 +01:00
Arpit Jalan
3006de39d1
REVERT "FIX: do not show private group flair on user avatars" (#13991)
This reverts commit fe3e18f981 and 0d8fd9ace6
2021-08-10 17:25:11 +05:30
Martin Brennan
6774c600a4
DEV: Fix uploads controller flaky presigned put spec (#13985)
Was missing RateLimiter.clear_all!, leading to 403 errors
2021-08-10 14:30:22 +10:00
Martin Brennan
93af139a76
Revert "DEV: Move rate limiter disable to after :each for tests (#13986)" (#13987)
This reverts commit 52035dc665.
2021-08-10 14:12:36 +10:00
Martin Brennan
52035dc665
DEV: Move rate limiter disable to after :each for tests (#13986)
We weren't calling clear_all! for the rate limiter which
was the first problem, and the second problem was that it
is very odd to do state cleanup before tests instead of after,
so moved the disabling and clear_all! to after.
2021-08-10 13:38:35 +10:00
Andrei Prigorshnev
09ad3ed41d
FEATURE: revert disallowing putting URLs in titles for TL0 users (#13970)
This reverts a part of changes introduced by https://github.com/discourse/discourse/pull/13947

In that PR I:
1. Disallowed topic feature links for TL-0 users
2. Additionally, disallowed just putting any URL in topic titles for TL-0 users

Actually, we don't need the second part. It introduced unnecessary complexity for no good reason. In fact, it tries to do the job that anti-spam plugins (like Akismet plugin) should be doing.

This PR reverts this second change.
2021-08-06 20:07:42 +04:00
Arpit Jalan
0d8fd9ace6
FIX: do not show flair bg color if flair is not visible (#13969)
follow up to fe3e18f981
2021-08-06 20:53:23 +05:30
Andrei Prigorshnev
2ec0a17bef
DEV: clarify the slow mode specs names (#13962) 2021-08-05 22:07:29 +04:00
Andrei Prigorshnev
0c0a11b66a
FEATURE: Disallow putting urls in the title for TL-0 users (#13947)
This disallows putting URLs in topic titles for TL0 users, which means that:

If a TL-0 user puts a link into the title, a topic featured link won't be generated (as if it was disabled in the site settings)
Server methods for creating and updating topics will be refusing featured links when they are called by TL-0 users
TL-0 users won't be able to put any link into the topic title. For example, the title "Hey, take a look at https://my-site.com" will be rejected.

Also, it improves a bit server behavior when creating or updating feature links on topics in the categories with disabled featured links. Before the server just silently ignored a featured link field that was passed to him, now it will be returning 422 response.
2021-08-05 13:38:39 +04:00
Alan Guo Xiang Tan
0bf27242ec FIX: Group inbox new filter not accounting for dismissed topics.
Follow-up to 2c046cc670
2021-08-05 16:53:12 +08:00
Alan Guo Xiang Tan
3f59ccefd7 FIX: Remove limit on dismissing unread and new messages.
Follow-up to 2c046cc670
2021-08-05 14:55:38 +08:00
Alan Guo Xiang Tan
2c046cc670 FEATURE: Dismiss new and unread for PM inboxes. 2021-08-05 12:56:15 +08:00
Krzysztof Kotlarek
354c939656
FIX: remove Nokogumbo references (#13951)
Specs broken after f4720205c0
2021-08-05 11:46:25 +10:00
jbrw
fb14e50741
SECURITY: Destroy EmailToken when EmailChangeRequest is destroyed (#13950) 2021-08-04 19:14:56 -04:00
Jarek Radosz
07c6b720bc
DEV: Remove PostProcessed trigger option (#13916)
It was deprecated 5 years ago in e55e2aff94

I've seen it still being used in the wild, even though it doesn't do anything anymore as I understand it.
2021-08-04 22:24:47 +02:00
Bianca Nenciu
d9843d757a
FIX: Update draft count when sequence is increased (#13940)
* FIX: Update draft count when sequence is increased

Sometimes users ended up having a draft count higher than the actual
number of drafts.

* FIX: Do not update draft count twice

The call to DraftSequence.next! above already does it.
2021-08-04 13:30:37 +03:00
Vinoth Kannan
1da0aa838f
FIX: use update_attribute method to trigger callbacks. (#13930)
Group flair is not removed while removing a user from the group since the `before_save` callback methods are not triggered while using the `update_columns` method.
2021-08-04 11:54:46 +05:30
Bianca Nenciu
54e2b95539
FIX: Remove 'reply above line' marker (#13927)
This marker is automatically added by Discourse's group SMTP mailer.
When user responded to emails sent by it, Discourse did not trim this
marker.
2021-08-03 20:08:19 +03:00
Bianca Nenciu
e2c415457c
FEATURE: Attach backup log as upload (#13849)
Discourse automatically sends a private message after backup or
restore finished. The private message used to contain the log inline
even when it was very long. A very long log can create issues because
the length of the post will be over the maximum allowed length of a
post. When that happens, Discourse will try to create an upload with
the logs. If that fails, it will trim the log and inline it.
2021-08-03 20:06:50 +03:00
Bianca Nenciu
52520638ca
FIX: Inline secure images with duplicated names (#13926)
Inlining secure images with the same name was not possible because they
were indexed by filename. If an email contained two files with the same
name, only the first image was used for both of them. The other file
was still attached to the email.
2021-08-03 18:58:34 +03:00
Blake Erickson
5c6e7e3401
DEV: Document some of the badge api endpoints (#13919)
Adding some api documentation for the badge routes.
2021-08-03 06:25:12 -06:00
Vinoth Kannan
4ec2c1e9a9
DEV: don't merge email address if target user is not human. (#13915)
While merging two user accounts don't merge the source user's email address if the target user is not a human.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2021-08-03 10:04:35 +05:30
Martin Brennan
b88d8c8894
FIX: Use reply-to address for incoming emails if present (#13896)
When the Reply-To header is present for incoming emails we
want to use it instead of the from address. This is usually the
case when forwarding an email via a mailing list into Discourse.

For now we are only using the Reply-To header if the email has
been forwarded via Google Groups, which is why we are checking the
X-Original-From header too. In future we may want to use the Reply-To
header in more cases.
2021-08-03 08:01:17 +10:00
Jean
e7b8e75583
FEATURE: Add post edits count to user activity (#13495) 2021-08-02 10:15:53 -04:00
Bianca Nenciu
fbf7627c8e
FIX: Make search work with sub-sub-categories (#13901)
Searching in a category looked only one level down, ignoring the site
setting max_category_nesting. The user interface did not support the
third level of categories and did not display them in the "Categorized"
input of the advanced search options.
2021-08-02 14:04:13 +03:00
Alan Guo Xiang Tan
016efeadf6
FEATURE: New and Unread messages for user personal messages. (#13603)
* FEATURE: New and Unread messages for user personal messages.

Co-authored-by: awesomerobot <kris.aubuchon@discourse.org>
2021-08-02 12:41:41 +08:00
Arpit Jalan
fe3e18f981
FIX: do not show private group flair on user avatars (#13872)
Meta ref: https://meta.discourse.org/t/visible-flair-for-invisible-groups-is-that-on-purpose/167674
2021-08-02 06:21:00 +05:30
Jean
ac777440fd
FIX: Validate value of custom dropdown user fields - dropdowns and multiple selects (#13890) 2021-07-30 13:50:47 -04:00
jbrw
2f28ba318c
FEATURE: Onebox can match engines based on the content_type (#13876)
* FEATURE: Onebox can match engines based on the content_type

`FinalDestination` now returns the `content_type` of a resolved URL.

`Oneboxer` passes this value to `Onebox` itself. Onebox engines can now specify a `matches_content_type` regex of content_types that the engine can handle, regardless of the URL.

`ImageOnebox` will match URLs with a content type of `image/png`, `jpg`, `gif`, `bmp`, `tif`, etc.

This will allow images that exist at a URL without a file type extension to be correctly rendered, assuming a valid `content_type` is returned.
2021-07-30 13:36:30 -04:00
Blake Erickson
1c60be7658
DEV: Document anonymize user api endpoint (#13893)
Adding the anonymize user endpoint to the api docs.

See: https://meta.discourse.org/t/158704/7
2021-07-29 17:40:41 -06:00
Bianca Nenciu
300db3d3fa
FIX: Update draft count after creating a post (#13884)
When a post is created, the draft sequence is increased and then older
drafts are automatically executing a raw SQL query. This skipped the
Draft model callbacks and did not update user's draft count.

I fixed another problem related to a raw SQL query from Draft.cleanup!
method.
2021-07-29 17:06:11 +03:00
Joffrey JAFFEUX
74f0631acd
FIX: allows authentication data to be present in bootstrap (#13885) 2021-07-29 15:01:11 +02:00
Alan Guo Xiang Tan
2b5625bbf0
FIX: Avoid creating a post revision when topic tags have not changed. (#13881)
Co-authored-by: jmperez127 <jmperez127@gmail.com>
2021-07-29 08:14:25 -04:00
Alan Guo Xiang Tan
543a2d70b2 FIX: PM tags route should work for usernames with a period. 2021-07-29 13:54:29 +08:00
Andrei Prigorshnev
5a2ad7e386
DEV: remove calls to guardian from GroupActionLogger (#13835)
We shouldn't be checking if a user is allowed to do an action in the logger. We should be checking it just before we perform the action. In fact, guardians in the logger can make things even worse in case of a security bug. Let's say we forgot to check user's permissions before performing some action, but we still have a call to the guardian in the logger. In this case, a user would perform the action anyway, and this action wouldn't even be logged!

I've checked all cases and I confirm that we're safe to delete this calls from the logger.

I've added two calls to guardians in admin/user_controller. We didn't have security bugs there, because regular users can't access admin/... routes at all. But it's good to have calls to guardian in these methods anyway, neighboring methods have them.
2021-07-28 15:04:04 +04:00
Alan Guo Xiang Tan
32951ca2f4 FIX: User can change name when auth_overrides_name is enabled. 2021-07-28 14:40:57 +08:00
Martin Brennan
b500949ef6
FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787)
This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads`  settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader.

A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used. 

### Starting a direct S3 upload

When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded.

Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage.

### Completing a direct S3 upload

Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`.

1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this.

2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues.

    We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large.

3. Finally we return the serialized upload record back to the client

There are several errors that could happen that are handled by `UploadsController` as well.

Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
2021-07-28 08:42:25 +10:00
Gerhard Schlager
4a37612fd5
FIX: Changing the post owner didn't update the reply_to_user_id of replies (#13862) 2021-07-27 20:49:08 +02:00
Andrew Schleifer
461cb96532 use more appropriate labels for chinese UI option 2021-07-27 22:47:59 +08:00
Bianca Nenciu
760c9a5698
FEATURE: Show draft count in user menu and activity (#13812)
This commit adds the number of drafts a user has next to the "Draft"
label in the user preferences menu and activity tab. The count is
updated via MessageBus when a draft is created or destroyed.
2021-07-27 14:05:33 +03:00
wbhouston
efe38efb0a
FIX: Don't grant sharing badges to users who don't exist (#13851)
In badge queries for 'First Share' and 'Nice/Good/Great Share' badges,
check that the user exists.

For 'Nice+ Share' badges, also grant badges if the number of shares is
equal to the threshhold count to better match the descriptions.
2021-07-27 16:32:59 +10:00
Alan Guo Xiang Tan
1780961e70 DEV: Fix flaky topics invite spec due to id collision.
The hardcoded group id eventually became a valid group id leading to 2
groups being attached to the invite.
2021-07-27 13:59:07 +08:00
Vinoth Kannan
5a93893b08
FIX: use correct URL in schema markup for post images. (#13847)
Currently, it wrongly adds Discourse base URL in prefix even for CDN URLs.
2021-07-26 21:39:51 +05:30
Martin Brennan
3d25572970
DEV: Remove flaky topic_view_serializer spec (#13845)
We tried fixing this one in b579e9a7de
but it continues to cause nothing but problems.
2021-07-26 13:17:53 +10:00
Krzysztof Kotlarek
88aa0136e3
FIX: do not raise exception when svg path is nil (#13844)
Bug was introduced here: f7ab852e12

If path is nil, it should not raise an exception and continue logging the error.
2021-07-26 12:35:27 +10:00
Bianca Nenciu
6db93e86d4
FIX: Show Uncategorized when unsubscribing (#13832)
If user tried to unsubscribe from a post from category Uncategorized,
the category name was not displayed. It said only "Stop watching all
topics in".
2021-07-26 12:19:30 +10:00
Alan Guo Xiang Tan
92d62eb2a4 DEV: Remove unncessary fabrication in tests. 2021-07-26 09:14:23 +08:00
Blake Erickson
6ac3f1f7b5
DEV: Return 400 instead of 500 for invalid top period (#13828)
* DEV: Return 400 instead of 500 for invalid top period

This change will prevent a fatal 500 error when passing in an invalid
period param value to the `/top` route.

* Check if the method exists first

I couldn't get `ListController.respond_to?` to work, but was still able
to check if the method exists with
`ListController.action_methods.include?`. This way we can avoid relying
on the `NoMethodError` exception which may be raised during the course
of executing the method.

* Just check if the period param value is valid

* Use the new TopTopic.validate_period method
2021-07-23 14:58:10 -06:00
Robin Ward
7b45a5ce55 FIX: Better and more secure validation of periods for TopicQuery
Co-authored-by: Martin Brennan <mjrbrennan@gmail.com>
2021-07-23 14:24:44 -04:00
Alan Guo Xiang Tan
53082e011a DEV: Make rubocop happy. 2021-07-23 16:04:41 +08:00
Alan Guo Xiang Tan
2923abdec7 SECURITY: Don't leak user of previous whisper post when deleting a topic.
A topic's last poster can be incorrectly set to a user of a whisper post
if the whisper post is before the last post and the last post is
deleted.
2021-07-23 15:42:23 +08:00
Alan Guo Xiang Tan
303e9e42b6 SECURITY: Do not reveal post whisperer in personal messages.
Prior to this fix, post whisperer in personal messages are revealed in
the topic's participants list even though non-staff users are unable to
see the whisper.
2021-07-23 15:42:23 +08:00
Andrei Prigorshnev
8bc01c1bb5
DEV: extract leave_group method from the group#remove_member method (#13823)
* Copy remove_member to new `leave` method

* Remove unneeded code from the leave method

* Rearrange the leave method

* Remove unneeded code from the remove_member method

* Add tests

* Implement on the client side
2021-07-22 20:14:18 +04:00
Bianca Nenciu
18c32a809b
FIX: Validate email_accent_bg_color color (#13778)
Using an invalid value was allowed. This commit tries to automatically
fix the color by adding missing # symbol or will show an error to the
user if it is not possible and it is not a CSS color either.
2021-07-22 17:42:47 +03:00
Andrei Prigorshnev
3cf7a3766a
DEV: extract join_group method from groups#add_members method (#13807)
* Copy the add_members method to the new join method

* Remove unneeded code from the join method

* Rearrange the join method

* Remove unneeded stuff from the add_members method

* Extract add_user_to_group method

* Implement of the client side

* Tests

* Doesn't inline users.uniq

* Return promise from join.then()

* Remove unnecessary begin and end

* Revert "Return promise from join.then()"

This reverts commit bda84d8d

* Remove variable already_in_group
2021-07-22 11:11:23 +04:00
Martin Brennan
f41908ad5b
SECURITY: Validate period param for top topic routes (#13818)
Fixes a possible SQL injection vector
2021-07-22 16:31:53 +10:00
Joffrey JAFFEUX
5eb6e9281a
FIX: manually adds frowning_face_with_open_mouth for apple (#13528) 2021-07-21 23:27:20 +02:00
Penar Musaraj
2ce2c83bc9
FIX: Show user filter hints when typing @ in search (#13799)
Will show the last 6 seen users as filtering suggestions when typing @ in quick search. (Previously the user suggestion required a character after the @.)

This also adds a default limit of 6 to the user search query, previously the backend was returning 20 results but a maximum of 6 results was being shown anyway.
2021-07-21 09:14:53 -04:00
Bianca Nenciu
4da0a33524
FIX: Allow only groups with flairs to be selected (#13744)
It used the same permission check as for primary groups which is
wrong because not all groups that can be primary have a flair.
2021-07-21 14:41:04 +03:00
mintsaxon
7162ecfb04 FEATURE: Per-category default slow mode duration for topics.
When configured, all topics in the category inherits the slow mode
duration from the category's default.

Note that currently there is no way to remove the slow mode from the
topics once it has been set.
2021-07-21 12:32:07 +08:00
Gerhard Schlager
0ea58fe51f DEV: Spec shouldn't depend on translation 2021-07-21 12:24:54 +08:00
Andrei Prigorshnev
1a8c949900
UX: suspend forever time period messages (#13776)
When the Forever option is selected for suspending a user, the user is suspended for 1000 years. Without customizing the site’s text, this time period is displayed to the user in the suspension email that is sent to the user, and if the user attempts to log back into the site. Telling someone that they have been suspended for 1000 years seems likely to come across as a bad attempt at humour.

This PR special case messages when a user suspended or silenced forever.
2021-07-20 14:42:08 +04:00
Bianca Nenciu
3d387d7891
FIX: Serialize flair group ID for preferences page (#13741)
Staff viewing the user preferences page of other users did not see the
selected flair because that information was not serialized for them.
2021-07-20 13:06:40 +03:00
Martin Brennan
0dadd61d27
FIX: Change email from to not have via site_name for group SMTP (#13788)
We now use the group's full name in group SMTP emails, so we are dropping the via #{site_name}. If group owners still want this they can just change the full name of the group.
2021-07-20 11:56:04 +10:00
Mark VanLandingham
80950d7b28
DEV: Add chat_mention notification type (#13784) 2021-07-19 14:52:12 -05:00
Michael Brown
76a11e6dc9 DEV: fix test (missed a reference to master) 2021-07-19 12:47:45 -04:00
Michael Brown
aa12d12c0b discourse/discourse change from 'master' to 'main': update fixture data 2021-07-19 11:46:15 -04:00
Alan Guo Xiang Tan
a1047f5ef4
FEATURE: Add new plugin API to allow plugins to extend Site#categories (#13773) 2021-07-19 13:54:19 +08:00
Neil Lalonde
b0f06b8ed0
FIX: don't allow category and tag tracking settings on staged users (#13688)
Configuring staged users to watch categories and tags is a way to sign
them up to get many emails. These emails may be unwanted and get marked
as spam, hurting the site's email deliverability.
Users can opt-in to email notifications by logging on to their
account and configuring their own preferences.

If staff need to be able to configure these preferences on behalf of
staged users, the "allow changing staged user tracking" site setting
can be enabled. Default is to not allow it.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2021-07-16 14:50:40 -04:00
Rafael dos Santos Silva
e12b00eab7
FEATURE: Stop checking referer for embeds (#13756)
Flips content_security_policy_frame_ancestors default to enabled, and
removes HTTP_REFERER checks on embed requests, as the new referer
privacy options made the check fragile.
2021-07-16 15:25:49 -03:00
Penar Musaraj
361c8be547
PERF: Add scheduled job to delete old stylesheet cache rows (#13747) 2021-07-16 10:58:01 -04:00
Andrei Prigorshnev
c4d7545f35
FIX: when updating timestamps on topic set a correct bump date (#13746)
There was a bug with changing timestamps using the topic wrench button. Under some circumstances, a topic was disappearing from the top of the latest tab after changing timestamps. Steps to reproduce:
- Choose a topic on the latest tab (the topic should be created some time ago, but has recent posts)
- Change topic timestamps (for example, move them one day forward):
- Go back to the latest tab and see that topic has disappeared.

This PR fixes this. We were setting topic.bumped_at to the timestamp user specified on the modal. This is incorrect. Instead, we should be setting topic.bumped_at to the created_at timestamp of the last regular (not a whisper and so on) post on the topic.
2021-07-16 11:56:51 +04:00
Andrei Prigorshnev
1cadae3879
FIX: simplify and improve choosing favorite badges (#13743)
* No need to return anything except a status code from the server

* Switch a badge state before sending a request and then switch it back in case of an error
2021-07-16 11:13:00 +08:00
David Taylor
8b89787426
SECURITY: Sanitize YouTube Onebox data (#13748)
CVE-2021-32764
2021-07-15 19:31:50 +01:00
Penar Musaraj
a23153fdca
FIX: Add order to outputted stylesheet link tags (#13735)
See PR for details. (Disabled by default in this commit.)
2021-07-15 12:51:46 -04:00
Vinoth Kannan
d6fc39c886
FEATURE: update existing users when group default notifications changed. (#13434)
Currently, the changes will only affect the users added after.
2021-07-15 19:53:57 +05:30
Andrei Prigorshnev
5cd447695e
FIX: problems with choosing favorite badges (#13731) 2021-07-15 14:51:44 +04:00
Osama Sayegh
31aa701518
FEATURE: Add option to grant badge multiple times to users using Bulk Award (#13571)
Currently when bulk-awarding a badge that can be granted multiple times, users in the CSV file are granted the badge once no matter how many times they're listed in the file and only if they don't have the badge already.

This PR adds a new option to the Badge Bulk Award feature so that it's possible to grant users a badge even if they already have the badge and as many times as they appear in the CSV file.
2021-07-15 05:53:26 +03:00
Osama Sayegh
2484abddb6
FIX: Assets for the theme tests page are not compressed (#13736)
A couple of weeks we made a change that skipped compressing assets used by the theme qunit page: https://github.com/discourse/discourse/pull/13619. This is a follow-up PR to stop the application helper from generating the assets for the theme qunit page with `.br` or `.gzip` extensions when a site uses S3 as a CDN.
2021-07-14 22:52:35 +03:00
Penar Musaraj
f7ab852e12
FIX: Issues with custom icons in themes (#13732)
Fixes two issues:
- ignores invalid XML in custom icon sprite SVG file (and outputs an error if sprite was uploaded via admin UI)
- clears SVG sprite cache when deleting an `icons-sprite` upload in a theme
2021-07-14 15:18:29 -04:00
Martin Brennan
068889cb5f
FIX: Email threads sometimes not grouping for group SMTP (#13727)
This PR fixes a couple of issues related to group SMTP:

1. When running the group SMTP job, we were exiting early if the email was for the OP because of an IMAP race condition. However this causes issues when replying as a new topic for an existing SMTP topic, as the recipient does not get the OP email which can cause threading problems.
2. When sending emails for a new topic spun out like the issue in 1., we are not maintaining the original subject/topic title because that is based on the incoming email record, which we were not doing because the group SMTP email was never sent because of issue 1.
2021-07-14 14:23:14 +10:00
jbrw
a64aea38b7
FIX: Don’t use user_generated images as avatar images in Oneboxed Twitter content (#13712)
By default, Twitter will return the URL for the avatar image of the tweet poster as the `og:image` value.

However, if the `user_generated` attribute is true, we should not use this as the avatar URL as this will be an URL of an image in the tweet itself (e.g., an image belonging to a tweeted news story).
2021-07-13 14:54:28 -04:00
Guhyoun Nam
0dc96ce817
FEATURE: Setting to allow moderators to change post ownership (#13708) 2021-07-13 09:40:11 -05:00
David Taylor
800c6e1a68 PERF: Improve topic_user.liked update performance when moving posts
Previously we would re-calculate topic_user.liked for all users who have ever viewed the source or destination topic. This can be very expensive on large sites. Instead, we can use the array of moved post ids to find which users are actually affected by the move, and restrict the update query to only check/update their records.

On an example site this reduced the `update_post_action_cache` time from ~27s to 300ms
2021-07-13 12:30:38 +01:00
Martin Brennan
7911124d3d
FEATURE: Uppy image uploader with UppyUploadMixin (#13656)
This PR adds the first use of Uppy in our codebase, hidden behind a enable_experimental_image_uploader site setting. When the setting is enabled only the user card background uploader will use the new uppy-image-uploader component added in this PR.

I've introduced an UppyUpload mixin that has feature parity with the existing Upload mixin, and improves it slightly to deal with multiple/single file distinctions and validations better. For now, this just supports the XHRUpload plugin for uppy, which keeps our existing POST to /uploads.json.
2021-07-13 12:22:00 +10:00
Dan Ungureanu
49090c3524
FIX: Suggest current username for staged users (#13706)
If user had a staged account and logged in using a third party service
a different username was suggested. This change will try to use the
username given by the authentication provider first, then the current
staged username and last suggest a new one.
2021-07-13 09:15:06 +10:00
Jeff Wong
ee539632ad
FIX: admin_user_response schema (#13711) 2021-07-12 15:00:57 -07:00
Bianca Nenciu
f74640ac58
FIX: Show correct dates in topic timelines (#13670)
The date shown in topic timeline was one day later if the post at that
position was made near midnight. This happened because the days number
was rounded down.
2021-07-12 19:35:24 +03:00
Arpit Jalan
953fd2cb50
FEATURE: add title tag for group detail page (#13702) 2021-07-12 20:05:57 +05:30
Dan Ungureanu
2d904ade6d
FIX: Let staged users choose their username (#13678)
When a staged user tried to redeem an invite, a different username was
suggested and manually typing the staged username failed because the
username was not available.
2021-07-12 07:57:38 +10:00
Bianca Nenciu
43d4859d30
FIX: Let user clear their flair group (#13669)
Users were able to select their flair, but were not able to clear it
by selecting (none).
2021-07-09 12:41:26 +08:00
Martin Brennan
9f275c12ab
FIX: Handle storage providers not implementing ACLs (#13675)
When secure media is enabled or when upload secure status
is updated, we also try and update the upload ACL. However
if the object storage provider does not implement this we
get an Aws::S3::Errors::NotImplemented error. This PR handles
this error so the update_secure_status method does not error
out and still returns whether the secure status changed.
2021-07-09 11:31:44 +10:00
Kane York
371fba6ae0 DEV: Commit failing test for email substution bug 2021-07-08 15:56:09 -07:00
Bianca Nenciu
87c1e98571
FEATURE: Let users select flair (#13587)
User flair was given by user's primary group. This PR separates the
two, adds a new field to the user model for flair group ID and users
can select their flair from user preferences now.
2021-07-08 10:46:21 +03:00
Martin Brennan
c6f2459cc4
FIX: Do not prevent other topic timers running on error (#13665)
There was an issue with the TopicTimerEnqueuer where any timer
that failed to enqueue_typed_job with an error would prevent
all other pending timers after the one that errored from running.

To mitigate this we just capture the error and log it (so we can
still fix it if needed for bug crushing) and proceed with the
rest of the timer enqueues.

The commit https://github.com/discourse/discourse/pull/13544 highlighted
this issue originally in hosted sites.

<!-- NOTE: All pull requests should have tests (rspec in Ruby, qunit in JavaScript). If your code does not include test coverage, please include an explanation of why it was omitted. -->
2021-07-08 12:49:58 +10:00
Jarek Radosz
48b92d8897
DEV: Isolate multisite specs (#13634)
Mixing multisite and standard specs can lead to issues (e.g. when using `fab!`)
Disabled the (upcoming https://github.com/discourse/rubocop-discourse/pull/11) rubocop rule for two files that have thoroughly tangled both types of specs.
2021-07-07 18:57:42 +02:00
Mark VanLandingham
14a13dc192
FIX: Check type of existing reviewables when new reviewable is created (#13662) 2021-07-07 11:45:00 -05:00
Dan Ungureanu
9969631cef
FEATURE: Make invites work with existing users (#13532)
* FEATURE: Redirect logged in user to invite topic

Users who were already logged in and were given an invite link to a
topic used to see an error message saying that they already have an
account and cannot redeem the invite. This commit amends that behavior
and redirects the user directly to the topic, if they can see it.

* FEATURE: Add logged in user to invite groups

Users who were already logged in and were given an invite link to a
group used to see an error message saying that they already have an
account and cannot redeem the invite. This commit amends that behavior
and adds the user to the group.
2021-07-07 19:42:42 +03:00
Penar Musaraj
35110f6681
FIX: Set CSP base-uri to self (#13654) 2021-07-07 09:43:48 -04:00
David Taylor
a1e5a6bbe0
DEV: Remove fsl cookie from specs (#13657)
This cookie has not been used for a number of years, and has no effect. This commit removes it from the specs. (diff is almost entirely whitespace)
2021-07-07 10:54:18 +01:00
dependabot[bot]
dda41cf253
Build(deps): Bump rubocop from 1.18.2 to 1.18.3 (#13653)
* Build(deps): Bump rubocop from 1.18.2 to 1.18.3

Bumps [rubocop](https://github.com/rubocop/rubocop) from 1.18.2 to 1.18.3.
- [Release notes](https://github.com/rubocop/rubocop/releases)
- [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rubocop/rubocop/compare/v1.18.2...v1.18.3)

---
updated-dependencies:
- dependency-name: rubocop
  dependency-type: indirect
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix offences

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2021-07-07 01:51:43 +02:00
Penar Musaraj
95b5794331
DEV: Compile core and plugin stylesheets independently of themes (#13638)
Take 2 of https://github.com/discourse/discourse/pull/13466. 

Fixes a few issues with the original PR: 

- color definition stylesheet target now includes the theme id, to avoid themes set to use the default color scheme loading the same stylesheet 
- changes the internal cache key for color definition stylesheet to reset the pre-existing cache
2021-07-06 13:11:10 -04:00
Vinoth Kannan
33eae4cbd8
FEATURE: add period filter in top topics route for tags. (#13415)
And also move all the "top topics by period" routes to query string param.

/top/monthly => /top?period=monthly
/c/:slug/:id/l/top/monthly => /c/:slug/:id/l/top?period=monthly
/tag/:slug/l/top/daily => /tag/:slug/l/top?period=daily (new)
2021-07-06 15:25:11 +05:30
Dan Ungureanu
34387c5a38
FEATURE: Warn if invited user cannot see topic (#13548)
Users can invite people to topics from secured category, but they will
not be redirected to the topic after signing up unless they have the
permissions to view the topic. This commit shows a warning when invite
is saved if the topic is in a secured category and none of the invite
groups are allowed to see it.
2021-07-06 12:49:26 +03:00