Commit Graph

995 Commits

Author SHA1 Message Date
Rafael dos Santos Silva
6888eb5c2d
Revert "Revert "FEATURE: Preload resources via link header (#18475)" (#18511)" (#18531)
* Revert "Revert "FEATURE: Preload resources via link header (#18475)" (#18511)"

This reverts commit 95a57f7e0c.

* put behind feature flag

* env -> global setting

* declare global setting

* forgot one spot
2022-10-11 20:11:44 -03:00
Jacob Michalskie
36f7fbebdc
DEV: Allow ManagedAuthenticator classes to match by username (#18517) 2022-10-11 11:25:13 +01:00
Martin Brennan
cab173d3b5
DEV: Add specs for hashtags related code (#18526)
Adding a couple of small specs to code around hashtags
for tags and categories since I will be working in this
area soon and these were not covered.
2022-10-11 09:04:51 +10:00
Blake Erickson
e320bbe513
FEATURE: Adds seeded default categories to the sidebar (#18512) 2022-10-07 16:02:03 -06:00
Rafael dos Santos Silva
95a57f7e0c
Revert "FEATURE: Preload resources via link header (#18475)" (#18511)
This reverts commit 2d1dbc6f96.

We need to increase nginx proxy buffer to land this.
2022-10-07 15:08:40 -03:00
Rafael dos Santos Silva
2d1dbc6f96
FEATURE: Preload resources via link header (#18475)
Experiment moving from preload tags in the document head to preload information the the response headers.

While this is a minor improvement in most browsers (headers are parsed before the response body), this allows smart proxies like Cloudflare to "learn" from those headers and build HTTP 103 Early Hints for subsequent requests to the same URI, which will allow the user agent to download and parse our JS/CSS while we are waiting for the server to generate and stream the HTML response.

Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
2022-10-07 13:19:50 -03:00
Krzysztof Kotlarek
e6b5b6eae3
DEV: update to patched discourse-seed-fu (#18493)
Original seed-fu was cloned and patched with David's fix 576b69a368
2022-10-07 09:16:04 +11:00
Bianca Nenciu
73e9875a1d
FEATURE: Handle oneboxes for complex GitHub URLs (#18474)
GitHub PR URLs can link to a commit of the PR, a comment or a review
discussion.
2022-10-06 20:26:04 +03:00
Alan Guo Xiang Tan
f3392a5a81
DEV: Avoid configuring Rails configuration in tests (#18499) 2022-10-06 15:45:19 +08:00
Arpit Jalan
8ae1edeb79
FIX: do not prefill default site title value on wizard introduction step (#18496) 2022-10-06 12:02:48 +05:30
Martin Brennan
6d7abc1c85
FIX: Make sure first admin users are added to auto groups (#18494)
When a user with an email matching those inside the
DISCOURSE_DEVELOPER_EMAILS env var log in, we make
them into admin users if they are not already. This
is used when setting up the first admin user for
self-hosters, since the discourse-setup script sets
the provided admin emails into DISCOURSE_DEVELOPER_EMAILS.

The issue being fixed here is that the new admins were
not being automatically added to the staff and admins
automatic groups, which was causing issues with the site
settings that are group_list based that don't have an explicit
staff override. All we need to do is refresh the automatic
staff, admin groups when admin is granted for the user.
2022-10-06 15:16:38 +10:00
Martin Brennan
f5194aadd3
DEV: Remove usages of enable_personal_messages (#18437)
cf. e62e93f83a

This PR also makes it so `bot` (negative ID) and `system` users are always allowed
to send PMs, since the old conditional was just based on `enable_personal_messages`
2022-10-05 10:50:20 +10:00
Osama Sayegh
2d391565e4
FIX: Skip quality title validations for static topics when edited by admin (#18468)
Static topics are the seeded topics that are automatically created for every Discourse instance to hold the content for the FAQ, ToS and Privacy pages. These topics are allowed to bypass the minimum title length checks when they're edited by admins:

ba27ee1637/app/assets/javascripts/discourse/app/models/composer.js (L487-L496)

However, on the server-side, the "quality title" validations aren't skipped for static topics and that can cause confusion for admins when they change the title of a static topic to something that's short enough to fail the quality title validations. This commit ignores all quality title validations on static topics when they're edited by admins.

Internal topic: t/75745.
2022-10-04 21:55:21 +03:00
Selase Krakani
0c38757250
FIX: Revert recursively tag lookup with missing ancestor tags (#18439)
This reverts commit 049f8569d8.

To be revisited with a more comprehensive solution covering parent
selection when multiple parents exist.
2022-09-30 08:28:09 +00:00
Vinoth Kannan
6ebd2cecda
FIX: missing theme upload should not break precompile process. (#18431)
Previously, if an active default theme's upload record went missing then it will break the site and cause downtime.
2022-09-30 10:48:26 +05:30
Bianca Nenciu
f60e6837c6
FEATURE: Add setting to always confirm old email (#18417)
By default, only staff members have to confirm their old email when
changing it. This commit adds a site setting that when enabled will
always ask the user to confirm old email.
2022-09-30 00:49:17 +03:00
Jarek Radosz
b27d5626d2
SECURITY: Prevent arbitrary file write when decompressing files (#18421)
* SECURITY: Prevent arbitrary file write when decompressing files
* FIX: Allow decompressing files into symlinked directories

Co-authored-by: OsamaSayegh <asooomaasoooma90@gmail.com>
Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
2022-09-29 20:00:38 +02:00
Martin Brennan
8ebd5edd1e
DEV: Rename secure_media to secure_uploads (#18376)
This commit renames all secure_media related settings to secure_uploads_* along with the associated functionality.

This is being done because "media" does not really cover it, we aren't just doing this for images and videos etc. but for all uploads in the site.

Additionally, in future we want to secure more types of uploads, and enable a kind of "mixed mode" where some uploads are secure and some are not, so keeping media in the name is just confusing.

This also keeps compatibility with the `secure-media-uploads` path, and changes new
secure URLs to be `secure-uploads`.

Deprecated settings:

* secure_media -> secure_uploads
* secure_media_allow_embed_images_in_emails -> secure_uploads_allow_embed_images_in_emails
* secure_media_max_email_embed_image_size_kb -> secure_uploads_max_email_embed_image_size_kb
2022-09-29 09:24:33 +10:00
Arpit Jalan
2ee721f8aa
FEATURE: add composer warning when user haven't been seen in a long time (#18340)
* FEATURE: add composer warning when user haven't been seen in a long time

When a user creates a PM and adds a recipient that hasn't been seen in a
long time then we'll now show a warning in composer indicating that the
user hasn't been seen in a long time.
2022-09-27 22:06:40 +05:30
Selase Krakani
049f8569d8
FIX: Recursively tag topics with missing ancestor tags (#18344)
* FIX: Recursively tag topics with missing ancestor tags

Given only a child tag, walk up the ancestry chain, get all of it's
ancestors for use in tagging a topic

* FIX: Ensure only one parent tag is returned for topic tagging

Current implementation selects and return first parent tag if child tag
has multiple parents.

This change updates recursive parent tag implementation to only return
parent tags via only one ancestry line.

* DEV: Add test case for tag cycles

Given we aren't performing a strict graph traversal to get a tag's
parent, cycles do not have any effect on the tags returned for topic
tagging.
2022-09-27 12:04:16 +00:00
Vinoth Kannan
076abe46fa
FEATURE: new site setting to set locale from cookie for anonymous users. (#18377)
This new hidden default-disabled site setting `set_locale_from_cookie` will set locale from anonymous user's cookie value.
2022-09-27 14:26:06 +05:30
Martin Brennan
7152345ee7
FIX: list_suggested_for conditional for personal_message_enabled_groups (#18373)
Follow-up to e62e93f83a,
misplaced a bracket and changed the meaning of the conditional.
2022-09-27 16:54:44 +10:00
Martin Brennan
e62e93f83a
FEATURE: Introduce personal_message_enabled_groups setting (#18042)
This will replace `enable_personal_messages` and
`min_trust_to_send_messages`, this commit introduces
the setting `personal_message_enabled_groups`
and uses it in all places that `enable_personal_messages`
and `min_trust_to_send_messages` currently apply.

A migration is included to set `personal_message_enabled_groups`
based on the following rules:

* If `enable_personal_messages` was false, then set
  `personal_message_enabled_groups` to `3`, which is
  the staff auto group
* If `min_trust_to_send_messages` is not default (1)
  and the above condition is false, then set the
  `personal_message_enabled_groups` setting to
  the appropriate auto group based on the trust level
* Otherwise just set `personal_message_enabled_groups` to
  11 which is the TL1 auto group

After follow-up PRs to plugins using these old settings, we will be
able to drop the old settings from core, in the meantime I've added
 DEPRECATED notices to their descriptions and added them
to the deprecated site settings list.

This commit also introduces a `_map` shortcut method definition
for all `group_list` site settings, e.g. `SiteSetting.personal_message_enabled_groups`
also has `SiteSetting.personal_message_enabled_groups_map` available,
which automatically splits the setting by `|` and converts it into
an array of integers.
2022-09-26 13:58:40 +10:00
Martin Brennan
e3d495850d
FEATURE: Overhaul email threading (#17996)
See https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499
for thorough reasoning.

This commit changes how we generate Message-IDs and do email
threading for emails sent from Discourse. The main changes are
as follows:

* Introduce an outbound_message_id column on Post that
  is either a) filled with a Discourse-generated Message-ID
  the first time that post is used for an outbound email
  or b) filled with an original Message-ID from an external
  mail client or service if the post was created from an
  incoming email.
* Change Discourse-generated Message-IDs to be more consistent
  and static, in the format `discourse/post/:post_id@:host`
* Do not send References or In-Reply-To headers for emails sent
  for the OP of topics.
* Make sure that In-Reply-To is filled with either a) the OP's
  Message-ID if the post is not a direct reply or b) the parent
  post's Message-ID
* Make sure that In-Reply-To has all referenced post's Message-IDs
* Make sure that References is filled with a chain of Message-IDs
  from the OP down to the parent post of the new post.

We also are keeping X-Discourse-Post-Id and X-Discourse-Topic-Id,
headers that we previously removed, for easier visual debugging
of outbound emails.

Finally, we backfill the `outbound_message_id` for posts that have
a linked `IncomingEmail` record, using the `message_id` of that record.
We do not need to do that for posts that don't have an incoming email
since they are backfilled at runtime if `outbound_message_id` is missing.
2022-09-26 09:14:24 +10:00
Loïc Guitaut
26fe047724 DEV: Use AR enums in reviewables related code
This is the first patch of many to replace our custom enums in Ruby by
the ones provided by `ActiveRecord`.
2022-09-22 14:44:27 +02:00
David Taylor
d0243f741e
UX: Use dominant color as image loading placeholder (#18248)
We previously had a system which would generate a 10x10px preview of images and add their URLs in a data-small-upload attribute. The client would then use that as the background-image of the `<img>` element. This works reasonably well on fast connections, but on slower connections it can take a few seconds for the placeholders to appear. The act of loading the placeholders can also break or delay the loading of the 'real' images.

This commit replaces the placeholder logic with a new approach. Instead of a 10x10px preview, we use imagemagick to calculate the average color of an image and store it in the database. The hex color value then added as a `data-dominant-color` attribute on the `<img>` element, and the client can use this as a `background-color` on the element while the real image is loading. That means no extra HTTP request is required, and so the placeholder color can appear instantly.

Dominant color will be calculated:
1. When a new upload is created
2. During a post rebake, if the dominant color is missing from an upload, it will be calculated and stored
3. Every 15 minutes, 25 old upload records are fetched and their dominant color calculated and stored. (part of the existing PeriodicalUpdates job)

Existing posts will continue to use the old 10x10px placeholder system until they are next rebaked
2022-09-20 10:28:17 +01:00
Gerhard Schlager
f91426a7fa
FIX: Empty inline BBCodes were broken (#18276)
Upgrading to Markdown.it v13 broke empty inline BBCodes. This works around the problem by adding an empty token before a closing token if the previous token was a BBCode token.

It also removes the unused `jump` attribute which was removed in Markdown.it v12.3
2022-09-20 09:50:22 +10:00
Jarek Radosz
08e63ddab2
DEV: Fix spec file name (#18227)
Match the impl file name
2022-09-12 14:03:23 +02:00
David Taylor
7e74dd0afe
DEV: Use DiscourseJsProcessor for theme template compilation (#18135)
Previously we were relying on a highly-customized version of the unmaintained Barber gem for theme template compilation. This commit switches us to use our own DiscourseJsProcessor, which makes use of more modern patterns and will be easier to maintain going forward.

In summary:
- Refactors DiscourseJsProcessor to move multiline JS heredocs into a companion `discourse-js-processor.js` file
- Use MiniRacer's `.call` method to avoid manually escaping JS strings
- Move Theme template AST transformers into DiscourseJsProcessor, and formalise interface for extending RawHandlebars AST transformations
- Update Ember template compilation to use a babel-based approach, just like Ember CLI. This gives each template its own ES6 module rather than directly assigning `Ember.TEMPLATES` values
- Improve testing of template compilation (and move some tests from `theme_javascript_compiler_spec.rb` to `discourse_js_processor_spec.rb`
2022-09-01 11:50:46 +01:00
Arpit Jalan
3aaf4dcfd0
FIX: hide welcome topic banner as soon as the welcome topic is edited (#18125)
* FIX: hide welcome topic banner as soon as the welcome topic is edited

This commit adds a message bus listener on client to hide the welcome
topic banner as soon as the welcome topic is edited.

* update test

* only subscribe when show_welcome_topic_banner is true

* Do not lookup for messageBus service if it's not required

* Remove unneeded code

* Cache result for Site.show_welcome_topic_banner

* Update tests per latest changes

* Changes per PR review
2022-09-01 09:44:21 +05:30
Jarek Radosz
8fa9f0cf92
DEV: Fix a flaky spec (#18146)
In some cases the topic of the fabricated post can be titled "This is a test topic 777" which matches the search query "#777"
2022-08-31 20:52:57 +02:00
David Taylor
7eaa4f6ad3
DEV: Add test for EmberCli#ember_version (#18140)
Followup to 6570fed175
2022-08-30 20:33:08 +01:00
Blake Erickson
00539307e1
FIX: Only seed general category on new sites (#18130)
* FIX: Only seed general category on new sites

If the site already has human users (users with an id > 0) don't seed
the categories.

Follow up to: a6ad74c759

* use human_users scope
2022-08-29 18:23:14 -06:00
Martin Brennan
b58d168f05
FIX: Do not save default auto_delete_preference for bookmark (#18119)
We don't want to save the auto_delete_preference for bookmarks to the
user options if it was passed through as nil from the frontend,
this leads to confusion for the end user since they did not explicitly set it.

It's fine to create the bookmark with the default of "never" if no
auto_delete_preference is provided since it applies only to the
single bookmark, not future bookmarks.
2022-08-30 09:21:41 +10:00
Blake Erickson
eba8b8d34d
FEATURE: Replace Lounge with General Category (#18097)
- Seed the General category so that the general chat channel will have
  a home
- Do not seed the Lounge category anymore
- Move the "Welcome to Site" topic to the General category
2022-08-29 13:05:41 -06:00
David Taylor
e16c8ea2e7
DEV: Support inline-hbs compilation in themes (#18112)
This commit makes a number of improvements to the DiscourseJsProcessor:

1. Remove dependence on the out-of-date Ember template compiler from the ember-rails gem; switch to modern template compiler
2. Refactor to make use of a proper module system with `define`/`require`
3. Introduce `babel-plugin-ember-template-compilation` to enable inline hbs compilation

The `mini-loader` is upgraded to support relative lookup and `require.has`, so that these new JS packages work correctly.
2022-08-29 19:53:42 +01:00
Bianca Nenciu
33ea8b4756
FIX: Skip topic allowed user for small actions (#18075)
Topic allowed user records were created for small actions, which lead to
the system user being invited in many private topics when the user
removed themselves or if a group was invited but some members already
had access.

This commits skips creating topic allowed user. They are already skipped
for the whisper posts.
2022-08-29 13:01:16 +03:00
Arpit Jalan
c4bb15441d
FIX: reload styling changes for wizard styling step (#18121) 2022-08-29 15:10:47 +05:30
Alan Guo Xiang Tan
acb6222caf
Revert "DEV: Upgrade Redis to 4.8" (#18099)
This reverts commit 0b5a0fd857.

Pending release of Sidekiq 6.5.6 so that we don't get depreciation
warnings
2022-08-26 15:16:50 +10:00
Alan Guo Xiang Tan
0b5a0fd857 DEV: Upgrade Redis to 4.8 2022-08-25 11:11:08 +08:00
David Taylor
e7a84948b9
FIX: Avoid duplicate topic-list requests (#18073)
When preloading topic_list data we were giving it a 'preload key' which was loosely based on the parameters of the list. However, it did not include all parameters, and mismatches between client/server-side logic would cause the preloaded data to be ignored.

This commit simplifies things by using a single key for all topic_list preloading. This works on the assumption that "The first topic_list the JS app will load is the one which was preloaded". That assumption also existed to some extent in the old design, so we don't expect any regressions here.
2022-08-24 11:54:01 +01:00
Gerhard Schlager
9ff13cee14
FIX: Backup/Restore didn't use correct Redis namespace in multisite (#18060)
In a multisite Discourse reported that no backup is running after 60 seconds because the Redis key expired. Also, the thread that listens for a shutdown signal stopped running immediately because it didn't detect a running operation.
2022-08-24 01:43:42 +02:00
Sam
df04462475
FIX: ignore malformed HTML for title extraction (#18040)
Certain HTML can be rejected by nokogumbo, specifically cases where there
are enormous amounts of attributes

This ensures that malformed HTML is simply skipped instead of leaking out
an exception and terminating downstream processes.
2022-08-23 15:03:57 +10:00
Bianca Nenciu
ec8306835d
FIX: Improvements to like button for archived topics (#17951)
* FIX: Do not allow to remove like if topic is archived

* FIX: Always show like button

The like button used to be hidden if the topic was archived and it had
no likes. This commit changes that to always show the like button, but
with a not-allowed cursor if the topic is archived.
2022-08-22 14:58:02 +03:00
Jarek Radosz
aa59229c80
DEV: Fix busted twitter onebox spec (#18009)
Added in #17519. Was causing spec failures.
2022-08-20 11:19:19 +02:00
jbrw
73b2522261
FIX: Allow match_all_tags to be passed as a URL param (#17972)
`TopicQueryParams` allows for `match_all_tags` to be passed as a query parameter. `TagsController` forces the value to be true.

This change allows a value to be passed, and only sets it to true if no value has been set. It then uses `ActiveModel::Type::Boolean.new.cast` to compare the value.
2022-08-19 15:41:56 -04:00
Bianca Nenciu
707034bc75
FIX: Limit new and existent staged users for email topics (#17970)
The maximum_staged_users_per_email site setting controls how many
staged users will be invited to the topic created from an incoming
email. Previously, it counted only the new staged users.
2022-08-18 18:19:20 +03:00
Bianca Nenciu
b082f459c9
FEATURE: Limit maximum recipients for group emails (#17971)
New maximum_recipients_per_new_group_email site setting can be used to
prevent spam group emails with many recipients.
2022-08-18 18:18:58 +03:00
Bianca Nenciu
626d50c15c
FIX: Disable Twitter onebox without API support (#17519)
Twitter removed OpenGraph tags from their pages. We can no longer
extract all the information (for example, the quoted tweet) we need
to render Oneboxes without using their API.
2022-08-17 18:32:48 +03:00
Arpit Jalan
3a21618e4e
FEATURE: allow wizard checkbox field to be disabled (#17916)
* FEATURE: allow wizard checkbox field to be disabled

* Changes per review feedback
2022-08-15 05:52:07 +05:30
Bianca Nenciu
c789c689c2
FIX: Remove dead and large images from oneboxes (#17868)
Dead and large images are replaced with a placeholder, either a broken
chain icon or a short text. This commit no longer applies this
transformation for images inside Oneboxes, but removes them instead.
2022-08-11 19:09:48 +03:00
Loïc Guitaut
afe7785141 FIX: Swallow SSL errors when generating oneboxes 2022-08-09 16:54:45 +02:00
David Taylor
3c81683955 DEV: Rename UriHelper.escape_uri to .normalized_encode
This is a much better description of its function. It performs idempotent normalization of a URL. If consumers truly need to `encode` a URL (including double-encoding of existing encoded entities), they can use the existing `.encode` method.
2022-08-09 11:55:25 +01:00
Sam Saffron
f0a0252526 FIX: broken onebox images due to url normalization bugs
normalized_encode in addressable has a number of issues, including https://github.com/sporkmonger/addressable/issues/472

To temporaily work around those issues for the majority of cases, we try parsing with `::URI`. If that fails (e.g. due to non-ascii characters) then we will fall back to addressable.

Hopefully we can simplify this back to `Addressable::URI.normalized_encode` in the future.

This commit also adds support for unicode domain names and emoji domain names with escape_uri.

This removes an unneeded hack checking for pre-signed urls, which are now handled by the general case due to starting off valid and only being minimally normalized. Previous test case continues to pass.

UrlHelper.s3_presigned_url? which was somewhat wide was removed.
2022-08-09 11:55:25 +01:00
Loïc Guitaut
00b3f0e2c4 DEV: Make the first argument to the top-level describe a constant in specs 2022-08-08 18:07:49 +02:00
Loïc Guitaut
3eaac56797 DEV: Use proper wording for contexts in specs 2022-08-04 11:05:02 +02:00
Daniel Waterworth
83d3543e33
DEV: Guardians aren't active record objects, so shouldn't use fab! (#17789) 2022-08-03 19:17:40 -05:00
Martin Brennan
7cab189b1e
FIX: Handle actor not having preferences in UserCommScreener (#17790)
Followup to d66115d918

* Makes sure the `actor_preferences` all initialize with an empty array instead of nil if there are no preferences e.g. the actor is not ignoring anyone
* If the actor has disabled all PMs make `actor_disallowing_pms?` always return true
2022-08-04 10:16:54 +10:00
Sam
28968d9977
FIX: don't memoize site setting in guardian (#17788)
* FIX: don't memoize site setting in guardian

Memoizing site settings can make tests more fragile and harder to debug


Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2022-08-04 10:07:12 +10:00
Martin Brennan
d66115d918
DEV: UserCommScreener fine-grained actor improvements (#17737)
This commit introduces several fine-grained methods
to UserCommScreener which can be used to show the actor
who they are ignoring/muting/blocking DMs from in order
to prevent them initiating conversation with those users
or to display relevant information in the UI to the
actor.

This will be used in a companion PR in discourse-chat,
and is a follow up to 74584ff3ca

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Co-authored-by: Osama Sayegh <asooomaasoooma90@gmail.com>
2022-08-04 09:06:51 +10:00
Sam
ff53f2c7bc
FIX: scope memoization of category moderation per category (#17786) 2022-08-04 08:43:19 +10:00
Sam
3b42e69174
FIX: avoid usage of dig when looking for job class (#17772)
`{a: "a"}.dig(:a, :b)` will result in an exception, since ruby assumes that `"a"` will be another hash it can look up the `:b` key on.
2022-08-03 14:28:46 +10:00
Sam
bfe502012d
FEATURE: track stats around failing scheduled jobs (#17769)
* FEATURE: track stats around failing scheduled jobs

Discourse.job_exception_stats can now be used to gather stats around how
many regular scheduled jobs failed in the current process.

This will be consumed by the Prometheus plugin and potentially other
monitoring plugins.
2022-08-03 12:53:26 +10:00
Arpit Jalan
5cbf0255a6
FIX: do not overwrite top_menu site setting in wizard styling step (#17743) 2022-08-02 19:29:34 +05:30
Selase Krakani
862007fb18
FEATURE: Add support for case-sensitive Watched Words (#17445)
* FEATURE: Add case-sensitivity flag to watched_words

Currently, all watched words are matched case-insensitively. This flag
allows a watched word to be flagged for case-sensitive matching.
To allow allow for backwards compatibility the flag is set to false by
default.

* FEATURE: Support case-sensitive creation of Watched Words via API

Extend admin creation and upload of Watched Words to support case
sensitive flag. This lays the ground work for supporting
case-insensitive matching of Watched Words.

Support for an extra column has also been introduced for the Watched
Words upload CSV file. The new column structure is as follows:

 word,replacement,case_sentive

* FEATURE: Enable case-sensitive matching of Watched Words

WordWatcher's word_matcher_regexp now returns a list of regular
expressions instead of one case-insensitive regular expression.

With the ability to flag a Watched Word as case-sensitive, an action
can have words of both sensitivities.This makes the use of the global
Regexp::IGNORECASE flag added to all words problematic.

To get around platform limitations around the use of subexpression level
switches/flags, a list of regular expressions is returned instead, one for each
case sensitivity.

Word matching has also been updated to use this list of regular expressions
instead of one.

* FEATURE: Use case-sensitive regular expressions for Watched Words

Update Watched Words regular expressions matching and processing to handle
the extra metadata which comes along with the introduction of
case-sensitive Watched Words.

This allows case-sensitive Watched Words to matched as such.

* DEV: Simplify type casting of case-sensitive flag from uploads

Use builtin semantics instead of a custom method for converting
string case flags in uploaded Watched Words to boolean.

* UX: Add case-sensitivity details to Admin Watched Words UI

Update Watched Word form to include a toggle for case-sensitivity.
This also adds support for, case-sensitive testing and matching of  Watched Word
in the admin UI.

* DEV: Code improvements from review feedback

 - Extract watched word regex creation out to a utility function
 - Make JS array presence check more explicit and readable

* DEV: Extract Watched Word regex creation to utility function

Clean-up work from review feedback. Reduce code duplication.

* DEV: Rename word_matcher_regexp to word_matcher_regexp_list

Since a list is returned now instead of a single regular expression,
change `word_matcher_regexp` to `word_matcher_regexp_list` to better communicate
this change.

* DEV:  Incorporate WordWatcher updates from upstream

Resolve conflicts and ensure apply_to_text does not remove non-word characters in matches
that aren't at the beginning of the line.
2022-08-02 10:06:03 +02:00
Martin Brennan
f4b45df83f
FIX: UserCommScreener filter acting user ID from target user IDs (#17702)
Fixes edge case from fa5f3e228c.
In case the acting user is sent in with the target_user_ids,
we do not need to load those preferences, because even if the
acting user is preventing PMs or muting etc they need to always be able to
send themselves messages.
2022-07-28 13:04:24 +10:00
Phil Pirozhkov
493d437e79
Add RSpec 4 compatibility (#17652)
* Remove outdated option

04078317ba

* Use the non-globally exposed RSpec syntax

https://github.com/rspec/rspec-core/pull/2803

* Use the non-globally exposed RSpec syntax, cont

https://github.com/rspec/rspec-core/pull/2803

* Comply to strict predicate matchers

See:
 - https://github.com/rspec/rspec-expectations/pull/1195
 - https://github.com/rspec/rspec-expectations/pull/1196
 - https://github.com/rspec/rspec-expectations/pull/1277
2022-07-28 10:27:38 +08:00
Loïc Guitaut
296aad430a DEV: Use describe for methods in specs 2022-07-27 16:35:27 +02:00
Loïc Guitaut
c9d22b643f DEV: Fix flaky FinalDestination specs 2022-07-27 14:52:55 +02:00
Bianca Nenciu
bc476978e8
FIX: Support for group everyone in tag setting (#17669)
The "everyone" group is an automatic group and GroupUser records do not
exist for it. This commit allows all users if the group everyone is one
of the groups in the setting "pm_tags_allowed_for_groups".
2022-07-27 15:44:41 +03:00
Blake Erickson
8b08b9a763
FIX: Rejected emails should not be cleaned up before their logs (#17648)
* FIX: Rejected emails should not be cleaned up before their logs

If we delete the rejected emails before we delete their associated logs
we will receive 404 errors trying to inspect an email message for that
log.

* don't add a blank line

* test for max value as well

* pr cleanup and add migration

* Fix failing test
2022-07-27 07:28:44 +01:00
Arpit Jalan
10f200a5d3
FEATURE: revamped wizard (#17477)
* FEATURE: revamped wizard

* UX: Wizard redesign (#17381)

* UX: Step 1-2

* swap out images

* UX: Finalize all steps

* UX: mobile

* UX: Fix test

* more test

* DEV: remove unneeded wizard components

* DEV: fix wizard tests

* DEV: update rails tests for new wizard

* Remove empty hbs files that were created because of rebase

* Fixes for rebase

* Fix wizard image link

* More rebase fixes

* Fix rails tests

* FIX: Update preview for new color schemes: (#17481)

* UX: make layout more responsive, update images

* fix typo

* DEV: move discourse logo svg to template only component

* DEV: formatting improvements

* Remove unneeded files

* Add tests for privacy step

* Fix banner image height for step "ready"

Co-authored-by: Jordan Vidrine <30537603+jordanvidrine@users.noreply.github.com>
Co-authored-by: awesomerobot <kris.aubuchon@discourse.org>
2022-07-27 06:53:01 +05:30
Loïc Guitaut
91b6b5eee7 DEV: Don’t use change { … }.by(0) in specs 2022-07-26 10:34:15 +02:00
Jarek Radosz
59686c1cb8
DEV: Fix a flaky FinalDestination spec (#17655) 2022-07-26 01:28:53 +02:00
Loïc Guitaut
db9245d188
DEV: Fix FinalDestination specs (#17642) 2022-07-25 11:38:54 +01:00
Loïc Guitaut
1f5682b7d7 FIX: Don’t raise an error on onebox timeouts
Currently when generating oneboxes if the connection timeouts and we’re
using the `FinalDestination#get` method, then it raises an exception.

We already catch this exception when using the
`FinalDestination#resolve` method so this patch just applies the same
logic to `FinalDestination#get`.
2022-07-25 10:41:46 +02:00
Jarek Radosz
02a96a53fb
DEV: Be specific about time units (#17629)
All other tests that are setting grade_period use either unitless `0`, `1.minute` or `5.minutes` so it wasn't clear if `5` was meant to be seconds (it was)
2022-07-25 02:32:15 +02:00
Jarek Radosz
6cae26c499
DEV: Always await for publishToMessageBus (#17626)
(+ typos)
2022-07-24 19:55:58 +02:00
Osama Sayegh
306dca3a48
DEV: Add a helper method to clear every possible theme cache (#17585)
Our theme system is very complex and it can take a while to figure out how to invalidate the various types of caches that are used throughout the theme system. So, having a single helper method that invalidates everything can be useful in emergency situations where there is no time to read through the code and figure out how to clear the various caches.

Internal ticket: t64732.
2022-07-22 09:46:52 +03:00
Vinoth Kannan
eef58f510c
FEATURE: users with no posts shouldn't able to edit username after the allowed period. (#17583)
We are no longer going to let users change their username after the allowed `username_change_period` when there are no posts created by the user.
2022-07-21 04:46:15 +05:30
Bianca Nenciu
f75a99e932
FIX: Notify tag watchers when publishing topic (#17576)
When a topic was published from a shared draft and it had tags, the
users watching the tags were not notified. The problem was that the
topics are usually created in a secret category and publishing it just
moves an existent topic to the target category, without making any
changes to the tags.
2022-07-20 19:07:18 +03:00
Bianca Nenciu
e7f04a8674
FIX: Use URI#merge to merge base and relative URLs (#17454)
The old implementation did not handle all cases, such as the case when
`src` is a relative URL that starts with `..`.
2022-07-18 14:17:54 +03:00
David Taylor
fab1c00c8f
DEV: Drop the deprecated themeSettings.blah syntax (#17394)
This syntax has been printing deprecation messages since 880311dd4d
2022-07-18 10:10:23 +01:00
Bianca Nenciu
e8d802eb86
FIX: Check if tags edit notifications are disabled (#17499)
Tag edit notifications are either created by PostActionNotifier or
PostRevisor. PostActionNotifier already checks if the site setting is
enabled. but PostRevisor scheduled a NotifyTagChange job without
checking disable_tags_edit_notifications.
2022-07-15 16:14:46 +10:00
Martin Brennan
098ab29d41
FEATURE: Add plugin API to register About stat group (#17442)
This commit introduces a new plugin API to register
a group of stats that will be included in about.json
and also conditionally in the site about UI at /about.

The usage is like this:

```ruby
register_about_stat_group("chat_messages", show_in_ui: true) do
  {
    last_day: 1,
    "7_days" => 10,
    "30_days" => 100,
    count: 1000,
    previous_30_days: 120
  }
end
```

In reality the stats will be generated any way the implementer
chooses within the plugin. The `last_day`, `7_days`, `30_days,` and `count`
keys must be present but apart from that additional stats may be added.
Only those core 4 stat keys will be shown in the UI, but everything will be shown
in about.json.

The stat group name is used to prefix the stats in about.json like so:

```json
"chat_messages_last_day": 2322,
"chat_messages_7_days": 2322,
"chat_messages_30_days": 2322,
"chat_messages_count": 2322,
```

The `show_in_ui` option (default false) is used to determine whether the
group of stats is shown on the site About page in the Site Statistics
table. Some stats may be needed purely for reporting purposes and thus
do not need to be shown in the UI to admins/users. An extension to the Site
serializer, `displayed_about_plugin_stat_groups`, has been added so this
can be inspected on the client-side.
2022-07-15 13:16:00 +10:00
Martin Brennan
6b2ea1b47b
FIX: Change UserCommScreener to use user_ids (#17489)
It makes more sense to use user_ids for the UserCommScreener
introduced in fa5f3e228c since
in most cases the ID will be available, not the username. This
was discovered while starting work on a plugin that will
use this. In the cases where only usernames are available
the extra query is negligble.
2022-07-14 15:23:09 +10:00
Martin Brennan
fa5f3e228c
DEV: Refactor user mute/ignore/disallow PM checks into central class (#17366)
The idea behind this refactor is to centralise all of the user ignoring / muting / disallow PM checks in a single place, so they can be used consistently in core as well as for plugins like chat, while improving the main bulk of the checks to run in a single fast non-AR query.

Also fixed up the invite error when someone is muting/ignoring the user that is trying to invite them to the topic.
2022-07-13 13:58:01 +10:00
Jarek Radosz
87353faac6
DEV: Implement distributed mutex in lua (#16228)
The rationale behind this was mostly to stop using `redis.synchronize` (now removed in redis gem 4.6)
2022-07-11 14:16:37 +02:00
Jordan Vidrine
fec3df60cb
DEV: Add more color schemes (#17398) 2022-07-08 16:46:32 -05:00
David Taylor
8e0b1ee74d
DEV: Future-proof ThemeJavascriptCompiler spec (#17393)
Previously the spec was hardcoded to expect a certain compiled result. The precise result will change as we update to more recent Ember versions.

Instead, we can compile via the special theme compiler, and then compile our expected result via the normal compiler. If they match, then things are working as intended. This technique should be safe across template-compiler upgrades.
2022-07-08 16:28:46 +01:00
Rafael dos Santos Silva
75e40baa64
FIX: min/max username length limits weren't validated (#17382)
* FIX: min/max username length limits weren't validated

The custom validators introduced in e0d7cda made so we ignored the mix
and max values set on site_settings.yml. That change allowed admins to
set values outside of the range defined on the yaml file.

Related to https://meta.discourse.org/t/group-names-with-more-than-60-characters-broken/232115?u=falco

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2022-07-08 12:00:47 -03:00
Gerhard Schlager
8bdbefe0e0
FIX: Logout could fail due to cached user (#17325)
Logging out failed when the current user was cached by an instance of `Auth::DefaultCurrentUserProvider` and `#log_off_user` was called on a different instance of that class.

Co-authored-by: Sam <sam.saffron@gmail.com>
2022-07-04 17:01:19 +02:00
Jarek Radosz
c5f0aa2f32
DEV: Remove handlebars from the old package.json (#17319) 2022-07-04 15:05:03 +02:00
Gerhard Schlager
caa0247f5c
FIX: Incorrect currentUser could be cached for requests with API key (#17279)
This happened when a middleware accessed the `currentUser` before a controller had a chance to populate the `action_dispatch.request.path_parameters` env variable. In that case Discourse would always cache `nil` as `currentUser`.
2022-07-01 10:18:24 +02:00
Krzysztof Kotlarek
09932738e5
FEATURE: whispers available for groups (#17170)
Before, whispers were only available for staff members.

Config has been changed to allow to configure privileged groups with access to whispers. Post migration was added to move from the old setting into the new one.

I considered having a boolean column `whisperer` on user model similar to `admin/moderator` for performance reason. Finally, I decided to keep looking for groups as queries are only done for current user and didn't notice any N+1 queries.
2022-06-30 10:18:12 +10:00
Sam
6ecfdc8f55
FIX: keep unique post checks separate for PMs vs topics (#17272)
This allows for people to use PMs for drafting and then post them on topics.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2022-06-29 15:35:07 +10:00
Gerhard Schlager
9d870f151c
FIX: Uploading large files (> 5GB) failed when enable_direct_s3_uploads is enabled (#16724)
Larger files require a multipart copy.
2022-06-28 21:30:00 +02:00
Gerhard Schlager
0bcc478635 DEV: Run some specs with fake S3 implementation instead of stubs 2022-06-28 21:27:52 +02:00
Sérgio Saquetim
5840fb5c62
UX: Update topics stats automatically (#17135)
Updates automatically data on the stats section of the topic.

It will update automatically the following information: likes, replies and last reply (timestamp and user)
2022-06-27 18:21:05 -03:00
Rafael dos Santos Silva
f56c44d1c7
FEATURE: Validate tags in WatchedWords (#17254)
* FEATURE: Validate tags in WatchedWords

We didn't validate watched words automatic tagging, so it was possible
for an admin to created watched words with an empty tag list which would
result in an exception when users tried to create a new topic that
matched the misconfigured watched word.

Bug report: https://meta.discourse.org/t/lib-topic-creator-fails-when-the-word-math-appears-in-the-topic-title-or-text/231018?u=falco
2022-06-27 16:16:33 -03:00
Penar Musaraj
3baefa25b5
FIX: Use first supported type item when JSON-LD returns array (#17217) 2022-06-23 13:02:01 -04:00
Joe
e82a2ce9ae
UX: Introduces a splash screen behind a hidden site setting (#17094)
This PR introduces a new hidden site setting that allows admins to display a splash screen while site assets load.

The splash screen can be enabled via the `splash_screen` hidden site setting.

This is what the splash screen currently looks like

5ceb72f085.mp4

Once site assets load, the splash screen is automatically removed.

To control the loading text that shows in the splash screen, you can change the preloader_text translation string in admin > customize > text
2022-06-22 04:35:46 +08:00
Michael Brown
beda4708f6 FIX: 99999 is not a valid TCP port
Future versions of redis will validate this port number causing the tests
relying on this to fail with:

```
Redis::CommandError:
ERR Invalid master port
```

Also change from an IPv4 address that might feasibly be in use to an IPv6
random ULA address that almost *certainly* won't be.
2022-06-21 14:55:19 -04:00
Roman Rizzi
e0ba35350e
FEATURE: Custom unsubscribe options (#17090)
With this change, plugins can create custom unsubscribe keys, extend the unsubscribe view with custom preferences, and decide how they are updated.
2022-06-21 15:49:47 -03:00
Alan Guo Xiang Tan
3562c598ef
DEV: Upgrade Sidekiq to 6.5 (#17142) 2022-06-21 09:23:36 +08:00
Osama Sayegh
5176c689e9
UX: Change wording for 'regular' categories to 'normal' (#17134)
At some point in the past we decided to rename the 'regular' notification state of topics/categories to 'normal'. However, some UI copy was missed when the initial renaming was done so this commit changes the spots that were missed to the new name.
2022-06-20 06:49:33 +03:00
Jarek Radosz
fddd6fd5e0
DEV: Fix an rspec warning (#17123)
```
WARNING: Using `expect { }.not_to raise_error(SpecificErrorClass)` risks false positives, since literally any other error would cause the expectation to pass, including those raised by Ruby (e.g. `NoMethodError`, `NameError` and `ArgumentError`), meaning the code you are intending to test may not even get reached. Instead consider using `expect { }.not_to raise_error` or `expect { }.to raise_error(DifferentSpecificErrorClass)`. This message can be suppressed by setting: `RSpec::Expectations.configuration.on_potential_false_positives = :nothing`. Called from /var/www/discourse/spec/lib/retrieve_title_spec.rb:155:in `block (3 levels) in <main>'.
```
2022-06-17 13:45:16 +02:00
David Taylor
e0ff367b49
DEV: Stop logging error response body in FileHelper (#17120)
This doesn't cope well with gzipped, binary, or large responses. Ideally we would teach FinalDestination to safely retrieve and decode some of the response body. But for now, let's remove the broken implementation.
2022-06-17 11:50:30 +01:00
Alan Guo Xiang Tan
f618fdf17f
Revert "DEV: Centralize user updates to a single MessageBus channel. (#17058)" (#17115)
This reverts commit 94c3bbc2d1.

At this current point in time, we do not have enough data on whether
this centralisation is the trade-offs of coupling features into a single
channel.
2022-06-17 12:24:15 +08:00
Jarek Radosz
f723b4c322
FIX: Handle sites with more than 1 JSON-LD element (#17095)
A followup to #17007
2022-06-15 02:55:55 +02:00
Martin Brennan
4d3c1ceb44
FEATURE: Log the SMTP response in EmailLog (#17056)
When sending emails with delivery_method_options -> return_response
set to true, the SMTP sending code inside Mail will return the SMTP
response when calling deliver! for mail within the app. This commit
ensures that Email::Sender captures this response if it is returned
and stores it against the EmailLog created for the sent email.

A follow up PR will make this visible within the admin email UI.
2022-06-15 10:28:30 +10:00
sansnumero
f0c6dd5682
Add support for JSON LD in Onebox (#17007)
* FIX: Fix a bug that is accessing the values in a hash wrongly and write tests

I decided to write tests in order to be confident in my refactor that's in the next commit.
Meanwhile I have discovered a potential bug. The `title_attr` key was accessed as a string,
but all the keys are actually symbols so it was never evaluated to be true.

irb(main):025:0> d = {key: 'value'}
=> {:key=>"value"}
irb(main):026:0> d['key']
=> nil
irb(main):027:0> d[:key]
=> "value"

* DEV: Extract methods for readability

I will be adding a new method following the conventions in place for adding a new normalizer. And this will make the readability of the `raw` block even more difficult; so I am extracting self contained private methods beforehand.

* FEATURE: Parse JSON-LD and introduce Movie object

JSON LD data is very easily transferable to Ruby objects because they contain types. If these types are mapped to Ruby objects, it is also better to make all the parsed data very explicit and easily extendable.

JSON-LD has many more standardized item types, with a full list here: https://schema.org/docs/full.html
However in order to decrease the scope, I only adapted the movie type.

* DEV: Change inheritance between normalizers

Normalizers are not supposed to have an inheritance relationships amongst each other. They are all normalizers, but all normalizing separate protocols. This is why I chose to extract a parent class and relieve Open Graph off that responsibility. Removing the parent class altogether could also a possibility, but I am keeping the scope limited to having a more accurate representation of the normalizers while making it easier to add a new one.

* Lint changes

* Bring back the Oembed OpenGraph inheritance

There is one test that caught that this inheritance was necessary. I still think modelling wise this inheritance shouldn't exist, but this can be tackled separately.

* Return empty hash if the json received is invalid

Before this change if there was a parsing error with JSON it would throw an exception. The goal of this commit is to rescue that exception and then log a warning. I chose to use Discourse's logger wrapper `warn_exception` to have the backtrace and not just used Rails logger. I considered raising an `InvalidParameters` error however if the JSON here is invalid it should not block showing of the Onebox, so logging is enough.

* Prep to support more JSONLD schema types with case

* Extract mustache template object created from JSONLD
2022-06-13 17:32:34 +02:00
Alan Guo Xiang Tan
94c3bbc2d1
DEV: Centralize user updates to a single MessageBus channel. (#17058)
Introduces an interface to publish user updates on the server side and
helps to reduce the growing number of subscriptions on the client side.
2022-06-13 14:27:43 +08:00
Sérgio Saquetim
300f835703
DEV: Supress logs when RetrieveTitle.crawl fails with Net::ReadTimeout errors (#16971)
This PR changes the rescue block to rescue only Net::TimeoutError exceptions and removes the log line to prevent clutter the logs with errors that are ignored. Other errors can bubble up because they're errors we probably want to know about
2022-06-09 16:30:22 -03:00
Bianca Nenciu
9db8f00b3d
FEATURE: Create upload_references table (#16146)
This table holds associations between uploads and other models. This can be used to prevent removing uploads that are still in use.

* DEV: Create upload_references
* DEV: Use UploadReference instead of PostUpload
* DEV: Use UploadReference for SiteSetting
* DEV: Use UploadReference for Badge
* DEV: Use UploadReference for Category
* DEV: Use UploadReference for CustomEmoji
* DEV: Use UploadReference for Group
* DEV: Use UploadReference for ThemeField
* DEV: Use UploadReference for ThemeSetting
* DEV: Use UploadReference for User
* DEV: Use UploadReference for UserAvatar
* DEV: Use UploadReference for UserExport
* DEV: Use UploadReference for UserProfile
* DEV: Add method to extract uploads from raw text
* DEV: Use UploadReference for Draft
* DEV: Use UploadReference for ReviewableQueuedPost
* DEV: Use UploadReference for UserProfile's bio_raw
* DEV: Do not copy user uploads to upload references
* DEV: Copy post uploads again after deploy
* DEV: Use created_at and updated_at from uploads table
* FIX: Check if upload site setting is empty
* DEV: Copy user uploads to upload references
* DEV: Make upload extraction less strict
2022-06-09 09:24:30 +10:00
Loïc Guitaut
fff3ff11c0 FIX: Make disabling TLS in mail possible again
Following the Rails 7 upgrade, the `DISCOURSE_SMTP_ENABLE_START_TLS`
setting doesn’t work anymore. This is because Rails upgraded the
`net-smtp` gem to the 0.3.1 version which enables `starttls` by default.
The `mail` gem doesn’t support this new behavior yet and doesn’t know
how to disable TLS. This should be fixed in an upcoming release.

Meanwhile applying this patch allows us to get back the previous
behavior which is expected by many.
2022-06-08 14:04:05 +02:00
Mayfield
99b0578b4c
FIX: escape youtube title when constructing onebox preview html (#16999) 2022-06-08 13:42:37 +08:00
Alan Guo Xiang Tan
82ac698d4f
FIX: Missing tracked sub category topics from tracked topic list (#17034)
Follow-up to 7ae647d092
2022-06-08 10:45:59 +08:00
Jarek Radosz
1a5dbbf430
FIX: Correctly handle invalid auth cookies (#16995)
Previously it would blow up on invalid utf byte sequences. This was a source of spec flakiness.
2022-06-07 13:00:25 +02:00
Alan Guo Xiang Tan
7da074d591
DEV: Implement "My Posts" section link for experimental sidebar (#17008) 2022-06-07 10:52:54 +08:00
Blake Erickson
35e17ce115
FIX: Email Send post has already been taken error (#16992)
* FIX: Email Send post has already been taken error

Adding a failing test first before coming up with a good solution.

Related: 357011eb3b

The above commit changed

```
PostReplyKey.find_or_create_by_safe!
```

to

```
PostReplyKey.create_or_find_by!
```

But I don't think it is working as a 1-1 replacement because of the
`Validation failed: Post has already been taken` error we are receiving
with this change. Also we need to make sure we don't re-introduce any
concurrency issues.

Reported: https://meta.discourse.org/t/224706/13

* Remove rails unique constraint and rely on db index

I believe this is what is causing `create_or_find_by!` to fail. Because
we have a unique constraint in the db I think we can remove this rails
unique constraint?

* clean up spec wording
2022-06-06 13:13:26 -06:00
Martin Brennan
f94682e2c4
FIX: Do not use SVGs for twitter:image metadata (#16973)
Twitter does not allow SVGs to be used for twitter:image
metadata (see https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/markup)
so we should fall back to the site logo if the image option
provided to `crawlable_meta_data` or SiteSetting.site_twitter_summary_large_image_url
is an SVG, and do not add the meta tag for twitter:image at all
if the site logo is an SVG.
2022-06-03 09:02:57 +10:00
Alan Guo Xiang Tan
7ae647d092
FIX: tracked filter did not account for max_category_nesting of 3 (#16963) 2022-06-01 12:09:58 +08:00
Martin Brennan
61b9e3ee30
FIX: InlineOneboxer watched word censor error (#16921)
In 7328a2bfb0 we changed the
InlineOneboxer#onebox_for method to run the title of the
onebox through WatchedWord#censor_text. However, it is
allowable for the title to be nil, which was causing this
error in production:

> NoMethodError : undefined method gsub for nil:NilClass

We just need to check whether the title is nil before trying
to censor it.
2022-05-26 14:01:44 +10:00
Martin Brennan
641c4e0b7a
FEATURE: Make S3 presigned GET URL expiry configurable (#16912)
Previously we hardcoded the DOWNLOAD_URL_EXPIRES_AFTER_SECONDS const
inside S3Helper to be 5 minutes (300 seconds). For various reasons,
some hosted sites may need this to be longer for other integrations.

The maximum expiry time for presigned URLs is 1 week (which is
604800 seconds), so that has been added as a validation on the
setting as well. The setting is hidden because 99% of the time
it should not be changed.
2022-05-26 09:53:01 +10:00
Bianca Nenciu
7328a2bfb0
FIX: Apply censored words to inline onebox (#16873)
Censored watched words were not censored inside the title of an inline
oneboxes. Malicious users could exploit this behaviour to insert bad
words. The same issue has been fixed for regular Oneboxes in commit
d184fe59ca.
2022-05-25 14:51:47 +03:00
Penar Musaraj
8222810099
FIX: Limits for PM and group header search (#16887)
When searching for PMs or PMs in a group inbox, results in the header search were not being limited to 5 with a "More" link to the full page search. This PR fixes that.

It also simplifies the logic and updates the search API docs to include recently added `in:messages` and `group_messages:groupname` options.
2022-05-24 11:31:24 -04:00
Loïc Guitaut
0403a8633b DEV: Apply Rails 6.1 defaults
We never applied `config.load_defaults` since its inception (Rails 5.0)
and doing so is necessary to properly upgrade to all the Rails 7 new
defaults.
2022-05-24 17:13:44 +02:00
Martin Brennan
1ee973e6e2
DEV: Add save_user_preferences option to BookmarkManager (#16894)
When saving / creating bookmarks, we have code to save
the user's preference of bookmark_auto_delete_preference
to their user_options.

Unfortunately this can cause weirdness when plugins
have code using BookmarkManager to set the auto delete preference for
only a specific bookmark.

This commit introduces a save_user_preferences option (false
by default) so that this user preference is not saved unless
specified by the consumer of BookmarkManager, so plugins will
not have to worry about it.
2022-05-24 11:13:21 +10:00
Bianca Nenciu
6c8f491dc3
DEV: Allow plugins to register Onebox handlers (#16870)
This targets only the local Oneboxes and allows plugins to customize
regular or inline Oneboxes for routes inside the site.
2022-05-23 20:02:02 +03:00
David Taylor
bf6f8299a7 FEATURE: Pull hotlinked images immediately after posting
Previously, with the default `editing_grace_period`, hotlinked images were pulled 5 minutes after a post is created. This delay was added to reduce the chance of automated edits clashing with user edits.

This commit refactors things so that we can pull hotlinked images immediately. URLs are immediately updated in the post's `cooked` HTML. The post's raw markdown is updated later, after the `editing_grace_period`.

This involves a number of behind-the-scenes changes including:

- Schedule Jobs::PullHotlinkedImages immediately after Jobs::ProcessPost. Move scheduling to after the `update_column` call to avoid race conditions

- Move raw changes into a separate job, which is delayed until after the ninja-edit window

- Move disable_if_low_on_disk_space logic into the `pull_hotlinked_images` job

- Move raw-parsing/replacing logic into `InlineUpload` so it can be easily be shared between `UpdateHotlinkedRaw` and `PullUserProfileHotlinkedImages`
2022-05-23 14:28:02 +01:00
Osama Sayegh
d15867463f
FEATURE: Site setting for blocking onebox of URLs that redirect (#16881)
Meta topic: https://meta.discourse.org/t/prevent-to-linkify-when-there-is-a-redirect/226964/2?u=osama.

This commit adds a new site setting `block_onebox_on_redirect` (default off) for blocking oneboxes (full and inline) of URLs that redirect. Note that an initial http → https redirect is still allowed if the redirect location is identical to the source (minus the scheme of course). For example, if a user includes a link to `http://example.com/page` and the link resolves to `https://example.com/page`, then the link will onebox (assuming it can be oneboxed) even if the setting is enabled. The reason for this is a user may type out a URL (i.e. the URL is short and memorizable) with http and since a lot of sites support TLS with http traffic automatically redirected to https, so we should still allow the URL to onebox.
2022-05-23 13:52:06 +03:00
Martin Brennan
fcc2e7ebbf
FEATURE: Promote polymorphic bookmarks to default and migrate (#16729)
This commit migrates all bookmarks to be polymorphic (using the
bookmarkable_id and bookmarkable_type) columns. It also deletes
all the old code guarded behind the use_polymorphic_bookmarks setting
and changes that setting to true for all sites and by default for
the sake of plugins.

No data is deleted in the migrations, the old post_id and for_topic
columns for bookmarks will be dropped later on.
2022-05-23 10:07:15 +10:00
Johannes Faigle
bf987af3ca
FEATURE Add reset bump dates bulk action (#16885) 2022-05-22 12:32:55 -04:00
David Taylor
166fe3bb34
FIX: Apply 'allowed_href_schemes' to all src/srcset attributes (#16860)
Previously we were only applying the restriction to `a[href]` and `img[src]`. This commit ensures we apply the same logic to all allowlisted media src attributes.
2022-05-19 11:18:30 +01:00
Alan Guo Xiang Tan
de9fe907ee
DEV: Readonly Redis support for DiscourseRedis#multi/pipelined (#16744)
Follow-up to 2df3c65ba9
2022-05-13 16:18:13 +08:00
Isaac Janzen
20740f196c
FIX: handle quote rendering for external Discourse instance (#16722)
Gracefully handle quotes from an external discourse instance by stripping quote-controls and including username in the title
2022-05-12 10:07:43 -05:00
Martin Brennan
8e9164fb60
DEV: Minor bookmark tweaks for polymorphism (#16728)
* Make the modal for bookmarks display more consistently
* Make sure bookmark query can handle empty results for certain
  bookmarkable queries
2022-05-12 10:29:01 +10:00
David Taylor
476bd1d237
DEV: Fix production sourcemaps with Ember CLI (#16707)
22a7905f restructured how we load Ember CLI assets in production. Unfortunately, it also broke sourcemaps for those assets. This commit fixes that regression via a couple of changes:

- It adds the necessary `.map` paths to `config.assets.precompile`
- It swaps Sprockets' default `SourcemappingUrlProcessor` with an extended version which maintains relative URLs of maps
2022-05-11 10:23:32 +01:00
Bianca Nenciu
4760cf604c
FEATURE: Allow category moderators to post consecutively (#16706)
First posters and staff are already allowed to have unlimited
consecutive posts. This adds the same capabilities to category
moderators.
2022-05-10 22:18:17 +03:00
Isaac Janzen
1a12e4cfc8
FEATURE: Introduce site setting to allow for non staff pm tagging (#16671)
Currently the only way to allow tagging on pms is to use the `allow_staff_to_tag_pms` site setting.  We are removing that site setting and replacing it with `pm_tags_allowed_for_groups` which will allow for non staff tagging. It will be group based permissions instead of requiring the user to be staff.

If the existing value of `allow_staff_to_tag_pms` is `true` then we include the `staff` groups as a default for `pm_tags_allowed_for_groups`.
2022-05-10 10:02:28 -05:00
Martin Brennan
fbcc35b417
DEV: Remove PostAction/UserAction bookmark refs (#16681)
We have not used anything related to bookmarks for PostAction
or UserAction records since 2020, bookmarks are their own thing
now. Deleting all this is just cleaning up old cruft.
2022-05-10 10:42:18 +10:00
Martin Brennan
955d47bbd0
FIX: Use polymorphic bookmarks for in:bookmarks search (#16684)
This commit makes sure the in:bookmarks post advanced
search filter works with polymorphic bookmarks.
2022-05-10 09:08:01 +10:00
Sam
2df3c65ba9
FIX: add support for pipelined and multi redis commands (#16682)
Latest redis interoduces a block form of multi / pipelined, this was incorrectly
passed through and not namespaced.

Fix also updates logster, we held off on upgrading it due to missing functions
2022-05-10 08:19:02 +10:00
Rafael dos Santos Silva
919f71537e
FIX: Background like count update didn't account for own user actions (#16688)
This fixes a corner case of the perf optimization in d4e35f5.

When you have the the same post showing in multiple tab/devices and like
said post in one place, we updated the like count but didn't flip the
`acted` bool in the front-end. This caused a small visual desync.

Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
2022-05-09 17:23:39 -03:00
Martin Brennan
222c8d9b6a
FEATURE: Polymorphic bookmarks pt. 3 (reminders, imports, exports, refactors) (#16591)
A bit of a mixed bag, this addresses several edge areas of bookmarks and makes them compatible with polymorphic bookmarks (hidden behind the `use_polymorphic_bookmarks` site setting). The main ones are:

* ExportUserArchive compatibility
* SyncTopicUserBookmarked job compatibility
* Sending different notifications for the bookmark reminders based on the bookmarkable type
* Import scripts compatibility
* BookmarkReminderNotificationHandler compatibility

This PR also refactors the `register_bookmarkable` API so it accepts a class descended from a `BaseBookmarkable` class instead. This was done because we kept having to add more and more lambdas/properties inline and it was very messy, so a factory pattern is cleaner. The classes can be tested independently as well.

Some later PRs will address some other areas like the discourse narrative bot, advanced search, reports, and the .ics endpoint for bookmarks.
2022-05-09 09:37:23 +10:00
Rafael dos Santos Silva
94cfe98ee4
FEATURE: Validate setting combination between exif strip and img opt (#16662)
Admins won't be able to disable strip_image_metadata if they don't
disable composer_media_optimization_image_enabled first since the later
will strip the same metadata on client during upload, making disabling
the former have no effect.

Bug report at https://meta.discourse.org/t/-/223350
2022-05-05 15:13:17 -03:00
Jarek Radosz
3f0e767106
DEV: Use FakeLogger in RequestTracker specs (#16640)
`TestLogger` was responsible for some flaky specs runs:

```
Error during failsafe response: undefined method `debug' for #<TestLogger:0x0000556c4b942cf0 @warnings=1>
Did you mean?  debugger
```

This commit also cleans up other uses of `FakeLogger`
2022-05-05 09:53:54 +08:00
Jarek Radosz
fb1a3a1dbb
DEV: Drop TrackingLogger for FakeLogger (#16642) 2022-05-05 09:50:43 +08:00
Gerhard Schlager
28e8ae553d
FIX: Missing translation when translation override contained a %{key} (#16625)
This happened only for languages other than "en" and when `I18n.t` was called without any interpolation keys. The lib still tried to interpolate keys because it interpreted the `overrides` option as interpolation key.
2022-05-04 17:35:22 +02:00
Bianca Nenciu
8695449cfc
FIX: Validate permalink_normalizations setting (#16604)
When an admin enters a badly formed regular expression in the
permalink_normalizations site setting, a RegexpError exception is
generated everytime a URL is normalized (see Permalink.normalize_url).

The new validator validates every regular expression present in the
setting value (delimited by '|').
2022-05-04 14:33:06 +03:00
David Taylor
485fc4636a
FEATURE: Add fallback to suggested value when auth_overrides_username (#16612)
If the identity provider does not provide a precise username value, then we should use our UserNameSuggester to generate one and use it for the override. This makes the override consistent with initial account creation.
2022-05-04 11:22:09 +01:00
David Taylor
c1db968740
DEV: Move hotlinked image information into a dedicated table (#16585)
This will make future changes to the 'pull hotlinked images' system easier. This commit should not introduce any functional change.

For now, the old post_custom_field data is kept in the database. This will be dropped in a future commit.
2022-05-03 13:53:32 +01:00
Penar Musaraj
b266a36967
FEATURE: Add group_messages: keyword to advanced search (#16584) 2022-04-28 10:47:40 -04:00
Loïc Guitaut
008b700a3f DEV: Upgrade to Rails 7
This patch upgrades Rails to version 7.0.2.4.
2022-04-28 11:51:03 +02:00
Penar Musaraj
eebce8f80a
FEATURE: Add in:messages search modifier (#16567)
This adds `in:messages` as a synonym for `in:personal` and sets it up as our default nomenclature (`in:personal` will still work).
2022-04-26 16:47:01 -04:00
Roman Rizzi
068e93534c
FIX: Check 2FA is disabled before enabling DiscourseConnect. (#16542)
Both settings are incompatible. We validated that DiscourseConnect is disabled before enabling 2FA but were missing the other way around.
2022-04-25 14:49:36 -03:00
David Taylor
922fbe82da
DEV: Ensure custom_fields_clean? returns false when values change (#16552)
We were calling `dup` on the hash and using that to check for changes. However, we were not duplicating the values, so changes to arrays or nested hashes would not be detected.
2022-04-25 17:19:39 +01:00
Gerhard Schlager
3d62e335f7
FEATURE: Detect emoji from Emoji 14.0 (#16531) 2022-04-22 08:42:15 +02:00
Martin Brennan
3e4621c2cb
FEATURE: Polymorphic bookmarks pt. 2 (lists, search) (#16335)
This pull request follows on from https://github.com/discourse/discourse/pull/16308. This one does the following:

* Changes `BookmarkQuery` to allow for querying more than just Post and Topic bookmarkables
* Introduces a `Bookmark.register_bookmarkable` method which requires a model, serializer, fields and preload includes for searching. These registered `Bookmarkable` types are then used when validating new bookmarks, and also when determining which serializer to use for the bookmark list. The `Post` and `Topic` bookmarkables are registered by default.
* Adds new specific types for Post and Topic bookmark serializers along with preloading of associations in `UserBookmarkList`
* Changes to the user bookmark list template to allow for more generic bookmarkable types alongside the Post and Topic ones which need to display in a particular way

All of these changes are gated behind the `use_polymorphic_bookmarks` site setting, apart from the .hbs changes where I have updated the original `UserBookmarkSerializer` with some stub methods.

Following this PR will be several plugin PRs (for assign, chat, encrypt) that will register their own bookmarkable types or otherwise alter the bookmark serializers in their own way, also gated behind `use_polymorphic_bookmarks`.

This commit also removes `BookmarkQuery.preloaded_custom_fields` and the functionality surrounding it. It was added in 0cd502a558 but only used by one plugin (discourse-assign) where it has since been removed, and is now used by no plugins. We don't need it anymore.
2022-04-22 08:23:42 +10:00
David Taylor
1551eaab01
FIX: Do not error when json-serialized cookies are used (#16522)
We intend to switch to the `:json` serializer, which will stringify all keys. However, we need a clean revert path. This commit ensures that our `_t` cookie handling works with both marshal (the current default) and json (the new default) serialization.
2022-04-20 19:15:40 +01:00
Isaac Janzen
692e0140e2
FEATURE: Enables support for dark mode emails (#16520)
This PR enables custom email dark mode styles by default that were added here.

There is currently poor support for dark mode queries in mail clients. The main beneficiary of these changes will be Apple Mail and Outlook.

Enjoy the darkness 🕶️
2022-04-20 13:00:04 -05:00
Isaac Janzen
196b791365
DEV: Prioritize full name when display_name_on_posts active (#16078)
See: https://meta.discourse.org/t/display-full-name-not-username-when-attributing-quote-or-reply/203533?u=isaacjanzen for context

The initial release [broke quoting](https://meta.discourse.org/t/quoting-broken-when-name-matches-username/217633?u=isaacjanzen) but we now pass the username when 
```
siteSettings.display_name_on_posts && !siteSettings.prioritize_username_in_ux && post.name
```
as well as the full name to guarantee that we are not getting any mismatches when querying for user / avatar.

eg. 
```
[quote="Isaac Janzen, post:3, topic:7, full:true, username:isaac.janzen"]
bing bong
[/quote]
```
2022-04-20 10:07:51 -05:00
Alan Guo Xiang Tan
935838adb6 DEV: Upgrade Sidekiq to 6.4.2 2022-04-20 16:26:38 +08:00
Martin Brennan
154afa60eb
FIX: Skip upload extension validation when changing security (#16498)
When changing upload security using `Upload#update_secure_status`,
we may not have the context of how an upload is being created, because
this code path can be run through scheduled jobs. When calling
update_secure_status, the normal ActiveRecord validations are run,
and ours include validating extensions. In some cases the upload
is created in an automated way, such as user export zips, and the
security is applied later, with the extension prohibited from
use when normally uploading.

This caused the upload to fail validation on `update_secure_status`,
causing the security change to silently fail. This fixes the issue
by skipping the file extension validation when the upload security
is being changed.
2022-04-20 14:11:39 +10:00
Martin Brennan
c6c633e041
FIX: Issues with incorrect unread and private message topic tracking state (#16474)
This commit fixes two issues at play. The first was introduced
in f6c852b (or maybe not introduced
but rather revealed). When a user posted a new message in a topic,
they received the unread topic tracking state MessageBus message,
and the Unread (X) indicator was incremented by one, because with the
aforementioned perf commit we "guess" the correct last read post
for the user, because we no longer calculate individual users' read
status there. This meant that every time a user posted in a topic
they tracked, the unread indicator was incremented. To get around
this, we can just exclude the user who created the post from the
target users of the unread state message.

The second issue was related to the private message topic tracking
state, and was somewhat similar. Whenever a user created a new private
message, the New (X) indicator was incremented, and could not be
cleared until the page was refreshed. To solve this, we just don't
update the topic state for the user when the new_topic tracking state
message comes through if the user who created the topic is the
same as the current user.

cf. https://meta.discourse.org/t/bottom-of-topic-shows-there-is-1-unread-remaining-when-there-are-actually-0-unread-topics-remaining/220817
2022-04-19 11:37:01 +10:00
David Taylor
8f786268be
SECURITY: Ensure user-agent-based responses are cached separately (#16475) 2022-04-14 14:25:52 +01:00
Osama Sayegh
eb5a3cfded
FEATURE: Add 2FA support to the Discourse Connect Provider protocol (#16386)
Discourse has the Discourse Connect Provider protocol that makes it possible to
use a Discourse instance as an identity provider for external sites. As a
natural extension to this protocol, this PR adds a new feature that makes it
possible to use Discourse as a 2FA provider as well as an identity provider.

The rationale for this change is that it's very difficult to implement 2FA
support in a website and if you have multiple websites that need to have 2FA,
it's unrealistic to build and maintain a separate 2FA implementation for each
one. But with this change, you can piggyback on Discourse to take care of all
the 2FA details for you for as many sites as you wish.

To use Discourse as a 2FA provider, you'll need to follow this guide:
https://meta.discourse.org/t/-/32974. It walks you through what you need to
implement on your end/site and how to configure your Discourse instance. Once
you're done, there is only one additional thing you need to do which is to
include `require_2fa=true` in the payload that you send to Discourse.

When Discourse sees `require_2fa=true`, it'll prompt the user to confirm their
2FA using whatever methods they've enabled (TOTP or security keys), and once
they confirm they'll be redirected back to the return URL you've configured and
the payload will contain `confirmed_2fa=true`. If the user has no 2FA methods
enabled however, the payload will not contain `confirmed_2fa`, but it will
contain `no_2fa_methods=true`.

You'll need to be careful to re-run all the security checks and ensure the user
can still access the resource on your site after they return from Discourse.
This is very important because there's nothing that guarantees the user that
will come back from Discourse after they confirm 2FA is the same user that
you've redirected to Discourse.

Internal ticket: t62183.
2022-04-13 15:04:09 +03:00
Isaac Janzen
6c0abe15e0
FEATURE: Add email dark mode (#16104)
implement dark mode emails when `SiteSetting.dark_mode_emails_active` is active.
2022-04-11 12:27:50 -05:00
Sam
cedcdb0057
FEATURE: allow for local theme js assets (#16374)
Due to default CSP web workers instantiated from CDN based assets are still
treated as "same-origin" meaning that we had no way of safely instansiating
a web worker from a theme.

This limits the theme system and adds the arbitrary restriction that WASM
based components can not be safely used.

To resolve this limitation all js assets in about.json are also cached on
local domain.

{
  "name": "Header Icons",
  "assets" : {
    "worker" : "assets/worker.js"
  }
}

This can then be referenced in JS via:

settings.theme_uploads_local.worker

local_js_assets are unconditionally served from the site directly and
bypass the entire CDN, using the pre-existing JavascriptCache

Previous to this change this code was completely dormant on sites which
used s3 based uploads, this reuses the very well tested and cached asset
system on s3 based sites.

Note, when creating local_js_assets it is highly recommended to keep the
assets lean and keep all the heavy working in CDN based assets. For example
wasm files can still live on the CDN but the lean worker that loads it can
live on local.

This change unlocks wasm in theme components, so wasm is now also allowed
in `theme_authorized_extensions`

* more usages of upload.content

* add a specific test for upload.content

* Adjust logic to ensure that after upgrades we still get a cached local js
on save
2022-04-07 07:58:10 +10:00
Jarek Radosz
8e809149d2
DEV: Fix "overridden" typos (#16399)
There are still some, but those are in actual code that's used outside core, so the change there would need to go through the deprecation cycle. That's a task for another day.
2022-04-06 23:17:20 +02:00
David Taylor
68c74e9b93
FEATURE: Allow multiple required tag groups for a category (#16381)
Previously we only supported a single 'required tag group' for a category. This commit allows admins to specify multiple required tag groups, each with their own minimum tag count.

A new category_required_tag_groups database table replaces the existing columns on the categories table. Data is automatically migrated.
2022-04-06 14:08:06 +01:00
Sam
254f48e568
FIX: include crawler content on old mobile browsers (#16387)
Previous to this change an optimisation stripped crawler content from
all mobile browsers.

This had a side effect that meant that when we dropped support for an old
mobile platform we would stop rendering topic and topic list pages.

The new implementation ensures we only perform the optimisation on modern
mobile browsers.
2022-04-06 11:09:12 +01:00
Loïc Guitaut
357011eb3b DEV: Clean up freedom patches
This patch removes some of our freedom patches that have been deprecated
for some time now.
Some of them have been updated so we’re not shipping code based on an
old version of Rails.
2022-04-06 10:07:14 +02:00
Jarek Radosz
3c44bed545
DEV: Make DistributedMemoizer use DistributedMutex (#16229)
Its implementation was already distributed-mutex-like, with slight differences that did not seem necessary.
2022-04-05 19:29:58 +02:00
Martin Brennan
b8828d4a2d
FEATURE: Polymorphic bookmarks pt. 1 (CRUD) (#16308)
This commit introduces a new use_polymorphic_bookmarks site setting
that is default false and hidden, that will be used to help continuous
development of polymorphic bookmarks. This setting **should not** be
enabled anywhere in production yet, it is purely for local development.

This commit uses the setting to enable create/update/delete actions
for polymorphic bookmarks on the server and client side. The bookmark
interactions on topics/posts are all usable. Listing, searching,
sending bookmark reminders, and other edge cases will be handled
in subsequent PRs.

Comprehensive UI tests will be added in the final PR -- we already
have them for regular bookmarks, so it will just be a matter of
changing them to be for polymorphic bookmarks.
2022-03-30 12:43:11 +10:00
David Taylor
ff93833fdf
UX: Use committed date for GitHub oneboxes (#16318)
Our copy says 'committed {date}`, but we were previously using the commit's authored date
2022-03-30 09:16:28 +08:00
David Taylor
b2a8dc4c0f
FIX: Maintain HTML <img when downloading remote images (#16278)
Under some conditions, replacing an `<img` with `![]()` can break rendering, and make the image disappear.

Context at https://meta.discourse.org/t/152801
2022-03-29 10:55:10 +01:00
Alan Guo Xiang Tan
8e5614b1bf
DEV: Remove hardcoded ID in test fabrication. (#16313)
This hardcoded ID can cause fabrication to fail once we create 999
users across the entire test suite.
2022-03-29 15:23:55 +08:00
Sam
24f327f7ad
DEV: add testing for multi del on keys (#16271)
* DEV: add testing for multi del on keys

Following #15905 we were missing some tests, this covers cases where
del is used in the form of .del(key1,key2)
2022-03-29 09:52:17 +11:00
Osama Sayegh
e40c4bb7f9
FIX: Validate category tag restrictions before sending new topics to review (#16292)
Tags (and tag groups) can be configured so that they can only be used in specific categories and (optionally) restrict topics in these categories to be able to add/use only these tags. These restrictions work as expected when a topic is created without going through the review queue; however, if the topic has to be reviewed by a moderator then these restrictions currently aren't checked before the topic is sent to the review queue, but they're checked later when a moderator tries to approve the topic. This is because if a user manages to submit a topic that doesn't meet the restrictions, moderators won't be able to approve and it'll be stuck in the review queue.

This PR prevents topics that don't meet the tags requirements from being sent to the review queue and shows the poster an error message that indicates which tags that cannot be used.

Internal ticket: t60562.
2022-03-28 21:25:26 +03:00
Penar Musaraj
b1211bee97
FIX: Resetting selectable avatars was failing (#16302) 2022-03-28 14:15:28 -04:00
David Taylor
720e1ca9e7
FEATURE: Support upload:// urls in img tags (#16277)
Previously, our `upload://` protocol urls were only supported in markdown image tags. This meant that our PullHotlinkedImages job was forced to convert `<img` tags to markdown. Depending on the exact syntax, this can actually cause the image to break.

This commit adds support for `upload://` inside regular HTML `<img` tags. In a future commit, we'll be able to use this to make our PullHotlinkedImages job much more robust.

Context at https://meta.discourse.org/t/152801
2022-03-28 16:46:47 +01:00
Bianca Nenciu
cbaf7c949b
FIX: Make sure max_oneboxes_per_post is enforced (#16215)
PostAnalyzer and CookedPostProcessor both replace URLs with oneboxes.
PostAnalyzer did not use the max_oneboxes_per_post site and setting and
CookedPostProcessor replaced at most max_oneboxes_per_post URLs ignoring
the oneboxes that were replaced already by PostAnalyzer.
2022-03-23 17:36:08 +02:00
Dan Ungureanu
8e9cbe9db4
FIX: Do not raise if title cannot be crawled (#16247)
If the crawled page returned an error, `FinalDestination#safe_get`
yielded `nil` for `uri` and `chunk` arguments. Another problem is that
`get` did not handle the case when `safe_get` failed and did not return
the `location` and `set_cookie` headers.
2022-03-22 20:13:27 +02:00
Jarek Radosz
be3dceccfa
DEV: Merge two spec files (#16244)
Also reenabled two specs on macOS as they're green now.
2022-03-22 09:23:06 +08:00
Vinoth Kannan
8a9a7a77d8
FIX: staff should not be able to PM groups that "Nobody" can message (#16163)
If a group's messageable_level is set to nobody then staff can't should not be able to send PMs to it.

Co-authored-by: Martin Brennan <martin@discourse.org>
2022-03-22 10:23:14 +10:00
Osama Sayegh
b0656f3ed0
FIX: Apply onebox blocked domain checks on every redirect (#16150)
The `blocked onebox domains` setting lets site owners change what sites
are allowed to be oneboxed. When a link is entered into a post,
Discourse checks the domain of the link against that setting and blocks
the onebox if the domain is blocked. But if there's a chain of
redirects, then only the final destination website is checked against
the site setting.

This commit amends that behavior so that every website in the redirect
chain is checked against the site setting, and if anything is blocked
the original link doesn't onebox at all in the post. The
`Discourse-No-Onebox` header is also checked in every response and the
onebox is blocked if the header is set to "1".

Additionally, Discourse will now include the `Discourse-No-Onebox`
header with every response if the site requires login to access content.
This is done to signal to a Discourse instance that it shouldn't attempt
to onebox other Discourse instances if they're login-only. Non-Discourse
websites can also use include that header if they don't wish to have
Discourse onebox their content.

Internal ticket: t59305.
2022-03-11 09:18:12 +03:00
Bianca Nenciu
6d422a8033
FEATURE: Highlight expired bookmark reminders (#15317)
The user can select what happens with a bookamrk after it expires. New
option allow bookmark's reminder to be kept even after it has expired.
After a bookmark's reminder notification is created, the reminder date
will be highlighted in red until the user resets the reminder date.
User can do that using the new Clear Reminder button from the dropdown.
2022-03-08 19:44:18 +02:00
Bianca Nenciu
34b4b53bac
FEATURE: Use Postgres unaccent to ignore accents (#16100)
The search_ignore_accents site setting can be used to make the search
indexer remove the accents before indexing the content. The unaccent
function from PostgreSQL is better than Ruby's unicode_normalize(:nfkd).
2022-03-07 23:03:10 +02:00
jbrw
fc30669db2
FIX: Support new layout on Amazon product pages (#16091)
Some product pages on Amazon are using a new HTML structure, meaning the previous Onebox engine was unable to gather the price and/or description. This change should allow these pages to be Oneboxed.
2022-03-04 18:31:53 -05:00
Osama Sayegh
8c71878ff5
UX: Add description to the 2FA page when adding new admins (#16098)
This PR adds an extra description to the 2FA page when granting a user admin access. It also introduces a general system for adding customized descriptions that can be used by future actions.

(Follow-up to dd6ec65061)
2022-03-04 06:43:06 +03:00
Dan Ungureanu
39ab14531a
FEATURE: API to create user's associated account (#15737)
Discourse users and associated accounts are created or updated when a
user logins or connects the account using their account preferences.
This new API can be used to create associated accounts and users too,
if necessary.
2022-03-03 18:17:02 +02:00
Arpit Jalan
567be512c9
FIX: save email rejection error class names for incoming email logs (#16095) 2022-03-03 18:58:13 +05:30
Joffrey JAFFEUX
a558c5bd30
FIX: correctly notifies subscribers with post_action_destroyer (#16084) 2022-03-03 09:49:36 +01:00
David Taylor
c9dab6fd08
DEV: Automatically require 'rails_helper' in all specs (#16077)
It's very easy to forget to add `require 'rails_helper'` at the top of every core/plugin spec file, and omissions can cause some very confusing/sporadic errors.

By setting this flag in `.rspec`, we can remove the need for `require 'rails_helper'` entirely.
2022-03-01 17:50:50 +00:00
Jarek Radosz
2fc70c5572
DEV: Correctly tag heredocs (#16061)
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Jarek Radosz
4020738eed
DEV: Fix specs deprecations (#16059) 2022-02-26 03:51:39 +01:00
Roman Rizzi
54ad50eda1
FIX: Respect the cooldown window when editing a flagged topic. (#16046)
When staff decides to hide a flagged post, and it's the first post on the topic, the post owner shouldn't be able to edit either of them until the cooldown finishes. Edit either of them automatically, unhides the post, and makes the topic visible when there's a flag involved.

Reported on meta: https://meta.discourse.org/t/users-can-edit-flagged-topic-title-when-they-should-not-be-able-to/217796
2022-02-25 11:09:31 -03:00
Sam
c71afdfdb0
FIX: avoid validations when destroying posts (#16049)
Previously email validations could fire when deleting posts if for
certain reasons any user validations fail on the user objects

This kind of condition could happen in core due to a corruption of a
user record, or via a plugin that introduces a new validation on User
2022-02-25 11:20:54 +11:00
Jeff Wong
85f1ec643d
FIX: Move selectable_avatars_mode_validator_spec to new directory (#16048)
Lib specs moved in 45cc16098d
Move the new selectable_avatars_mode_validator_spec to the new location
Remove the old selectable_avatars_enabled_validator_spec

follow-up of d1bdb6c65d
2022-02-24 13:57:26 -08:00
Jarek Radosz
cecfc7ce79
FIX: Don't warn on empty .discourse-compatibility (#16032) 2022-02-23 09:44:37 +01:00
Osama Sayegh
586d572e05
FIX: Don't advance draft sequence when editing topic title (#16002)
This commit handles the edge case where a draft is lost with no warnings if the user edits the title (or category/tags) of a topic while they're replying.to the same topic. Repro steps are as follows:

1. Start replying to a topic and type enough to get a draft saved.
2. Scroll up to the topic title and click the pencil icon next to the topic title, change the title, category and/or tags, and then save the changes.
3. Reload the page and you'll see that the draft is gone.

This happens because we only allow 1 draft per topic per user and when you edit the title of a topic that you're replying to, from the server perspective it'll look like as if you've submitted your reply so it will advance the draft sequence for the topic and delete the draft.

The fix in this commit makes `PostRevisor` skip advancing the draft sequence when a topic's title is edited using the pencil button next to the title.

Internal ticket: t60854.

Co-authored-by: Robin Ward <robin.ward@gmail.com>
2022-02-23 10:39:54 +03:00
Martin Brennan
599a72768c
DEV: Add force_quote_link option to PrettyText (#16034)
This option will make it so the [quote] bbcode will always
include the HTML link to the quoted post, even if a topic_id
is not provided in the PrettyText#cook options. This is so
[quote] bbcode can be used in other places, like chat messages,
that always need the link and do not have an "off-topic" ID
to use.
2022-02-23 16:13:46 +10:00
Sam
d4d3580761
PERF: perform all cached counting in background (#15991)
Previously cached counting made redis calls in main thread and performed
the flush in main thread.

This could lead to pathological states in extreme heavy load.

This refactor reduces load and cleans up the interface
2022-02-22 16:45:25 +00:00
David Taylor
f6c852bf8e
PERF: Use a shared message for replies to tracked topics (#16022)
Previously we were publishing one messagebus message per user which was 'tracking' a topic. On large sites, this can easily be 1000+ messages. The important information in the message is common between all users, so we can manage with a single message on a shared channel, which will be much more efficient.

For user-specific values (notification_level and last_read_post_number), the JS app can infer values which are 'good enough'. Correct values will be loaded as soon as a topic-list containing the topic is visited.
2022-02-22 15:27:46 +00:00
Bianca Nenciu
90c3695ab0
FEATURE: Rename Reset Read bulk action to Defer (#15972)
It is enabled only if defer is enabled in user options too and if the
button shows up in the topic's footer.
2022-02-21 22:45:01 +02:00
Jarek Radosz
45cc16098d
DEV: Move spec/components to spec/lib (#15987)
Lib specs were inexplicably split into two directories (`lib` and `components`)

This moves them all into `lib`.
2022-02-18 19:41:54 +01:00
David Taylor
ffcd2e9faf
FIX: Handle nil values in DistributedCache#defer_get_set (#15978)
Themes often cache `nil` values in a DistributedCache. This bug meant that we were re-calculating some values on every request, AND triggering message-bus publishing on every request.

This fix should provide a significant performance improvement for busy sites.
2022-02-17 14:52:14 +00:00
Osama Sayegh
dd6ec65061
FEATURE: Centralized 2FA page (#15377)
2FA support in Discourse was added and grown gradually over the years: we first
added support for TOTP for logins, then we implemented backup codes, and last
but not least, security keys. 2FA usage was initially limited to logging in,
but it has been expanded and we now require 2FA for risky actions such as
adding a new admin to the site.

As a result of this gradual growth of the 2FA system, technical debt has
accumulated to the point where it has become difficult to require 2FA for more
actions. We now have 5 different 2FA UI implementations and each one has to
support all 3 2FA methods (TOTP, backup codes, and security keys) which makes
it difficult to maintain a consistent UX for these different implementations.
Moreover, there is a lot of repeated logic in the server-side code behind these
5 UI implementations which hinders maintainability even more.

This commit is the first step towards repaying the technical debt: it builds a
system that centralizes as much as possible of the 2FA server-side logic and
UI. The 2 main components of this system are:

1. A dedicated page for 2FA with support for all 3 methods.
2. A reusable server-side class that centralizes the 2FA logic (the
`SecondFactor::AuthManager` class).

From a top-level view, the 2FA flow in this new system looks like this:

1. User initiates an action that requires 2FA;

2. Server is aware that 2FA is required for this action, so it redirects the
user to the 2FA page if the user has a 2FA method, otherwise the action is
performed.

3. User submits the 2FA form on the page;

4. Server validates the 2FA and if it's successful, the action is performed and
the user is redirected to the previous page.

A more technically-detailed explanation/documentation of the new system is
available as a comment at the top of the `lib/second_factor/auth_manager.rb`
file. Please note that the details are not set in stone and will likely change
in the future, so please don't use the system in your plugins yet.

Since this is a new system that needs to be tested, we've decided to migrate
only the 2FA for adding a new admin to the new system at this time (in this
commit). Our plan is to gradually migrate the remaining 2FA implementations to
the new system.

For screenshots of the 2FA page, see PR #15377 on GitHub.
2022-02-17 12:12:59 +03:00
Martin Brennan
4b4f2330da
DEV: Expand UploadMarkdown generation capabilities (#15930)
The chat quoting mechanism will need to be able to generate
markdown for all kinds of uploads. The UploadMarkdown class
was missing generation for video and audio uploads. This
commit adds that in, and also expands the server-side regex
recognition of FileHelper types to match those in uploads.js,
and adds a spec for UploadMarkdown
2022-02-14 15:48:27 +10:00
Dan Ungureanu
1fb97f8bba
FIX: Replace Twitter handles one at a time (#15870)
Previously, all handles and hashtags were replaced in one go which could
result in a wrong result if a handle was a substring of another one.
2022-02-09 13:54:02 +02:00
Martin Brennan
59343c3057
DEV: Add Windows Hello webauthn authentication spec (#15871)
Follow up to 6f7364e48b to add a spec
that tests the full authentication of a Windows Hello algorithm (-257)
webauthn verification. The test added in that commit only tested that
we know about that algorithm, not whether it was actually usable.
2022-02-09 12:47:47 +10:00
Dan Ungureanu
6f7364e48b
FEATURE: RS512, RS384 and RS256 COSE algorithms (#15804)
* FEATURE: RS512, RS384 and RS256 COSE algorithms

These algorithms are not implemented by cose-ruby, but used in the web
authentication API and were marked as supported.

* FEATURE: Use all algorithms supported by cose-ruby

Previously only a subset of the algorithms were allowed.
2022-02-08 14:07:47 +02:00
Alan Guo Xiang Tan
7afe768d60
DEV: Add tests for wistia onebox. (#15860)
Follow-up to 4ef56b0ca4
2022-02-08 13:04:32 +08:00
Alan Guo Xiang Tan
930f51e175 FEATURE: Split up text segmentation for Chinese and Japanese.
* Chinese segmenetation will continue to rely on cppjieba
* Japanese segmentation will use our port of TinySegmenter
* Korean currently does not rely on segmentation which was dropped in c677877e4f
* SiteSetting.search_tokenize_chinese_japanese_korean has been split
into SiteSetting.search_tokenize_chinese and
SiteSetting.search_tokenize_japanese respectively
2022-02-07 09:21:14 +08:00
Rafael dos Santos Silva
5b5cbbfe5c
FEATURE: Onebox for news.ycombinator.com (#15781) 2022-02-03 13:39:21 -03:00
Martin Brennan
82cb67e67b
FIX: Canonical Message-ID was incorrect for some cases (#15701)
When creating a direct message to a group with group SMTP
set up, and adding another person to that message in the OP,
we send an email to the second person in the OP via the group_smtp
job. This in turn creates an IncomingEmail record to guard against
IMAP double sync.

The issue with this was that this IncomingEmail (which is essentialy
a placeholder/dummy one) was having its Message-ID used as the canonical
References Message-ID for subsequent emails sent out to user_private_message
recipients (such as members of the group), causing threading issues in
the mail client. The canonical <topic/ID@HOST> format should be used
instead for these cases.

This commit fixes the issue by only using the IncomingEmail for the
OP's Message-ID if the OP was created via our handle_mail email receiver
pipeline. It does not make sense to use it in other cases.
2022-02-03 10:36:32 +10:00
Natalie Tay
aac9f43038
Only block domains at the final destination (#15689)
In an earlier PR, we decided that we only want to block a domain if 
the blocked domain in the SiteSetting is the final destination (/t/59305). That 
PR used `FinalDestination#get`. `resolve` however is used several places
 but blocks domains along the redirect chain when certain options are provided.

This commit changes the default options for `resolve` to not do that. Existing
users of `FinalDestination#resolve` are
- `Oneboxer#external_onebox`
- our onebox helper `fetch_html_doc`, which is used in amazon, standard embed 
and youtube
  - these folks already go through `Oneboxer#external_onebox` which already
  blocks correctly
2022-01-31 15:35:12 +08:00
Bianca Nenciu
376799b1a4
FIX: Hide excerpt of binary files in GitHub onebox (#15639)
Oneboxer did not know if a file is binary or not and always tried to
show an excerpt of the file.
2022-01-19 14:45:36 +02:00
jbrw
2909b8b820
FIX: origins_to_regexes should always return an array (#15589)
If the SiteSetting `allowed_onebox_iframes` contains a value of `*`, it will use the values of `all_iframe_origins` during the Oneboxing process. If `all_iframe_origins` itself contains a value of `*`, `origins_to_regexes` will try to return a "catch-all" regex.

Other code assumes `origins_to_regexes`will return an array, so this change ensures the `*` case will return an array containing only the catch-all regex.
2022-01-17 12:48:41 -05:00
Jarek Radosz
31b27b3712
FIX: Broken GitHub folder onebox logic (#15612)
1. `html_doc.css('.Box.md')` always returns a truthy value (e.g. `[]`) so the second branch of the if-elsif never ran
2. `node&.css('text()')` was invalid code that would raise an error
3. Matching on h3 elements is no longer correct with the current html structure returned by GitHub
2022-01-17 18:32:07 +01:00
Jarek Radosz
19fcb0b5ea
DEV: Prevent extraneous log message in specs (#15504)
Yo dawg, I put `silence_stdout` in your `silence_stdout` so you can still write to stdout? 🤔
2022-01-09 20:26:52 +01:00
Martin Brennan
04c7776650
DEV: Rolling back bookmarkable column changes (#15482)
It is too close to release of 2.8 for incomplete
feature shenanigans. Ignores and drops the columns and drops
the trigger/function introduced in
e21c640a3c.
Will pick this feature back up post-release.
2022-01-07 12:16:43 +10:00
Martin Brennan
e21c640a3c
DEV: Add polymorphic bookmarkable columns (#15454)
We are planning on attaching bookmarks to more and
more other models, so it makes sense to make a polymorphic
relationship to handle this. This commit adds the new
columns and backfills them in the bookmark table, and
makes sure that any new bookmark changes fill in the columns
via DB triggers.

This way we can gradually change the frontend and backend
to use these new columns, and eventually delete the
old post_id and for_topic columns in `bookmarks`.
2022-01-06 08:56:05 +10:00
Peter Zhu
c5fd8c42db
DEV: Fix methods removed in Ruby 3.2 (#15459)
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
2022-01-05 18:45:08 +01:00
Martin Brennan
20fe5eceb8
FEATURE: Scheduled group email credential problem check (#15396)
This commit adds a check that runs regularly as per
2d68e5d942 which tests the
credentials of groups with SMTP or IMAP enabled. If any issues
are found with those credentials a high priority problem is added to the
admin dashboard.

This commit also formats the admin dashboard differently if
there are high priority problems, bringing them to the top of
the list and highlighting them.

The problem will be cleared if the issue is fixed before the next
problem check, or if the group's settings are updated with a valid
credential.
2022-01-04 10:14:33 +10:00
David Taylor
cdf4d7156e
DEV: Introduce Auth::Result API for overrides_* (#15378)
This allows authenticators to instruct the Auth::Result to override attributes without using the general site settings. This provides an easy migration path for auth plugins which offer their own "overrides email", "overrides username" or "overrides name" settings. With this new api, they can set `overrides_*` on the result object, and the attribute will be overriden regardless of the general site setting.

ManagedAuthenticator is updated to use this new API. Plugins which consume ManagedAuthenticator will instantly take advantage of this change.
2021-12-23 10:53:17 +00:00
jbrw
6e925fee6f
FIX: Use basic meta description if other description tags are missing (#15356)
When attempting to Onebox a page if there is no `meta property="og:description"` tag but there is a  `meta name="description"` tag, Onebox should try to use that value.
2021-12-17 19:36:54 -05:00
Gerhard Schlager
e19a7a7c8d FIX: translation precedence was different on client and server
As an example, the lookup order for German was:

1. override for de
2. override for en
3. value from de
4. value from en

After this change the lookup order is the same as on the client:
1. override for de
2. value from de
3. override for en
4. value from en

see /t/16381
2021-12-17 14:03:35 +01:00
Blake Erickson
b93b6c4299
FIX: Blurry onebox favicon images (#15258)
This is a fix to address blurry onebox favicon images if the site you
are linking to happens to have a favicon.ico file that contains multiple
images.

This fix detects of we are trying to create an upload for a favicon.ico
file. We then convert it to a png and not a jpeg like we were doing. We
want a png because it will preserve transparency, otherwise if we
convert it to a jpeg we lose that and it looks bad on dark themed sites.

This fix also addresses the fact that .ico files can include multiple
images. The blurry images we were producing was caused by the
ImageMagick `-flatten` option when the .ico file had multiple images
which then squishes them all together. So for .ico files we are no
longer flattening them and instead we are grabbing the last image in the
.ico bundle and converting that single image to a png.
2021-12-10 12:25:50 -07:00
David Taylor
f799b8bfb1
FIX: Ensure MessageIdService can handle hostname changes and multisite (#15231) 2021-12-08 11:17:20 +00:00
Martin Brennan
f26b8b448d
FIX: References header leading to broken email threading (#15206)
Since 3b13f1146b the email threading
in mail clients has been broken, because the random suffix meant
that the References header would always be different for non-group
SMTP email notifications sent out.

This commit fixes the issue by always using the "canonical" topic
reference ID inside the References header in the format:

topic/TOPIC_ID@HOST

Which was the old format. We also add the References header to
notifications sent for the first post arriving, so the threading
works for subsequent emails. The Message-ID header is still random
as per the previous change.
2021-12-08 08:14:48 +10:00
Martin Brennan
3b13f1146b
FIX: Add random suffix to outbound Message-ID for email (#15179)
Currently the Message-IDs we send out for outbound email
are not unique; for a post they look like:

topic/TOPIC_ID/POST_ID@HOST

And for a topic they look like:

topic/TOPIC_ID@HOST

This commit changes the outbound Message-IDs to also have
a random suffix before the host, so the new format is
like this:

topic/TOPIC_ID/POST_ID.RANDOM_SUFFIX@HOST

Or:

topic/TOPIC_ID.RANDOM_SUFFIX@HOST

This should help with email deliverability. This change
is backwards-compatible, the old Message-ID format will
still be recognized in the mail receiver flow, so people
will still be able to reply using Message-IDs, In-Reply-To,
and References headers that have already been sent.

This commit also refactors Message-ID related logic
to a central location, and adds judicious amounts of
tests and documentation.
2021-12-06 10:34:39 +10:00
Osama Sayegh
7bd3986b21
FEATURE: Replace Crawl-delay directive with proper rate limiting (#15131)
We have a couple of site setting, `slow_down_crawler_user_agents` and `slow_down_crawler_rate`, that are meant to allow site owners to signal to specific crawlers that they're crawling the site too aggressively and that they should slow down.

When a crawler is added to the `slow_down_crawler_user_agents` setting, Discourse currently adds a `Crawl-delay` directive for that crawler in `/robots.txt`. Unfortunately, many crawlers don't support the `Crawl-delay` directive in `/robots.txt` which leaves the site owners no options if a crawler is crawling the site too aggressively.

This PR replaces the `Crawl-delay` directive with proper rate limiting for crawlers added to the `slow_down_crawler_user_agents` list. On every request made by a non-logged in user, Discourse will check the User Agent string and if it contains one of the values of the `slow_down_crawler_user_agents` list, Discourse will only allow 1 request every N seconds for that User Agent (N is the value of the `slow_down_crawler_rate` setting) and the rest of requests made within the same interval will get a 429 response. 

The `slow_down_crawler_user_agents` setting becomes quite dangerous with this PR since it could rate limit lots if not all of anonymous traffic if the setting is not used appropriately. So to protect against this scenario, we've added a couple of new validations to the setting when it's changed:

1) each value added to setting must 3 characters or longer
2) each value cannot be a substring of tokens found in popular browser User Agent. The current list of prohibited values is: apple, windows, linux, ubuntu, gecko, firefox, chrome, safari, applewebkit, webkit, mozilla, macintosh, khtml, intel, osx, os x, iphone, ipad and mac.
2021-11-30 12:55:25 +03:00
Natalie Tay
4c46c7e334
DEV: Remove xlink hrefs (#15059) 2021-11-25 15:22:43 +11:00
David Taylor
13fdc979a8
DEV: Improve multisite testing (#14884)
This commit adds the RailsMultisite middleware in test mode when Rails.configuration.multisite is true. This allows for much more realistic integration testing. The `multisite_spec.rb` file is rewritten to avoid needing to simulate a middleware stack.
2021-11-11 16:44:58 +00:00
Martin Brennan
fc98d1edfa
DEV: Improve s3:ensure_cors_rules logging (#14832) 2021-11-08 11:44:12 +10:00
Martin Brennan
9a72a0945f
FIX: Ensure CORS rules exist for S3 using rake task (#14802)
This commit introduces a new s3:ensure_cors_rules rake task
that is run as a prerequisite to s3:upload_assets. This rake
task calls out to the S3CorsRulesets class to ensure that
the 3 relevant sets of CORS rules are applied, depending on
site settings:

* assets
* direct S3 backups
* direct S3 uploads

This works for both Global S3 settings and Database S3 settings
(the latter set directly via SiteSetting).

As it is, only one rule can be applied, which is generally
the assets rule as it is called first. This commit changes
the ensure_cors! method to be able to apply new rules as
well as the existing ones.

This commit also slightly changes the existing rules to cover
direct S3 uploads via uppy, especially multipart, which requires
some more headers.
2021-11-08 09:16:38 +10:00
jbrw
aec125b617
FIX: Display Instagram Oneboxes in an iframe (#14789)
We are no longer able to display the image returned by Instagram directly within a Discourse site (either in the composer, or within a cooked post within a topic), so:

- Display an image placeholder in the composer preview
- A cooked post should use an iframe to display the Instagram 'embed' content
2021-11-02 14:34:51 -04:00
Joffrey JAFFEUX
b18c01e3c6
DEV: prevents flakky spec when deleting plugin (#14701)
Not reseting the registry could lead to assets still being registered for example.

This flakky spec was reprdocible with this call: `bundle exec rspec --seed 9472 spec/components/discourse_plugin_registry_spec.rb spec/components/svg_sprite/svg_sprite_spec.rb`

Which would trigger the following error:

```
Failures:

  1) DiscoursePluginRegistry#register_asset registers vendored_core_pretty_text properly
     Failure/Error: expect(registry.javascripts.count).to eq(0)

       expected: 0
            got: 1

       (compared using ==)
     # ./spec/components/discourse_plugin_registry_spec.rb:248:in `block (3 levels) in <top (required)>'
     # ./spec/rails_helper.rb:280:in `block (2 levels) in <top (required)>'
     # /Users/joffreyjaffeux/.gem/ruby/2.7.3/gems/webmock-3.14.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'
```
2021-10-25 10:24:21 +02:00
Yasuo Honda
9a083a550c FIX: BackupRestore::DatabaseRestorer failures with Ruby 3
Implemented workaround suggested at
https://github.com/freerange/mocha/issues/445#issuecomment-644944003
2021-10-12 17:25:51 -04:00
Alan Guo Xiang Tan
34cebfd867
FIX: Exclude PMs that user sent to themselves. (#14496)
Regression from 016efeadf6

Follow-up to 016efeadf6
2021-10-04 11:55:35 +08:00
Alan Guo Xiang Tan
9d5da2b383
PERF: Revert all inboxes from messages route. (#14445)
The all inboxes was introduced in
016efeadf6 but we decided to roll it back
for performance reasons. The main performance challenge here is that PG
has to basically loop through all the PMs that a user is allowed to view
before being able to order by `Topic#bumped_at`. The all inboxes was not
planned as part of the new/unread filter so we've decided not to tackle
the performance issue for the upcoming release.

Follow-up to 016efeadf6
2021-09-28 11:58:04 +08:00
Martin Brennan
dba6a5eabf
FEATURE: Humanize file size error messages (#14398)
The file size error messages for max_image_size_kb and
max_attachment_size_kb are shown to the user in the KB
format, regardless of how large the limit is. Since we
are going to support uploading much larger files soon,
this KB-based limit soon becomes unfriendly to the end
user.

For example, if the max attachment size is set to 512000
KB, this is what the user sees:

> Sorry, the file you are trying to upload is too big (maximum
size is 512000KB)

This makes the user do math. In almost all file explorers that
a regular user would be familiar width, the file size is shown
in a format based on the maximum increment (e.g. KB, MB, GB).

This commit changes the behaviour to output a humanized file size
instead of the raw KB. For the above example, it would now say:

> Sorry, the file you are trying to upload is too big (maximum
size is 512 MB)

This humanization also handles decimals, e.g. 1536KB = 1.5 MB
2021-09-22 07:59:45 +10:00
Martin Brennan
27699648ef
FEATURE: Go to last unread for topic-level bookmark links (#14396)
Instead of going to the OP of the topic for topic-level bookmarks
(which are bookmarks where for_topic is true) when clicking on the
bookmark in the quick access menu or on the user bookmark list,
this commit takes the user to the last unread post in
the topic instead. This should be generally more useful than landing
on the unchanging OP.

To make this work nicely, I needed to add the last_read_post_number to
the BookmarkQuery based on the TopicUser association. It should not add
too much extra weight to the query, because it is limited to the user
that we are fetching bookmarks for.

Also fixed an issue where the bookmark serializer highest_post_number was
not taking into account whether the user was staff, which is when we
should use highest_staff_post_number instead.
2021-09-21 13:49:56 +10:00
Alan Guo Xiang Tan
7a8b5cdd5c
DEV: Improve tests coverage when listing private messages. (#14385)
This is in response to the security incident published in
https://github.com/discourse/discourse/security/advisories/GHSA-vm3x-w6jm-j9vv.

The security incident highlighted a gap in our test suite so we're
adding more test cases to ensure that personal and group messages do not
leak between users in the future.
2021-09-21 10:39:59 +08:00
Martin Brennan
0c42a1e5f3
FEATURE: Topic-level bookmarks (#14353)
Allows creating a bookmark with the `for_topic` flag introduced in d1d2298a4c set to true. This happens when clicking on the Bookmark button in the topic footer when no other posts are bookmarked. In a later PR, when clicking on these topic-level bookmarks the user will be taken to the last unread post in the topic, not the OP. Only the OP can have a topic level bookmark, and users can also make a post-level bookmark on the OP of the topic.

I had to do some pretty heavy refactors because most of the bookmark code in the JS topics controller was centred around instances of Post JS models, but the topic level bookmark is not centred around a post. Some refactors were just for readability as well.

Also removes some missed reminderType code from the purge in 41e19adb0d
2021-09-21 08:45:47 +10:00
Martin Brennan
41e19adb0d
DEV: Ignore reminder_type for bookmarks (#14349)
We don't actually use the reminder_type for bookmarks anywhere;
we are just storing it. It has no bearing on the UI. It used
to be relevant with the at_desktop bookmark reminders (see
fa572d3a7a)

This commit marks the column as readonly, ignores it, and removes
the index, and it will be dropped in a later PR. Some plugins
are relying on reminder_type partially so some stubs have been
left in place to avoid errors.
2021-09-16 09:56:54 +10:00
Alan Guo Xiang Tan
ddb458343d
PERF: Improve query performance all inbox private messages. (#14304)
First reported in https://meta.discourse.org/t/-/202482/19

There are two optimizations being applied here:

1. Fetch a user's group ids in a seperate query instead of including it
   as a sub-query. When I tried a subquery, the query plan becomes very
inefficient.

1. Join against the `topic_allowed_users` and `topic_allowed_groups`
   table instead of doing an IN against a subquery where we UNION the
`topic_id`s from the two tables. From my profiling, this enables PG to
do a backwards index scan on the `index_topics_on_timestamps_private`
index.

This commit fixes a bug where listing all messages was incorrectly
excluding topics if a topic has been archived by a group even if the
user did not belong to the group.

This commit also fixes another bug where dismissing private messages
selectively was subjected to the default limit of 30.
2021-09-15 10:29:42 +08:00
Martin Brennan
22208836c5
DEV: Ignore bookmarks.topic_id column and remove references to it in code (#14289)
We don't need no stinkin' denormalization! This commit ignores
the topic_id column on bookmarks, to be deleted at a later date.
We don't really need this column and it's better to rely on the
post.topic_id as the canonical topic_id for bookmarks, then we
don't need to remember to update both columns if the bookmarked
post moves to another topic.
2021-09-15 10:16:54 +10:00
Jean
34ff7bfeeb
FEATURE: Hide suspended users from site-wide search to regular users (#14245) 2021-09-06 09:59:35 -04:00
Martin Brennan
e43a8af3bd
FIX: Do not send emails to mailing_list_mode subscribers for PMs (#14159)
This bug was introduced by f66007ec83.

In PostJobsEnqueuer we previously did not fire the after_post_create
event and after_topic_create event for private message topics. This was
changed in the above commit in order to publish message bus messages
for topic tracking state updates. Unfortunately this caused the
NotifyMailingListSubscribers job to be enqueued for all posts including
private messages, and admins and the users involved in the PMs got
emailed the contents of the PMs if they had mailing list mode enabled.

Luckily the impact of this was mitigated by a Guardian#can_see? check
for each mailing list mode user in the NotifyMailingListSubscribers job.
We never want to notify mailing list mode subscribers for private messages
so an early return has been added there, plus the logic in PostJobsEnqueuer
has been fixed, and tests have been added to that class where there were
none before.
2021-08-26 15:16:35 +10:00
Alan Guo Xiang Tan
d13716286c
FIX: Unread group PMs should use GroupUser#first_unread_pm_at. (#14075)
This bug was causing unread PMs for groups to appear inaccurate.
2021-08-18 11:23:28 +08:00
Chema Balsas
745b99edbf TEST: Adds test for urls with url-encoded section hash 2021-08-12 10:43:50 -04:00
Chema Balsas
6b8ee4d5ef TEST: Adds test for urls with section hash 2021-08-12 10:43:50 -04:00
Alan Guo Xiang Tan
0bf27242ec FIX: Group inbox new filter not accounting for dismissed topics.
Follow-up to 2c046cc670
2021-08-05 16:53:12 +08:00
Alan Guo Xiang Tan
2c046cc670 FEATURE: Dismiss new and unread for PM inboxes. 2021-08-05 12:56:15 +08:00
Bianca Nenciu
e2c415457c
FEATURE: Attach backup log as upload (#13849)
Discourse automatically sends a private message after backup or
restore finished. The private message used to contain the log inline
even when it was very long. A very long log can create issues because
the length of the post will be over the maximum allowed length of a
post. When that happens, Discourse will try to create an upload with
the logs. If that fails, it will trim the log and inline it.
2021-08-03 20:06:50 +03:00
Bianca Nenciu
fbf7627c8e
FIX: Make search work with sub-sub-categories (#13901)
Searching in a category looked only one level down, ignoring the site
setting max_category_nesting. The user interface did not support the
third level of categories and did not display them in the "Categorized"
input of the advanced search options.
2021-08-02 14:04:13 +03:00
Alan Guo Xiang Tan
016efeadf6
FEATURE: New and Unread messages for user personal messages. (#13603)
* FEATURE: New and Unread messages for user personal messages.

Co-authored-by: awesomerobot <kris.aubuchon@discourse.org>
2021-08-02 12:41:41 +08:00
jbrw
2f28ba318c
FEATURE: Onebox can match engines based on the content_type (#13876)
* FEATURE: Onebox can match engines based on the content_type

`FinalDestination` now returns the `content_type` of a resolved URL.

`Oneboxer` passes this value to `Onebox` itself. Onebox engines can now specify a `matches_content_type` regex of content_types that the engine can handle, regardless of the URL.

`ImageOnebox` will match URLs with a content type of `image/png`, `jpg`, `gif`, `bmp`, `tif`, etc.

This will allow images that exist at a URL without a file type extension to be correctly rendered, assuming a valid `content_type` is returned.
2021-07-30 13:36:30 -04:00
Joffrey JAFFEUX
5eb6e9281a
FIX: manually adds frowning_face_with_open_mouth for apple (#13528) 2021-07-21 23:27:20 +02:00
Michael Brown
76a11e6dc9 DEV: fix test (missed a reference to master) 2021-07-19 12:47:45 -04:00
Michael Brown
aa12d12c0b discourse/discourse change from 'master' to 'main': update fixture data 2021-07-19 11:46:15 -04:00
David Taylor
8b89787426
SECURITY: Sanitize YouTube Onebox data (#13748)
CVE-2021-32764
2021-07-15 19:31:50 +01:00
jbrw
a64aea38b7
FIX: Don’t use user_generated images as avatar images in Oneboxed Twitter content (#13712)
By default, Twitter will return the URL for the avatar image of the tweet poster as the `og:image` value.

However, if the `user_generated` attribute is true, we should not use this as the avatar URL as this will be an URL of an image in the tweet itself (e.g., an image belonging to a tweeted news story).
2021-07-13 14:54:28 -04:00
Jarek Radosz
48b92d8897
DEV: Isolate multisite specs (#13634)
Mixing multisite and standard specs can lead to issues (e.g. when using `fab!`)
Disabled the (upcoming https://github.com/discourse/rubocop-discourse/pull/11) rubocop rule for two files that have thoroughly tangled both types of specs.
2021-07-07 18:57:42 +02:00
Penar Musaraj
35110f6681
FIX: Set CSP base-uri to self (#13654) 2021-07-07 09:43:48 -04:00
Arpit Jalan
05bdbd9f97
SECURITY: Onebox canonical links bypassing FinalDestination checks (#13605) 2021-07-01 20:09:29 +05:30
Arpit Jalan
b63c9febe8
FIX: ignore canonical link to localhost (#13577) 2021-06-30 13:55:17 +05:30
Krzysztof Kotlarek
a6363170e9 FIX: flaky search-spec
More precise expectations for search spec
2021-06-29 10:06:44 +08:00
Jarek Radosz
046a875222
DEV: Improve script/downsize_uploads.rb (#13508)
* Only shrink images that are used in Posts and no other models
* Don't save the upload if the size is the same
2021-06-24 00:09:40 +02:00
David Taylor
d2c5165052 FIX: Check all migrations for dropped columns/tables during restore
Previously only post-deploy migrations were being checked for DROPPED_(COLUMNS|TABLES) constants
2021-06-23 17:43:38 +01:00
Alan Guo Xiang Tan
44aa46ca05 Code review comments. 2021-06-21 11:06:58 +08:00
Alan Guo Xiang Tan
8e3691d537 PERF: Eager load Theme associations in Stylesheet Manager.
Before this change, calling `StyleSheet::Manager.stylesheet_details`
for the first time resulted in multiple queries to the database. This is
because the code was modelled in a way where each `Theme` was loaded
from the database one at a time.

This PR restructures the code such that it allows us to load all the
theme records in a single query. It also allows us to eager load the
required associations upfront. In order to achieve this, I removed the
support of loading multiple themes per request. It was initially added
to support user selectable theme components but the feature was never
completed and abandoned because it wasn't a feature that we thought was
worth building.
2021-06-21 11:06:58 +08:00
jbrw
fbfd1fd80b
FIX: Allow SVG uploads if dimensions are a fraction of a unit (#13409)
* FIX: Allow SVG uploads if dimensions are a fraction of a unit

`UploadCreator` counts the number of pixels in an file to determine if it is valid. `pixels` is calculated by multiplying the width and height of the image, as determined by FastImage.

SVG files can have their width/height expressed in a variety of different units of measurement. For example, ‘px’, ‘in’, ‘cm’, ‘mm’, ‘pt’, ‘pc’, etc are all valid within SVG files. If an image has a width of `0.5in`, FastImage may interpret this as being a width of `0`, meaning it will report the `size` as being `0`.

However, we don’t need to concern ourselves with the number of ‘pixels’ in a SVG files, as that is irrelevant for this file format, so we can skip over the check for `pixels == 0` when processing this file type.

* DEV: Speed up getting SVG dimensions

The `-ping` flag prevents the entire image from being rasterized before a result is returned. See:

https://imagemagick.org/script/command-line-options.php#ping
2021-06-17 15:56:11 -04:00
Jarek Radosz
e36377d9ab
DEV: Don't user before(:all)/after(:all) (#13389)
Leaking state and non-obvious order (before :all runs *before* RailsHelper.test_setup) are not worth it.
A replacement PR for #13370. Fixes some flaky specs, e.g.
```
bin/rspec './spec/components/freedom_patches/translate_accelerator_spec.rb[1:3]' './spec/jobs/clean_up_user_export_topics_spec.rb[1:1]' --tag ~type:multisite --seed 35994
```

Also included:
* DEV: No need for locale reset (we do it anyway in rails_helper in `test_setup`)
2021-06-15 17:25:06 +02:00
Roman Rizzi
fa57316a4e
FIX: Validate upload is still valid after calling the "before_upload_creation" event (#13091)
Since we use the event to perform additional validations on the file, we should check if it added any errors to the upload before saving it. This change makes the UploadCreator more consistent since we no longer have to rely on exceptions.
2021-06-15 10:10:03 -03:00
Penar Musaraj
6f76479054
FEATURE: Add upgrade-insecure-requests to CSP when force_https is enabled (#13348)
If force_https is enabled all resource (including markdown preview and so on) will be accessed using HTTPS

If for any reason you attempt to link to non HTTPS reachable content content may appear broken
2021-06-10 10:53:10 +10:00
Penar Musaraj
8336e732d3
DEV: Add manifest-src to CSP (#13319)
Defaults to `manifest-src: 'self'` and allows plugins/themes to extend it.
2021-06-08 09:32:31 -04:00
Penar Musaraj
f90c4bd6a1
DEV: Allow plugins to extend frame-ancestors (#13316) 2021-06-07 14:59:15 -04:00
jbrw
09bc95d46b
FIX: Quoting Oneboxed content should exclude formatting (#13296)
* FIX: Quoting Oneboxed content should exclude formatting

When a post is quoted that includes Oneboxed content, we should not include the formatting generated by the Onebox. Rather, we should attempt to collapse the link referenced by the Onebox to a single line text link.

* DEV: fix tests
2021-06-07 13:03:53 -04:00
Sam
435c4817cb
FEATURE: enable tagging by default (#13175)
Over the years we have found that a few communities never discovered tags.

Instead of having them default off we now have them default on, ensuring
that everyone finds out about them.

Co-authored-by: Dan Ungureanu <dan@ungureanu.me>
2021-06-07 18:07:46 +03:00
Arpit Jalan
2e4f07678e
FIX: IMDb links were being oneboxed as posters (#13310)
IMDb movie links were being rendered as posters. This was because
IMDb was sending `og:type` as `image` randomly in some cases. To
fix this we'll now default all IMDb links as article type. This will
ensure that the IMDb onebox link includes all the information instead
of showing just a poster without any context.
2021-06-07 18:45:59 +05:30
jbrw
461a2c334b
FIX: return an empty result if response from Amazon is missing expected attributes (#13173)
* FIX: return an empty result if response from Amazon is missing attributes

Check we have the basic attributes requires to construct a Onebox for Amazon.

This is an attempt to handle scenarios where we receive a valid 200-status response from an Amazon request that does not include the data we’re expecting.

* Update lib/onebox/engine/amazon_onebox.rb

Co-authored-by: Régis Hanol <regis@hanol.fr>

Co-authored-by: Régis Hanol <regis@hanol.fr>
2021-06-01 16:23:18 -04:00
Gerhard Schlager
3df928d609
DEV: Fix flaky specs (#13226)
Some specs failed when `LOAD_PLUGINS=1` was set while migrating the test DB and the narrative-bot plugin disabled the `send_welcome_message` site setting.
2021-06-01 14:38:55 +02:00
Penar Musaraj
06e1af2b1d
FIX: Giphy oneboxing when the response is an image (#13199) 2021-05-28 15:10:32 -04:00
Penar Musaraj
47e09700fe
FIX: Support pausing GIFs for giphy/tenor oneboxes (#13194) 2021-05-28 08:40:30 -04:00
Martin Brennan
501de809da
FIX: Do not mark badge image uploads as secure (#13193)
* FIX: Do not mark badge image uploads as secure

We do not need badge_image upload types to be marked as secure.
Post migration is the same as
https://github.com/discourse/discourse/pull/12081.

See
https://meta.discourse.org/t/secure-media-uploads/140017/122?u=martin
2021-05-28 12:35:52 +10:00
Martin Brennan
964da21817
FEATURE: Improve group email settings UI (#13083)
This overhauls the user interface for the group email settings management, aiming to make it a lot easier to test the settings entered and confirm they are correct before proceeding. We do this by forcing the user to test the settings before they can be saved to the database. It also includes some quality of life improvements around setting up IMAP and SMTP for our first supported provider, GMail. This PR does not remove the old group email config, that will come in a subsequent PR. This is related to https://meta.discourse.org/t/imap-support-for-group-inboxes/160588 so read that if you would like more backstory.

### UI

Both site settings of `enable_imap` and `enable_smtp` must be true to test this. You must enable SMTP first to enable IMAP.

You can prefill the SMTP settings with GMail configuration. To proceed with saving these settings you must test them, which is handled by the EmailSettingsValidator.

If there is an issue with the configuration or credentials a meaningful error message should be shown.

IMAP settings must also be validated when IMAP is enabled, before saving.

When saving IMAP, we fetch the mailboxes for that account and populate them. This mailbox must be selected and saved for IMAP to work (the feature acts as though it is disabled until the mailbox is selected and saved):

### Database & Backend

This adds several columns to the Groups table. The purpose of this change is to make it much more explicit that SMTP/IMAP is enabled for a group, rather than relying on settings not being null. Also included is an UPDATE query to backfill these columns. These columns are automatically filled when updating the group.

For GMail, we now filter the mailboxes returned. This is so users cannot use a mailbox like Sent or Trash for syncing, which would generally be disastrous.

There is a new group endpoint for testing email settings. This may be useful in the future for other places in our UI, at which point it can be extracted to a more generic endpoint or module to be included.
2021-05-28 09:28:18 +10:00
Dan Ungureanu
723d7de18c
Various GitHub Onebox improvements (#13163)
* FIX: Improve GitHub folder regexp in Onebox

It used to match any GitHub URL that was not matched by the other GitHub
Oneboxes and it did not do a good job at handling those. With this
change, the generic Onebox will handle the remaining URLs.

* FEATURE: Add Onebox for GitHub Actions

* FEATURE: Add Onebox for PR check runs

* FIX: Remove image from GitHub folder Oneboxes

It is a generic, auto-generated image which does not provide any value.

* DEV: Add tests

* FIX: Strip HTML comments from PR body
2021-05-27 12:38:42 +03:00
Arpit Jalan
283b08d45f
DEV: Absorb onebox gem into core (#12979)
* Move onebox gem in core library

* Update template file path

* Remove warning for onebox gem caching

* Remove onebox version file

* Remove onebox gem

* Add sanitize gem

* Require onebox library in lazy-yt plugin

* Remove onebox web specific code

This code was used in standalone onebox Sinatra application

* Merge Discourse specific AllowlistedGenericOnebox engine in core

* Fix onebox engine filenames to match class name casing

* Move onebox specs from gem into core

* DEV: Rename `response` helper to `onebox_response`

Fixes a naming collision.

* Require rails_helper

* Don't use `before/after(:all)`

* Whitespace

* Remove fakeweb

* Remove poor unit tests

* DEV: Re-add fakeweb, plugins are using it

* Move onebox helpers

* Stub Instagram API

* FIX: Follow additional redirect status codes (#476)

Don’t throw errors if we encounter 303, 307 or 308 HTTP status codes in responses

* Remove an empty file

* DEV: Update the license file

Using the copy from https://choosealicense.com/licenses/gpl-2.0/#

Hopefully this will enable GitHub to show the license UI?

* DEV: Update embedded copyrights

* DEV: Add Onebox copyright notice

* DEV: Add MIT license, convert COPYRIGHT.txt to md

* DEV: Remove an incorrect copyright claim

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Co-authored-by: jbrw <jamie@goatforce5.org>
2021-05-26 15:11:35 +05:30
Penar Musaraj
6e9ee3db03
FIX: Sort user bookmarks by reminder date (#13145) 2021-05-25 15:23:46 -04:00
Josh Soref
59097b207f
DEV: Correct typos and spelling mistakes (#12812)
Over the years we accrued many spelling mistakes in the code base. 

This PR attempts to fix spelling mistakes and typos in all areas of the code that are extremely safe to change 

- comments
- test descriptions
- other low risk areas
2021-05-21 11:43:47 +10:00
Arpit Jalan
130160537c
FEATURE: add support for "skip_validations" option in UploadCreator (#13094)
FIX: do not validate uploads when running `uploads:fix_missing_s3` task
2021-05-19 20:54:52 +05:30
Blake Erickson
23de5e959e
DEV: Remove duplicate test (#12989)
I think this test was duplicated on accident. We test for the
exact thing a few lines up.
2021-05-07 15:34:14 -06:00
Krzysztof Kotlarek
e29605b79f
FEATURE: the ability to search users by custom fields (#12762)
When the admin creates a new custom field they can specify if that field should be searchable or not.

That setting is taken into consideration for quick search results.
2021-04-27 15:52:45 +10:00
David Taylor
3436fef5e3
FIX: Update database_restorer to avoid shell use (#12731)
Follow-up to 0ec5fd5262
2021-04-16 13:39:45 +01:00
Osama Sayegh
cd24eff5d9
FEATURE: Introduce theme/component QUnit tests (take 2) (#12661)
This commit allows themes and theme components to have QUnit tests. To add tests to your theme/component, create a top-level directory in your theme and name it `test`, and Discourse will save all the files in that directory (and its sub-directories) as "tests files" in the database. While tests files/directories are not required to be organized in a specific way, we recommend that you follow Discourse core's tests [structure](https://github.com/discourse/discourse/tree/master/app/assets/javascripts/discourse/tests).

Writing theme tests should be identical to writing plugins or core tests; all the `import` statements and APIs that you see in core (or plugins) to define/setup tests should just work in themes.

You do need a working Discourse install to run theme tests, and you have 2 ways to run theme tests:

* In the browser at the `/qunit` route. `/qunit` will run tests of all active themes/components as well as core and plugins. The `/qunit` now accepts a `theme_name` or `theme_url` params that you can use to run tests of a specific theme/component like so: `/qunit?theme_name=<your_theme_name>`.

* In the command line using the `themes:qunit` rake task. This take is meant to run tests of a single theme/component so you need to provide it with a theme name or URL like so: `bundle exec rake themes:qunit[name=<theme_name>]` or `bundle exec rake themes:qunit[url=<theme_url>]`.

There are some refactors to how Discourse processes JavaScript that comes with themes/components, and these refactors may break your JS customizations; see https://meta.discourse.org/t/upcoming-core-changes-that-may-break-some-themes-components-april-12/186252?u=osama for details on how you can check if your themes/components are affected and what you need to do to fix them.

This commit also improves theme error handling in Discourse. We will now be able to catch errors that occur when theme initializers are run and prevent them from breaking the site and other themes/components.
2021-04-12 15:02:58 +03:00
Osama Sayegh
2b9ab3a0d9
Revert "FEATURE: Introduce theme/component QUnit tests (#12517)" (#12632)
This reverts commit a53d8d3e61 and 105634435f.

Reverted because the change broke some components. Will be added back in a few days.
2021-04-07 17:45:49 +03:00
Osama Sayegh
105634435f
FIX: Prevent double slashes in Ember templates paths (#12630)
Follow-up to https://github.com/discourse/discourse/pull/12517
2021-04-07 14:08:29 +03:00
Penar Musaraj
5096920500
FEATURE: Implement nonces for Google Tag Manager integration (#12531) 2021-03-26 11:19:31 -04:00
Sam
5a1318e3c3
DEV: upgrade mini_sql (#12465)
* DEV: upgrade mini_sql

Even though we are not planning on using this quite yet, mini_sql now supports
prepared statements.

Would like this upgrade merged so we can do some benchmarking.

Note, this will not work with pg_bouncer, but sites that are not using it
may benefit from the feature.

* implement multisite friendly prepared statements
2021-03-24 08:48:04 +11:00
Rafael dos Santos Silva
fb4486d5f1
FEATURE: Add CSP frame-ancestors support (#12404) 2021-03-22 16:00:25 -03:00
Martin Brennan
49f4c548ef
FEATURE: Bookmark pinning (#12431)
Users can now pin bookmarks from their bookmark list. This will anchor the bookmark to the top of the list, and show a pin icon next to it. This also applies in the nav bookmarks panel. If there are multiple pinned bookmarks they sort by last updated order.
2021-03-22 09:50:22 +10:00
Blake Erickson
44153cde18
FIX: Be able to handle long file extensions (#12375)
* FIX: Be able to handle long file extensions

Some applications have really long file extensions, but if we truncate
them weird behavior ensues.

This commit changes the file extension size from 10 characters to 255
characters instead.

See:

https://meta.discourse.org/t/182824

* Keep truncation at 10, but allow uppercase and dashes
2021-03-17 12:01:29 -06:00
Arpit Jalan
64d2f260a9
FIX: do not convert format for site setting uploads (#12410) 2021-03-16 20:14:41 +05:30
Gerhard Schlager
58c218a4bf
FIX: Remap old S3 endpoints during backup restore (#12276)
It also starts outputting exceptions on the console.
2021-03-03 21:10:09 +01:00
jbrw
a9b6f4d829
FIX - use ImageMagick to determine size of svg images (#12230)
SVG files can have dimensions expressed in inches, centimeters, etc., which may lead to the dimensions being misinterpreted (e.g. “8in” ends up as 8 pixels).

If the file type is `svg`, ask ImageMagick to work out what size the SVG file should be rendered on screen.

NOTE: The `pencil.svg` file was obtained from https://freesvg.org/1534028868, which has placed the file in to the public domain.
2021-03-01 11:44:00 -05:00
Martin Brennan
9f0f801ae3
FIX: Do not mark group_flair images as secure on upload (#12081)
See https://meta.discourse.org/t/secure-media-uploads-breaks-group-flair-image/173671/4

Group flair image uploads definitely do not need to be secure.
2021-02-16 12:34:03 +10:00
jbrw
70050a8ba3
FIX: should_alter_quality should respect png_to_jpg_quality (#12055)
`convert_to_jpeg!` is only called if `convert_png_to_jpeg?` and/or `should_alter_quality?` is true.

`convert_png_to_jpeg?` can be disabled by setting `SiteSetting.png_to_jpg_quality` to 100.

However, `should_alter_quality?` could be true if `SiteSetting.recompress_original_jpg_quality` was lower than the quality of the uploaded file, regardless of file type.

This commits changes `should_alter_quality?` so that uploaded png files will use the `SiteSetting.png_to_jpg_quality` value, rather than ``SiteSetting.recompress_original_jpg_quality` value.
2021-02-12 13:37:35 -05:00
Gerhard Schlager
0b05302cfe FIX: Restoring could fail due to missing path 2021-02-09 17:28:03 +01:00
Gerhard Schlager
4d719725c8
FEATURE: Allow overriding the backup location when restoring via CLI (#12015)
You can use `discourse restore --location=local FILENAME` if you want to restore a backup that is stored locally even though the `backup_location` has the value `s3`.
2021-02-09 16:02:44 +01:00
David Taylor
821bb1e8cb
FEATURE: Rename 'Discourse SSO' to DiscourseConnect (#11978)
The 'Discourse SSO' protocol is being rebranded to DiscourseConnect. This should help to reduce confusion when 'SSO' is used in the generic sense.

This commit aims to:
- Rename `sso_` site settings. DiscourseConnect specific ones are prefixed `discourse_connect_`. Generic settings are prefixed `auth_`
- Add (server-side-only) backwards compatibility for the old setting names, with deprecation notices
- Copy `site_settings` database records to the new names
- Rename relevant translation keys
- Update relevant translations

This commit does **not** aim to:
- Rename any Ruby classes or methods. This might be done in a future commit
- Change any URLs. This would break existing integrations
- Make any changes to the protocol. This would break existing integrations
- Change any functionality. Further normalization across DiscourseConnect and other auth methods will be done separately

The risks are:
- There is no backwards compatibility for site settings on the client-side. Accessing auth-related site settings in Javascript is fairly rare, and an error on the client side would not be security-critical.
- If a plugin is monkey-patching parts of the auth process, changes to locale keys could cause broken error messages. This should also be unlikely. The old site setting names remain functional, so security-related overrides will remain working.

A follow-up commit will be made with a post-deploy migration to delete the old `site_settings` rows.
2021-02-08 10:04:33 +00:00
Martin Brennan
f49e3e5731
DEV: Add security_last_changed_at and security_last_changed_reason to uploads (#11860)
This PR adds security_last_changed_at and security_last_changed_reason to uploads. This has been done to make it easier to track down why an upload's secure column has changed and when. This necessitated a refactor of the UploadSecurity class to provide reasons why the upload security would have changed.

As well as this, a source is now provided from the location which called for the upload's security status to be updated as they are several (e.g. post creator, topic security updater, rake tasks, manual change).
2021-01-29 09:03:44 +10:00
Joffrey JAFFEUX
21d6603245
DEV: followup to 8edd2b38cb to use existing spec (#11830)
This commit also better explains in spec why max_values might be off by one.
2021-01-25 12:04:27 +01:00
Joffrey JAFFEUX
8edd2b38cb
FIX: ensures timeline_lookup includes last tuple (#11829)
A simplified version of the logic used in the function before my fix is as follow:

```ruby
result = []
things = [0,1,2,3]
max_values = 2
every = (things.size.to_f / max_values).ceil

things.each_with_index do |t, index|
  next unless (t % every) === 0
  result << t
end

p result # [0, 2]
# 3 doesn’t get included
```

The problem is that if you get unlucky two times you won't get last tuple(s) and might get a very erroneous date.

Double unlucky:
- last tuple index % computed every !== 0 and you don't get the last tuple
- the last tuple is related to a post with a very different date than the previous tuples (on year difference in our case)
2021-01-25 11:30:59 +01:00
Gerhard Schlager
3b2f6e129a
FEATURE: Add English (UK) as locale (#11768)
* "English" gets renamed into "English (US)"
* "English (UK)" replaces "English"

@discourse-translator-bot keep_translations_and_approvals
2021-01-20 21:32:22 +01:00
Bianca Nenciu
74b95c88ac
FIX: Use ImageMagick to detect animated images (#11702)
This is a fallback when FastImage cannot be used (animated WEBP images).
2021-01-13 19:01:30 +02:00
Bianca Nenciu
499a594754
FIX: Do not downsize or crop GIF images (#10989)
It was a problem because during this operation only the first frame
is kept. This commit removes the alternative solution to check if a GIF
image is animated.
2021-01-12 17:07:07 +02:00
David Taylor
b0088361a4
FIX: Do not include URL query in auto-generated CSP header (#11673) 2021-01-09 13:52:53 +00:00
Mark VanLandingham
521934f163
FIX: Only cache reports with exceptions for 1 minute (#11447) 2020-12-09 10:54:41 -06:00
David Taylor
ed91385c18
DEV: Update DB.after_commit to be compatible with 'real' transactions (#11294)
Previously it matched the behavior of standard ActiveRecord after_commit callbacks. They do not work well within `joinable: false` nested transactions. Now `DB.after_commit` callbacks will only be run when the outermost transaction has been committed.

Tests always run inside transactions, so this also introduces some logic to run callbacks once the test-wrapping transaction is reached.
2020-12-08 11:03:31 +11:00
Bianca Nenciu
2910996feb
FIX: Do not optimize uploaded custom emoji (#11203)
Animated emojis were converted to static images. This commit moves the
responsability on site admins to optimize their animated emojis before
uploading them (gifsicle is no longer used).
2020-11-12 11:22:38 +11:00
Mark VanLandingham
be07853cc1
DEV: Add plugins client/server translation yml file priority structure (#11194)
Plugin client.en.yml and server.en.yml can now be client/server-(1-100).en.yml. 1 is the lowest priority, and 100 is the highest priority. This allows plugins to set their priority higher than other plugins, so that they can override eachothers' translations.
2020-11-11 09:44:01 -06:00
Justin DiRose
09b8a61f65
FEATURE: Add Google Universal Analytics v4 as an option (#11123)
Per Google, sites are encouraged to upgrade from Universal Analytics v3 `analytics.js` to v4 `gtag.js` for Google Analytics tracking. We're giving admins the option to stay on the v3 API or migrate to v4. Admins can change the implementation they're using via the `ga_version` site setting. Eventually Google will deprecate v3, but our implementation gives admins the choice on what to use for now.

We chose this implementation to make the change less error prone, as many site admins are using custom events via the v3 UA API. With the site stetting defaulted to `v3_analytics`, site analytics won't break until the admin is ready to make the migration.

Additionally, in the v4 implementation, we do not enable automatic pageview tracking (on by default in the v4 API). Instead we rely on Discourse's page change API to report pageviews on transition to avoid double-tracking.
2020-11-06 14:15:36 -06:00
Martin Brennan
00c8f520e9
FIX: Do not enable published page if secure media enabled (#11131)
There are issues around displaying images on published pages when secure media is enabled. This PR temporarily makes it appear as if published pages are enabled if secure media is also enabled.
2020-11-06 10:33:19 +10:00
Vinoth Kannan
72810853ea
FIX: strip the trailing slash (/) of cors origins. (#10996)
Strips trailing `/` from global settings
Provides a validation for site settings to ensure a trailing `/` is not added
2020-10-29 13:01:06 +11:00
jbrw
aeb24bd4b5
FIX - don't attempt to optimized animated images (#11031)
* FIX - don't attempt to optimized animated images

* ensure_safe_paths before calling ImageMagick
2020-10-26 15:10:19 -04:00
jbrw
2bcca46cc5
FEATURE - ImageMagick jpeg quality (#11004)
* FEATURE - Add SiteSettings to control JPEG image quality

`recompress_original_jpg_quality` - the maximum quality of a newly
uploaded file.

`image_preview_jpg_quality` - the maximum quality of OptimizedImages
2020-10-23 12:38:28 -04:00
Gerhard Schlager
d5ef6188ed
PERF: Disable Sidekiq only during database restore (#10857)
It pauses Sidekiq, clears Redis (namespaced to the current site), clears Sidekiq jobs for the current site, restores the database and unpauses Sidekiq. Previously it stayed paused until the end of the restore.

Redis is cleared because we don't want any old data lying around (e.g. old Sidekiq jobs). Most data in Redis is prefixed with the name of the multisite, but Sidekiq jobs in a multisite are all stored in the same keys. So, deleting those jobs requires a little bit more logic.
2020-10-16 15:19:02 +02:00
Gerhard Schlager
1febf11362 FIX: Backup didn't work anymore after a running backup was canceled 2020-10-13 19:48:53 +02:00
Justin DiRose
8c77b84aac
Revert "FEATURE: Upgrade analytics.js to gtag.js (#10893)" (#10910)
Reverting due to a few unforseen issues with customizations.
2020-10-13 12:20:41 -05:00
Justin DiRose
f4034226c2
FEATURE: Upgrade analytics.js to gtag.js (#10893)
Per Google, sites are encouraged to upgrade from `analytics.js` to `gtag.js` for Google Analytics tracking. This commit updates core Discourse to use the new `gtag.js` API Google is asking sites to use. This API has feature parity with `analytics.js` but does not use trackers.
2020-10-13 11:24:06 -05:00
Arpit Jalan
4326827a4e
FIX: second factor cannot be enabled if SSO is enabled (#10880)
* FIX: second factor cannot be enabled if SSO is enabled

If `enable_sso` setting is enabled then admin should not be able to
enable `enforce_second_factor` setting as that will lock users out.

Co-authored-by: Robin Ward <robin.ward@gmail.com>
2020-10-09 22:36:38 +05:30
Jarek Radosz
891987a284
DEV: Recover missing files of existing uploads (#10757)
UploadRecovery only worked on missing Upload records. Now it also works with existing ones that have an invalid_etag status.

The specs (first that test the S3 path) are a bit of stub-a-palooza, but that's how much this class interacts with the the outside world. 🤷‍♂️
2020-10-01 14:54:45 +02:00
Jarek Radosz
e00abbe1b7 DEV: Clean up S3 specs, stubs, and helpers
Extracted commonly used spec helpers into spec/support/uploads_helpers.rb, removed unused stubs and let definitions. Makes it easier to write new S3-related specs without copy and pasting setup steps from other specs.
2020-09-28 12:02:25 +01:00
Penar Musaraj
c664ba8fa8
DEV: Fix flaky UploadSecurity spec 2020-09-18 14:36:41 -04:00
Martin Brennan
14b324e5ed
FIX: Provide better API for registering custom upload public types (#10697)
With secure media and the UploadSecurity class, we need a nice way for plugins to register custom upload types that should be considered public and never secure.
2020-09-18 11:54:33 +10:00
Krzysztof Kotlarek
cb58cbbc2c
FEATURE: allow to extend topic_eager_loads in Search (#10625)
This additional interface is required by encrypt plugin
2020-09-14 11:58:28 +10:00
Gerhard Schlager
ac70c48be4 FIX: Prevent "uploads are missing in S3" alerts after restoring a backup
After restoring a backup it takes up to 48 hours for uploads stored on S3 to appear in the S3 inventory. This change prevents alerts about missing uploads by preventing the EnsureS3UploadsExistence job from running in the first 48 hours after a restore. During the restore it  deletes the count of missing uploads from the PluginStore, so that an alert isn't triggered by an old number.
2020-09-10 21:37:48 +02:00
Martin Brennan
431bd84dec
FIX: Make deleted topic post bookmarks more resilient (#10619)
This PR ensures that new bookmarks cannot be created for deleted posts and topics, and also makes sure that if a bookmark was created and then the topic deleted that the show topic page does not error from trying to retrieve the bookmark reminder at.
2020-09-07 14:52:14 +10:00
Dan Ungureanu
38c9c87128
FIX: Add to tags result set only visible tags (#10580) 2020-09-02 13:24:40 +03:00
Jarek Radosz
188d224303
DEV: Restore UploadRecovery specs (#10543)
Originally disabled in 0c0192e. Upload specs now use separate paths for each spec worker.

Fixes an issue in UploadRecovery#recover_from_local – it didn't take into account the testing infix (e.g. test_0) in the uploads/tombstone paths.
2020-08-27 15:57:10 +02:00
Martin Brennan
e8a842ab8c
FIX: Stop category logo + background being marked secure (#10513)
Meta topic: https://meta.discourse.org/t/secure-media-uploads-breaks-category-logos/161693

Category backgrounds and logos are public uploads and should not be marked as secure.

I also discovered that a lot of the UploadSecurity specs for public types were returning false positives; this has been fixed.
2020-08-24 17:12:28 +10:00
Gerhard Schlager
f51ccea028
FIX: Backups should use relative paths for local uploads
This also ensures that restoring a backup works when it was created with the wrong upload paths in the time between ab4c0a4970 (shortly after v2.6.0.beta1) and this fix.
2020-08-21 15:22:28 +02:00
Mark VanLandingham
a3c0d4a8b5
FIX: Error message when setting enforce 2fa with social logins (#10479) 2020-08-19 13:16:31 -05:00
David Taylor
8ac85f54fb
DEV: Remove blob: workers from CSP (#10440)
Ace editor is reconfigured to load workers directly from their JS URL. Workers must be on the same origin as the site, so they will not use the CDN.
2020-08-14 18:15:30 +01:00
David Taylor
3cf93e9a8f
Revert "DEV: Disable ACE editor worker blobs"
This reverts commit d5463d2a4d.

With S3 assets, CORS prevents loading worker assets directly.
2020-08-14 14:12:11 +01:00
David Taylor
a5608025aa
FIX: Add script asset locations to worker-src CSP directives
We no longer need :blob worker src since d5463d2a. But we do want to allow workers to be loaded from all our existing script-src options.
2020-08-14 12:31:50 +01:00
Martin Brennan
95b71b35d6
FEATURE: IMAP delete email sync for group inboxes (#10392)
Adds functionality to reflect topic delete in Discourse to IMAP inbox (Gmail only for now) and reflecting Gmail deletes in Discourse.

Adding lots of tests, various refactors and code improvements.

When Discourse topic is destroyed in PostDestroyer mark the topic incoming email as imap_sync: true, and do the opposite when post is recovered.
2020-08-12 10:16:26 +10:00
Penar Musaraj
86cb5803ba
Convert HEIC uploads to JPG by default (#10395) 2020-08-07 11:17:50 -04:00
Martin Brennan
5a3494b1e1
FIX: IMAP archive fix and group list mailbox code unification (#10355)
* Fixed an issue I introduced in the last PR where I am just archiving everything regardless of whether it is actually archived in Discourse man_facepalming
* Refactor group list_mailboxes IMAP code to use providers, add specs, and add provider code to get the correct prodivder
2020-08-04 14:19:57 +10:00
Martin Brennan
2920988b3a
FIX: IMAP sync email update uniqueness across groups and minor improvements (#10332)
Adds a imap_group_id column to IncomingEmail to deal with an issue where we were trying to update emails in the mailbox, calling IncomingEmail.where(imap_sync: true). However UID and UIDVALIDITY could be the same across accounts. So if group A used IMAP details for Gmail account A, and group B used IMAP details for Gmail account B, and both tried to sync changes to an email with UID of 3 (e.g. changing Labels), one account could affect the other. This even applied to Archiving!

Also in this PR:

* Fix error occurring if we do a uid_fetch and no emails are returned
* Allow for creating labels within the target mailbox (previously we would not do this, only use existing labels)
* Improve consistency for log messages
* Add specs for generic IMAP provider (Gmail specs still to come)
* Add custom archiving support for Gmail
* Only use Message-ID for uniqueness of IncomingEmail if it was generated by us
* Various refactors and improvements
2020-08-03 13:10:17 +10:00
Martin Brennan
9e5b213089
FIX: Ensure topic user bookmarked synced on bookmark auto-delete (#10323)
For the following conditions, the TopicUser.bookmarked column was not updated correctly:

* When a bookmark was auto-deleted because the reminder was sent
* When a bookmark was auto-deleted because the owner of the bookmark replied to the topic

This adds another migration to fix the out-of-sync column and also some refactors to BookmarkManager to allow for more of these delete cases. BookmarkManager is used instead of directly destroying the bookmark in PostCreator and BookmarkReminderNotificationHandler.
2020-07-29 09:43:32 +10:00
Krzysztof Kotlarek
e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Guo Xiang Tan
181c4eb760 PERF: Avoid parsing Post#cooked with Nokogiri for every search. 2020-07-24 10:43:09 +08:00
Penar Musaraj
7559758e10
FEATURE: Support converting HEIF images to JPEG (#10079) 2020-07-22 21:40:09 -04:00
Martin Brennan
41b43a2a25
FEATURE: Add "delete on owner reply" bookmark functionality (#10231)
This adds an option to "delete on owner reply" to bookmarks. If you select this option in the modal, then reply to the topic the bookmark is in, the bookmark will be deleted on reply.

This PR also changes the checkboxes for these additional bookmark options to an Integer column in the DB with a combobox to select the option you want.

The use cases are:

* Sometimes I will bookmark the topics to read it later. In this case we definitely don’t need to keep the bookmark after I replied to it.
* Sometimes I will read the topic in mobile and I will prefer to reply in PC later. Or I may have to do some research before reply. So I will bookmark it for reply later.
2020-07-21 10:00:39 +10:00
Martin Brennan
716ccf7fe4
FIX: Bookmark search fixes (#10239)
* Remove unneeded bookmark name index.
* Change bookmark search query to use post_search_data. This allows searching on topic title and post content
* Tweak the style/layout of the bookmark list so the search looks better and the whole page fits better on mobile.
2020-07-17 15:55:07 +10:00
Martin Brennan
bcc80e0ea8
FEATURE: Add search to user bookmark list (#10230)
User bookmarks can now be searched by name or post raw content. The q querystring param is hooked up from the Ember router as well.
2020-07-14 14:43:41 +10:00
Kane York
79b52b1e9a
DEV: Add SVG tests for 31e31ef44 (#10205) 2020-07-09 14:02:25 -07:00
Martin Brennan
6be7a66ba7
FIX: Cap bookmark name at 100 chars and truncate existing names (#10189)
We have a couple of examples of enormous amounts of text being entered in the name column of bookmarks. This is not desirable...it is just meant to be a short note / reminder of why you bookmarked this.

This PR caps the column at 100 characters and truncates existing names in the database to 100 characters.
2020-07-08 17:19:01 +10:00
Gerhard Schlager
859d9b75a7 FIX: Restoring backup from PG12 could fail on PG10
The `EXECUTE FUNCTION` syntax for `CREATE TRIGGER` statements was introduced in PostgreSQL 11. We need to replace `EXECUTE FUNCTION` with `EXECUTE PROCEDURE` in order to be able to restore backups created with PG12 on PG10.
2020-06-16 16:04:14 +02:00
Jarek Radosz
3d55f2e3b7
FIX: Improvements and fixes to the image downsizing script (#9950)
Fixed bugs, added specs, extracted the upload downsizing code to a class, added support for non-S3 setups, changed it so that images aren't downloaded twice.

This code has been tested on production and successfully resized ~180k uploads.

Includes:

* DEV: Extract upload downsizing logic
* DEV: Add support for non-S3 uploads
* DEV: Process only images uploaded by users
* FIX: Incorrect usage of `count` and `exist?` typo
* DEV: Spec S3 image downsizing
* DEV: Avoid downloading images twice
* DEV: Update filesizes earlier in the process
* DEV: Return false on invalid upload
* FIX: Download images that currently above the limit (If the image size limit is decreased, then there was no way to resize those images that now fall outside the allowed size range)
* Update script/downsize_uploads.rb (Co-authored-by: Régis Hanol <regis@hanol.fr>)
2020-06-11 14:47:59 +02:00
Andrew Schleifer
74d28a43d1
new S3 backup layout (#9830)
* DEV: new S3 backup layout

Currently, with $S3_BACKUP_BUCKET of "bucket/backups", multisite backups
end up in "bucket/backups/backups/dbname/" and single-site will be in
"bucket/backups/".

Both _should_ be in "bucket/backups/dbname/"

- remove MULTISITE_PREFIX,
- always include dbname,
- method to move to the new prefix
- job to call the method

* SPEC: add tests for `VacateLegacyPrefixBackups` onceoff job.

Co-authored-by: Vinoth Kannan <vinothkannan@vinkas.com>
2020-05-29 00:28:23 +05:30
Michael Brown
d9a02d1336
Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eee.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
Jeff Atwood
20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6f, reversing
changes made to 2660c2e21d.
2020-05-22 20:25:56 -07:00
Guo Xiang Tan
96c02caba7
DEV: Change use of Redis flushall to flushdb.
FLUSHALL removes all keys from all databases. Instead we only want to
remove keys from the current Redis database.
2020-05-19 10:20:00 +08:00
Sam Saffron
862773ec83
FIX: do not remove stop words when using English locale
PG already handles English stop words, the list in cppjieba is
bigger than the list PG uses, which in turn causes confusion cause
words such as "volume" are stripped using cppijieba stop word list

We will follow up with another commit here to apply the Chinese
word stopwords, but for now to eliminate the confusion we are
skipping applying the stopword list when the dictionary in PG is
in English.
2020-05-18 10:54:56 +10:00
Guo Xiang Tan
858735b5da
DEV: Fix broken test due to 9ab5801a1b. 2020-05-15 15:00:04 +08:00
Martin Brennan
22dffe6f69 FIX: Do not allow null options for bookmark manager 2020-05-08 15:24:59 +00:00
Martin Brennan
6fb0f36ce1
FEATURE: Optionally delete bookmark when reminder sent (#9637)
We now show an options gear icon next to the bookmark name.

When expanded we show the "delete bookmark when reminder sent" option. The value of this checkbox is saved in local storage for the user.

If this is ticked, when a reminder is sent for the bookmark the bookmark itself is deleted. This is so people can use the reminder functionality by itself.

Also remove the blue alert reminder section from the "Edit Bookmark" modal as it just added clutter, because the user can already see they had a reminder set:

Adds a default false boolean column `delete_when_reminder_sent` to bookmarks.
2020-05-07 13:37:39 +10:00
Gerhard Schlager
ec2f3169ff FIX: Using the default_locale in locale fallbacks caused problems
Locale files get precompiled after deployment and they contained translations from the `default_locale`. That's especially bad in multisites, because the initial `default_locale` is `en_US`. Sites where the `default_locale` isn't `en_US` could see missing translations. The same thing could happen when users are allowed to chose a different locale.

This change simplifies the logic by not using the `default_locale` in the locale chain. It always falls back to `en` in case of missing translations.
2020-05-06 22:59:07 +02:00
Martin Brennan
fa572d3a7a
DEV: Remove all code referencing at_desktop bookmark reminders (#9650)
We have found no need for these reminder types, so we are removing the code for them.
2020-05-06 15:22:43 +10:00
Robin Ward
612284cef3
DEV: Remove Discourse.RAW_TEMPLATES (#9630)
We were sharing `Discourse` both as an application object and a
namespace which complicated things for Ember CLI. This patch
moves raw templates into `__DISCOURSE_RAW_TEMPLATES` and adds
a couple helper methods to create/remove them.
2020-05-05 12:15:03 -04:00
David Taylor
03818e642a
FEATURE: Include optimized thumbnails for topics (#9215)
This introduces new APIs for obtaining optimized thumbnails for topics. There are a few building blocks required for this:

- Introduces new `image_upload_id` columns on the `posts` and `topics` table. This replaces the old `image_url` column, which means that thumbnails are now restricted to uploads. Hotlinked thumbnails are no longer possible. In normal use (with pull_hotlinked_images enabled), this has no noticeable impact

- A migration attempts to match existing urls to upload records. If a match cannot be found then the posts will be queued for rebake

- Optimized thumbnails are generated during post_process_cooked. If thumbnails are missing when serializing a topic list, then a sidekiq job is queued

- Topic lists and topics now include a `thumbnails` key, which includes all the available images:
   ```
   "thumbnails": [
   {
     "max_width": null,
     "max_height": null,
     "url": "//example.com/original-image.png",
     "width": 1380,
     "height": 1840
   },
   {
     "max_width": 1024,
     "max_height": 1024,
     "url": "//example.com/optimized-image.png",
     "width": 768,
     "height": 1024
   }
   ]
  ```

- Themes can request additional thumbnail sizes by using a modifier in their `about.json` file:
   ```
    "modifiers": {
      "topic_thumbnail_sizes": [
        [200, 200],
        [800, 800]
      ],
      ...
  ```
  Remember that these are generated asynchronously, so your theme should include logic to fallback to other available thumbnails if your requested size has not yet been generated

- Two new raw plugin outlets are introduced, to improve the customisability of the topic list. `topic-list-before-columns` and `topic-list-before-link`
2020-05-05 09:07:50 +01:00
Krzysztof Kotlarek
9bff0882c3
FEATURE: Nokogumbo (#9577)
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
David Taylor
5901717531
DEV: Allow DB.after_commit to be used outside of a transaction
In this case, it will execute the given block immediately
2020-05-04 09:42:41 +01:00
Martin Brennan
bcc9ad6f57
FIX: Bookmark UI tweaks (#9604)
* When hovering over the bookmark icon for a post, show the name of the bookmark at the end of the tooltip _if_ it has been set.
* Order bookmarks by `updated_at DESC` in the user list and show that instead of created at.
2020-05-01 16:14:20 +10:00
David Taylor
ba616ffb50
DEV: Use a tmp directory for storing uploads in tests (#9554)
This avoids development-mode upload files from polluting the test environment
2020-04-28 14:03:04 +01:00
Martin Brennan
5108cf8ddf
FIX: Topic user bookmarked column logic was not correct (#9563)
Make sure the topic_user.bookmarked column is set correctly when user bookmarks/unbookmarks any post in a topic. For example, you bookmarked a post in the topic that was not the OP, the bookmark icon in the topic list would not be shown. Same if deleting a bookmark for the last bookmarked post in a topic, the bookmark icon in the topic list would not be removed.

Previously this was only setting it to true if bookmarking the OP/topic, which was not correct -- we want to show the icon on the topic list if any post is bookmarked.
Also set to false if unbookmarking the last bookmarked post in the topic.
Also in this PR is a migration to correct any out of sync topic_user.bookmarked columns, based on the new logic.
2020-04-28 16:19:25 +10:00
David Taylor
f95609ae23
FIX: Add additional checks for automatic theme script CSP
- Skip absolute URLs
- Skip CDN URLs
- Skip invalid URLs
- Properly format protocol-less URLs
2020-04-27 15:56:29 +01:00
David Taylor
8a112b7464
DEV: Automatically extend CSP when themes link to external scripts (#9531) 2020-04-24 09:47:01 +01:00
Martin Brennan
628ba9d1e2
FEATURE: Promote bookmarks with reminders to core functionality (#9369)
The main thrust of this PR is to take all the conditional checks based on the `enable_bookmarks_with_reminders` away and only keep the code from the `true` path, making bookmarks with reminders the core bookmarks feature. There is also a migration to create `Bookmark` records out of `PostAction` bookmarks for a site.

### Summary

* Remove logic based on whether enable_bookmarks_with_reminders is true. This site setting is now obsolete, the old bookmark functionality is being removed. Retain the setting and set the value to `true` in a migration.
* Use the code from the rake task to create a database migration that creates bookmarks from post actions.
* Change the bookmark report to read from the new table.
* Get rid of old endpoints for bookmarks
* Link to the new bookmarks list from the user summary page
2020-04-22 13:44:19 +10:00
Martin Brennan
e18aeb799e
FIX: Make sure reminder_type is parsed on bookmark update (#9503)
Otherwise we are trying to update the reminder type with a string which often evaluates to 0 (At Desktop) which causes reminders to come through early.
2020-04-22 10:44:04 +10:00
Martin Brennan
344ef5226c
FEATURE: Edit bookmark reminders from post and explicit delete button (#9455)
There is now an explicit "Delete Bookmark" button in the edit modal. A confirmation is shown before deleting.

Along with this, when the bookmarked post icon is clicked the modal is now shown instead of just deleting the bookmark. Also, the "Delete Bookmark" button from the user bookmark list now confirms the action.

Add a `d d` shortcut in the modal to delete the bookmark.
2020-04-20 13:30:04 +10:00
Martin Brennan
8f0544137a
FEATURE: Allow editing bookmark reminders (#9437)
Users can now edit the bookmark name and reminder time from their list of bookmarks.

We use "Custom" for the date and time in the modal because if the user set a reminder for "tomorrow" then edit the reminder "tomorrow", the definition of what "tomorrow" is has changed.
2020-04-17 11:08:07 +10:00
Martin Brennan
51672b9121
FIX: Minor bookmark with reminder issue cleanup (#9436)
* Count user summary bookmarks from new Bookmark table if bookmarks with reminders enabled
* Update topic user bookmarked column when new topic bookmark changed
* Make in:bookmarks search work with new bookmarks
* Fix batch inserts for bookmark rake task (and thus migration). We were only inserting one bookmark at a time, completely defeating the purpose of batching!
2020-04-16 11:32:21 +10:00
Penar Musaraj
724d2e99de
DEV: Only include "report-sample" CSP directive when reporting is enabled (#9337) 2020-04-02 11:16:38 -04:00
Neil Lalonde
90fcede832
FIX: replace default welcome topic post with new value from wizard
Previously the text entered in the wizard would be prepended onto the
default first paragraph.
2020-04-01 15:42:45 -04:00
Jarek Radosz
7ff889574d
DEV: Add rubocop-rspec (#9288)
This adds rubocop-rspec, and enables some cops that were either already passing or are passing now, after fixing them in this commit.

Some new cops are disabled for now, with annotation: "TODO" or "To be decided". Those either need to be discussed first, or require manual changes, or the number of found and fixed offenses is too large to bundle them up in a single PR.

Includes:

* DEV: Update rubocop's `TargetRubyVersion` to 2.6
* DEV: Enable RSpec/VoidExpect
* DEV: Enable RSpec/SharedContext
* DEV: Enable RSpec/EmptyExampleGroup (Removed an obsolete empty spec file)
* DEV: Enable RSpec/ItBehavesLike
* DEV: Remove RSpec/ScatteredLet (It's too strict, as it doesn't recognize fab! as a let-like)
* DEV: Remove RSpec/MultipleExpectations
2020-03-27 17:35:40 +01:00
Jarek Radosz
d21d80198c
DEV: Update rubocop-discourse (#9270)
Includes:
* DEV: Use `eq_time` matcher
2020-03-26 16:32:41 +01:00
Martin Brennan
097851c135
FIX: Change secure media to encompass attachments as well (#9271)
If the “secure media” site setting is enabled then ALL files uploaded to Discourse (images, video, audio, pdf, txt, zip etc. etc.) will follow the secure media rules. The “prevent anons from downloading files” setting will no longer have any bearing on upload security. Basically, the feature will more appropriately be called “secure uploads” instead of “secure media”.

This is being done because there are communities out there that would like all attachments and media to be secure based on category rules but still allow anonymous users to download attachments in public places, which is not possible in the current arrangement.
2020-03-26 07:16:02 +10:00
David Taylor
5db41cd578
SECURITY: Respect topic permissions when loading bookmark metadata
Co-authored-by: Martin Brennan <martin@discourse.org>
Co-authored-by: Sam Saffron <sam.saffron@gmail.com>
2020-03-23 11:30:48 +00:00
David Taylor
19814c5e81
FIX: Allow CSP to work correctly for non-default hostnames/schemes (#9180)
- Define the CSP based on the requested domain / scheme (respecting force_https)
- Update EnforceHostname middleware to allow secondary domains, add specs
- Add URL scheme to anon cache key so that CSP headers are cached correctly
2020-03-19 19:54:42 +00:00
Martin Brennan
0cd502a558
FIX: N1 issues for bookmark list (#9236)
* Preload custom fields for BookmarkQuery and add preload callback. Copy TopicQuery preload methodology to allow plugins to preload data for the BookmarkQuery. This fixes assigned plugin custom fields N1
* Include topic tags in initial query to avoid tags N1

Related: discourse/discourse-assign#63
2020-03-19 15:48:23 +10:00
Kane York
8e98f39d9f Add spec for CSP subfolder 2020-03-17 19:09:06 -07:00
Gerhard Schlager
8022e51179 FIX: Failed to restore backups from versions without translation overrides
Rails calls I18n.translate during initialization and by default translation overrides are used. Database migrations would fail if the system tried to migrate from an old version that didn't have the `translation_overrides` table with all its columns yet.

This makes restoring really old backups work again. Running `DISABLE_TRANSLATION_OVERRIDES=1 rake db:migrate` will allow you to upgrade such an old database as well.
2020-03-14 00:00:22 +01:00
David Taylor
3723c64257
DEV: Correct references to theme flags
Followup to d1474e94
2020-03-13 16:45:55 +00:00
Martin Brennan
af92444948
FIX: Make sure bookmark serializer works with deleted topics + posts (#9195) 2020-03-13 10:44:39 +10:00
Martin Brennan
e1eb5fb9b3
FEATURE: MVP Bookmarks with reminders user list changes (#8999)
* This PR changes the user activity bookmarks stream to show a new list of bookmarks based on the Bookmark record.
* If a bookmark has a name or reminder it will be shown as metadata above the topic title in the list
* The categories, tags, topic status, and assigned show for each bookmarked post based on the post topic
* Bookmarks can be deleted from the [...] menu in the list
* As well as this, the list of bookmarks from the quick access panel is now drawn from the Bookmarks table for a user:
* All of this new functionality is gated behind the enable_bookmarks_with_reminders site setting
The /bookmarks/ route now redirects directly to /user/:username/activity/bookmarks-with-reminders
* The structure of the Ember for the list of bookmarks is not ideal, this is an MVP PR so we can start testing this functionality internally. There is a little repeated code from topic.js.es6. There is an ongoing effort to start standardizing these lists that will be addressed in future PRs.
* This PR also fixes issues with feature detection for at_desktop bookmark reminders
2020-03-12 15:20:56 +10:00
Martin Brennan
849631188f
FEATURE: Allow custom date + time for bookmark reminders (#9185)
A custom date and time can now be selected for a bookmark reminder

The reminder will not happen at the exact time but rather at the next 5 minute interval of the bookmark reminder schedule.

This PR also fixes issues with bulk deleting topic bookmarks.
2020-03-12 10:52:15 +10:00
Martin Brennan
793f39139a
FEATURE: Send notifications for time-based and At Desktop bookmark reminders (#9071)
* This PR implements the scheduling and notification system for bookmark reminders. Every 5 minutes a schedule runs to check any reminders that need to be sent before now, limited to **300** reminders at a time. Any leftover reminders will be sent in the next run. This is to avoid having to deal with fickle sidekiq and reminders in the far-flung future, which would necessitate having a background job anyway to clean up any missing `enqueue_at` reminders.

* If a reminder is sent its `reminder_at` time is cleared and the `reminder_last_sent_at` time is filled in. Notifications are only user-level notifications for now.

* All JavaScript and frontend code related to displaying the bookmark reminder notification is contained here. The reminder functionality is now re-enabled in the bookmark modal as well.

* This PR also implements the "Remind me next time I am at my desktop" bookmark reminder functionality. When the user is on a mobile device they are able to select this option. When they choose this option we set a key in Redis saying they have a pending at desktop reminder. The next time they change devices we check if the new device is desktop, and if it is we send reminders using a DistributedMutex. There is also a job to ensure consistency of these reminders in Redis (in case Redis drops the ball) and the at desktop reminders expire after 20 days.

* Also in this PR is a fix to delete all Bookmarks for a user via `UserDestroyer`
2020-03-12 10:16:00 +10:00
Robin Ward
a3f0543f99
Support for transpiling .js files (#9160)
* Remove some `.es6` from comments where it does not matter

* Use a post processor for transpilation

This will allow us to eventually use the directory structure to
transpile rather than the extension.

* FIX: Some errors and clean up in confirm-new-email

It would throw an error if the webauthn element wasn't present.
Also I changed things so that no-module is not explicitly
referenced.

* Remove `no-module`

Instead we allow a magic comment: `// discourse-skip-module` to prevent
the asset pipeline from creating a module.

* DEV: Enable babel transpilation based on directory

If it's in `app/assets/javascripts/dicourse` it will be transpiled
even without the `.es6` extension.

* REFACTOR: Remove Tilt/ES6ModuleTranspiler
2020-03-11 09:43:55 -04:00
David Taylor
d1474e94a1
FEATURE: Allow themes to specify modifiers in their about.json file (#9097)
There are three modifiers:
- serialize_topic_excerpts (boolean)
- csp_extensions (array of strings)
- svg_icons (array of strings)

When multiple themes are active, the values will be combined. The combination method varies based on the setting. CSP/SVG arrays will be combined. serialize_topic_excerpts will use `Enumerable#any`.
2020-03-11 13:30:45 +00:00
Mark VanLandingham
475f545557
DEV: Test theme js compiler append_raw_template (#8950)
* DEV: Test for append_raw_template

* remove empty line
2020-03-06 11:35:52 -05:00
Robin Ward
a5f61729e0 Revert "Revert "FIX: Don't allow people to clear the upload bucket while it's enabled""
This reverts commit d4fc76b335.
2020-03-06 09:35:55 -05:00
Robin Ward
d4fc76b335 Revert "FIX: Don't allow people to clear the upload bucket while it's enabled"
This reverts commit 4bb8db024c.
2020-03-05 16:29:59 -05:00
Robin Ward
4bb8db024c FIX: Don't allow people to clear the upload bucket while it's enabled 2020-03-05 16:04:52 -05:00
Gerhard Schlager
8fa8bab9ff FIX: Don't optimize icons during db:migrate when restoring backup
Uploads are extracted after the DB migration, so this could lead to a failure during the restore. Site icons get optimized after extracting uploads.
2020-03-04 16:59:49 +01:00
Martin Brennan
0388653a4d
DEV: Upload and secure media retroactive rake task improvements (#9027)
* Add uploads:sync_s3_acls rake task to ensure the ACLs in S3 are the correct (public-read or private) setting based on upload security

* Improved uploads:disable_secure_media to be more efficient and provide better messages to the user.

* Rename uploads:ensure_correct_acl task to uploads:secure_upload_analyse_and_update as it does more than check the ACL

* Many improvements to uploads:secure_upload_analyse_and_update

* Make sure that upload.access_control_post is unscoped so deleted posts are still fetched, because they still affect the security of the upload.

* Add escape hatch for capture_stdout in the form of RAILS_ENABLE_TEST_STDOUT. If provided the capture_stdout code will be ignored, so you can see the output if you need.
2020-03-03 10:03:58 +11:00
Martin Brennan
3af2670bd5
FIX: Consider webp a supported image format for upload (#9015)
* Also fixes an issue where if webp was a downloaded hotlinked
  image and then secure + sent in an email, it was not being
  redacted because webp was not a supported media format in
  FileHelper
* Webp originally removed as an image format in
  https://github.com/discourse/discourse/pull/6377
  and there was a spec to make sure a .bin webp
  file did not get renamed from its type to webp.

  However we want to support webp images now to make
  sure they are properly redacted if secure media is
  on, so change the example in the spec to use tiff,
  another banned format, instead
2020-02-21 13:08:01 +10:00
Martin Brennan
04df3bd46d
FIX: Only mark attachments as secure media if SiteSetting.secure_media? (#9009)
* Attachments (non media files) were being marked as secure if just
SiteSetting.prevent_anons_from_downloading_files was enabled. this
was not correct as nothing should be marked as actually "secure" in
the DB without that site setting enabled
* Also add a proper standalone spec file for the upload security class
2020-02-21 09:35:16 +10:00
Jarek Radosz
53529a3427
DEV: Upgrade Ember to version 3.12.2 (#8753)
* DEV: Use Ember 3.12.2
* Add Ember version to ThemeField's DEPENDENT_CONSTANTS
* DEV: Use `id` instead of `elementId` (See: https://github.com/emberjs/ember.js/issues/18147)
* FIX: Don't leak event listeners (bug introduced in 999e2ff)
2020-02-05 14:51:00 +01:00
Martin Brennan
1b3b0708c0
FEATURE: Update upload security status on post move, topic conversion, category change (#8731)
Add TopicUploadSecurityManager to handle post moves. When a post moves around or a topic changes between categories and public/private message status the uploads connected to posts in the topic need to have their secure status updated, depending on the security context the topic now lives in.
2020-01-23 12:01:10 +10:00
Martin Brennan
2583aedd42
FIX: Change additional public uploads to not be secure (#8738)
Custom emoji, profile background, and card background were being set to secure, which we do not want as they are always in a public context and result in a 403 error from the ACL if linked directly.
2020-01-17 13:16:27 +10:00
Gerhard Schlager
f216c6d60b FEATURE: Drop "backup" schema 7 days after restore
The "backup" schema is used to rollback a failed restore. It isn't useful after a longer period of time and turns into a waste of disk space.
2020-01-16 17:48:47 +01:00
Martin Brennan
7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
Gerhard Schlager
5e3fc31f2c DEV: Less hacky way of rolling back DB changes
Some specs use psql to test database restores and dropping the table after the test needs to happen outside of rspec because of transactions. The previous attempt lead to some changes to be stored in the test database.
2020-01-15 23:37:42 +01:00
Gerhard Schlager
68a7ae3091 REFACTOR: Simplify backup version check
Adds specs for inalid version number in metadata file.
Follow-up to c3cd2389fe
2020-01-15 23:37:40 +01:00
Martin Brennan
66f2db4ea4 SECURITY: 2FA with U2F / TOTP 2020-01-15 11:27:12 +01:00
Gerhard Schlager
e474cda321 REFACTOR: Restoring of backups and migration of uploads to S3 2020-01-14 11:41:35 +01:00
Martin Brennan
cb660ef952 SECURITY: Improve second factor auth logic 2020-01-10 10:45:56 +10:00
Martin Brennan
c031434b86
FIX: Catch error when unknown COSE algorithm is supplied for Security Key (#8649)
Added a fix to gracefully error with a Webauthn::SecurityKeyError if somehow a user provides an unkown COSE algorithm when logging in with a security key.

If `COSE::Algorithm.find` returns nil we now fail gracefully and log the algorithm used along with the user ID and the security key params for debugging, as this will help us find other common algorithms to implement for webauthn
2020-01-02 10:14:22 +10:00
David Taylor
bc4c40abd4
DEV: Remove unsafe-eval from development CSP (#8569)
- Refactor source_url to avoid using eval in development
- Precompile handlebars in development
- Include template compilers when running qunit
- Remove unsafe-eval in development CSP
- Include unsafe-eval only for qunit routes in development
2019-12-30 12:17:12 +00:00
Gerhard Schlager
2e8c23f0d0 Fix spec 2019-12-13 12:40:52 +01:00
dependabot-preview[bot]
863d3f89de Build(deps): Bump aws-sdk-s3 from 1.36.1 to 1.59.0 (#8502)
* Build(deps): Bump aws-sdk-s3 from 1.36.1 to 1.59.0

Bumps [aws-sdk-s3](https://github.com/aws/aws-sdk-ruby) from 1.36.1 to 1.59.0.
- [Release notes](https://github.com/aws/aws-sdk-ruby/releases)
- [Changelog](https://github.com/aws/aws-sdk-ruby/blob/master/gems/aws-sdk-s3/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-ruby/compare/v1.36.1...v1.59.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Dependabot/bundler/aws sdk s3 1.59.0 (#8532)
2019-12-11 10:18:11 -08:00
Dan Ungureanu
4e130f1e03
SECURITY: Remove event handlers from SVG files 2019-12-11 17:04:50 +02:00
Martin Brennan
afb5533581
FEATURE: Add timezone to core user_options (#8380)
* Add timezone to user_options table

* Also migrate existing timezone values from UserCustomField,
  which is where the discourse-calendar plugin is storing them

* Allow user to change their core timezone from Profile

* Auto guess & set timezone on login & invite accept & signup

* Serialize user_options.timezone for group members. this is so discourse-group-timezones can access the core user timezone, as it is being removed in discourse-calendar.

* Annotate user_option with timezone

* Validate timezone values
2019-11-25 10:49:27 +10:00
Martin Brennan
5ad6dc01c1
FEATURE: Support RS256 algorithm for webauthn (#8385)
* Add support for RS256 webauthn algorithm (-257)
2019-11-21 15:45:36 +10:00
Martin Brennan
d45bd7f131
FIX: Abort CensoredWordsValidator early if censored_words_regexp nil (#8375)
* Abort CensoredWordsValidator early if censored_words_regexp nil. Sometimes censored_words_regex can end up nil, erroring the validator. This handles the nil condition and also adds a spec for the validator
2019-11-20 20:05:06 +10:00
Martin Brennan
02cb01406e
FIX: Allow secure uploads if global s3 setting active and enable_s3_uploads validations (#8373)
The secure media functionality relied on `SiteSetting.enable_s3_uploads?` which, as we found in dev, did not take into account global S3 settings via `GlobalSetting.use_s3?`. We now use `SiteSetting.Upload.enable_s3_uploads` instead to be more consistent.

Also, we now validate `enable_s3_uploads` changes, because if `GlobalSetting.use_s3?` is true users should NOT be enabling S3 uploads manually.
2019-11-20 07:46:44 +10:00
Penar Musaraj
102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Martin Brennan
5c59247c3a Block enabling force 2FA if local logins disabled & vice-versa (#8355) 2019-11-15 17:05:10 +11:00
Sam Saffron
0c0192e7da DEV: disable all upload recovery specs
These specs are inherently fragile when they are run in a concurrent mode
in CI.

One process will create an upload, another will destroy the upload on disk
at the same time. We need a new safe mechanism here.
2019-11-11 16:04:18 +11:00
Penar Musaraj
0dfc594784 FIX: skip invalid URLs when checking for audio/video in search blurbs
Fixes 500 errors on search queries introduced in 580a4a8
2019-11-06 10:32:15 -05:00
Krzysztof Kotlarek
5ad0dd0c23 FIX: tweak restorer spec to make it stableish (#8300) 2019-11-06 17:05:33 +11:00
Daniel Waterworth
f9a05e4ae7 DEV: Switch flaky blame to restorer_spec.rb 2019-11-01 12:27:38 +00:00
Penar Musaraj
f8b72d9835 DEV: Refactor excluding audio/video URLs from search result blurbs
Followup to 580a4a82
2019-10-31 09:13:24 -04:00
Penar Musaraj
580a4a827b Exclude audio/video URLs from search result blurbs
Displays translatable "[audio]" or "[video]" placeholders instead of ugly (and often long) URLs.
2019-10-30 13:07:16 -04:00
Daniel Waterworth
29e41dc511 Revert "DEV: Mark flaky tests as pending"
This reverts commit 0c47529713.
2019-10-29 13:32:42 +00:00
Krzysztof Kotlarek
c5bcad10a6
FIX: more flaky jobs_base_spec.rb (#8259)
This commit is follow up of 417be323a2

We need more steps to properly switch from multi-site to single-site
2019-10-29 21:32:43 +11:00
Daniel Waterworth
0c47529713 DEV: Mark flaky tests as pending 2019-10-29 10:00:00 +00:00
Krzysztof Kotlarek
417be323a2 FIX: flaky jobs_base_spec.rb (#8258)
I was searching for a reason for randomly failing jobs_base_spec.rb. The reason was that after restorer_spec, the database is not restored to default.
After restorer spec RailsMultisite::ConnectionManagement.all_dbs is returning array of ['default', 'second']

Then base job execution is evaluated twice
```
dbs = RailsMultisite::ConnectionManagement.all_dbs
dbs.each do |db|
     execute(opts)
end
```
2019-10-29 18:25:00 +11:00
Krzysztof Kotlarek
b8688c4af7
FIX: Rubocop rule on restorer spec (#8238) 2019-10-24 12:03:27 +11:00
Krzysztof Kotlarek
f530378df3 FIX: Restore for non-multisite is not raising an error on reconnect step (#8237)
That commit introduced a bug to the system: f69dacf979

Restore works fine for multisite, however, stopped working for non-multisite.

Reason for that was that `establish_connection` method got a check if the multisite instance is available:
```
    def self.instance
      @instance
    end

    def self.establish_connection(opts)
      @instance.establish_connection(opts) if @instance
    end
```
However, the reload method don't have that check
```
    def self.reload
      @instance = new(instance.config_filename)
    end
```

To solve it, let's ensure we are in a multisite environment before call reload
2019-10-24 11:46:22 +11:00
Daniel Waterworth
1a72a61822 FIX: Fixed testsuite
The backup restorer tests weren't cleaning up after themselves and
playing poorly with prefabrication.
2019-10-23 09:43:05 +01:00
Krzysztof Kotlarek
f69dacf979 FIX: Reconnect in restore process connects to correct DB (#8218)
Simplified flow of restore is like that
```
migrate_database
reconnect
extract_uploads
```

Problem with incorrect current database started with this fix https://github.com/discourse/discourse/commit/025d4ee91f4

Dump task is reconnecting to default database https://github.com/rails/rails/blob/master/activerecord/lib/active_record/railties/databases.rake#L429

And then, we are trying to reconnect to the original database with that code:
```
def reconnect_database
  log "Reconnecting to the database..."
  RailsMultisite::ConnectionManagement::establish_connection(db: @current_db)
end
```

This reconnect is not switching us back to correct database because of that check
https://github.com/discourse/rails_multisite/blob/master/lib/rails_multisite/connection_management.rb#L181
Basically, it finds existing handler and it thinks that we are connected to correct DB and this step can be skipped.

To solve it, we can reload RailsMultisite::ConnectionManagement which creates a new instance of that class
https://github.com/discourse/rails_multisite/blob/master/lib/rails_multisite/connection_management.rb#L38
2019-10-23 17:23:50 +11:00
Roman Rizzi
01bc465db8
DEV: Split max decompressed setting for themes and backups (#8179) 2019-10-11 14:38:10 -03:00
Roman Rizzi
5357ab3324
SECURITY: Safely decompress backups when restoring. (#8166)
* SECURITY: Safely decompress backups when restoring.

* Fix tests and update theme_controller_spec to work with zip files instead of .tar.gz
2019-10-09 11:41:16 -03:00
Jarek Radosz
bfe0bc0cbd
FIX: Check for category conflicts in SiteSetting validations (#8137)
It was possible to add a category to more than one default group, e.g. "default categories muted" and "default categories watching first post".

The bug was caused by category validations inadvertently comparing strings and numbers.
2019-10-06 20:50:07 +02:00
Roman Rizzi
10565e4623
SECURITY: Safely decompress files. (#8124)
* FEATURE: Adds an extra protection layer when decompressing files.

* Rename exporter/importer to zip importer. Update old locale

* Added a new composite class to decompress a file with multiple strategies

* Set max file size inside a site setting

* Ensure that file is deleted after compression

* Sanitize path and files before compressing/decompressing
2019-10-03 10:19:35 -03:00
Sam Saffron
55ee9abecb DEV: clean up dependencies in spec
Follow up on zeitwork we needed to be a bit more explicit about a few
dependencies internally.

On certain orders the test suite could fail.
2019-10-02 14:50:54 +10:00
Krzysztof Kotlarek
427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Martin Brennan
68d35b14f4 FEATURE: Webauthn authenticator management with 2FA login (Security Keys) (#8099)
Adds 2 factor authentication method via second factor security keys over [web authn](https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API).

Allows a user to authenticate a second factor on login, login-via-email, admin-login, and change password routes. Adds registration area within existing user second factor preferences to register multiple security keys. Supports both external (yubikey) and built-in (macOS/android fingerprint readers).
2019-10-01 19:08:41 -07:00
Daniel Waterworth
7f8cdea924 FIX: Cleanup DiscoursePluginRegistry state after tests that use it
This was causing some heisentests
2019-09-20 13:32:54 +01:00
Dan Ungureanu
8aa5df69f0
FIX: Detect DiscourseHub user agent.
It used to show up as an unknown device and browser in user preferences
(Recently Used Devices).
2019-08-09 11:58:15 +03:00
Penar Musaraj
b5705348b3 FIX: Correctly identify Chromium-based Edge
Chromium-based Edge browsers has only "Edg" in user agent string.
2019-07-29 16:51:08 -04:00
Blake Erickson
b0c92bb0b9 REFACTOR: Clean up parameterized title
Follow up to [FIX: Empty backup names with unicode site titles][1]

- Use .presence - "It's cleaner"
- Update spec to use System.system_user so it is more readable

[1]: c8661674d4
2019-07-18 15:49:16 -06:00
Blake Erickson
c8661674d4 FIX: Empty backup names with unicode site titles
If a site title contains unicode it may end up with an empty backup
filename because of the rails `parameterize` method we are calling.

This fix ensures that the backup filenames default to "discourse" if the
parameterized site title is empty.

Bug reported [here][1].

[1]: https://meta.discourse.org/t/backup-checksum-and-backup-name-missing-when-unicode-site-name/123192?u=blake
2019-07-17 17:07:10 -06:00
Joffrey JAFFEUX
b3eb67976d
DEV: Upgrades to Ember 3.10 (#7871)
Co-Authored-By: majakomel <maja.komel@gmail.com>
2019-07-16 12:45:15 +02:00
Michael Brown
08b286808a
FIX: backups taken by pg_dump >= 11 are nonportable (#7893) 2019-07-15 18:07:44 -04:00
Robin Ward
1d38040579 SECURITY: SQL injection with default categories
This is a low severity security fix because it requires a logged in
admin user to update a site setting via the API directly to an invalid
value.

The fix adds validation for the affected site settings, as well as a
secondary fix to prevent injection in the event of bad data somehow
already exists.
2019-07-11 13:41:51 -04:00
Gerhard Schlager
19edc4abb8 FIX: English locale must not fall back to any other locale 2019-06-07 21:53:01 +02:00
Penar Musaraj
f00275ded3 FEATURE: Support private attachments when using S3 storage (#7677)
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
2019-06-06 13:27:24 +10:00
Bianca Nenciu
7716617c0d DEV: Add spec for BrowserDetection and Chromebook. 2019-05-30 16:31:28 +03:00
Dan Ungureanu
6bd082feab
FIX: Update mapping between locales and Postgres dictionaries. (#7606) 2019-05-27 16:52:09 +03:00
Gerhard Schlager
b788948985 FEATURE: English locale with international date formats
Makes en_US the new default locale
2019-05-20 13:47:20 +02:00
Régis Hanol
b5c8f5f900 SPEC: ensure never remap readonly columns
Follow-up to bfcbfd78
2019-05-09 18:01:35 +02:00
Daniel Waterworth
e219588142 DEV: Prefabrication (test optimization) (#7414)
* Introduced fab!, a helper that creates database state for a group

It's almost identical to let_it_be, except:

 1. It creates a new object for each test by default,
 2. You can disable it using PREFABRICATION=0
2019-05-07 13:12:20 +10:00
Sam Saffron
9be70a22cd DEV: introduce new API to look up dynamic site setting
This removes all uses of both `send` and `public_send` from consumers of
SiteSetting and instead introduces a `get` helper for dynamic lookup

This leads to much cleaner and safer code long term as we are always explicit
to test that a site setting is really there before sending an arbitrary
string to the class

It also removes a couple of risky stubs from the auth provider test
2019-05-07 11:00:30 +10:00
Guo Xiang Tan
24347ace10 FIX: Properly associate user_profiles background urls via upload id.
`Upload#url` is more likely and can change from time to time. When it
does changes, we don't want to have to look through multiple tables to
ensure that the URLs are all up to date. Instead, we simply associate
uploads properly to `UserProfile` so that it does not have to replicate
the URLs in the table.
2019-05-02 14:58:24 +08:00
Sam Saffron
4ea21fa2d0 DEV: use #frozen_string_literal: true on all spec
This change both speeds up specs (less strings to allocate) and helps catch
cases where methods in Discourse are mutating inputs.

Overall we will be migrating everything to use #frozen_string_literal: true
it will take a while, but this is the first and safest move in this direction
2019-04-30 10:27:42 +10:00
Vinoth Kannan
d0fe42e2ef FIX: should look through posts for image markdown
Downloaded onebox images only included in the cooked HTML content. So we have to check 'post.cooked' instead of 'raw'. bfdd0fe64c
2019-04-10 13:52:35 +05:30
Guo Xiang Tan
6a95d3fded FIX: UploadRecovery should look through posts for img src and bbcode. 2019-04-02 11:41:00 +08:00
Sam Saffron
9ebabc1de8 FEATURE: unconditionally update Topic updated_at when posts change in topic
Previously we would bypass touching `Topic.updated_at` for whispers and post
recovery / deletions.

This meant that certain types of caching can not be done where we rely on
this information for cache accuracy.

For example if we know we have zero unread topics as of yesterday and whisper
is made I need to bump this date so the cache remains accurate

This is only half of a larger change but provides the groundwork.

Confirmed none of our serializers leak out Topic.updated_at so this is safe
spot for this info

At the moment edits still do not change this but it is not relevant for the
unread cache.

This commit also cleans up some specs to use the new `eq_time` matcher for
millisecond fidelity comparison of times

Previously `freeze_time` would fudge this which is not that clean.
2019-03-28 17:28:01 +11:00
Gerhard Schlager
dc90133d29 FIX: Forcing permissions of seeded categories shouldn't fail
Less restrictive permissions of subcategories could make the seeding of categories fail.
2019-03-26 10:39:07 +01:00
Guo Xiang Tan
19c3c25db1 FIX: Handle BBCode in migrate_to_s3 task as well. 2019-03-22 16:47:06 +08:00
Gerhard Schlager
3fd04df781
FEATURE: Locale support for seeded categories and topics (#7110) 2019-03-18 21:09:13 +01:00
Robin Ward
fa5a158683 REFACTOR: Move queue_jobs out of SiteSetting
It is not a setting, and only relevant in specs. The new API is:

```
Jobs.run_later!        # jobs will be thrown on the queue
Jobs.run_immediately!  # jobs will run right away, avoid the queue
```
2019-03-14 10:47:38 -04:00
Robin Ward
d1d9a4f128 Add new run_jobs_synchronously! helper for tests
Previously if you wanted to have jobs execute in test mode, you'd have
to do `SiteSetting.queue_jobs = false`, because the opposite of queue
is to execute.

I found this very confusing, so I created a test helper called
`run_jobs_synchronously!` which is much more clear about what it does.
2019-03-11 16:58:35 -04:00
David Taylor
705c898c21
FEATURE: Calculate CSP based on active themes (#6976) 2019-02-11 12:32:04 +00:00
Gerhard Schlager
84a10f8212 Fix failing spec
The Report class created its own BackupStore without a stubbed S3 client
2019-02-08 21:38:10 +01:00
Robin Ward
b75a3e391a Use skip not pending 2019-02-08 13:48:29 -05:00
Robin Ward
fb18c57372 For now, make spec pending since it is broken. 2019-02-08 12:46:19 -05:00
David Taylor
95b5c5898e FIX: Prevent duplicate params for raw template theme handlebars helpers 2019-02-08 12:54:00 +00:00
Gerhard Schlager
45b056b615 FIX: Do not show backups stored in subfolder of bucket 2019-01-24 22:28:03 +01:00
David Taylor
0b6be2cebc FIX: Correct deprecated theme settings in handlebars templates
Was overriding the incorrect `themeSetting.blah` syntax, instead of `themeSettings.blah`
2019-01-18 15:39:50 +00:00
David Taylor
880311dd4d
FEATURE: Support for localized themes (#6848)
- Themes can supply translation files in a format like `/locales/{locale}.yml`. These files should be valid YAML, with a single top level key equal to the locale being defined. For now these can only be defined using the `discourse_theme` CLI, importing a `.tar.gz`, or from a GIT repository.

- Fallback is handled on a global level (if the locale is not defined in the theme), as well as on individual keys (if some keys are missing from the selected interface language).

- Administrators can override individual keys on a per-theme basis in the /admin/customize/themes user interface.

- Theme developers should access defined translations using the new theme prefix variables:
  JavaScript: `I18n.t(themePrefix("my_translation_key"))`
  Handlebars: `{{theme-i18n "my_translation_key"}}` or `{{i18n (theme-prefix "my_translation_key")}}`

- To design for backwards compatibility, theme developers can check for the presence of the `themePrefix` variable in JavaScript

- As part of this, the old `{{themeSetting.setting_name}}` syntax is deprecated in favour of `{{theme-setting "setting_name"}}`
2019-01-17 11:46:11 +00:00
Penar Musaraj
e11c6ffa89 FEATURE: allow extending CSP base-uri and object-src
Plus, ensure :none is stripped, it cannot be combined with other sources
2019-01-09 15:34:14 -05:00
Robin Ward
9ba8bfb1aa FIX: Multisite DB was leaving old data in test mode
This commit introduces a new helper to enable transactional fixtures
when testing multisite. This would show up as tests that passed the
first time then failed the second time due to stale data being leftover.
2019-01-09 15:20:37 -05:00
Kyle Zhao
dec8e5879a FEATURE: set CSP base-uri and object-src to none (#6863) 2019-01-09 15:04:50 -05:00
Vinoth Kannan
75dbb98cca FEATURE: Add S3 etag value to uploads table (#6795) 2019-01-04 14:16:22 +08:00
Sam
766e67ce57 FEATURE: introduce lossy color optimization on resized pngs
This feature ensures optimized images run via pngquant, this results extreme amounts of savings for resized images. Effectively the only impact is that the color palette on small resized images is reduced to 256.

To ensure safety we only apply this optimisation to images smaller than 500k.

This commit also makes a bunch of image specs less fragile.
2019-01-02 17:20:02 +11:00
Régis Hanol
5381096bfd PERF: new 'migrate_to_s3' rake task 2018-12-26 17:34:49 +01:00
Gerhard Schlager
2bdbca3801 DEV: Remove unnecessary to_not raise_error from specs
Follow-up to 01cdbd3a13
2018-12-17 16:10:10 +01:00
Gerhard Schlager
01cdbd3a13 FEATURE: Prohibit S3 bucket reusage
This validation makes sure that the s3_upload_bucket and the
s3_backup_bucket have different values. The backup bucket is
allowed to be a subfolder of the upload bucket. The other way
around is forbidden because the backup system searches by
prefix and would return all files stored within the backup
bucket and its subfolders.
2018-12-17 11:35:28 +01:00
Gerhard Schlager
1a8ca68ea3 FEATURE: Improve backup stats on admin dashboard
* Dashboard doesn't timeout anymore when Amazon S3 is used for backups
* Storage stats are now a proper report with the same caching rules
* Changing the backup_location, s3_backup_bucket or creating and deleting backups removes the report from the cache
* It shows the number of backups and the backup location
* It shows the used space for the correct backup location instead of always showing used space on local storage
* It shows the date of the last backup as relative date
2018-12-17 11:35:11 +01:00