In some cases (e.g. user notification emails) we
are passing an excerpted/stripped version of the
post HTML to Email::Styles, at which point the
<span> elements surrounding the hashtag text have
been stripped. This caused an error when trying to
remove that element to replace the text.
Instead we can just remove all elements inside
a.hashtag-cooked and replace with the raw #hashtag
text which will work in more cases.
This is closer to git's redirect following behaviour. We prevented git
following redirects when we clone in order to prevent SSRF attacks.
Follow-up-to: 291bbc4fb9
When doing local oneboxes we sometimes want to allow
SVGs in the final preview HTML. The main case currently
is for the new cooked hashtags, which include an SVG
icon.
SVGs will be included in local oneboxes via `ExcerptParser` _only_
if they have the d-icon class, and if the caller for `post.excerpt`
specifies the `keep_svg: true` option.
Adds stats for API and user API requests similar to regular page views.
This comes with a new report to visualize API requests per day like the
consolidated page views one.
This commit introduce a new API for registering callbacks, which we'll execute when a user gets destroyed, and the `delete_posts` opt is true. The chat plugin registers one callback and queues a job to destroy every message from that user in batches.
This reverts commit a71f6cf09b.
The github UI had an error I didn't notice which resulted
in a security commit being merged _after_ the bump, now
I have to redo the bump.
Previously we would unconditionally fetch all images via HTTP to grab
original sizing from cooked post processor in 2 different spots.
This was wasteful as we already calculate and cache this info in upload records.
This also simplifies some specs and reduces use of mocks.
This hasn't been necessary for many years, and is no longer supported following 84bec1cb. Only extremely old plugins might be trying to do this. All the affected open-source plugins I can find have already been updated.
We now use Ember CLI (core/plugins) and DiscourseJSProcessor (themes) for all Ember and template compilation. This commit removes the remnants of the legacy Sprockets-based Ember compilation system.
Sprockets, and its DiscourseJSProcess-based Babel transformations, is still in use for a few assets. Ideally that will be removed/replaced in the near future.
Raw paths like `/test/path` are not supported natively in the CSP. This commit prepends the site's base URL to these paths. This allows plugins to add 'local' assets to the CSP without needing to hardcode the site's hostname.
We're going to change the default return value of the `primary_email_verified?` method of `Auth::ManagedAuthenticator` to false, so we need to explicitly define the method on authenticators to return true where it makes sense to do so.
Internal topic: t/82084.
In some setups the keys start with "original/" and "optimized/" and in some setups the key is something like "foo/original/", so lets make the filter less strict.
This will be used by plugins to handle the client side of their custom
post validations without having to overwrite the whole composer save
action as it was done in other plugins.
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
This commit fleshes out and adds functionality for the new `#hashtag` search and
lookup system, still hidden behind the `enable_experimental_hashtag_autocomplete`
feature flag.
**Serverside**
We have two plugin API registration methods that are used to define data sources
(`register_hashtag_data_source`) and hashtag result type priorities depending on
the context (`register_hashtag_type_in_context`). Reading the comments in plugin.rb
should make it clear what these are doing. Reading the `HashtagAutocompleteService`
in full will likely help a lot as well.
Each data source is responsible for providing its own **lookup** and **search**
method that returns hashtag results based on the arguments provided. For example,
the category hashtag data source has to take into account parent categories and
how they relate, and each data source has to define their own icon to use for the
hashtag, and so on.
The `Site` serializer has two new attributes that source data from `HashtagAutocompleteService`.
There is `hashtag_icons` that is just a simple array of all the different icons that
can be used for allowlisting in our markdown pipeline, and there is `hashtag_context_configurations`
that is used to store the type priority orders for each registered context.
When sending emails, we cannot render the SVG icons for hashtags, so
we need to change the HTML hashtags to the normal `#hashtag` text.
**Markdown**
The `hashtag-autocomplete.js` file is where I have added the new `hashtag-autocomplete`
markdown rule, and like all of our rules this is used to cook the raw text on both the clientside
and on the serverside using MiniRacer. Only on the server side do we actually reach out to
the database with the `hashtagLookup` function, on the clientside we just render a plainer
version of the hashtag HTML. Only in the composer preview do we do further lookups based
on this.
This rule is the first one (that I can find) that uses the `currentUser` based on a passed
in `user_id` for guardian checks in markdown rendering code. This is the `last_editor_id`
for both the post and chat message. In some cases we need to cook without a user present,
so the `Discourse.system_user` is used in this case.
**Chat Channels**
This also contains the changes required for chat so that chat channels can be used
as a data source for hashtag searches and lookups. This data source will only be
used when `enable_experimental_hashtag_autocomplete` is `true`, so we don't have
to worry about channel results suddenly turning up.
------
**Known Rough Edges**
- Onebox excerpts will not render the icon svg/use tags, I plan to address that in a follow up PR
- Selecting a hashtag + pressing the Quote button will result in weird behaviour, I plan to address that in a follow up PR
- Mixed hashtag contexts for hashtags without a type suffix will not work correctly, e.g. #ux which is both a category and a channel slug will resolve to a category when used inside a post or within a [chat] transcript in that post. Users can get around this manually by adding the correct suffix, for example ::channel. We may get to this at some point in future
- Icons will not show for the hashtags in emails since SVG support is so terrible in email (this is not likely to be resolved, but still noting for posterity)
- Additional refinements and review fixes wil
The UPDATE statement could lock the `uploads` table for a very long time
when the `verification_status` of lots of uploads changed. Splitting up
and simplifying the UPDATE solves that problem.
Also, this change ensures that only the needed data from the inventory
gets inserted into the `TEMP TABLE`. For example, there's no need to
have records for optimized images in that table when the `uploads` table
gets updated.
The hidden site setting `suppress_secured_categories_from_admin` will
suppress visibility of categories without explicit access from admins
in a few key areas (category drop downs and topic lists)
It is not intended to be a security wall since admins can amend any site
setting. Instead it is feature that allows hiding the categories from the
UI.
Admins will still be able to see topics in categories without explicit
access using direct URLs or flags.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
These two dropdown fields in the setup wizard are pre-populated, and
there is no way to de-select a value, you can only change it. So we can
remove the required attribute so that an asterisk doesn't show up in the
UI.
Previously we were forcing node's max-old-space-size to be 2GB. This override was added in a01b1dd6 to avoid issues caused by a lower default node heap_size_limit on machines with less memory.
This commit makes that `max-old-space-size` override more specific so that it only applies to machines with less memory. Other machines will go use Node's defaults.
The override is also lowered to 1GB. This is still high enough for the build to complete, while reducing memory usage.
https://meta.discourse.org/t/245547
* FEATURE: Default Composer Category Site Setting
- Create the default_composer_category site setting
- Replace general_category_id logic for auto selecting the composer
category
- Prevent Uncategorized from being selected if not allowed
- Add default_composer_category option to seeded categories
- Create a migration to populate the default_composer_category site
setting if there is a general_category_id populated
- Added some tests
* Add missing translation for the new site setting
* fix some js tests
* Just check that the header value is null
Currently, moderators are able to set primary group for users
irrespective of the of the `moderators_manage_categories_and_groups` site
setting value.
This change updates Guardian implementation to honour it.
* Remove old bookmark column ignores to follow up b22450c7a8
* Change some group site setting checks to use the _map helper
* Remove old secure_media helper stub for chat
* Change attr_accessor to attr_reader for preloaded_custom_fields to follow up 70af45055a
Previously the stylesheet cachebusting hash was based on the maximum mtime of files. This works well in development and during in-container updates (e.g. via docker_manager). However, when a fresh docker image is created for each deploy, the file mtimes will change even if the contents has not.
This commit changes the production logic to calculate the cachebuster from the filenames and contents of the relevant assets. This should be consistent across deploys, thereby improving cache hits and improving page load times.
- Ensure it works with prefixed S3 buckets
- Perform a sanity check that all current assets are present on S3 before starting deletion
- Remove the lifecycle rule configuration and delete expired assets immediately. This task should be run post-deploy anyway, so adding a 10-day window is not required
Since the system user is a regular user, it can have its
`allow_private_messages` user option turned off, which
with our current `can_send_private_message?(Discourse.system_user)`
check inside the CurrentUserSerializer, will prevent any
user from sending messages in the UI if the system user is not
accepting PMs.
This commit adds a new `can_send_private_messages?` method to
the Guardian, which can be used in serializers and not depend
on the system user. When the user actually sends a message
we still rely on the old `can_send_private_message?(target)`
call to see if they are allowed to send the message to the target.
The new method is just to say they can "generally" send
private messages.
Previously, we didn't have a site-wide setting to set the default behavior for user profile visibility and user presence features. But we already have a user preference for that.
This task is supposed to skip uploading if the asset is already present in S3. However, when a bucket 'folder path' was configured, this logic was broken and so the assets would be re-uploaded every time.
This commit fixes that logic to include the bucket 'folder path' in the check
This should fix fetching from gitlab.
In order to get SSRF protection, we had to prevent redirects when cloning via git, but some repos are behind redirects and we want to support those too. We use `FinalDestination` before cloning to try to simulate git with redirects, but this isn't quite how git works, so there's some discrepancies between our SSRF protected cloning behavior and normal git behavior that I'm trying to work around.
This is temporary fix. It would be better to use `FinalDestination` to simulate the first request that git makes. I aim to make it work like that in the not too distant future, but this is better for now.
Depends on: #18806
We have a banner that prompts to edit the welcome topic, so let's not
show it in the topic list until it has been edited. Previously this
banner covered the welcome topic, now the banner will be above the topic
list, so we need to hide the welcome topic.
Before this commit, there was no way for us to efficiently check an
array of topics for which a user can see. Therefore, this commit
introduces the `TopicGuardian#can_see_topic_ids` method which accepts an
array of `Topic#id`s and filters out the ids which the user is not
allowed to see. The `TopicGuardian#can_see_topic_ids` method is meant to
maintain feature parity with `TopicGuardian#can_see_topic?` at all
times so a consistency check has been added in our tests to ensure that
`TopicGuardian#can_see_topic_ids` returns the same result as
`TopicGuardian#can_see_topic?`. In the near future, the plan is for us
to switch to `TopicGuardian#can_see_topic_ids` completely but I'm not
doing that in this commit as we have to be careful with the performance
impact of such a change.
This method is currently not being used in the current commit but will
be relied on in a subsequent commit.
Linking a commit from a GitHub pull request included the complete commit
message, instead of just the first line. The rest of the commit message
will be added to the body of the Onebox.
When PostRevisor is called with 'skip_validations: true' it can save
the post twice and one of the calls passes the correct 'validate: false'
argument, but the other one does not.
The filenames (minus the extensions) were being used as keys in a hash to pass to Terser, which meant that colocated connector files would overwrite each other. This commit moves the un-colocating earlier in the pipeline so that the fixed filenames are passed to Terser.
Followup to be3d6a56ce
This commit adds a new `/hashtag/search` endpoint and both
relevant JS and ruby plugin APIs to handle plugins adding their
own data sources and priority orders for types of things to search
when `#` is pressed.
A `context` param is added to `setupHashtagAutocomplete` which
a corresponding chat PR https://github.com/discourse/discourse-chat/pull/1302
will now use.
The UI calls `registerHashtagSearchParam` for each context that will
require a `#` search (e.g. the topic composer), for each type of record that
the context needs to search for, as well as a priority order for that type. Core
uses this call to add the `category` and `tag` data sources to the topic composer.
The `register_hashtag_data_source` ruby plugin API call is for plugins to
add a new data source for the hashtag searching endpoint, e.g. discourse-chat
may add a `channel` data source.
This functionality is hidden behind the `enable_experimental_hashtag_autocomplete`
flag, except for the change to `setupHashtagAutocomplete` since only core and
discourse-chat are using that function. Note this PR does **not** include required
changes for hashtag lookup or new styling.
Theme javascript is now minified using Terser, just like our core/plugin JS bundles. This reduces the amount of data sent over the network.
This commit also introduces sourcemaps for theme JS. Browser developer tools will now be able show each source file separately when browsing, and also in backtraces.
For theme test JS, the sourcemap is inlined for simplicity. Network load is not a concern for tests.
Previously, compiling theme 'extra_js' was done with a number of steps. Each theme_field would be compiled into its own value_baked column, and then the JavascriptCache content would be built by concatenating all of those compiled values.
This commit streamlines things by removing the value_baked step. The raw value of all extra_js theme_fields are passed directly to the ThemeJavascriptCompiler, and then the result is stored in the JavascriptCache.
In itself, this commit should not cause any behavior change. It is designed to open the door to more advanced compilation features which have interdependencies between different source files (e.g. template colocation, sourcemaps).
RS256 was added for Windows Hello and as a side effect we speculatively added
RS384 and RS512. These ciphers were not tested and are now failing on solo
keys. It may be the case that the ciphers are not configured correctly on
our side. It may be the case that this is a Solo key bug.
Regardless, we are removing the ciphers and will only consider adding them
again if absolutely needed.
The previous implementation would attempt to fetch groups using the end-user's Google auth token. This only worked for admin accounts, or users with 'delegated' access to the `admin.directory.group.readonly` API.
This commit changes the approach to use a single 'service account' for fetching the groups. This removes the need to add permissions to all regular user accounts. I'll be updating the [meta docs](https://meta.discourse.org/t/226850) with instructions on setting up the service account.
This is technically a breaking change in behavior, but the existing implementation was marked experimental, and is currently unusable in production google workspace environments.
Previously, when the array had both nil and string values it returned the error "comparison of NilClass with String failed". Now I added the `.compact` method to prevent this issue as per @martin-brennan's suggestion https://github.com/discourse/discourse/pull/18431#discussion_r984204788
* Revert "Revert "FEATURE: Preload resources via link header (#18475)" (#18511)"
This reverts commit 95a57f7e0c.
* put behind feature flag
* env -> global setting
* declare global setting
* forgot one spot
* FEATURE: Hide Privacy Policy and TOS topics
As a way to simplify new sites this change will hide the privacy policy
and the TOS topics from the topic list. They can still be accessed and
edited though.
* add tests
Experiment moving from preload tags in the document head to preload information the the response headers.
While this is a minor improvement in most browsers (headers are parsed before the response body), this allows smart proxies like Cloudflare to "learn" from those headers and build HTTP 103 Early Hints for subsequent requests to the same URI, which will allow the user agent to download and parse our JS/CSS while we are waiting for the server to generate and stream the HTML response.
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
Adds a new upload field for a second dark mode category logo.
This alternative will be used when the browser is in dark mode (similar to the global site setting for a dark logo).
These errors tend to indicate that the upload is missing on the remote store. This is bad, but we don't want it to block the dominant-color calculation process. This commit catches errors when there is an HTTP error, and fixes the `base_store.rb` implementation when `FileHelper.download` returns nil.
When a user with an email matching those inside the
DISCOURSE_DEVELOPER_EMAILS env var log in, we make
them into admin users if they are not already. This
is used when setting up the first admin user for
self-hosters, since the discourse-setup script sets
the provided admin emails into DISCOURSE_DEVELOPER_EMAILS.
The issue being fixed here is that the new admins were
not being automatically added to the staff and admins
automatic groups, which was causing issues with the site
settings that are group_list based that don't have an explicit
staff override. All we need to do is refresh the automatic
staff, admin groups when admin is granted for the user.
This PR enables the [`no-action-modifiers`](https://github.com/ember-template-lint/ember-template-lint/blob/master/docs/rule/no-action-modifiers.md) template lint rule and removes all usages of the `{{action}}` modifier in core.
In general, instances of `{{action "x"}}` have been replaced with `{{on "click" (action "x")}}`.
In many cases, such as for `a` elements, we also need to prevent default event handling to avoid unwanted side effects. While the `{{action}}` modifier internally calls `event.preventDefault()`, we need to handle these cases more explicitly. For this purpose, this PR also adds the [ember-event-helpers](https://github.com/buschtoens/ember-event-helpers) dependency so we can use the `prevent-default` handler. For instance:
```
<a href {{on "click" (prevent-default (action "x"))}}>Do X</a>
```
Note that `action` has not in general been refactored away as a helper yet. In general, all event handlers should be methods on the corresponding component and referenced directly (e.g. `{{on "click" this.doSomething}}`). However, the `action` helper is used extensively throughout the codebase and often references methods in the `actions` hash on controllers or routes. Thus this refactor will also be extensive and probably deserves a separate PR.
Note: This work was done to complement #17767 by minimizing the potential impact of the `action` modifier override, which uses private API and arguably should be replaced with an AST transform.
This is a followup to #18333, which had to be reverted because it did not account for the default treatment of modifier keys by the {{action}} modifier.
Commits:
* Enable `no-action-modifiers` template lint rule
* Replace {{action "x"}} with {{on "click" (action "x")}}
* Remove unnecessary action helper usage
* Remove ctl+click tests for user-menu
These tests now break in Chrome when used with addEventListener. As per the comment, they can probably be safely removed.
* Prevent default event handlers to avoid unwanted side effects
Uses `event.preventDefault()` in event handlers to prevent default event handling. This had been done automatically by the `action` modifier, but is not always desirable or necessary.
* Restore UserCardContents#showUser action to avoid regression
By keeping the `showUser` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showUser` argument that's been passed.
* Revert EditCategoryTab#selectTab -> EditCategoryTab#select
Avoid potential breaking change in themes / plugins
* Restore GroupCardContents#showGroup action to avoid regression
By keeping the `showGroup` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showGroup` argument that's been passed.
* Restore SecondFactorAddTotp#showSecondFactorKey action to avoid regression
By keeping the `showSecondFactorKey` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showSecondFactorKey` property that's maintained on the controller.
* Refactor away from `actions` hash in ChooseMessage component
* Modernize EmojiPicker#onCategorySelection usage
* Modernize SearchResultEntry#logClick usage
* Modernize Discovery::Categories#showInserted usage
* Modernize Preferences::Account#resendConfirmationEmail usage
* Modernize MultiSelect::SelectedCategory#onSelectedNameClick usage
* Favor fn over action in SelectedChoice component
* Modernize WizardStep event handlers
* Favor fn over action usage in buttons
* Restore Login#forgotPassword action to avoid possible regression
* Introduce modKeysPressed utility
Returns an array of modifier keys that are pressed during a given `MouseEvent` or `KeyboardEvent`.
* Don't interfere with click events on links with `href` values when modifier keys are pressed
cf. e62e93f83a
This PR also makes it so `bot` (negative ID) and `system` users are always allowed
to send PMs, since the old conditional was just based on `enable_personal_messages`
Static topics are the seeded topics that are automatically created for every Discourse instance to hold the content for the FAQ, ToS and Privacy pages. These topics are allowed to bypass the minimum title length checks when they're edited by admins:
ba27ee1637/app/assets/javascripts/discourse/app/models/composer.js (L487-L496)
However, on the server-side, the "quality title" validations aren't skipped for static topics and that can cause confusion for admins when they change the title of a static topic to something that's short enough to fail the quality title validations. This commit ignores all quality title validations on static topics when they're edited by admins.
Internal topic: t/75745.
This PR enables the [`no-action-modifiers`](https://github.com/ember-template-lint/ember-template-lint/blob/master/docs/rule/no-action-modifiers.md) template lint rule and removes all usages of the `{{action}}` modifier in core.
In general, instances of `{{action "x"}}` have been replaced with `{{on "click" (action "x")}}`.
In many cases, such as for `a` elements, we also need to prevent default event handling to avoid unwanted side effects. While the `{{action}}` modifier internally calls `event.preventDefault()`, we need to handle these cases more explicitly. For this purpose, this PR also adds the [ember-event-helpers](https://github.com/buschtoens/ember-event-helpers) dependency so we can use the `prevent-default` handler. For instance:
```
<a href {{on "click" (prevent-default (action "x"))}}>Do X</a>
```
Note that `action` has not in general been refactored away as a helper yet. In general, all event handlers should be methods on the corresponding component and referenced directly (e.g. `{{on "click" this.doSomething}}`). However, the `action` helper is used extensively throughout the codebase and often references methods in the `actions` hash on controllers or routes. Thus this refactor will also be extensive and probably deserves a separate PR.
Note: This work was done to complement #17767 by minimizing the potential impact of the `action` modifier override, which uses private API and arguably should be replaced with an AST transform.
Commits:
* Enable `no-action-modifiers` template lint rule
* Replace {{action "x"}} with {{on "click" (action "x")}}
* Remove unnecessary action helper usage
* Remove ctl+click tests for user-menu
These tests now break in Chrome when used with addEventListener. As per the comment, they can probably be safely removed.
* Prevent default event handlers to avoid unwanted side effects
Uses `event.preventDefault()` in event handlers to prevent default event handling. This had been done automatically by the `action` modifier, but is not always desirable or necessary.
* Restore UserCardContents#showUser action to avoid regression
By keeping the `showUser` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showUser` argument that's been passed.
* Revert EditCategoryTab#selectTab -> EditCategoryTab#select
Avoid potential breaking change in themes / plugins
* Restore GroupCardContents#showGroup action to avoid regression
By keeping the `showGroup` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showGroup` argument that's been passed.
* Restore SecondFactorAddTotp#showSecondFactorKey action to avoid regression
By keeping the `showSecondFactorKey` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showSecondFactorKey` property that's maintained on the controller.
* Refactor away from `actions` hash in ChooseMessage component
* Modernize EmojiPicker#onCategorySelection usage
* Modernize SearchResultEntry#logClick usage
* Modernize Discovery::Categories#showInserted usage
* Modernize Preferences::Account#resendConfirmationEmail usage
* Modernize MultiSelect::SelectedCategory#onSelectedNameClick usage
* Favor fn over action in SelectedChoice component
* Modernize WizardStep event handlers
* Favor fn over action usage in buttons
* Restore Login#forgotPassword action to avoid possible regression
This commit adds non-archived group messages and `group_message_summary` notifications in the messages tab in the user menu. With this change, the messages tab in the user menu now includes 3 types of items:
1. Unread `private_message` notifications (notifications when you receive a reply in a PM)
2. Unread and read `group_message_summary` notifications (notifications when there's a new message in a group inbox that you track)
3. Non-archived personal and group messages
Unread `private_message` notifications are always shown first, followed by unread `group_message_summary` notifications, and then everything else (messages and read `group_message_summary` notifications) sorted by recency (most recent first).
Internal topic: t/72976.
By default, only staff members have to confirm their old email when
changing it. This commit adds a site setting that when enabled will
always ask the user to confirm old email.
* REFACTOR: Improve reusability by Decoupling flag modal from flag target.
We want chat message's flags to have the same features as topic and posts' flags, but we prefer not having to duplicate core's logic. This PR moves target specific bits to different classes, allowing plugins to flag custom things by
providing their own.
* A couple of fixes for the flag modal:
- Make sure buttons are disabled until a flag type is selected.
- Don't throw an error when checking if the user can undo an action on a deleted topic.
- Disable flagging on deleted topics.
This commit introduces an icon to all links in the sidebar. If an icon has not been configured, we will fall back to a generic "link" icon. As part of this commit, we also standardised the size of each prefix to 20px by 20px and set a fix margin. This is to allow sufficient space for text prefixes and image prefixes to be displayed.
Tests have been intentionally left out for now as I don't feel like asserting for the icons will bring much value at this point. Time shall prove me wrong.
Co-authored-by: awesomerobot <kris.aubuchon@discourse.org>
This commit renames all secure_media related settings to secure_uploads_* along with the associated functionality.
This is being done because "media" does not really cover it, we aren't just doing this for images and videos etc. but for all uploads in the site.
Additionally, in future we want to secure more types of uploads, and enable a kind of "mixed mode" where some uploads are secure and some are not, so keeping media in the name is just confusing.
This also keeps compatibility with the `secure-media-uploads` path, and changes new
secure URLs to be `secure-uploads`.
Deprecated settings:
* secure_media -> secure_uploads
* secure_media_allow_embed_images_in_emails -> secure_uploads_allow_embed_images_in_emails
* secure_media_max_email_embed_image_size_kb -> secure_uploads_max_email_embed_image_size_kb
* FEATURE: add composer warning when user haven't been seen in a long time
When a user creates a PM and adds a recipient that hasn't been seen in a
long time then we'll now show a warning in composer indicating that the
user hasn't been seen in a long time.
* FIX: Recursively tag topics with missing ancestor tags
Given only a child tag, walk up the ancestry chain, get all of it's
ancestors for use in tagging a topic
* FIX: Ensure only one parent tag is returned for topic tagging
Current implementation selects and return first parent tag if child tag
has multiple parents.
This change updates recursive parent tag implementation to only return
parent tags via only one ancestry line.
* DEV: Add test case for tag cycles
Given we aren't performing a strict graph traversal to get a tag's
parent, cycles do not have any effect on the tags returned for topic
tagging.
This will replace `enable_personal_messages` and
`min_trust_to_send_messages`, this commit introduces
the setting `personal_message_enabled_groups`
and uses it in all places that `enable_personal_messages`
and `min_trust_to_send_messages` currently apply.
A migration is included to set `personal_message_enabled_groups`
based on the following rules:
* If `enable_personal_messages` was false, then set
`personal_message_enabled_groups` to `3`, which is
the staff auto group
* If `min_trust_to_send_messages` is not default (1)
and the above condition is false, then set the
`personal_message_enabled_groups` setting to
the appropriate auto group based on the trust level
* Otherwise just set `personal_message_enabled_groups` to
11 which is the TL1 auto group
After follow-up PRs to plugins using these old settings, we will be
able to drop the old settings from core, in the meantime I've added
DEPRECATED notices to their descriptions and added them
to the deprecated site settings list.
This commit also introduces a `_map` shortcut method definition
for all `group_list` site settings, e.g. `SiteSetting.personal_message_enabled_groups`
also has `SiteSetting.personal_message_enabled_groups_map` available,
which automatically splits the setting by `|` and converts it into
an array of integers.
See https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499
for thorough reasoning.
This commit changes how we generate Message-IDs and do email
threading for emails sent from Discourse. The main changes are
as follows:
* Introduce an outbound_message_id column on Post that
is either a) filled with a Discourse-generated Message-ID
the first time that post is used for an outbound email
or b) filled with an original Message-ID from an external
mail client or service if the post was created from an
incoming email.
* Change Discourse-generated Message-IDs to be more consistent
and static, in the format `discourse/post/:post_id@:host`
* Do not send References or In-Reply-To headers for emails sent
for the OP of topics.
* Make sure that In-Reply-To is filled with either a) the OP's
Message-ID if the post is not a direct reply or b) the parent
post's Message-ID
* Make sure that In-Reply-To has all referenced post's Message-IDs
* Make sure that References is filled with a chain of Message-IDs
from the OP down to the parent post of the new post.
We also are keeping X-Discourse-Post-Id and X-Discourse-Topic-Id,
headers that we previously removed, for easier visual debugging
of outbound emails.
Finally, we backfill the `outbound_message_id` for posts that have
a linked `IncomingEmail` record, using the `message_id` of that record.
We do not need to do that for posts that don't have an incoming email
since they are backfilled at runtime if `outbound_message_id` is missing.
This PR makes some updates to the prior keyboard accessibility commit (eb98746):
- Makes `tabindex` attribute only appear on emoji markup in the emoji picker.
- After pressing the Esc key, focus returns to the <textarea/> input (composer editor or chat input)
At some point moved from hljs pure source to their CDN assets, but we
did not change the way we created the HLJS bundle. The CDN asset comes
with their "common" languages already included, so we were duplicating
around 35 languagues in the bundle we create.
This patch includes a list of their current common langs so we never
double bundle those.
Changes in size are significant. Numbers before minimization/compression
are 747.53 KB before and 117.57 KB after.
We previously had a system which would generate a 10x10px preview of images and add their URLs in a data-small-upload attribute. The client would then use that as the background-image of the `<img>` element. This works reasonably well on fast connections, but on slower connections it can take a few seconds for the placeholders to appear. The act of loading the placeholders can also break or delay the loading of the 'real' images.
This commit replaces the placeholder logic with a new approach. Instead of a 10x10px preview, we use imagemagick to calculate the average color of an image and store it in the database. The hex color value then added as a `data-dominant-color` attribute on the `<img>` element, and the client can use this as a `background-color` on the element while the real image is loading. That means no extra HTTP request is required, and so the placeholder color can appear instantly.
Dominant color will be calculated:
1. When a new upload is created
2. During a post rebake, if the dominant color is missing from an upload, it will be calculated and stored
3. Every 15 minutes, 25 old upload records are fetched and their dominant color calculated and stored. (part of the existing PeriodicalUpdates job)
Existing posts will continue to use the old 10x10px placeholder system until they are next rebaked
This commit adds to the experimental user menu a new "other notifications" tab that's very similar to the "all notifications" tab, but with the main difference being that it doesn't show notification types that do have dedicated tabs in the menu (e.g. mentions, likes, replies etc.).
The rationale behind this is that the notification types that do have dedicated tabs tend to dominate the "all notifications" tab, leaving very small chances for the user to notice rarer or infrequent notification types. Adding a tab for all the other types gives the user a way to review those infrequent notification types.
Internal ticket: t72978.
Co-authored-by: OsamaSayegh <asooomaasoooma90@gmail.com>
Previously we were relying on a highly-customized version of the unmaintained Barber gem for theme template compilation. This commit switches us to use our own DiscourseJsProcessor, which makes use of more modern patterns and will be easier to maintain going forward.
In summary:
- Refactors DiscourseJsProcessor to move multiline JS heredocs into a companion `discourse-js-processor.js` file
- Use MiniRacer's `.call` method to avoid manually escaping JS strings
- Move Theme template AST transformers into DiscourseJsProcessor, and formalise interface for extending RawHandlebars AST transformations
- Update Ember template compilation to use a babel-based approach, just like Ember CLI. This gives each template its own ES6 module rather than directly assigning `Ember.TEMPLATES` values
- Improve testing of template compilation (and move some tests from `theme_javascript_compiler_spec.rb` to `discourse_js_processor_spec.rb`
It used to return the next URL anyway which lead to an additional
request. On the frontend, if the result set was empty, it kept retrying
until at least one result was returned. This bug is fixed in this commit
too.
This lets us use all our normal JS tooling like prettier, esline and babel on the splash screen JS. At runtime the JS file is read and inlined into the HTML. This commit also switches us to use a CSP hash rather than a nonce for the splash screen.
* FIX: hide welcome topic banner as soon as the welcome topic is edited
This commit adds a message bus listener on client to hide the welcome
topic banner as soon as the welcome topic is edited.
* update test
* only subscribe when show_welcome_topic_banner is true
* Do not lookup for messageBus service if it's not required
* Remove unneeded code
* Cache result for Site.show_welcome_topic_banner
* Update tests per latest changes
* Changes per PR review
The order in which Onebox engines are loaded is not guaranteed. Occasionally during tests, the twitter engine would be loaded before the instagram engine, and cause the Instagram Onebox spec to fail due to the lack of `Onebox.options.twitter_client`.
This commit makes the load order of Onebox engines consistent, and fixes the issue in the twitter_status_onebox.
The source-of-truth for our ember version is now the installed node_module. The `ember_source` gem carries an old version of Ember and so the constant is no longer useful. We'll be dropping the gem soon.
* FIX: Only seed general category on new sites
If the site already has human users (users with an id > 0) don't seed
the categories.
Follow up to: a6ad74c759
* use human_users scope
We don't want to save the auto_delete_preference for bookmarks to the
user options if it was passed through as nil from the frontend,
this leads to confusion for the end user since they did not explicitly set it.
It's fine to create the bookmark with the default of "never" if no
auto_delete_preference is provided since it applies only to the
single bookmark, not future bookmarks.
- Seed the General category so that the general chat channel will have
a home
- Do not seed the Lounge category anymore
- Move the "Welcome to Site" topic to the General category
This commit makes a number of improvements to the DiscourseJsProcessor:
1. Remove dependence on the out-of-date Ember template compiler from the ember-rails gem; switch to modern template compiler
2. Refactor to make use of a proper module system with `define`/`require`
3. Introduce `babel-plugin-ember-template-compilation` to enable inline hbs compilation
The `mini-loader` is upgraded to support relative lookup and `require.has`, so that these new JS packages work correctly.
We were already compiling the markdown bundle via ember-cli, but that version was only being used in the test environment. This commit improves the implementation, and updates the filename so it's also used in production.
This commit also
- Removes the vendored copy of `markdown-it.js` and fetches from node_modules instead
- Updates `pretty_text.rb` to remove the custom sprockets-manifest-parsing
- Removes `pretty-text-bundle.js`, which was only being used by `pretty_text.rb`
Topic allowed user records were created for small actions, which lead to
the system user being invited in many private topics when the user
removed themselves or if a group was invited but some members already
had access.
This commits skips creating topic allowed user. They are already skipped
for the whisper posts.
If a user was granted a trust level, joined a group that granted a trust
level and left the group, the trust level was reset. This commit tries
to restore the last known trust level before joining the group by
looking into staff logs.
This commit also migrates old :change_trust_level user history records
to use previous_value and new_value fields.
The previous sprockets implementation was including admin-specific JS in the plugin's main JS file, which would be served to all users regardless of admin status. This commit achieves the same result under the ember-cli plugin asset compiler with one difference: the admin js is compiled into a separate file. That means that in future, we'll be able to make it loaded only for admins. For now though, it's loaded for everyone, just like before.
In a multisite Discourse reported that no backup is running after 60 seconds because the Redis key expired. Also, the thread that listens for a shutdown signal stopped running immediately because it didn't detect a running operation.
Certain HTML can be rejected by nokogumbo, specifically cases where there
are enormous amounts of attributes
This ensures that malformed HTML is simply skipped instead of leaking out
an exception and terminating downstream processes.
* FIX: Do not allow to remove like if topic is archived
* FIX: Always show like button
The like button used to be hidden if the topic was archived and it had
no likes. This commit changes that to always show the like button, but
with a not-allowed cursor if the topic is archived.
When `EMBER_CLI_PLUGIN_ASSETS=1`, plugin application JS will be compiled via Ember CLI. In this mode, the existing `register_asset` API will cause any registered JS files to be made available in `/plugins/{plugin-name}_extra.js`. These 'extra' files will be loaded immediately after the plugin app JS file, so this should not affect functionality.
Plugin compilation in Ember CLI is implemented as an addon, similar to the existing 'admin' addon. We bypass the normal Ember CLI compilation process (which would add the JS to the main app bundle), and reroute the addon Broccoli tree into a separate JS file per-plugin. Previously, Sprockets would add compiled templates directly to `Ember.TEMPLATES`. Under Ember CLI, they are compiled into es6 modules. Some new logic in `discourse-boot.js` takes care of remapping the new module names into the old-style `Ember.TEMPLATES`.
This change has been designed to be a like-for-like replacement of the old plugin compilation system, so we do not expect any breakage. Even so, the environment variable flag will allow us to test this in a range of environments before enabling it by default.
A manual silence implementation is added for the build-time `ember-glimmer.link-to.positional-arguments` deprecation while we work on a better story for plugins.
`TopicQueryParams` allows for `match_all_tags` to be passed as a query parameter. `TagsController` forces the value to be true.
This change allows a value to be passed, and only sets it to true if no value has been set. It then uses `ActiveModel::Type::Boolean.new.cast` to compare the value.
The maximum_staged_users_per_email site setting controls how many
staged users will be invited to the topic created from an incoming
email. Previously, it counted only the new staged users.
Twitter removed OpenGraph tags from their pages. We can no longer
extract all the information (for example, the quoted tweet) we need
to render Oneboxes without using their API.
This also adds an optional `ticket` parameter to `Backuper` which allows identifying the backup in `backup_complete` and `backup_failed` events. Both events contain the logs as payload and moving some methods around ensures that all errors are included in the logs.
Instead of relying on another help to generate the icons, we want to
rely on the interface for adding prefix icons. This ensures that prefix
icons are consistent across the section links in Sidebar
Previously, for every bookmarked topic, all topic_user records were being preloaded. Only the current user's record is actually required.
This commit introduces a new `perform_custom_preload!` API which bookmarkables can use to add custom preloading logic. We use this in topic_bookmarkable to load just the topic_user data we need (in the same way as `topic_list.rb`).
Co-authored-by: Blake Erickson <o.blakeerickson@gmail.com>
Fixes warning:
```
Deprecation notice: Jobs::SendSystemMessage was enqueued with argument values which do not cleanly serialize to/from JSON. This means that the job will be run with slightly different values than the ones supplied to `enqueue`. Argument values should be strings, booleans, numbers, or nil (or arrays/hashes of those value types). (deprecated since Discourse 2.9) (removal in Discourse 3.0)
At /var/www/discourse/lib/post_destroyer.rb:335:in `notify_deletion`
```
Polling every 0.001s can cause extreme load on the redis instance, especially in scenarios where multiple app instances are waiting on the same lock. This commit introduces an exponential backoff starting from 0.001s and reaching a maximum interval of 1s.
Previously `CHECK_READONLY_ATTEMPTS` was 10, and resulted in a block for 0.01s. Under the new logic, 10 attempts take more than 1s. Therefore CHECK_READONLY_ATTEMPTS is reduced to 5, bringing its total time to around 0.031s
Dead and large images are replaced with a placeholder, either a broken
chain icon or a short text. This commit no longer applies this
transformation for images inside Oneboxes, but removes them instead.
Hard deleting topics that contained soft deleted posts or small actions
used to create orphan posts because only the first post was hard
deleted. This commit adds an error message if there are still posts left
in the topic that must be hard deleted first or hard deletes all small
actions too immediately (there is no other way of hard deleting a small
action because there is no wrench menu).
Some of the changes in this PR are extracted from https://github.com/discourse/discourse/pull/17379.
Similar to the bookmarks tab in the new user menu, the messages tab also displays a mix of notifications and messages. When there are unread message notifications, the tab displays all of these notifications at the top and fills the remaining space in the menu with a list of the user's messages. The bubble/badge count on the messages tab indicates how many unread message notifications there are.
Previously we were only `yarn install`ing for linting and qunit runs. The Rails app now relies on a number of `node_modules` dependencies (e.g. for pretty_text, and discourse_js_processor), so we need to make sure they're available.
This is a much better description of its function. It performs idempotent normalization of a URL. If consumers truly need to `encode` a URL (including double-encoding of existing encoded entities), they can use the existing `.encode` method.
normalized_encode in addressable has a number of issues, including https://github.com/sporkmonger/addressable/issues/472
To temporaily work around those issues for the majority of cases, we try parsing with `::URI`. If that fails (e.g. due to non-ascii characters) then we will fall back to addressable.
Hopefully we can simplify this back to `Addressable::URI.normalized_encode` in the future.
This commit also adds support for unicode domain names and emoji domain names with escape_uri.
This removes an unneeded hack checking for pre-signed urls, which are now handled by the general case due to starting off valid and only being minimally normalized. Previous test case continues to pass.
UrlHelper.s3_presigned_url? which was somewhat wide was removed.
The new plugin list is based on the ones currently used in our ember-cli pipeline, and are based on our official browser support policy.
This commit includes an update to the raw-handlebars compiler to remove the 'very hacky but lets us use ES6' code. It's served us well for the last 6 years, but the babel config changes broke it (`const` -> `let`). This commit takes the opportunity to refactor it to take a similar approach to PrettyText, by leaning on `mini-loader.js`.
Some of the changes in this commit are extracted from https://github.com/discourse/discourse/pull/17379.
The bookmarks tab in the new user menu is different from the other tabs in that it can display a mixture of notifications and bookmarks. When there are unread bookmark reminder notifications, the tab displays all of these notifications at the top and fills the remaining space in the menu with the rest of the bookmarks. The bubble/badge count on the bookmarks tab indicates how many unread bookmark reminder notifications there are.
On the technical aspect, since this commit introduces a new `bookmark-item` component, we've done some refactoring so that all 3 "item" components (`notification-item`, `reviewable-item` and the new `bookmark-item`) inherit from a base component and get identical HTML structure so they all look consistent.
Internal tickets: t70584 and t65045.
Our internal implementation of #perform on jobs performs remapping.
This happens cause we do "exception aggregation".
Scheduled jobs run on every site in the multisite cluster, and we report
one error per site that failed. During this aggregation we reshape the
context from the original object shape returned by mini_scheduler
The new integration test ensures this interface will remain stable even if
decoupled parts of the code change shapes.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Followup to d66115d918
* Makes sure the `actor_preferences` all initialize with an empty array instead of nil if there are no preferences e.g. the actor is not ignoring anyone
* If the actor has disabled all PMs make `actor_disallowing_pms?` always return true
* FIX: don't memoize site setting in guardian
Memoizing site settings can make tests more fragile and harder to debug
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
This commit introduces several fine-grained methods
to UserCommScreener which can be used to show the actor
who they are ignoring/muting/blocking DMs from in order
to prevent them initiating conversation with those users
or to display relevant information in the UI to the
actor.
This will be used in a companion PR in discourse-chat,
and is a follow up to 74584ff3ca
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Co-authored-by: Osama Sayegh <asooomaasoooma90@gmail.com>
* FEATURE: track stats around failing scheduled jobs
Discourse.job_exception_stats can now be used to gather stats around how
many regular scheduled jobs failed in the current process.
This will be consumed by the Prometheus plugin and potentially other
monitoring plugins.
Allow users to specify the seed of the tests using the env variable RSPEC_SEED
Example:
bundle exec rake "plugin:spec[plugin-name]" RSPEC_SEED=65536
This is useful while fixing flaky tests.
* FEATURE: Add case-sensitivity flag to watched_words
Currently, all watched words are matched case-insensitively. This flag
allows a watched word to be flagged for case-sensitive matching.
To allow allow for backwards compatibility the flag is set to false by
default.
* FEATURE: Support case-sensitive creation of Watched Words via API
Extend admin creation and upload of Watched Words to support case
sensitive flag. This lays the ground work for supporting
case-insensitive matching of Watched Words.
Support for an extra column has also been introduced for the Watched
Words upload CSV file. The new column structure is as follows:
word,replacement,case_sentive
* FEATURE: Enable case-sensitive matching of Watched Words
WordWatcher's word_matcher_regexp now returns a list of regular
expressions instead of one case-insensitive regular expression.
With the ability to flag a Watched Word as case-sensitive, an action
can have words of both sensitivities.This makes the use of the global
Regexp::IGNORECASE flag added to all words problematic.
To get around platform limitations around the use of subexpression level
switches/flags, a list of regular expressions is returned instead, one for each
case sensitivity.
Word matching has also been updated to use this list of regular expressions
instead of one.
* FEATURE: Use case-sensitive regular expressions for Watched Words
Update Watched Words regular expressions matching and processing to handle
the extra metadata which comes along with the introduction of
case-sensitive Watched Words.
This allows case-sensitive Watched Words to matched as such.
* DEV: Simplify type casting of case-sensitive flag from uploads
Use builtin semantics instead of a custom method for converting
string case flags in uploaded Watched Words to boolean.
* UX: Add case-sensitivity details to Admin Watched Words UI
Update Watched Word form to include a toggle for case-sensitivity.
This also adds support for, case-sensitive testing and matching of Watched Word
in the admin UI.
* DEV: Code improvements from review feedback
- Extract watched word regex creation out to a utility function
- Make JS array presence check more explicit and readable
* DEV: Extract Watched Word regex creation to utility function
Clean-up work from review feedback. Reduce code duplication.
* DEV: Rename word_matcher_regexp to word_matcher_regexp_list
Since a list is returned now instead of a single regular expression,
change `word_matcher_regexp` to `word_matcher_regexp_list` to better communicate
this change.
* DEV: Incorporate WordWatcher updates from upstream
Resolve conflicts and ensure apply_to_text does not remove non-word characters in matches
that aren't at the beginning of the line.
Fixes edge case from fa5f3e228c.
In case the acting user is sent in with the target_user_ids,
we do not need to load those preferences, because even if the
acting user is preventing PMs or muting etc they need to always be able to
send themselves messages.
The "everyone" group is an automatic group and GroupUser records do not
exist for it. This commit allows all users if the group everyone is one
of the groups in the setting "pm_tags_allowed_for_groups".
* FIX: Rejected emails should not be cleaned up before their logs
If we delete the rejected emails before we delete their associated logs
we will receive 404 errors trying to inspect an email message for that
log.
* don't add a blank line
* test for max value as well
* pr cleanup and add migration
* Fix failing test
* FEATURE: revamped wizard
* UX: Wizard redesign (#17381)
* UX: Step 1-2
* swap out images
* UX: Finalize all steps
* UX: mobile
* UX: Fix test
* more test
* DEV: remove unneeded wizard components
* DEV: fix wizard tests
* DEV: update rails tests for new wizard
* Remove empty hbs files that were created because of rebase
* Fixes for rebase
* Fix wizard image link
* More rebase fixes
* Fix rails tests
* FIX: Update preview for new color schemes: (#17481)
* UX: make layout more responsive, update images
* fix typo
* DEV: move discourse logo svg to template only component
* DEV: formatting improvements
* Remove unneeded files
* Add tests for privacy step
* Fix banner image height for step "ready"
Co-authored-by: Jordan Vidrine <30537603+jordanvidrine@users.noreply.github.com>
Co-authored-by: awesomerobot <kris.aubuchon@discourse.org>
Currently when generating oneboxes if the connection timeouts and we’re
using the `FinalDestination#get` method, then it raises an exception.
We already catch this exception when using the
`FinalDestination#resolve` method so this patch just applies the same
logic to `FinalDestination#get`.
Our theme system is very complex and it can take a while to figure out how to invalidate the various types of caches that are used throughout the theme system. So, having a single helper method that invalidates everything can be useful in emergency situations where there is no time to read through the code and figure out how to clear the various caches.
Internal ticket: t64732.
When the experimental Sidebar is enabled, the hamburger drop down is replaced by a sidebar drop down. A user is given the ability to dock and undock the sidebar depending on their personal preference.
Do also note that the experimental sidebar is well, considered experimental at this point so I do not intend for the features here to be perfect. What I aim to do here is to ship the changes fast so that it can be used internally by the team to provide feedback. Custom links added by plugins and dark mode toggle has not been implemented as part of this commit as I aim to tackle it in another commit.
Co-authored-by: awesomerobot <kris.aubuchon@discourse.org>
When a topic was published from a shared draft and it had tags, the
users watching the tags were not notified. The problem was that the
topics are usually created in a secret category and publishing it just
moves an existent topic to the target category, without making any
changes to the tags.
Also, the change in insert-hyperlink (from `this.linkUrl.indexOf("http") === -1` to `!this.linkUrl.startsWith("http")`) was intentional fix: we don't want to prevent users from looking up topics with http in their titles.
Tag edit notifications are either created by PostActionNotifier or
PostRevisor. PostActionNotifier already checks if the site setting is
enabled. but PostRevisor scheduled a NotifyTagChange job without
checking disable_tags_edit_notifications.
This commit introduces a new plugin API to register
a group of stats that will be included in about.json
and also conditionally in the site about UI at /about.
The usage is like this:
```ruby
register_about_stat_group("chat_messages", show_in_ui: true) do
{
last_day: 1,
"7_days" => 10,
"30_days" => 100,
count: 1000,
previous_30_days: 120
}
end
```
In reality the stats will be generated any way the implementer
chooses within the plugin. The `last_day`, `7_days`, `30_days,` and `count`
keys must be present but apart from that additional stats may be added.
Only those core 4 stat keys will be shown in the UI, but everything will be shown
in about.json.
The stat group name is used to prefix the stats in about.json like so:
```json
"chat_messages_last_day": 2322,
"chat_messages_7_days": 2322,
"chat_messages_30_days": 2322,
"chat_messages_count": 2322,
```
The `show_in_ui` option (default false) is used to determine whether the
group of stats is shown on the site About page in the Site Statistics
table. Some stats may be needed purely for reporting purposes and thus
do not need to be shown in the UI to admins/users. An extension to the Site
serializer, `displayed_about_plugin_stat_groups`, has been added so this
can be inspected on the client-side.
It makes more sense to use user_ids for the UserCommScreener
introduced in fa5f3e228c since
in most cases the ID will be available, not the username. This
was discovered while starting work on a plugin that will
use this. In the cases where only usernames are available
the extra query is negligble.
The idea behind this refactor is to centralise all of the user ignoring / muting / disallow PM checks in a single place, so they can be used consistently in core as well as for plugins like chat, while improving the main bulk of the checks to run in a single fast non-AR query.
Also fixed up the invite error when someone is muting/ignoring the user that is trying to invite them to the topic.
* FIX: min/max username length limits weren't validated
The custom validators introduced in e0d7cda made so we ignored the mix
and max values set on site_settings.yml. That change allowed admins to
set values outside of the range defined on the yaml file.
Related to https://meta.discourse.org/t/group-names-with-more-than-60-characters-broken/232115?u=falco
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
All files under `app/assets/javascripts/scripts` will be run through babel, given sourcemaps, and output under `/assets/{name}.js`. This is another step towards removing our sprockets dependence.
## Without multisite.yml config
No change. `bin/rails db:create` / `db:migrate` / `db:drop` should work the same.
## With multisite.yml config
### db:create
`bin/rails db:create` creates development, test, and all databases from the multisite config
`RAILS_DB=[site] bin/rails db:create` creates the database for the specified site from the multisite config
### db:migrate
`bin/rails db:migrate` migrates the development database and all databases from the multisite config
`RAILS_ENV=test bin/rails db:migrate` migrates the test database and `discourse_test_multisite`
`RAILS_DB=[site] bin/rails db:migrate` migrates the database for the specified site from the multisite config
### db:drop
`bin/rails db:drop` drops development, test, and all databases from the multisite config
`RAILS_DB=[site] bin/rails db:create` drops the database for the specified site from the multisite config
This commit adds preload links for core/plugin/theme CSS stylesheets in the head.
Preload links are non-blocking and run in parallel. This means that they should have already been downloaded by the time we use the actual stylesheets (in the <body> tag).
Google is currently complaining about this here and this PR will address that warning.
This commit will also fix an issue in the splash screen where it sometimes doesn't respect the theme colors - causing a slightly jarring experience on dark themes.
Note that I opted not to add new specs because the underlying work required already has a lot of coverage. The new methods only change the output HTML so we can chuck that in the document <head>
This change also means that we can make all the stylesheets non-render blocking, but that will follow in a separate commit.
Logging out failed when the current user was cached by an instance of `Auth::DefaultCurrentUserProvider` and `#log_off_user` was called on a different instance of that class.
Co-authored-by: Sam <sam.saffron@gmail.com>
This happened when a middleware accessed the `currentUser` before a controller had a chance to populate the `action_dispatch.request.path_parameters` env variable. In that case Discourse would always cache `nil` as `currentUser`.
We previously relied on CSS animation-delay for the splash. This means that we can get inconsistent results based on device/network conditions.
This PR moves us to a more consistent timing based on {request time + 2 seconds}
Internal topic: /t/65378/65
Before, whispers were only available for staff members.
Config has been changed to allow to configure privileged groups with access to whispers. Post migration was added to move from the old setting into the new one.
I considered having a boolean column `whisperer` on user model similar to `admin/moderator` for performance reason. Finally, I decided to keep looking for groups as queries are only done for current user and didn't notice any N+1 queries.
Updates automatically data on the stats section of the topic.
It will update automatically the following information: likes, replies and last reply (timestamp and user)
- Sets `https://www.mixcloud.com` as a `requires_iframe_origins` to allow the iframe content to be displayed
- Attempts to render something approximating the Mixcloud content in the preview pane of the Composer, rather than just displaying a large version of the artwork associated with the link
Now that we've switched to Ember CLI, these things are no longer used.
- These sprockets manifests are superceded by the assets generated by ember cli
- These vendored scripts are now fetched by ember-auto-import at compile time
At some point in the past we decided to rename the 'regular' notification state of topics/categories to 'normal'. However, some UI copy was missed when the initial renaming was done so this commit changes the spots that were missed to the new name.
Anyone still using `EMBER_CLI_PROD_ASSETS=0` in development or production will be gracefully switched to Ember CLI. In development, a repeated message will be logged to STDERR.
Similarly, passing `QUNIT_EMBER_CLI=0` to the qunit rake task will now do nothing. A warning will be printed, and ember-cli mode will be used. Note that we've chosen not to fail the task, so that existing plugin/theme CI jobs don't immediately start failing. We may switch to a hard fail in the coming days/weeks.
This doesn't cope well with gzipped, binary, or large responses. Ideally we would teach FinalDestination to safely retrieve and decode some of the response body. But for now, let's remove the broken implementation.
* The `javascript:update` rake task failed because recent versions of chart.js use a lowercase filename (`chart.min.js` instead of `Chart.min.js`)
* Changed `loadScript()` to use lowercase keys to lookup scripts
* `svg-arrow.css` seems to have changed slightly (linebreak at the end of file)
When sending emails with delivery_method_options -> return_response
set to true, the SMTP sending code inside Mail will return the SMTP
response when calling deliver! for mail within the app. This commit
ensures that Email::Sender captures this response if it is returned
and stores it against the EmailLog created for the sent email.
A follow up PR will make this visible within the admin email UI.
* FIX: Fix a bug that is accessing the values in a hash wrongly and write tests
I decided to write tests in order to be confident in my refactor that's in the next commit.
Meanwhile I have discovered a potential bug. The `title_attr` key was accessed as a string,
but all the keys are actually symbols so it was never evaluated to be true.
irb(main):025:0> d = {key: 'value'}
=> {:key=>"value"}
irb(main):026:0> d['key']
=> nil
irb(main):027:0> d[:key]
=> "value"
* DEV: Extract methods for readability
I will be adding a new method following the conventions in place for adding a new normalizer. And this will make the readability of the `raw` block even more difficult; so I am extracting self contained private methods beforehand.
* FEATURE: Parse JSON-LD and introduce Movie object
JSON LD data is very easily transferable to Ruby objects because they contain types. If these types are mapped to Ruby objects, it is also better to make all the parsed data very explicit and easily extendable.
JSON-LD has many more standardized item types, with a full list here: https://schema.org/docs/full.html
However in order to decrease the scope, I only adapted the movie type.
* DEV: Change inheritance between normalizers
Normalizers are not supposed to have an inheritance relationships amongst each other. They are all normalizers, but all normalizing separate protocols. This is why I chose to extract a parent class and relieve Open Graph off that responsibility. Removing the parent class altogether could also a possibility, but I am keeping the scope limited to having a more accurate representation of the normalizers while making it easier to add a new one.
* Lint changes
* Bring back the Oembed OpenGraph inheritance
There is one test that caught that this inheritance was necessary. I still think modelling wise this inheritance shouldn't exist, but this can be tackled separately.
* Return empty hash if the json received is invalid
Before this change if there was a parsing error with JSON it would throw an exception. The goal of this commit is to rescue that exception and then log a warning. I chose to use Discourse's logger wrapper `warn_exception` to have the backtrace and not just used Rails logger. I considered raising an `InvalidParameters` error however if the JSON here is invalid it should not block showing of the Onebox, so logging is enough.
* Prep to support more JSONLD schema types with case
* Extract mustache template object created from JSONLD
This PR changes the rescue block to rescue only Net::TimeoutError exceptions and removes the log line to prevent clutter the logs with errors that are ignored. Other errors can bubble up because they're errors we probably want to know about
This table holds associations between uploads and other models. This can be used to prevent removing uploads that are still in use.
* DEV: Create upload_references
* DEV: Use UploadReference instead of PostUpload
* DEV: Use UploadReference for SiteSetting
* DEV: Use UploadReference for Badge
* DEV: Use UploadReference for Category
* DEV: Use UploadReference for CustomEmoji
* DEV: Use UploadReference for Group
* DEV: Use UploadReference for ThemeField
* DEV: Use UploadReference for ThemeSetting
* DEV: Use UploadReference for User
* DEV: Use UploadReference for UserAvatar
* DEV: Use UploadReference for UserExport
* DEV: Use UploadReference for UserProfile
* DEV: Add method to extract uploads from raw text
* DEV: Use UploadReference for Draft
* DEV: Use UploadReference for ReviewableQueuedPost
* DEV: Use UploadReference for UserProfile's bio_raw
* DEV: Do not copy user uploads to upload references
* DEV: Copy post uploads again after deploy
* DEV: Use created_at and updated_at from uploads table
* FIX: Check if upload site setting is empty
* DEV: Copy user uploads to upload references
* DEV: Make upload extraction less strict
Following the Rails 7 upgrade, the `DISCOURSE_SMTP_ENABLE_START_TLS`
setting doesn’t work anymore. This is because Rails upgraded the
`net-smtp` gem to the 0.3.1 version which enables `starttls` by default.
The `mail` gem doesn’t support this new behavior yet and doesn’t know
how to disable TLS. This should be fixed in an upcoming release.
Meanwhile applying this patch allows us to get back the previous
behavior which is expected by many.
This commit introduces a new site setting: `block_hotlinked_media`. When enabled, all attempts to hotlink media (images, videos, and audio) will fail, and be replaced with a linked placeholder. Exceptions to the rule can be added via `block_hotlinked_media_exceptions`.
`download_remote_image_to_local` can be used alongside this feature. In that case, hotlinked images will be blocked immediately when the post is created, but will then be replaced with the downloaded version a few seconds later.
This implementation is purely server-side, and does not impact the composer preview.
Technically, there are two stages to this feature:
1. `PrettyText.sanitize_hotlinked_media` is called during `PrettyText.cook`, and whenever new images are introduced by Onebox. It will iterate over all src/srcset attributes in the post HTML and check if they're allowed. If not, the attributes will be removed and replaced with a `data-blocked-hotlinked-src(set)` attribute
2. In the `CookedPostProcessor`, we iterate over all `data-blocked-hotlinked-src(set)` attributes and check whether we have a downloaded version of the media. If yes, we update the src to use the downloaded version. If not, the entire media element is replaced with a placeholder. The placeholder is labelled 'external media', and is a link to the offsite media.
Twitter does not allow SVGs to be used for twitter:image
metadata (see https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/markup)
so we should fall back to the site logo if the image option
provided to `crawlable_meta_data` or SiteSetting.site_twitter_summary_large_image_url
is an SVG, and do not add the meta tag for twitter:image at all
if the site logo is an SVG.
This commit seeks to only handle the `f=tracked` and `filter=tracked`
query params for a topic list. There are other "hidden" filters for a
topic list which can be activated by passing the right query param to
the request. However, they are hidden because there is no way to
activate those filters via the UI. We are handling the `f=tracked`
filter because we will soon be adding a link that allows a user to
quickly view their tracked topics.
Previously we limited Discourse Connect provider to 1 secret per domain.
This made it pretty awkward to cycle secrets in environments where config
takes time to propagate
This change allows for the same domain to have multiple secrets
Also fixes internal implementation on DiscourseConnectProvider which was
not thread safe as it leaned on class variables to ferry data around
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Co-authored-by: David Taylor <david@taylorhq.com>
In 7328a2bfb0 we changed the
InlineOneboxer#onebox_for method to run the title of the
onebox through WatchedWord#censor_text. However, it is
allowable for the title to be nil, which was causing this
error in production:
> NoMethodError : undefined method gsub for nil:NilClass
We just need to check whether the title is nil before trying
to censor it.
Previously we hardcoded the DOWNLOAD_URL_EXPIRES_AFTER_SECONDS const
inside S3Helper to be 5 minutes (300 seconds). For various reasons,
some hosted sites may need this to be longer for other integrations.
The maximum expiry time for presigned URLs is 1 week (which is
604800 seconds), so that has been added as a validation on the
setting as well. The setting is hidden because 99% of the time
it should not be changed.
Censored watched words were not censored inside the title of an inline
oneboxes. Malicious users could exploit this behaviour to insert bad
words. The same issue has been fixed for regular Oneboxes in commit
d184fe59ca.
When searching for PMs or PMs in a group inbox, results in the header search were not being limited to 5 with a "More" link to the full page search. This PR fixes that.
It also simplifies the logic and updates the search API docs to include recently added `in:messages` and `group_messages:groupname` options.
When saving / creating bookmarks, we have code to save
the user's preference of bookmark_auto_delete_preference
to their user_options.
Unfortunately this can cause weirdness when plugins
have code using BookmarkManager to set the auto delete preference for
only a specific bookmark.
This commit introduces a save_user_preferences option (false
by default) so that this user preference is not saved unless
specified by the consumer of BookmarkManager, so plugins will
not have to worry about it.
Previously, with the default `editing_grace_period`, hotlinked images were pulled 5 minutes after a post is created. This delay was added to reduce the chance of automated edits clashing with user edits.
This commit refactors things so that we can pull hotlinked images immediately. URLs are immediately updated in the post's `cooked` HTML. The post's raw markdown is updated later, after the `editing_grace_period`.
This involves a number of behind-the-scenes changes including:
- Schedule Jobs::PullHotlinkedImages immediately after Jobs::ProcessPost. Move scheduling to after the `update_column` call to avoid race conditions
- Move raw changes into a separate job, which is delayed until after the ninja-edit window
- Move disable_if_low_on_disk_space logic into the `pull_hotlinked_images` job
- Move raw-parsing/replacing logic into `InlineUpload` so it can be easily be shared between `UpdateHotlinkedRaw` and `PullUserProfileHotlinkedImages`
Previously this mapping of **cooked** images was only being run for oneboxes. Now it runs for all images, so we can transform hotlinked images without needing to immediately update `raw`
This feature only was only demuxing stdout, not stderr. That means that stdout and stderr output appears out-of-order, and makes debugging migrations very confusing.
In future we may want to add stderr support to the demuxing. But right now, the concurrency variable is hard-coded to 1. Therefore the easiest fix is to bypass the demuxing.
Meta topic: https://meta.discourse.org/t/prevent-to-linkify-when-there-is-a-redirect/226964/2?u=osama.
This commit adds a new site setting `block_onebox_on_redirect` (default off) for blocking oneboxes (full and inline) of URLs that redirect. Note that an initial http → https redirect is still allowed if the redirect location is identical to the source (minus the scheme of course). For example, if a user includes a link to `http://example.com/page` and the link resolves to `https://example.com/page`, then the link will onebox (assuming it can be oneboxed) even if the setting is enabled. The reason for this is a user may type out a URL (i.e. the URL is short and memorizable) with http and since a lot of sites support TLS with http traffic automatically redirected to https, so we should still allow the URL to onebox.
Incorporates learnings from /t/64227:
* Changes the code to set access control posts in the rake
task to be an efficient UPDATE SQL query.
The original version was timing out with 312017 post uploads,
the new query took ~3s to run.
* Changes the code to mark uploads as secure/not secure in
the rake task to be an efficient UPDATE SQL query rather than
using UploadSecurity. This took a very long time previously,
and now takes only a few seconds.
* Spread out ACL syncing for uploads into jobs with batches of
100 uploads at a time, so they can be parallelized instead
of having to wait ~1.25 seconds for each ACL to be changed
in S3 serially.
One issue that still remains is post rebaking. Doing this serially
is painfully slow. We have a way to do this in sidekiq via PeriodicalUpdates
but this is limited by max_old_rebakes_per_15_minutes. It would
be better to fan this rebaking out into jobs like we did for the
ACL sync, but that should be done in another PR.
This commit migrates all bookmarks to be polymorphic (using the
bookmarkable_id and bookmarkable_type) columns. It also deletes
all the old code guarded behind the use_polymorphic_bookmarks setting
and changes that setting to true for all sites and by default for
the sake of plugins.
No data is deleted in the migrations, the old post_id and for_topic
columns for bookmarks will be dropped later on.
22a7905f restructured how we load Ember CLI assets in production. Unfortunately, it also broke sourcemaps for those assets. This commit fixes that regression via a couple of changes:
- It adds the necessary `.map` paths to `config.assets.precompile`
- It swaps Sprockets' default `SourcemappingUrlProcessor` with an extended version which maintains relative URLs of maps
Enabled extensions are not saved in backups and thus not created when
restoring a newer backup (that has a new extension) to an old site
(that does not have the migration).
When we build and send emails using MessageBuilder and Email::Sender
we add custom headers defined in SiteSetting.email_custom_headers.
However this was causing errors in cases where the custom headers
defined a header that we already specify in outbound emails (e.g.
the Precedence: list header for topic/post emails).
This commit makes it so we always use the header value defined in Discourse
core if there is a duplicate, discarding the custom header value
from the site setting.
cf. https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/14
These validate/after_create/after_destroy methods were added
back in b8828d4a2d before
the RegisteredBookmarkable API and pattern was nailed down.
This commit updates BookmarkManager to call out to the
relevant bookmarkable for these and bookmark_metadata for
consistency.
Currently the only way to allow tagging on pms is to use the `allow_staff_to_tag_pms` site setting. We are removing that site setting and replacing it with `pm_tags_allowed_for_groups` which will allow for non staff tagging. It will be group based permissions instead of requiring the user to be staff.
If the existing value of `allow_staff_to_tag_pms` is `true` then we include the `staff` groups as a default for `pm_tags_allowed_for_groups`.
