Commit Graph

308 Commits

Author SHA1 Message Date
Alan Guo Xiang Tan
57f4176b57
DEV: Bump rubocop_discourse () 2024-11-06 06:27:49 +08:00
David Taylor
80b9c280ba
DEV: Switch to pnpm for JS dependencies ()
This will bring significant improvements to install speed & storage requirements. For information on how it may affect you, see https://meta.discourse.org/t/324521

This commit:
- removes the `yarn.lock` and replaces with `pnpm-lock.yaml`
- updates workspaces to pnpm format
- adjusts package dependencies to work with pnpm's stricter resolution strategy
- updates Rails app to load modules from more specific node_modules directories
- adds a `.pnpmfile` which automatically cleans up old yarn-managed `node_modules` directories
- updates various scripts to call `pnpm` instead of `yarn`
- updates patches to use pnpm's native patch system instead of patch-package
- adds a patch for licensee to support pnpm
2024-09-03 10:51:07 +01:00
Jarek Radosz
064332ef6e
DEV: Update ember-cli-deprecation-workflow from 2.2.0 to 3.0.1 ()
From dependabot PR:

<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/ember-cli/ember-cli-deprecation-workflow/releases">ember-cli-deprecation-workflow's releases</a>.</em></p>
<blockquote>
<h2>v3.0.1-ember-cli-deprecation-workflow</h2>
<h2>Release (2024-07-11)</h2>
<p>ember-cli-deprecation-workflow 3.0.1 (patch)</p>
<h4>🏠 Internal</h4>
<ul>
<li><code>ember-cli-deprecation-workflow</code>
<ul>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/192">#192</a> fix repository link in package.json (<a href="https://github.com/mansona"><code>@​mansona</code></a>)</li>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/191">#191</a> update release plan workflow (<a href="https://github.com/mansona"><code>@​mansona</code></a>)</li>
</ul>
</li>
</ul>
<h4>Committers: 1</h4>
<ul>
<li>Chris Manson (<a href="https://github.com/mansona"><code>@​mansona</code></a>)</li>
</ul>
<h2>v3.0.0</h2>
<h2>Release (2024-06-25)</h2>
<p>ember-cli-deprecation-workflow 3.0.0 (major)</p>
<h4>💥 Breaking Change</h4>
<ul>
<li><code>ember-cli-deprecation-workflow</code>
<ul>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/159">#159</a> [BREAKING] Convert to a module. Drops support for Ember &lt; 3.28, requires manual initialization (<a href="https://github.com/lolmaus"><code>@​lolmaus</code></a>)</li>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/175">#175</a> Node 16 is the minimum supported version (<a href="https://github.com/mixonic"><code>@​mixonic</code></a>)</li>
</ul>
</li>
</ul>
<h4>🐛 Bug Fix</h4>
<ul>
<li><code>ember-cli-deprecation-workflow</code>
<ul>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/181">#181</a> Remove unused broccoli magic (<a href="https://github.com/simonihmig"><code>@​simonihmig</code></a>)</li>
</ul>
</li>
</ul>
<h4>📝 Documentation</h4>
<ul>
<li><code>ember-cli-deprecation-workflow</code>
<ul>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/184">#184</a> Update configuration paths in documentation (<a href="https://github.com/backspace"><code>@​backspace</code></a>)</li>
</ul>
</li>
</ul>
<h4>🏠 Internal</h4>
<ul>
<li><code>ember-cli-deprecation-workflow</code>
<ul>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/189">#189</a> start using release-plan (<a href="https://github.com/mansona"><code>@​mansona</code></a>)</li>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/188">#188</a> start using pnpm (<a href="https://github.com/mansona"><code>@​mansona</code></a>)</li>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/178">#178</a> Upgrade Ember CLI to 5.4 (<a href="https://github.com/lolmaus"><code>@​lolmaus</code></a>)</li>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/170">#170</a> Bump Node, swap to npm, update CI pipeline (<a href="https://github.com/mixonic"><code>@​mixonic</code></a>)</li>
</ul>
</li>
</ul>
<h4>Committers: 5</h4>
<ul>
<li>Andrey Mikhaylov (lolmaus) (<a href="https://github.com/lolmaus"><code>@​lolmaus</code></a>)</li>
<li>Buck Doyle (<a href="https://github.com/backspace"><code>@​backspace</code></a>)</li>
<li>Chris Manson (<a href="https://github.com/mansona"><code>@​mansona</code></a>)</li>
<li>Matthew Beale (<a href="https://github.com/mixonic"><code>@​mixonic</code></a>)</li>
<li>Simon Ihmig (<a href="https://github.com/simonihmig"><code>@​simonihmig</code></a>)</li>
</ul>
</blockquote>
</details>
2024-08-07 17:59:42 +02:00
Blake Erickson
30048ab97f
FIX: Video thumbnails for missing videos ()
Skip trying to find a thumbnail if the video src cannot be found.

Bug report: https://meta.discourse.org/t/317423
2024-07-31 10:54:52 -06:00
锦心
5b05cdfbd9
FIX: Add post id to the anchor to prevent two identical anchors ()
* FIX: Add post id to the anchor to prevent two identical anchors

We generate anchors for headings in posts. This works fine if there is
only one post in a topic with anchors. The problem comes when you have
two or more posts with the same heading. PrettyText generates anchors
based on the heading text using the raw context of each post, so it is
entirely possible to generate the same anchor for two posts in the same
topic, especially for topics with template replies

    Post1:
    # heading
    context
    Post2:
    # heading
    context

When both posts are on the page at the same time, the anchor will only
work for the first post, according to the [HTML specification](https://html.spec.whatwg.org/multipage/browsing-the-web.html#scroll-to-the-fragment-identifier).

> If there is an a element in the document tree whose root is document
> that has a name attribute whose value is equal to fragment, then
> return the *first* such element in tree order.

This bug is particularly serious in forums with non-Latin languages,
such as Chinese. We do not generate slugs for Chinese, which results in
the heading anchors being completely dependent on their order.

```ruby
[2] pry(main)> PrettyText.cook("# 中文")
=> "<h1><a name=\"h-1\" class=\"anchor\" href=\"#h-1\"></a>中文</h1>"
```

Therefore, the anchors in the two posts must be in exactly the same by
order, causing almost all of the anchors in the second post to be
invalid.

This commit solves this problem by adding the `post_id` to the anchor.
The new anchor generation method will add `p-{post_id}` as a prefix when
post_id is available:

```ruby
[3] pry(main)> PrettyText.cook("# 中文", post_id: 1234)
=> "<h1><a name=\"p-1234-h-1\" class=\"anchor\" href=\"#p-1234-h-1\"></a>中文</h1>"
```

This way we can ensure that each anchor name only appears once on the
same topic. Using post id also prevents the potential possibility of the
same anchor name when splitting/merging topics.
2024-07-25 13:50:30 +08:00
Martin Brennan
964945a11f
DEV: Add case-converter common lib and use with AdminPlugin ()
The AdminPlugin JS model uses a similar pattern to chat models,
where it is a plain JS class manually converting provided
snake_case attributes from the serializer to JS camelCase.

However this doesn't work when it comes to using `add_to_serializer`
in plugins since core does not know about these new attributes.

Instead, we can use a JS function to convert snake_case to camelCase
and use that when initializing AdminPlugin. This commit also moves
similar functions to a new case-converter.js file in
discourse-common/lib.
2024-07-05 13:22:48 +10:00
Bianca Nenciu
2ce83b4d2e
FIX: Do not extract links for hotlinked images ()
When a post is cooked the links are extracted and `TopicLink` instances
are created for each of them. These links are used in various places,
including the topic view, user summary page, etc.

In previous commit 48e5d1a, hotlinked images from Oneboxes have been
ignored from the texts, but hotlinked images turned into Lightboxes
were still extracted.
2024-06-19 14:25:29 +03:00
Jarek Radosz
42a529f9ae
PERF: Avoid excessive object creations in watched words ()
Inline the helper functions, avoid creating and then immediately destructuring arrays, use complete strings instead of string interpolation, Map instead of a pojo.
2024-06-10 14:44:31 +02:00
Régis Hanol
8f7a3e5b29 FIX: subfolder absolute links in summaries
This fixes the `PrettyText.make_all_links_absolute` to better handle subfolder.

In subfolder, when given the cooked version of a post, links to mentions includes the `Discourse.base_path` prefix. Adding the `Discourse.base_url` was doubling the `Discourse.base_path`.

The issue was hidden behind the specs which was stubbing `Discourse.base_url` instead of relying on `Discourse.base_path`.

This fixes both the "algorithm" used in `PrettyText.make_all_links_absolute` to better handle this case and correct the specs to properly handle subfolder cases.

There are lots of changes in the specs due to a refactoring to use squiggly heredoc strings for easier reading and less escaping.
2024-05-22 15:38:18 +02:00
Martin Brennan
0d0dbd391a
DEV: Rename with_secure_uploads? to should_secure_uploads? on Post ()
This method name is a bit confusing; with_secure_uploads implies
it may return a block or something with the uploads of the post,
and has_secure_uploads implies that it's checking whether the post
is linked to any secure uploads.

should_secure_uploads? communicates the true intent of this method --
which is to say whether uploads attached to this post should be
secure or not.
2024-04-09 13:23:11 +10:00
Sam
34a14112a7
FIX: remove "fake" mentions from extract_mentions ()
```
<a class="mention" href="/u/test1">bsam</a>
```

Is not a mention of the user sam. We expect an @ in front always.
2024-03-20 12:20:15 +11:00
David Taylor
588a79c80c
DEV: Merge root JS packages ()
Before this commit, we had a yarn package set up in the root directory and also in `app/assets/javascripts`. That meant two `yarn install` calls and two `node_modules` directories. This commit merges them both into the root location, and updates references to node_modules.

A previous attempt can be found at https://github.com/discourse/discourse/pull/21172. This commit re-uses that script to merge the `yarn.lock` files.

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2024-02-26 13:45:58 +00:00
Blake Erickson
bb261094cf
FEATURE: Auto generate and display video preview image ()
This change will allow auto generated video thumbnails to be used
instead of the black video thumbnail that overlays videos.

Follow up to: 2443446e62
2024-02-14 13:43:53 -07:00
David Taylor
dea753a204
FIX: Restore support for .js.es6 files in PrettyText ()
Regressed in 1757a688c4

https://meta.discourse.org/t/294155
2024-02-07 09:34:31 +00:00
David Taylor
1757a688c4
DEV: Remove sprockets from plugin 'extra js' pipeline ()
JS assets added by plugins via `register_asset` will be outside the `assets/javascripts` directory, and are therefore exempt from being transpiled. That means that there isn't really any need to run them through DiscourseJsProcessor. Instead, we can just concatenate them together, and avoid the need for all the sprockets-wrangling.

This commit also takes the opportunity to clean up a number of plugin-asset-related codepaths which are no longer required (e.g. globs, handlebars)
2024-02-01 11:48:31 +00:00
Jarek Radosz
694b5f108b
DEV: Fix various rubocop lints ()
These (21 + 3 from previous PRs) are soon to be enabled in rubocop-discourse:

Capybara/VisibilityMatcher
Lint/DeprecatedOpenSSLConstant
Lint/DisjunctiveAssignmentInConstructor
Lint/EmptyConditionalBody
Lint/EmptyEnsure
Lint/LiteralInInterpolation
Lint/NonLocalExitFromIterator
Lint/ParenthesesAsGroupedExpression
Lint/RedundantCopDisableDirective
Lint/RedundantRequireStatement
Lint/RedundantSafeNavigation
Lint/RedundantStringCoercion
Lint/RedundantWithIndex
Lint/RedundantWithObject
Lint/SafeNavigationChain
Lint/SafeNavigationConsistency
Lint/SelfAssignment
Lint/UnreachableCode
Lint/UselessMethodDefinition
Lint/Void

Previous PRs:
Lint/ShadowedArgument
Lint/DuplicateMethods
Lint/BooleanSymbol
RSpec/SpecFilePathSuffix
2023-12-06 23:25:00 +01:00
Godfrey Chan
9a1695ccc1
DEV: remove markdown-it-bundle and custom build code ()
With Embroider, we can rely on async `import()` to do the splitting
for us.

This commit extracts from `pretty-text` all the parts that are
meant to be loaded async into a new `discourse-markdown-it` package
that is also a V2 addon (meaning that all files are presumed unused
until they are imported, aka "static").

Mostly I tried to keep the very discourse specific stuff (accessing
site settings and loading plugin features) inside discourse proper,
while the new package aims to have some resembalance of a general
purpose library, a MarkdownIt++ if you will. It is far from perfect
because of how all the "options" stuff work but I think it's a good
start for more refactorings (clearing up the interfaces) to happen
later.

With this, pretty-text and app/lib/text are mostly a kitchen sink
of loosely related text processing utilities.

After the refactor, a lot more code related to setting up the
engine are now loaded lazily, which should be a pretty nice win. I
also noticed that we are currently pulling in the `xss` library at
initial load to power the "sanitize" stuff, but I suspect with a
similar refactoring effort those usages can be removed too. (See
also ).

This PR does not attempt to fix the sanitize issue, but I think it
sets things up on the right trajectory for that to happen later.

Co-authored-by: David Taylor <david@taylorhq.com>
2023-11-06 16:59:49 +00:00
Bianca Nenciu
fd07c943ad
DEV: Refactor watched words ()
- Ignore only invalid words, not all words if one of them is invalid

- The naming scheme for methods was inconsistent

- Optimize regular expressions
2023-11-01 16:41:10 +02:00
Blake Erickson
2443446e62
DEV: Prevent videos from preloading metadata ()
Preloading just metadata is not always respected by browsers, and
sometimes the whole video will be downloaded. This switches to using a
placeholder image for the video and only loads the video when the play
button is clicked.
2023-10-12 13:47:48 -06:00
Jarek Radosz
94649565ce
DEV: Correct Style/RedundantReturn rubocop issues () 2023-08-10 02:03:38 +02:00
Martin Brennan
09223e5ae7
DEV: Remove enable_experimental_hashtag_autocomplete logic ()
This commit removes any logic in the app and in specs around
enable_experimental_hashtag_autocomplete and deletes some
old category hashtag code that is no longer necessary.

It also adds a `slug_ref` category instance method, which
will generate a reference like `parent:child` for a category,
with an optional depth, which hashtags use. Also refactors
PostRevisor which was using CategoryHashtagDataSource directly
which is a no-no.

Deletes the old hashtag markdown rule as well.
2023-08-08 11:18:55 +10:00
David Taylor
2fde58def4
DEV: Move avatar-utils into dedicated discourse-common module ()
These avatar-related helper functions are used in pretty-text, which currently means we load the entire `discourse/lib/utilities` module into the mini-racer when running pretty-text on the server side. This stops us adding any logic or imports to discourse/lib/utilities which may depend on other `discourse/` namespace features.

This commit moves the avatar-related utils into a dedicated module in the `discourse-common` namespace, adds backwards-compatibility shims, and updates the pretty-text config accordingly.
2023-07-12 09:06:16 +01:00
Sam
9e241e82e9
DEV: use HTML5 version of loofah ()
https://meta.discourse.org/t/markdown-preview-and-result-differ/263878

The result of this markdown had different results in the composer preview and the post. This is solved by updating Loofah to the latest version and using html5 fragments like our user had reported. While the change was only needed in cooked_post_processor.rb for this fix, other areas also had to be updated due to various side effects.
2023-06-20 09:49:22 +08:00
Sérgio Saquetim
4b22e67c8b
DEV: Added modifier to change mentions extracted from cooked text ()
Added a new modifier hook to allow plugins to modify the @mentions
extracted from a cooked text.

Use case: Some plugins may change how the mentions are cooked to prevent
them from being confused with user or group mentions and display the user
card.

This modifier hook allows the plugin to filter the mentions detected or add new ways
to add mentions into cooked text.
2023-06-15 10:52:52 -03:00
Joffrey JAFFEUX
7130047bd8
DEV: ensures __optInput is initialized ()
Test were sometimes failing with similar error to the following:

```
  1) UsernameChanger#override when unicode_usernames is off overrides the username if a new name has different case
     Failure/Error:
           protect { v8.eval(<<~JS) }
               __paths = #{paths_json};
               __utils.avatarImg({size: #{size.inspect}, avatarTemplate: #{avatar_template.inspect}}, __getURL);
             JS

     MiniRacer::RuntimeError:
       ReferenceError: __optInput is not defined
     # JavaScript at exports.helperContext (<anonymous>:21:17)
     # JavaScript at getRawAvatarSize (<anonymous>:108:49)
     # JavaScript at avatarUrl (<anonymous>:102:21)
     # JavaScript at Object.avatarImg (<anonymous>:129:15)
     # JavaScript at <anonymous>:2:9
     # ./lib/pretty_text.rb:259:in `block in avatar_img'
     # ./lib/pretty_text.rb:661:in `block in protect'
     # ./lib/pretty_text.rb:661:in `synchronize'
     # ./lib/pretty_text.rb:661:in `protect'
     # ./lib/pretty_text.rb:259:in `avatar_img'
     # ./app/jobs/regular/update_username.rb:14:in `execute'
```

This should not be needed as it should already have been initialised but that should stop the flakey-ness for now while being a safe change.
2023-06-01 12:58:15 +02:00
Sam
c2332d7505
FEATURE: reduce avatar sizes to 6 from 20 ()
* FEATURE: reduce avatar sizes to 6 from 20

This PR introduces 3 changes:

1. SiteSetting.avatar_sizes, now does what is says on the tin.
previously it would introduce a large number of extra sizes, to allow for
various DPIs. Instead we now trust the admin with the size list.

2. When `avatar_sizes` changes, we ensure consistency and remove resized
avatars that are not longer allowed per site setting. This happens on the
12 hourly job and limited out of the box to 20k cleanups per cycle, given
this may reach out to AWS 20k times to remove things.

3.Our default avatar sizes are now "24|48|72|96|144|288" these sizes were
very specifically picked to limit amount of bluriness introduced by webkit.
Our avatars are already blurry due to 1px border, so this corrects old blur.

This change heavily reduces storage required by forums which simplifies
site moves and more.

Co-authored-by: David Taylor <david@taylorhq.com>
2023-06-01 10:00:01 +10:00
Martin Brennan
0b3cf83e3c
FIX: Do not cook icon with hashtags ()
This commit makes some fundamental changes to how hashtag cooking and
icon generation works in the new experimental hashtag autocomplete mode.
Previously we cooked the appropriate SVG icon with the cooked hashtag,
though this has proved inflexible especially for theming purposes.

Instead, we now cook a data-ID attribute with the hashtag and add a new
span as an icon placeholder. This is replaced on the client side with an
icon (or a square span in the case of categories) on the client side via
the decorateCooked API for posts and chat messages.

This client side logic uses the generated hashtag, category, and channel
CSS classes added in a previous commit.

This is missing changes to the sidebar to use the new generated CSS
classes and also colors and the split square for categories in the
hashtag autocomplete menu -- I will tackle this in a separate PR so it
is clearer.
2023-05-23 09:33:55 +02:00
Bianca Nenciu
9a2780397f
FIX: Handle all UTF-8 characters ()
Watched words were converted to regular expressions containing \W, which
handled only ASCII characters. Using [^[:word]] instead ensures that
UTF-8 characters are also handled correctly.
2023-05-15 12:45:04 +03:00
Jarek Radosz
43e0025141
Revert "DEV: Merge package.json files ()" ()
This reverts commit 49a1e1cd0e.

Is causing issues in prod-adjacent environments (Jenkins)
2023-04-20 14:57:40 +02:00
Jarek Radosz
49a1e1cd0e
DEV: Merge package.json files ()
This means: a single yarn.lock and removing one of the package.json files
2023-04-20 12:46:12 +02:00
David Battersby
967010e545
FEATURE: Add an emoji deny list site setting ()
This feature will allow sites to define which emoji are not allowed. Emoji in this list should be excluded from the set we show in the core emoji picker used in the composer for posts when emoji are enabled. And they should not be allowed to be chosen to be added to messages or as reactions in chat.

This feature prevents denied emoji from appearing in the following scenarios:
- topic title and page title
- private messages (topic title and body)
- inserting emojis into a chat
- reacting to chat messages
- using the emoji picker (composer, user status etc)
- using search within emoji picker

It also takes into account the various ways that emojis can be accessed, such as:
- emoji autocomplete suggestions
- emoji favourites (auto populates when adding to emoji deny list for example)
- emoji inline translations
- emoji skintones (ie. for certain hand gestures)
2023-04-13 15:38:54 +08:00
Martin Brennan
e6feb3f136
DEV: Remove currentUser lookup in pretty-text ()
This was added in d3f02a1270
for hashtags but later removed usage in
b2acc416e7. It was removed because
serializing the user does not include things like their
secure_categories.

It is not used by any other plugins or themes, and can cause
issues where it will error when operating on a null user. Better
to just pass in the user_id and use it to look up a user
directly in a PrettyText::Helper
2023-03-30 14:50:36 +10:00
Jan Cernik
afe3e36363
DEV: Remove lazy-yt and replace with lazy-videos ()
- Refactors the old plugin to remove jquery usage
- Adds support for Vimeo videos (default on) and Tiktok (experimental and default off)
2023-03-29 11:54:25 -04:00
Ghassan Maslamani
96a6bb69b5
FIX: vimeo iframe url when data-original-href is missing () 2023-01-31 12:00:27 +01:00
Martin Brennan
63fdb6dd65
FIX: Do not add empty use/svg tags in ExcerptParser ()
There was an issue where if hashtag-cooked HTML was sent
to the ExcerptParser without the keep_svg option, we would
end up with empty </use> and </svg> tags on the parts of the
excerpt where the hashtag was, in this case when a post
push notification was sent.

Fixed this, and also added a way to only display a plaintext
version of the hashtag for cases like this via PrettyText#excerpt.
2023-01-24 14:40:24 +10:00
Daniel Waterworth
666536cbd1
DEV: Prefer \A and \z over ^ and $ in regexes () 2023-01-20 12:52:49 -06:00
David Taylor
6417173082
DEV: Apply syntax_tree formatting to lib/* 2023-01-09 12:10:19 +00:00
Martin Brennan
b2acc416e7
FIX: Server-side hashtag lookups of secure categories for a user ()
* FIX: Use Category.secured(guardian) for hashtag datasource

Follow up to comments in , changing the category
hashtag datasource to use Category.secured(guardian) instead
of Site.new(guardian).categories here since the latter does
more work for not much benefit, and the query time is the
same. Also eliminates some Hash -> Model back and forth
busywork. Add some more specs too.

* FIX: Server-side hashtag lookup cooking user loading

When we were using the PrettyText.options.currentUser
and parsing back and forth with JSON for the hashtag
lookups server-side, we had a bug where the user's
secure categories were not loaded since we never actually
loaded a User model from the database, only parsed it
from JSON.

This commit fixes the issue by instead using the
PretyText.options.userId and looking up the user directly
from the database when calling hashtag_lookup via the
PrettyText::Helpers code when cooking server-side. Added
the missing spec to check for this as well.
2022-12-09 10:34:25 +10:00
Andrei Prigorshnev
a76d864c51
FEATURE: Show live user status on inline mentions on posts ()
Note that we don't have a database table and a model for post mentions yet, and I decided to implement it without adding one to avoid heavy data migrations. Still, we may want to add such a model later, that would be convenient, we have such a model for mentions in chat.

Note that status appears on all mentions on all posts in a topic except of the case when you just posted a new post, and it appeared on the bottom of the topic. On such posts, status won't be shown immediately for now (you'll need to reload the page to see the status). I'll take care of it in one of the following PRs.
2022-12-06 19:10:36 +04:00
Martin Brennan
d3f02a1270
FEATURE: Generic hashtag autocomplete lookup and markdown cooking ()
This commit fleshes out and adds functionality for the new `#hashtag` search and
lookup system, still hidden behind the `enable_experimental_hashtag_autocomplete`
feature flag.

**Serverside**

We have two plugin API registration methods that are used to define data sources
(`register_hashtag_data_source`) and hashtag result type priorities depending on
the context (`register_hashtag_type_in_context`). Reading the comments in plugin.rb
should make it clear what these are doing. Reading the `HashtagAutocompleteService`
in full will likely help a lot as well.

Each data source is responsible for providing its own **lookup** and **search**
method that returns hashtag results based on the arguments provided. For example,
the category hashtag data source has to take into account parent categories and
how they relate, and each data source has to define their own icon to use for the
hashtag, and so on.

The `Site` serializer has two new attributes that source data from `HashtagAutocompleteService`.
There is `hashtag_icons` that is just a simple array of all the different icons that
can be used for allowlisting in our markdown pipeline, and there is `hashtag_context_configurations`
that is used to store the type priority orders for each registered context.

When sending emails, we cannot render the SVG icons for hashtags, so
we need to change the HTML hashtags to the normal `#hashtag` text.

**Markdown**

The `hashtag-autocomplete.js` file is where I have added the new `hashtag-autocomplete`
markdown rule, and like all of our rules this is used to cook the raw text on both the clientside
and on the serverside using MiniRacer. Only on the server side do we actually reach out to
the database with the `hashtagLookup` function, on the clientside we just render a plainer
version of the hashtag HTML. Only in the composer preview do we do further lookups based
on this.

This rule is the first one (that I can find) that uses the `currentUser` based on a passed
in `user_id` for guardian checks in markdown rendering code. This is the `last_editor_id`
for both the post and chat message. In some cases we need to cook without a user present,
so the `Discourse.system_user` is used in this case.

**Chat Channels**

This also contains the changes required for chat so that chat channels can be used
as a data source for hashtag searches and lookups. This data source will only be
used when `enable_experimental_hashtag_autocomplete` is `true`, so we don't have
to worry about channel results suddenly turning up.

------

**Known Rough Edges**

- Onebox excerpts will not render the icon svg/use tags, I plan to address that in a follow up PR
- Selecting a hashtag + pressing the Quote button will result in weird behaviour, I plan to address that in a follow up PR
- Mixed hashtag contexts for hashtags without a type suffix will not work correctly, e.g. #ux which is both a category and a channel slug will resolve to a category when used inside a post or within a [chat] transcript in that post. Users can get around this manually by adding the correct suffix, for example ::channel. We may get to this at some point in future
- Icons will not show for the hashtags in emails since SVG support is so terrible in email (this is not likely to be resolved, but still noting for posterity)
- Additional refinements and review fixes wil
2022-11-21 08:37:06 +10:00
David Taylor
f7fdaa97d1
DEV: Replace mini-loader with Ember's loader.js library ()
We already have this as a dependency, so it makes sense to use it rather than using our own not-quite-spec-compliant implementation
2022-10-18 14:53:06 +01:00
Martin Brennan
8ebd5edd1e
DEV: Rename secure_media to secure_uploads ()
This commit renames all secure_media related settings to secure_uploads_* along with the associated functionality.

This is being done because "media" does not really cover it, we aren't just doing this for images and videos etc. but for all uploads in the site.

Additionally, in future we want to secure more types of uploads, and enable a kind of "mixed mode" where some uploads are secure and some are not, so keeping media in the name is just confusing.

This also keeps compatibility with the `secure-media-uploads` path, and changes new
secure URLs to be `secure-uploads`.

Deprecated settings:

* secure_media -> secure_uploads
* secure_media_allow_embed_images_in_emails -> secure_uploads_allow_embed_images_in_emails
* secure_media_max_email_embed_image_size_kb -> secure_uploads_max_email_embed_image_size_kb
2022-09-29 09:24:33 +10:00
David Taylor
1bd1664ae0
DEV: Compile markdown-it-bundle with ember-cli ()
We were already compiling the markdown bundle via ember-cli, but that version was only being used in the test environment. This commit improves the implementation, and updates the filename so it's also used in production.

This commit also
- Removes the vendored copy of `markdown-it.js` and fetches from node_modules instead
- Updates `pretty_text.rb` to remove the custom sprockets-manifest-parsing
- Removes `pretty-text-bundle.js`, which was only being used by `pretty_text.rb`
2022-08-29 19:11:59 +01:00
David Taylor
3c81683955 DEV: Rename UriHelper.escape_uri to .normalized_encode
This is a much better description of its function. It performs idempotent normalization of a URL. If consumers truly need to `encode` a URL (including double-encoding of existing encoded entities), they can use the existing `.encode` method.
2022-08-09 11:55:25 +01:00
Selase Krakani
862007fb18
FEATURE: Add support for case-sensitive Watched Words ()
* FEATURE: Add case-sensitivity flag to watched_words

Currently, all watched words are matched case-insensitively. This flag
allows a watched word to be flagged for case-sensitive matching.
To allow allow for backwards compatibility the flag is set to false by
default.

* FEATURE: Support case-sensitive creation of Watched Words via API

Extend admin creation and upload of Watched Words to support case
sensitive flag. This lays the ground work for supporting
case-insensitive matching of Watched Words.

Support for an extra column has also been introduced for the Watched
Words upload CSV file. The new column structure is as follows:

 word,replacement,case_sentive

* FEATURE: Enable case-sensitive matching of Watched Words

WordWatcher's word_matcher_regexp now returns a list of regular
expressions instead of one case-insensitive regular expression.

With the ability to flag a Watched Word as case-sensitive, an action
can have words of both sensitivities.This makes the use of the global
Regexp::IGNORECASE flag added to all words problematic.

To get around platform limitations around the use of subexpression level
switches/flags, a list of regular expressions is returned instead, one for each
case sensitivity.

Word matching has also been updated to use this list of regular expressions
instead of one.

* FEATURE: Use case-sensitive regular expressions for Watched Words

Update Watched Words regular expressions matching and processing to handle
the extra metadata which comes along with the introduction of
case-sensitive Watched Words.

This allows case-sensitive Watched Words to matched as such.

* DEV: Simplify type casting of case-sensitive flag from uploads

Use builtin semantics instead of a custom method for converting
string case flags in uploaded Watched Words to boolean.

* UX: Add case-sensitivity details to Admin Watched Words UI

Update Watched Word form to include a toggle for case-sensitivity.
This also adds support for, case-sensitive testing and matching of  Watched Word
in the admin UI.

* DEV: Code improvements from review feedback

 - Extract watched word regex creation out to a utility function
 - Make JS array presence check more explicit and readable

* DEV: Extract Watched Word regex creation to utility function

Clean-up work from review feedback. Reduce code duplication.

* DEV: Rename word_matcher_regexp to word_matcher_regexp_list

Since a list is returned now instead of a single regular expression,
change `word_matcher_regexp` to `word_matcher_regexp_list` to better communicate
this change.

* DEV:  Incorporate WordWatcher updates from upstream

Resolve conflicts and ensure apply_to_text does not remove non-word characters in matches
that aren't at the beginning of the line.
2022-08-02 10:06:03 +02:00
Jarek Radosz
d1d6868325
DEV: Update xss.js ()
* DEV: Update xss.js

* Fix our hack
2022-06-24 10:28:05 +08:00
Jarek Radosz
c9d3c45ba9
DEV: Remove obsolete parts of our custom loader () 2022-06-20 18:40:25 +02:00
David Taylor
5238f6788c
FEATURE: Allow hotlinked media to be blocked ()
This commit introduces a new site setting: `block_hotlinked_media`. When enabled, all attempts to hotlink media (images, videos, and audio) will fail, and be replaced with a linked placeholder. Exceptions to the rule can be added via `block_hotlinked_media_exceptions`.

`download_remote_image_to_local` can be used alongside this feature. In that case, hotlinked images will be blocked immediately when the post is created, but will then be replaced with the downloaded version a few seconds later.

This implementation is purely server-side, and does not impact the composer preview.

Technically, there are two stages to this feature:

1. `PrettyText.sanitize_hotlinked_media` is called during `PrettyText.cook`, and whenever new images are introduced by Onebox. It will iterate over all src/srcset attributes in the post HTML and check if they're allowed. If not, the attributes will be removed and replaced with a `data-blocked-hotlinked-src(set)` attribute

2. In the `CookedPostProcessor`, we iterate over all `data-blocked-hotlinked-src(set)` attributes and check whether we have a downloaded version of the media. If yes, we update the src to use the downloaded version. If not, the entire media element is replaced with a placeholder. The placeholder is labelled 'external media', and is a link to the offsite media.
2022-06-07 15:23:04 +01:00
Jarek Radosz
3db4b3bc5e
DEV: Drop lodash () 2022-03-06 18:15:25 +01:00
Martin Brennan
599a72768c
DEV: Add force_quote_link option to PrettyText ()
This option will make it so the [quote] bbcode will always
include the HTML link to the quoted post, even if a topic_id
is not provided in the PrettyText#cook options. This is so
[quote] bbcode can be used in other places, like chat messages,
that always need the link and do not have an "off-topic" ID
to use.
2022-02-23 16:13:46 +10:00