This commit removes any logic in the app and in specs around
enable_experimental_hashtag_autocomplete and deletes some
old category hashtag code that is no longer necessary.
It also adds a `slug_ref` category instance method, which
will generate a reference like `parent:child` for a category,
with an optional depth, which hashtags use. Also refactors
PostRevisor which was using CategoryHashtagDataSource directly
which is a no-no.
Deletes the old hashtag markdown rule as well.
Generally follows the same pattern as #22520
There are some changes here, notably it uses the addon's babel
settings rather than the app's, and it goes through the same
treatment as the rest of the addon code (which may include more
than just babel).
However, this probably brings us closer to the normal expectations
you have around developing addon code, and in any case, does not
seem to have any effect on the final output:
```
$ diff dist/assets/markdown-it-bundle.js /tmp/dist-before/assets/markdown-it-bundle.js
```
Motivation: aligning us with JS/Ember practices (runtime deps in `dependencies`, build/dev-time deps in `devDependencies`)
1. Move deps to devDeps where applicable (rule of thumb: it's a devDep unless it's required at runtime by the rails app or it's imported in the addon's code)
2. Remove unused dependencies and add missing ones (in addons)
3. Remove empty `repository` fields
4. Move `engines` and `ember` fields to the bottom
While still in ember-cli new app blueprint, I don't think this package does much for us. It has support for older things like bower and npm-shrinkwrap, but doesn't support checking yarn.lock and doesn't necessarily work well with our project structure.
When we introduced unicode support in the regular expressions used in watched words (9a27803) we didn't realize the cost adding the `u` flag would be.
Turns out, it's pretty bad when you have lots of regular expressions to test. A customer had slightly less than 200 watched words, and it would freeze the browser for about 2s on the first check of those regular expressions (roughly 10ms per regular expression).
This commit introduces a new field (`word`) to the serialized watched words which is then converted to a very fast and cheap regular expression on the client-side. We use that regexp to quicly check whether a matcher is even worth trying so that we don't incure the cost of compiling the expensive unicode regexp.
This commit also busts the `WordWatcher` cache since we added a new field to be serialized.
One nice side effect of using `matchAll` instead of a `while / exec` loop is that the likeliness of having a bad regexp matching infinitely is vastly reduced 🙌
Adds a new `[grid]` tag that can arrange images (or other media) into a grid in posts.
The grid defaults to a 3-column with a few exceptions:
- if there are only 2 or 4 items, it defaults to a 2-column grid (because it generally looks better)
- on mobile, it defaults to a 2-column grid
- if there is only one item, the grid has no effect
This commit prevents unallowed URLs in iframe src by adding a relative path like `https://bob.com/abc/def/../ghi`. Currently, the iframe linking to the site uses the current_user, not the post's author, so users who have no access to a certain path are not able to view anything they shouldn't.
This commit makes some fundamental changes to how hashtag cooking and
icon generation works in the new experimental hashtag autocomplete mode.
Previously we cooked the appropriate SVG icon with the cooked hashtag,
though this has proved inflexible especially for theming purposes.
Instead, we now cook a data-ID attribute with the hashtag and add a new
span as an icon placeholder. This is replaced on the client side with an
icon (or a square span in the case of categories) on the client side via
the decorateCooked API for posts and chat messages.
This client side logic uses the generated hashtag, category, and channel
CSS classes added in a previous commit.
This is missing changes to the sidebar to use the new generated CSS
classes and also colors and the split square for categories in the
hashtag autocomplete menu -- I will tackle this in a separate PR so it
is clearer.
Due to the order we were parsing markdown, bbcode [url] elements were not
handled properly.
`[url]https://example.com/path[/url]` was not currectly parsing cause
linkify was detecting the url as: `https://example.com/path[/url]` which is
legit.
To resolve this I swapped url to use a replace rule, and instead re-parsed
the internal payload and injected the tokens in.
This fix is complex cause we support stuff like
`[url][b]test.com[/b][/url]`
So we need to parse the content inside url `[b]test.com[/b]`
It's backward compatible so still supports our 3.28 ember-source.
The visible change is finally getting rid of this message:
```
WARNING: Node v18.12.0 is not tested against Ember CLI on your platform. We recommend that you use the most-recent "Active LTS" version of Node.js. See https://git.io/v7S5n for details.
```
---
`@ember/string` dependency is added for future compatibility. See: https://github.com/ember-cli/ember-cli/pull/10125
---
`tests/helpers/index.js` is unused for now, but is a nice pattern. We could move some of our test setup into local `setupApplicationTest/setupRenderingTest/setupTest` helpers.
Co-authored-by: David Taylor <david@taylorhq.com>
This feature will allow sites to define which emoji are not allowed. Emoji in this list should be excluded from the set we show in the core emoji picker used in the composer for posts when emoji are enabled. And they should not be allowed to be chosen to be added to messages or as reactions in chat.
This feature prevents denied emoji from appearing in the following scenarios:
- topic title and page title
- private messages (topic title and body)
- inserting emojis into a chat
- reacting to chat messages
- using the emoji picker (composer, user status etc)
- using search within emoji picker
It also takes into account the various ways that emojis can be accessed, such as:
- emoji autocomplete suggestions
- emoji favourites (auto populates when adding to emoji deny list for example)
- emoji inline translations
- emoji skintones (ie. for certain hand gestures)
This reverts commit f6063c684b.
Videos on sites with a cdn enabled aren't playing w/ a default cdn
config. They are showing a "CORS request did not succeed" error.
* DEV: Add crossOrigin to video tag
This is a follow-up commit to f144c64e13
which enables the ability to generate thumbnail images for video
uploads.
In order for the html5 canvas element to create an image or blob the
source video element needs to to have the crossOrigin attribute set to
"anonymous" because a cdn is likely being used in production
environments.
We are already doing something similar in
e292c45924/app/assets/javascripts/discourse/app/lib/update-tab-count.js (L63)
We were only supporting the main name of each HighlightJS language. So, by default, you could not use `js` or `jsx` to highlight Javascript, given they are aliases for `javascript`.
This PR adds a list of aliases as a constant to core (built via a rake task), and then checks against the `highlighted_languages` site settings plus the list of aliases when processing a code block.
As of ba3f62f576, handlebars templates are colocated with js files so the path to hbs templates referenced by this rake task is no longer valid. This commit fixes the path to hbs templates and updates a couple of files that are generated by the rake task.
The way our markdown raw_html hoisting worked, we only
supported one level of hoisting the HTML content. However
when nesting [chat] transcript BBCode we need to allow
for multiple levels of it. This commit changes opts.discourse.hoisted
to be more constant, and the GUID keys that have the hoisted
content are only deleted by unhoistForCooked rather than
the cook function itself, which prematurely deletes them
when they are needed further down the line.
* FIX: Use Category.secured(guardian) for hashtag datasource
Follow up to comments in #19219, changing the category
hashtag datasource to use Category.secured(guardian) instead
of Site.new(guardian).categories here since the latter does
more work for not much benefit, and the query time is the
same. Also eliminates some Hash -> Model back and forth
busywork. Add some more specs too.
* FIX: Server-side hashtag lookup cooking user loading
When we were using the PrettyText.options.currentUser
and parsing back and forth with JSON for the hashtag
lookups server-side, we had a bug where the user's
secure categories were not loaded since we never actually
loaded a User model from the database, only parsed it
from JSON.
This commit fixes the issue by instead using the
PretyText.options.userId and looking up the user directly
from the database when calling hashtag_lookup via the
PrettyText::Helpers code when cooking server-side. Added
the missing spec to check for this as well.
* FEATURE: Enforce mention limits for chat messages
The first part of these changes adds a new setting called `max_mentions_per_chat_message`, which skips notifications when the message contains too many mentions. It also respects the `max_users_notified_per_group_mention` setting
and skips notifications if expanding a group mention would exceed it.
We also include a new component to display JIT warning for these limits to the user while composing a message.
* Simplify ignoring/muting filter in chat_notifier
* Post-send warnings for unsent warnings
* Improve pluralization
* Address review feedback
* Fix test
* Address second feedback round
* Third round of feedback
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
This fix changes the hashtag-raw hashtags, which are
the ones that do not actually match anything, back
to the old style which does not look like mentions.
Follow up from d3f02a1270
This commit fixes post quoting so that if the new
hashtag-cooked HTML is selected, we convert back to
a regular plain text #hashtag with the correct type and ref.
This commit fleshes out and adds functionality for the new `#hashtag` search and
lookup system, still hidden behind the `enable_experimental_hashtag_autocomplete`
feature flag.
**Serverside**
We have two plugin API registration methods that are used to define data sources
(`register_hashtag_data_source`) and hashtag result type priorities depending on
the context (`register_hashtag_type_in_context`). Reading the comments in plugin.rb
should make it clear what these are doing. Reading the `HashtagAutocompleteService`
in full will likely help a lot as well.
Each data source is responsible for providing its own **lookup** and **search**
method that returns hashtag results based on the arguments provided. For example,
the category hashtag data source has to take into account parent categories and
how they relate, and each data source has to define their own icon to use for the
hashtag, and so on.
The `Site` serializer has two new attributes that source data from `HashtagAutocompleteService`.
There is `hashtag_icons` that is just a simple array of all the different icons that
can be used for allowlisting in our markdown pipeline, and there is `hashtag_context_configurations`
that is used to store the type priority orders for each registered context.
When sending emails, we cannot render the SVG icons for hashtags, so
we need to change the HTML hashtags to the normal `#hashtag` text.
**Markdown**
The `hashtag-autocomplete.js` file is where I have added the new `hashtag-autocomplete`
markdown rule, and like all of our rules this is used to cook the raw text on both the clientside
and on the serverside using MiniRacer. Only on the server side do we actually reach out to
the database with the `hashtagLookup` function, on the clientside we just render a plainer
version of the hashtag HTML. Only in the composer preview do we do further lookups based
on this.
This rule is the first one (that I can find) that uses the `currentUser` based on a passed
in `user_id` for guardian checks in markdown rendering code. This is the `last_editor_id`
for both the post and chat message. In some cases we need to cook without a user present,
so the `Discourse.system_user` is used in this case.
**Chat Channels**
This also contains the changes required for chat so that chat channels can be used
as a data source for hashtag searches and lookups. This data source will only be
used when `enable_experimental_hashtag_autocomplete` is `true`, so we don't have
to worry about channel results suddenly turning up.
------
**Known Rough Edges**
- Onebox excerpts will not render the icon svg/use tags, I plan to address that in a follow up PR
- Selecting a hashtag + pressing the Quote button will result in weird behaviour, I plan to address that in a follow up PR
- Mixed hashtag contexts for hashtags without a type suffix will not work correctly, e.g. #ux which is both a category and a channel slug will resolve to a category when used inside a post or within a [chat] transcript in that post. Users can get around this manually by adding the correct suffix, for example ::channel. We may get to this at some point in future
- Icons will not show for the hashtags in emails since SVG support is so terrible in email (this is not likely to be resolved, but still noting for posterity)
- Additional refinements and review fixes wil
Our method of loading a subset of client settings into tests via
tests/helpers/site-settings.js can be improved upon. Currently we have a
hardcoded subset of the client settings, which may get out of date and not have
the correct defaults. As well as this plugins do not get their settings into the
tests, so whenever you need a setting from a plugin, even if it has a default,
you have to do needs.setting({ ... }) which is inconvenient.
This commit introduces an ember CLI build step to take the site_settings.yml and
all the plugin settings.yml files, pull out the client settings, and dump them
into a variable in a single JS file we can load in our tests, so we have the
correct selection of settings and default values in our JS tests. It also fixes
many, many tests that were operating under incorrect assumptions or old
settings.
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
This commit renames all secure_media related settings to secure_uploads_* along with the associated functionality.
This is being done because "media" does not really cover it, we aren't just doing this for images and videos etc. but for all uploads in the site.
Additionally, in future we want to secure more types of uploads, and enable a kind of "mixed mode" where some uploads are secure and some are not, so keeping media in the name is just confusing.
This also keeps compatibility with the `secure-media-uploads` path, and changes new
secure URLs to be `secure-uploads`.
Deprecated settings:
* secure_media -> secure_uploads
* secure_media_allow_embed_images_in_emails -> secure_uploads_allow_embed_images_in_emails
* secure_media_max_email_embed_image_size_kb -> secure_uploads_max_email_embed_image_size_kb
This PR makes some updates to the prior keyboard accessibility commit (eb98746):
- Makes `tabindex` attribute only appear on emoji markup in the emoji picker.
- After pressing the Esc key, focus returns to the <textarea/> input (composer editor or chat input)
* DEV: Make emoji elements focusable
Since emoji elements are of type `<img>` it requires a `tablindex="0"` in order to be focusable.
* WIP: Handle emoji focus/selection via arrow keys
Near completion, however, need a few fixes/improvements and overall code cleanup
* WIP: Testing
* DEV: Fixes and cleanup
* DEV: Follow conventions
* DEV: Improve up/down traversal when recents present
* DEV: Emoji markup in tests should include `tabindex`
* DEV: Add `tabindex` to topic tests
* DEV: Variable name as `searchInput` instead of `searchBar`
* DEV: Use appropriate method name (`_setNumEmojiPerRow`)
* DEV: Add comments and avoid nested if
* WIP: Adding test
* Fix first test
* DEV: Add assertions for arrow keys and escape key
* Some fixes for up/down navigation
This does not fix everything, when going from one section to another,
there are issues
* Fix a small regression
* FIX: Ability to focus on search results
Fixes regression
* Refactor calculating next up/down emoji
* Debugging test failure
* Skip stubborn CI test, add others
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
Upgrading to Markdown.it v13 broke empty inline BBCodes. This works around the problem by adding an empty token before a closing token if the previous token was a BBCode token.
It also removes the unused `jump` attribute which was removed in Markdown.it v12.3
* FEATURE: Add case-sensitivity flag to watched_words
Currently, all watched words are matched case-insensitively. This flag
allows a watched word to be flagged for case-sensitive matching.
To allow allow for backwards compatibility the flag is set to false by
default.
* FEATURE: Support case-sensitive creation of Watched Words via API
Extend admin creation and upload of Watched Words to support case
sensitive flag. This lays the ground work for supporting
case-insensitive matching of Watched Words.
Support for an extra column has also been introduced for the Watched
Words upload CSV file. The new column structure is as follows:
word,replacement,case_sentive
* FEATURE: Enable case-sensitive matching of Watched Words
WordWatcher's word_matcher_regexp now returns a list of regular
expressions instead of one case-insensitive regular expression.
With the ability to flag a Watched Word as case-sensitive, an action
can have words of both sensitivities.This makes the use of the global
Regexp::IGNORECASE flag added to all words problematic.
To get around platform limitations around the use of subexpression level
switches/flags, a list of regular expressions is returned instead, one for each
case sensitivity.
Word matching has also been updated to use this list of regular expressions
instead of one.
* FEATURE: Use case-sensitive regular expressions for Watched Words
Update Watched Words regular expressions matching and processing to handle
the extra metadata which comes along with the introduction of
case-sensitive Watched Words.
This allows case-sensitive Watched Words to matched as such.
* DEV: Simplify type casting of case-sensitive flag from uploads
Use builtin semantics instead of a custom method for converting
string case flags in uploaded Watched Words to boolean.
* UX: Add case-sensitivity details to Admin Watched Words UI
Update Watched Word form to include a toggle for case-sensitivity.
This also adds support for, case-sensitive testing and matching of Watched Word
in the admin UI.
* DEV: Code improvements from review feedback
- Extract watched word regex creation out to a utility function
- Make JS array presence check more explicit and readable
* DEV: Extract Watched Word regex creation to utility function
Clean-up work from review feedback. Reduce code duplication.
* DEV: Rename word_matcher_regexp to word_matcher_regexp_list
Since a list is returned now instead of a single regular expression,
change `word_matcher_regexp` to `word_matcher_regexp_list` to better communicate
this change.
* DEV: Incorporate WordWatcher updates from upstream
Resolve conflicts and ensure apply_to_text does not remove non-word characters in matches
that aren't at the beginning of the line.
Also, the change in insert-hyperlink (from `this.linkUrl.indexOf("http") === -1` to `!this.linkUrl.startsWith("http")`) was intentional fix: we don't want to prevent users from looking up topics with http in their titles.
Seems to only be a problem when a markdown.it rule inserts links without a attribute value. There's no test, because it's not reproducible with the markdown rules in core.
Updates markdown-it to v13.0.1
Noteworthy changes:
* `markdownit()` is now available on `globalThis` instead of `window`.
* The `text_collapse` rule was renamed to `fragments_join` which affected the `bbcode-inline` implementation.
* The `linkify` rule was added to the `inline` chain which affected the handling of the `[url]` BBCode. If available, our implementation reuses `link_open` and `link_close` tokens created by linkify in order to prevent duplicate links.
* The rendered HTML for code changed slightly. There's now a linebreak before the `</code>` tag. The tests were adjusted accordingly.
Previously we were only applying the restriction to `a[href]` and `img[src]`. This commit ensures we apply the same logic to all allowlisted media src attributes.
String.prototype.substr() is deprecated so we replace it with String.prototype.slice() which works similarily but isn't deprecated.
Signed-off-by: Tobias Speicher <rootcommander@gmail.com>
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Previously, our `upload://` protocol urls were only supported in markdown image tags. This meant that our PullHotlinkedImages job was forced to convert `<img` tags to markdown. Depending on the exact syntax, this can actually cause the image to break.
This commit adds support for `upload://` inside regular HTML `<img` tags. In a future commit, we'll be able to use this to make our PullHotlinkedImages job much more robust.
Context at https://meta.discourse.org/t/152801
When returning the customRenderFn from within buildCustomMarkdownCookFunction
for custom markdown engines (such as the one used by the [chat] transcripts)
we were not hoisting/unhoisting the `html_raw` tokens created by the
transcript, which meant that opts.discourse.hoisted could end up in
a state where it was null, and which caused errors and general unpleasantness.
Instead, we can just call the `cook` function that is already exported
from discourse-markdown-it, that takes care of what we did previously
plus the hoisting.
There is a companion chat commit that adds tests for this, there are
no custom markdown engine usages in core to test with.
This option will make it so the [quote] bbcode will always
include the HTML link to the quoted post, even if a topic_id
is not provided in the PrettyText#cook options. This is so
[quote] bbcode can be used in other places, like chat messages,
that always need the link and do not have an "off-topic" ID
to use.
This makes a small improvement to 'cold cache' ember-cli build times, and a large improvement to 'warm cache' build times
The ember-auto-import update means that vendor is now split into multiple files for efficiency. These are named `chunk.*`, and should be included immediately after the `vendor.js` file. This commit also updates the rails app to render script tags for these chunks.
This change was previously merged, and caused memory-related errors on RAM-constrained machines. This was because Webpack 5 switches from multiple worker processes to a single multi-threaded process. This meant that it was hitting node's default heap size limit (~500mb on a 1GB RAM server). Discourse's standard install procedure recommends adding 2GB swap to 1GB-RAM machines, so we can afford to override's Node's default via the `--max-old-space-size` flag.
Allows to write custom code blocks:
```
```mermaid height=200,foo=bar
test
```
```
Which will then get converted to:
```
<pre data-code-wrap="mermaid" data-code-height="200" data-code-foo="bar">
<code class="lang-nohighlight">
test
</code>
</pre>
```
This reverts commit f4c6a61855 and a8325c9016
This update of ember-auto-import and webpack causes significantly higher memory use during rebuilds. This made ember-cli totally unusable on 1GB RAM / 2GB swap environments. We don't have a specific need for this upgrade right now, so reverting for now.
This makes a small improvement to 'cold cache' ember-cli build times, and a large improvement to 'warm cache' build times
The ember-auto-import update means that vendor is now split into multiple files for efficiency. These are named `chunk.*`, and should be included immediately after the `vendor.js` file. This commit also updates the rails app to render script tags for these chunks
This commit adds a requestCustomMarkdownCookFunction function
to the `helper` that is provided to custom markdown rules
via their `setup` function.
The way this works is that once the default markdown engine that
we use for cooking posts has been set up, we loop through all
of the callbacks registered by `requestCustomMarkdownCookFunction`
and call `_buildCustomMarkdownCookFunction`. This creates
a new markdown engine using many of the same settings as the
default one, but will allow for the following options to be
changed by the markdown rule requesting the custom function:
* featuresOverride - The markdown-it features to allow for the engine
* markdownItRules - The markdown-it rules to allow for the engine
After this engine is set up a render function which renders + sanitizes
the output is returned for use by the markdown rule.
The use case for this API is mainly for block BBCode markdown rules
which want to render their content with a limited subset of the
markdown features/rules. Our initial use case for this is chat message
quoting.
This commit also does some minor refactoring of discourse-markdown-it
to accommodate this new engine building.
Sometimes plugins need to have additional data or options available
when rendering custom markdown features/rules that are not available
on the default opts.discourse object. These additional options should
be namespaced to the plugin adding them.
```
Site.markdown_additional_options["chat"] = { limited_pretty_text_markdown_rules: [] }
```
These are passed down to markdown rules on opts.discourse.additionalOptions.
The main motivation for adding this is the chat plugin, which currently stores
chat_pretty_text_features and chat_pretty_text_markdown_rules on
the Site object via additions to the serializer, and the Site object is
not accessible to import via markdown rules (either through
Site.current() or through container.lookup). So, to have this working
for both front + backend code, we need to attach these additional options
from the Site object onto the markdown options object.
Previously only `<div>one top element</div>` was allowed because we use `firstChild` instead of `children`.
We also want `<div>one</div><div>two</div>` to work with this method.
