Upgrading to Markdown.it v13 broke empty inline BBCodes. This works around the problem by adding an empty token before a closing token if the previous token was a BBCode token.
It also removes the unused `jump` attribute which was removed in Markdown.it v12.3
* FEATURE: Add case-sensitivity flag to watched_words
Currently, all watched words are matched case-insensitively. This flag
allows a watched word to be flagged for case-sensitive matching.
To allow allow for backwards compatibility the flag is set to false by
default.
* FEATURE: Support case-sensitive creation of Watched Words via API
Extend admin creation and upload of Watched Words to support case
sensitive flag. This lays the ground work for supporting
case-insensitive matching of Watched Words.
Support for an extra column has also been introduced for the Watched
Words upload CSV file. The new column structure is as follows:
word,replacement,case_sentive
* FEATURE: Enable case-sensitive matching of Watched Words
WordWatcher's word_matcher_regexp now returns a list of regular
expressions instead of one case-insensitive regular expression.
With the ability to flag a Watched Word as case-sensitive, an action
can have words of both sensitivities.This makes the use of the global
Regexp::IGNORECASE flag added to all words problematic.
To get around platform limitations around the use of subexpression level
switches/flags, a list of regular expressions is returned instead, one for each
case sensitivity.
Word matching has also been updated to use this list of regular expressions
instead of one.
* FEATURE: Use case-sensitive regular expressions for Watched Words
Update Watched Words regular expressions matching and processing to handle
the extra metadata which comes along with the introduction of
case-sensitive Watched Words.
This allows case-sensitive Watched Words to matched as such.
* DEV: Simplify type casting of case-sensitive flag from uploads
Use builtin semantics instead of a custom method for converting
string case flags in uploaded Watched Words to boolean.
* UX: Add case-sensitivity details to Admin Watched Words UI
Update Watched Word form to include a toggle for case-sensitivity.
This also adds support for, case-sensitive testing and matching of Watched Word
in the admin UI.
* DEV: Code improvements from review feedback
- Extract watched word regex creation out to a utility function
- Make JS array presence check more explicit and readable
* DEV: Extract Watched Word regex creation to utility function
Clean-up work from review feedback. Reduce code duplication.
* DEV: Rename word_matcher_regexp to word_matcher_regexp_list
Since a list is returned now instead of a single regular expression,
change `word_matcher_regexp` to `word_matcher_regexp_list` to better communicate
this change.
* DEV: Incorporate WordWatcher updates from upstream
Resolve conflicts and ensure apply_to_text does not remove non-word characters in matches
that aren't at the beginning of the line.
Also, the change in insert-hyperlink (from `this.linkUrl.indexOf("http") === -1` to `!this.linkUrl.startsWith("http")`) was intentional fix: we don't want to prevent users from looking up topics with http in their titles.
Seems to only be a problem when a markdown.it rule inserts links without a attribute value. There's no test, because it's not reproducible with the markdown rules in core.
Updates markdown-it to v13.0.1
Noteworthy changes:
* `markdownit()` is now available on `globalThis` instead of `window`.
* The `text_collapse` rule was renamed to `fragments_join` which affected the `bbcode-inline` implementation.
* The `linkify` rule was added to the `inline` chain which affected the handling of the `[url]` BBCode. If available, our implementation reuses `link_open` and `link_close` tokens created by linkify in order to prevent duplicate links.
* The rendered HTML for code changed slightly. There's now a linebreak before the `</code>` tag. The tests were adjusted accordingly.
Previously we were only applying the restriction to `a[href]` and `img[src]`. This commit ensures we apply the same logic to all allowlisted media src attributes.
String.prototype.substr() is deprecated so we replace it with String.prototype.slice() which works similarily but isn't deprecated.
Signed-off-by: Tobias Speicher <rootcommander@gmail.com>
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Previously, our `upload://` protocol urls were only supported in markdown image tags. This meant that our PullHotlinkedImages job was forced to convert `<img` tags to markdown. Depending on the exact syntax, this can actually cause the image to break.
This commit adds support for `upload://` inside regular HTML `<img` tags. In a future commit, we'll be able to use this to make our PullHotlinkedImages job much more robust.
Context at https://meta.discourse.org/t/152801
When returning the customRenderFn from within buildCustomMarkdownCookFunction
for custom markdown engines (such as the one used by the [chat] transcripts)
we were not hoisting/unhoisting the `html_raw` tokens created by the
transcript, which meant that opts.discourse.hoisted could end up in
a state where it was null, and which caused errors and general unpleasantness.
Instead, we can just call the `cook` function that is already exported
from discourse-markdown-it, that takes care of what we did previously
plus the hoisting.
There is a companion chat commit that adds tests for this, there are
no custom markdown engine usages in core to test with.
This option will make it so the [quote] bbcode will always
include the HTML link to the quoted post, even if a topic_id
is not provided in the PrettyText#cook options. This is so
[quote] bbcode can be used in other places, like chat messages,
that always need the link and do not have an "off-topic" ID
to use.
This makes a small improvement to 'cold cache' ember-cli build times, and a large improvement to 'warm cache' build times
The ember-auto-import update means that vendor is now split into multiple files for efficiency. These are named `chunk.*`, and should be included immediately after the `vendor.js` file. This commit also updates the rails app to render script tags for these chunks.
This change was previously merged, and caused memory-related errors on RAM-constrained machines. This was because Webpack 5 switches from multiple worker processes to a single multi-threaded process. This meant that it was hitting node's default heap size limit (~500mb on a 1GB RAM server). Discourse's standard install procedure recommends adding 2GB swap to 1GB-RAM machines, so we can afford to override's Node's default via the `--max-old-space-size` flag.
Allows to write custom code blocks:
```
```mermaid height=200,foo=bar
test
```
```
Which will then get converted to:
```
<pre data-code-wrap="mermaid" data-code-height="200" data-code-foo="bar">
<code class="lang-nohighlight">
test
</code>
</pre>
```
This reverts commit f4c6a61855 and a8325c9016
This update of ember-auto-import and webpack causes significantly higher memory use during rebuilds. This made ember-cli totally unusable on 1GB RAM / 2GB swap environments. We don't have a specific need for this upgrade right now, so reverting for now.
This makes a small improvement to 'cold cache' ember-cli build times, and a large improvement to 'warm cache' build times
The ember-auto-import update means that vendor is now split into multiple files for efficiency. These are named `chunk.*`, and should be included immediately after the `vendor.js` file. This commit also updates the rails app to render script tags for these chunks
This commit adds a requestCustomMarkdownCookFunction function
to the `helper` that is provided to custom markdown rules
via their `setup` function.
The way this works is that once the default markdown engine that
we use for cooking posts has been set up, we loop through all
of the callbacks registered by `requestCustomMarkdownCookFunction`
and call `_buildCustomMarkdownCookFunction`. This creates
a new markdown engine using many of the same settings as the
default one, but will allow for the following options to be
changed by the markdown rule requesting the custom function:
* featuresOverride - The markdown-it features to allow for the engine
* markdownItRules - The markdown-it rules to allow for the engine
After this engine is set up a render function which renders + sanitizes
the output is returned for use by the markdown rule.
The use case for this API is mainly for block BBCode markdown rules
which want to render their content with a limited subset of the
markdown features/rules. Our initial use case for this is chat message
quoting.
This commit also does some minor refactoring of discourse-markdown-it
to accommodate this new engine building.
Sometimes plugins need to have additional data or options available
when rendering custom markdown features/rules that are not available
on the default opts.discourse object. These additional options should
be namespaced to the plugin adding them.
```
Site.markdown_additional_options["chat"] = { limited_pretty_text_markdown_rules: [] }
```
These are passed down to markdown rules on opts.discourse.additionalOptions.
The main motivation for adding this is the chat plugin, which currently stores
chat_pretty_text_features and chat_pretty_text_markdown_rules on
the Site object via additions to the serializer, and the Site object is
not accessible to import via markdown rules (either through
Site.current() or through container.lookup). So, to have this working
for both front + backend code, we need to attach these additional options
from the Site object onto the markdown options object.
Previously only `<div>one top element</div>` was allowed because we use `firstChild` instead of `children`.
We also want `<div>one</div><div>two</div>` to work with this method.
This commit extends the options which can be passed to
`PrettyText.markdown` so that which Markdown-it rules and Discourse
Markdown plugins to be used when rendering a text can be customizable.
Currently, this extension is mainly used by plugins.
Watched words of type 'replace' or 'link' replaced the text inside
mentions or hashtags too, which broke these. These types of watched
words must skip any match that has an @ or # before it.
We rely on yarn workspaces so we don't want people using npm in the repo by accident.
Also updated the required node version to 12+.
~~Not sure about the min yarn version – the latest one could be missing in various CI-like envs, so I might change it yet.~~
Downgraded yarn to ">= 1.21.1" (the oldest of "current" versions, tagged "legacy")
The generated regular expressions did not contain \b which matched
every text that contained the word, even if it was only a substring of
a word.
For example, if "art" was a watched word a post containing word
"artist" matched.
It was not clear that replace watched words can be used to replace text
with URLs. This introduces a new watched word type that makes it easier
to understand.
1. It defaults to `cache: true` already
2. Setting it to `false` for non-GET request doesn't do anything
3. We were correcting `cache: false` GET requests to use `cache: true`
…so setting it to anything at all, for any type of request doesn't make sense (anymore)
Over the years we accrued many spelling mistakes in the code base.
This PR attempts to fix spelling mistakes and typos in all areas of the code that are extremely safe to change
- comments
- test descriptions
- other low risk areas
Watched words are always regular expressions, despite watched_words_
_regular_expressions being enabled or not. Internally, wildcard
characters are replaced with a regular expression that matches any non
whitespace character.
* FIX: Cache missing inline oneboxes
Some inline oneboxes were not cached when the server did not return an
answer for an URL and the queried URL and the absolute URL were
different.
For example, if user typed www.example.com, the client asked the server
for http://www.example.com and if the server returned an empty response,
then the client would keep requesting an inline onebox everytime the
composer changed.
In other words, the key used for reading (the absolute URL) and the one
used for writing (the URL as typed by the user) were not the same when
the server returned an empty response.
* DEV: Check cache before making request
There is another cache check in PrettyText, but that is not enough if
multiple requests are pending. This problem was made obvious in tests,
but can happen for users with slow connections.
Note that this commit is also fixing various mistakes in emojis.
Some of them have been fixed manually in db.json/data.js/groups.json and will need to be fixed in emoji-db gem.
Headings with the exact same name generated exactly the same heading
names, which was invalid. This replaces the old code for generating
names for non-English headings which were using URI encode and resulted
in unreadable headings.
* FIX: Use theme color for anchor icon
* FIX: Do not count anchor links
* FIX: Do not count hashtags links either
* DEV: Add tests for link_count
* FIX: Disable anchors in quotes and preview
* FIX: Try building some anchor slugs for unicode
* DEV: Fix tests
Applying oneboxes and replacing censored watched words does not happen
in a strict order which often lead to inconsistencies. This commit
fixes the behavior and will never censor oneboxes.
To make it always censor oneboxes implies significant changes to the
PrettyText pipeline.
We override the default replacements rule to no longer replace "(c)", "(p)", and "(p)". Additionally, we merged the custom arrows rule into the replacement function.
This is not a security issue because regular users are not allowed to insert FA icons anywhere in the app. Admins can insert icons via custom badges, but they do have the ability to create themes with JS.
This commit includes other various improvements to watched words.
auto_silence_first_post_regex site setting was removed because it overlapped
with 'require approval' watched words.
Previously we would always take the first image in a post to use as the
thumbnail. On media-heavy sites, users may want to manually select a
specific image as the topic thumbnail. This commit allows this to be
done via a `|thumbnail` attribute in markdown.
For example, in this case, bbb would be chosen as the thumbnail:
```
```
This encompasses a lot of work done over the last year, much of which
has already been merged into master. This is the final set of changes
required to get Ember CLI running locally for development.
From here on it will be bug fixes / enhancements.
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Co-authored-by: romanrizzi <rizziromanalejandro@gmail.com>
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Co-authored-by: romanrizzi <rizziromanalejandro@gmail.com>
We want to wrap the `Ember.run.debounce` function and internally call `Ember.run` instead when running tests.
This commit changes discourseDebounce to work the same way as `Ember.run.debounce`.
Now that `discourseDebounce` works exactly like `Ember.run.debounce`, let's replace it and only use `DiscourseDebounce` from now on.
Move debounce to discourse-common to be able to reuse it in different bundles
Keep old debounce file for backwards-compatibility
- frowning was using slighty_frowning
- slightly_frowning was using frowning
- grinning_face_with_smiling_eyes was not defined
- fronwing_face_with_open_mouth was not defined
Adding a video in composer and then continuing to type into it will make the
video element flicker and restart playback on every keystroke, as the preview
is rendered. In certain configurations, this can lead to some performance
problems too.
Onebox already does the same for external videos.
eslint --fix is capable of fix it automatically for you, ensure prettier is run after eslint as eslint --fix could leave the code in an invalid prettier state.
typographer can change " to ” leading to breakages in parser
At least codify this. Longer term we want to re-prioritize typographer so
it always runs after bbcode parsing.
Previously attributes such as `[test a='a"a' b="a'a"]` were not correctly
handled.
This amends the regex parser to ensure it correctly parses attributes
without breaking incorrectly on the first nested quote
Before this patch, discourse-markdown depended on the modules in its
bundle being defined in a specific order or it wouldn't load properly.
Now, any file in the bundle can export a `priority` const (default 0)
and files will be loaded in order of ascending priority instead. This
allows us to use a bundle packaged in any order we want.
The emoji-picker is a specific piece of code as it has very strong performance requirements which are almost not found anywhere else in the app, as a result it was using various hacks to make it work decently even on old browsers.
Following our drop of Internet Explorer, and various new features in Ember and recent browsers we can now take advantage of this to reduce the amount of code needed, this rewrite most importantly does the following:
- use loading="lazy" preventing the full list of emojis to be loaded on opening
- uses InterserctionObserver to find the active section
- limits the use of native event listentes only for hover/click emojis (for performance reason we track click on the whole emoji area and delegate events), everything else is using ember events
- uses popper to position the emoji picker
- no jquery code
Meta report: https://meta.discourse.org/t/sending-many-requests-for-video-audio-upload-while-editing-post/161487
When typing in the composer we are sending a lot of unnecessary load() requests for the video/audio elements. This line was added months ago before we improved previewing/video thumbnails, which have improved things, so it is no longer required. After removing this line everything still works and no more additional requests are sent.
This fixes an issue where sometimes when composing a post and uploading a video/audio file, _loadCachedShortUrls/the uploads controller would return a full URL with origin, instead of just the URL with the host e.g. http://localhost:3000/some/video.mp4 instead of just //localhost:3000/some/video.mp4. We were prepending window.location.origin onto the URL no matter what, and since http://localhost:3000/some/video.mp4 does not match the host URL regex, we were ending up with something like http://localhost:3000http://localhost:3000/some/video.mp4 which broke composer previews. This was only noticed with a video upload in a secure upload environment.
This reverts commit 7d289a4f3e.
Now that 36bad0c31f is in and we have video previews on all platforms, the commit that's being reverted is no longer needed. In the worst case scenario, the video description is clipped under the video poster if the video aspect ratio is other than 16:9. This commit removes descriptions and the custom style for the video elements.
# Conflicts:
# app/assets/javascripts/pretty-text/addon/engines/discourse-markdown-it.js
# test/javascripts/lib/pretty-text-test.js
It's a stop gap – ideally we would generate a thumbnail for uploaded videos. For now, a bit of intentionality in the style and a pinch of context should do.
- This function now requires an explicit scope. It will never run on the entire document.
- Previously debounce was being used with an anonymous function, which means it was having no effect.
There were two constants here, `INLINE_ONEBOX_LOADING_CSS_CLASS` and
`INLINE_ONEBOX_CSS_CLASS` that were both longer than the strings they
were DRYing up: `inline-onebox-loading` and `inline-onebox`
I normally appreciate constants, but in this case it meant that we had
a lot of JS imports resulting in many more lines of code (and CPU cycles
spent figuring them out.)
It also meant we had an `.erb` file and had to invoke Ruby to create the
JS file, which meant the app was harder to port to Ember CLI.
I removed the constants. It's less DRY but faster and simpler, and
arguably the loss of DRYness is not significant as you can still search
for the `inline-onebox-loading` and `inline-onebox` strings easily if
you are refactoring.
* Remove Handlebars.SafeString usage
* DEV: Support for `import Handlebars from 'handlebars'`;
* FIX: Sprockets was broken when `node_modules` was present
By default the old version of sprockets looks for application.js
anywhere, including in a node_modules folder if this exists
(which it will when we move to Ember CLI.)
Even though `type` is an alias for `method`, we have custom logic in `/discourse/lib/ajax` that checks only `type`, and ~200 other ajax calls in the codebase already use `type` param.
When secure media is enabled and an attachment is marked as secure we want to use the full url instead of the short-url so we get the same access control post protections as secure media uploads.
Meta report: https://meta.discourse.org/t/excessive-requests-to-uploads-lookup-urls-leading-to-429-response/143119
* The data-orig-src attribute was not being removed from cooked
video and audio so the composer was infinitely trying to get the
URLs for them, which would never resolve to anything
* Also the code that retrieved the short URL was unscoped, and was
getting everything on the page. if running from the composer we
now scope to the preview window
* Also fixed a minor issue where the element href for the video
and audio tags was not being set when the short URL was found
Meta topic: https://meta.discourse.org/t/secure-media-uploads-expire/140894
This fixes the issue where if secure media was enabled, audio
and video files would do an initial load using the presigned
URL for the media to get metadata information e.g. duration of
track/video. However this started the expiry countdown for the
URL, so when a user pressed play on the media after 15 seconds
the media would be expired and AWS would return a 403 error.
We do not preload media if secure media is enabled. Otherwise
we just set the preload type to "metadata" which is the browser
default anyway.
Commit aa24be1 made it possible to build data attributes from image's
Markdown and this changes ensure that the resize controls still work
when data attributes are present.
* DEV: Use Ember 3.12.2
* Add Ember version to ThemeField's DEPENDENT_CONSTANTS
* DEV: Use `id` instead of `elementId` (See: https://github.com/emberjs/ember.js/issues/18147)
* FIX: Don't leak event listeners (bug introduced in 999e2ff)
For consistency this PR introduces using custom markdown and short upload:// URLs for video and audio uploads, rather than just treating them as links and relying on the oneboxer. The markdown syntax for videos is  and for audio it is .
This is achieved in discourse-markdown-it by modifying the rules for images in mardown-it via md.renderer.rules.image. We return HTML instead of the token when we encounter audio or video after | and the preview renders that HTML. Also when uploading an audio or video file we insert the relevant markdown into the composer.
Let's say post #2 quotes post number #1. If a user decides to quote the
quote in post #2, it should keep the information of post #1
("user_1, post: 1, topic: X"), instead of replacing with current post
info ("user_2, post: 2, topic: X").
* DEV: Fix the function prototype observers deprecation
DEPRECATION: Function prototype extensions have been deprecated, please migrate from function(){}.observes('foo') to observer('foo', function() {}). [deprecation id: function-prototype-extensions.observes] See https://deprecations.emberjs.com/v3.x/#toc_function-prototype-extensions-observes for more details.
* DEV: Fix the function prototype event listeners deprecation
DEPRECATION: Function prototype extensions have been deprecated, please migrate from function(){}.on('foo') to on('foo', function() {}). [deprecation id: function-prototype-extensions.on] See https://deprecations.emberjs.com/v3.x/#toc_function-prototype-extensions-on for more details.
* DEV: Simplify `default as` imports
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
* DEV: Add API to alter uploads Markdown
* DEV: Extract data attributes from image / download Markdown
For example '[test|attachment|hello=world]' will generate an 'a' element
with a data attribute: 'data-hello=world'.
This commit also makes MarkdownIt to transform '|attachment' into
'class="attachment"'. This transformation used to be a part of the
process which resolves short URLs (i.e. upload://).
* DEV: Export imageNameFromFileName
* DEV: Provide radix 10 argument to parseInt
* DEV: Provide radix 16 argument to parseInt
* DEV: Remove unnecessary parseInt calls
* Fix year formatting
parseInt was used here to convert decimals to ints
* FIX: move attachment_css_class constant out of upload-short-url for discourse-markdown-it
* Use setTimeout instead of ember later
* WIP. Not sure if this worked.
* oneboxer cache in separate file
* Reset onebox cache still
* set functions for oneboxers cache
- Client-side censoring fixed for non-chrome browsers. (Regular expression rewritten to avoid lookback)
- Regex generation is now done on the server, to reduce repeated logic, and make it easier to extend in plugins
- Censor tests are moved to ruby, to ensure everything works end-to-end
- If "watched words regular expressions" is enabled, warn the admin when the generated regex is invalid
FIX: buildTranslationTree was erroring when translations overlapped (ie. ":-)" and ":-))")
FIX: emoji translations wasn't working properly when translations overlapped
This allows fidelity in controlling excerpt (text that shows up when you pin a topic or link to it externally):
```
I am some text
[excerpt]
This is some **custom** markdown that should be the excerpt
[/excerpt]
More text
```
Previous solution relied on DIVs, unfortunately DIVs do not play well,
by design with mixing markdown unless you have a preceding newline eg:
```
<div class='hello'>
this will be treated properly as markdown
</div>
```
This extra newline is not desirable.
I am also considering adding
```
[div class=excerpt]
[/div]
```
This would offer lots of flexibility to themes and plugins that do not want the extra annoying newline.
Previously users could control excerpt with `<span class='excerpt'>`
in Markdown, this is somewhat limited for plugins that need to define this
across a section. This adds support for DIV as well
Previously, the site setting was only effective on the client side of
things. Once the site setting was been reached, all oneboxes are not
rendered. This commit changes it such that the site setting is respected
both on the client and server side. The first N oneboxes are rendered and
once the limit has been reached, subsequent oneboxes will not be
rendered.
We were looking up each mention one by one without any form of caching and that results
in a problem somewhat similar to an N+1. When we have to do alot of DB
lookups, it also increased the time spent in the V8 context which may
eventually lead to a timeout. The change here makes it such that mention lookups only does a single
DB query per post that happens outside of the V8 context.
This was a result of:
c4ed353fae
jQuery map on a jQuery object, returns a jQuery object http://api.jquery.com/map/
So you need to convert it to an array prior to mapping
