There is a couple of layers of caching for theme JavaScript in Discourse:
The first layer is the `javascript_caches` table in the database. When a theme
with JavaScript files is installed, Discourse stores each one of the JavaScript
files in the `theme_fields` table, and then concatenates the files, compiles
them, computes a SHA1 digest of the compiled JavaScript and store the results
along with the SHA1 digest in the `javascript_caches` table.
Now when a request comes in, we need to render `<script>` tags for the
activated theme(s) of the site. To do this, we retrieve the `javascript_caches`
records of the activated themes and generate a `<script>` tag for each record.
The `src` attribute of these tags is a path to the `/theme-javascripts/:digest`
route which simply responds with the compiled JavaScript that has the requested
digest.
The second layer is a distributed cache whose purpose is to make rendering
`<script>` a lot more efficient. Without this cache, we'd have to query the
`javascript_caches` table to retrieve the SHA1 digests for every single
request. So we use this cache to store the `<script>` tags themselves so that
we only have to retrieve the `javascript_caches` records of the activated
themes for the first request and future requests simply get the cached
`<script>` tags.
What this commit does it ensures that the SHA1 digest in the
`javascript_caches` table stay the same across compilations by adding an order
by id clause to the query that loads the `theme_fields` records. Currently, we
specify no order when retrieving the `theme_fields` records so the order in
which they're retrieved can change across compilations and therefore cause the
SHA1 to change even though the individual records have not changed at all.
An inconsistent SHA1 digest across compilations can cause the database cache
and the distributed cache to have different digests and that causes the
JavaScript to fail to load (and if the theme heavily customizes the site, it
gives the impression that the site is broken) until the cache is cleared.
This can happen in busy sites when 2 concurrent requests recompile the
JavaScript files of a theme at the same time (this can happen when deploying a
new Discourse version) and request A updates the database cache after request B
did, and request B updates the distributed cache after request A did.
Internal ticket: t60783.
Co-authored-by: David Taylor <david@taylorhq.com>
We serve `service-worker.js` in an unusual way, which means that the sourcemap is not available on an adjacent path. This means that the browser fails to fetch the map, and shows an error in the console.
This commit re-writes the source map reference in the static_controller to be an absolute link to the asset (including the appropriate CDN, if enabled), and adds a spec for the behavior.
It's important to do this at runtime, rather than JS precompile time, so that changes to CDN configuration do not require re-compilation to take effect.
The `assets:precompile` rake task loads the full Ruby app, which can consume around 500mb of RAM by itself. Using `exec` to run `ember build` allows us to free up the Ruby memory and make more space for `ember build`
The old choose-topic component did not have the same style as the rest
of the create invite modal and was not very suitable to use in the modal
because it introduced the search results in modal's body.
The new topic-chooser is built using select-kit and provides a more
polished user experience.
This makes a small improvement to 'cold cache' ember-cli build times, and a large improvement to 'warm cache' build times
The ember-auto-import update means that vendor is now split into multiple files for efficiency. These are named `chunk.*`, and should be included immediately after the `vendor.js` file. This commit also updates the rails app to render script tags for these chunks.
This change was previously merged, and caused memory-related errors on RAM-constrained machines. This was because Webpack 5 switches from multiple worker processes to a single multi-threaded process. This meant that it was hitting node's default heap size limit (~500mb on a 1GB RAM server). Discourse's standard install procedure recommends adding 2GB swap to 1GB-RAM machines, so we can afford to override's Node's default via the `--max-old-space-size` flag.
- Update UI to improve contrast
- Make it clear that the message is only shown to administrators
- Add theme name and id to the console output
- Parse the error backtrace to identify the theme-id for post-decoration errors
- Improve console output to include the theme name / URL
- Add `?safe_mode=no_custom` to the admin panel link, so that it will work even if the theme is causing the site to break
The chat quoting mechanism will need to be able to generate
markdown for all kinds of uploads. The UploadMarkdown class
was missing generation for video and audio uploads. This
commit adds that in, and also expands the server-side regex
recognition of FileHelper types to match those in uploads.js,
and adds a spec for UploadMarkdown
When a site has secure media enabled and a post is with secure
media, we were incorrectly cooking custom emoji URLs and using the
secure URL for those emojis, even though they should not be
considered secure (their corresponding upload records in the
database are _not_ secure). Now instead of the blanket
post.with_secure_media? boolean for the secure: param, we also
want to make sure the image whose URL is being cooked is also
_not_ a custom emoji.
This will allow pretty text deprecations / errors / warnings to appear in the Rails logs, rather than disappearing silently.
(implementation adapted from `discourse_js_processor.rb`)
Inspired by [this suggestion](https://meta.discourse.org/t/what-is-a-here-mention/217878/3) from G Ann Campbell
> I suppose the description of the “here mention” field could be improved to something like:
> "Name used for @here mention to allow a privileged user to notify up to 10 people participating in the thread. Must not…"
This is intended for use by plugins which are building their own
topic lists, and want to include PMs alongside regular topics (e.g.
discourse-assign). It does not get used directly in core.
In 8e5b945b0f, we reverted the commit but
at the same time resulted in Theme::BASE_COMPILER_VERSION going
backwards which caused problems with themes caching.
This commit bumps the version to clear all the caches.
Follow-up to 8e5b945b0f
Redis supports deleting multiple keys at once using the `del` command and so does the `redis` gem: 21ec1dec06/lib/redis/commands/keys.rb (L188-L193). However, our wrapper around the `del` method currently accepts only one argument and expects it to be a string so it's impossible to delete multiple keys at once.
This PR changes the signature of the `DiscourseRedis#del` method so it accepts any number of arguments and makes sure all keys are properly namespaced before calling the `del` implementation of the `redis` gem.
* DEV: Document external topic id endpoints
This commit documents the existing Create Topic endpoint with the
`external_id` param and documents the new get topic by external id
endpoint.
It also refactors the existing topic show endpoint to use the new format
where we load the expected json schema response from a file.
See: 71f7f7ed49
* clean up unused test variables
Breakdown of fixes in this commit:
* `UserStat#topic_count` was not updated when visibility of
the topic changed.
* `UserStat#post_count` was not updated when post was hidden or
unhidden.
* `TopicConverter` was only incrementing or decrementing the counts by 1
even if a user has multiple posts in the topic.
* The commit turns off the verbose logging by default as it is just
noise to normal users who are not debugging this problem.
Another use case for focusComposer() is if the user is
already inside a topic but another component (such as the
floating chat window) needs to open the composer. This
commit also fixes the appendText option to only prepend
2 new lines if there is text before the text to be appended.
Follow up 7850ee318f
- Update UI to improve contrast
- Make it clear that the message is only shown to administrators
- Add theme name and id to the console output
- Parse the error backtrace to identify the theme-id for post-decoration errors
- Improve console output to include the theme name / URL
- Add `?safe_mode=no_custom` to the admin panel link, so that it will work even if the theme is causing the site to break
accept HTML attribute is not fully supported on iOS yet and can contain
only MIME types. This changes the input to allow all files and the
extension check is performed later in JavaScript.
Ember CLI gives sourcemaps their own digest. Our `s3.rake` logic assumes that the digest portion of sourcemap filenames remains the same.
The Ember CLI sourcemaps are included in the manifest file, so we can ensure they are uploaded by letting them past the MiniMime check.
Followup to abefb1beff
If no path is supplied, browsers will look for the map on the same path as the JS file itself. This fixes two problems that we see in production:
1. When compiling assets against one CDN, and then re-using them on a site with a different CDN, the sourceMappingUrls would be incorrect and print warnings in the console
2. If both an S3 CDN and an app CDN are configured, we were using the S3 CDN for the JS and the app CDN for the map. This commit will make sure we use the S3 CDN for both.
ember-cli already runs terser on its output. Running it through again provides no benefit, takes longer, and also breaks source mapping of these assets in production.
Instead of relaying on /timings request, we should cache last read post number. That should protect from having incorrect unread counter when going back to topic list.
This additional cache is very temporary as once /timings request is finished, serializer will have a correct result.
Simplified flow is:
1. Store in cache information about last seen post number before /timings request is sent
2. When getting back to topic list compare value of last seen post number returned by /latest request and information in cache. If cache number is higher, than use it instead of information returned by /latest. In addition delete cache item as there is high chance that `/timings` request already finished.
3. Optionally, delete cache when timings request is done and topic list was not yet visited.
Keeping cache reasonably small should not affect performance.
TopicTrackingState should correctly set filterCategory and filterTag for all different configurations.
When filterTag exists and new_topic message arrives, it ensures that filterTag is included in payload tags
If filterTag is part of payload tags, message that new topics are available is displayed and after click, new topics are included in the list.
