In our test suite, we sometimes see ChunkLoadErrors. This plugin should cause those failed requests to be retried seamlessly. It'll also help clients with flaky internet connections in production.
This will avoid warnings: "The code generator has deoptimised the styling of {path} as it exceeds the max of 500KB."
Should also provide a tiny improvement in build times
By default, the swc minifier seems to unwrap 'unneeded' IIFE. That means it was undoing the 'bugfix' transformation we have for class fields in Safari 15. Disabling the 'inline' and 'reduce_funcs' options seems to stop this behavior.
In an attempt to improve build performance, 9db5eafb mistakenly removed minimization for some of our JS assets, leading to a significant increase in the size of some files.
This commit restores minimization to those files. To avoid regressing on the build time improvements, this commit switches to using the `webpack-terser-plugin`'s "swcMinify" option. On an entry-level 1CPU/1GB-ram/2GB-swap DO droplet, this commit increases build time from ~16 minutes to ~18 minutes.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Disabling webpack minimize is a bug we are working to resolve but we
have to consider self-hosters that deploy on low cost hardware
and reenabling this for them drastically increases the build time.
For now, add a `DISCOURSE_WEBPACK_MINIMIZE` env to allow sites to opt
back in.
decorator-transforms (https://github.com/ef4/decorator-transforms) is a modern replacement for babel's plugin-proposal-decorators. It provides a decorator implementation using modern browser features, without needing to enable babel's full suite of class feature transformations. This improves the developer experience and performance.
In local testing with Google's 'tachometer' tool, this reduces Discourse's 'init-to-render' time by around 3-4% (230ms -> 222ms).
It reduces our initial gzip'd JS payloads by 3.2% (2.43MB -> 2.35MB), or 7.5% (14.5MB -> 13.4MB) uncompressed.
This was previously reverted in 97847f6. This version includes a babel transformation which works around the bug in Safari <= 15.
For Cloudflare compatibility issues, check https://meta.discourse.org/t/311390
This reverts commit 0f4520867b.
This has led to two problems:
1. An incompatibility with Cloudflare's "auto minify" feature. They've deprecated this feature because of incompatibility with modern JS syntax. But unfortunately it will remain enabled on existing properties until 2024-08-05.
2. Discourse fails to boot in Safari 15. This is strange, because Safari does support all the required features in our production JS bundles. Even more strangely, things start working as soon as you open the developer tools. That suggests the cause could be a Safari bug rather than a simple incompatibility.
Reverting while we work out a path forward on both those issues.
decorator-transforms (https://github.com/ef4/decorator-transforms) is a modern replacement for babel's plugin-proposal-decorators. It provides a decorator implementation using modern browser features, without needing to enable babel's full suite of class feature transformations. This improves the developer experience and performance.
In local testing with Google's 'tachometer' tool, this reduces Discourse's 'init-to-render' time by around 3-4% (230ms -> 222ms).
It reduces our initial gzip'd JS payloads by 3.2% (2.43MB -> 2.35MB), or 7.5% (14.5MB -> 13.4MB) uncompressed.
- Use 'cheap-source-map' webpack config on low-memory machines
This results in worse quality sourcemaps in browser dev tools, but it significantly reduces memory use in our webpack build. In approximate local testing it drops from 1100mb to 590mb. This should make the rebuild process on low-memory machines much faster and less likely to trigger OOM errors.
In development, and on higher-memory machines, the higher-quality 'source-map' option is maintained.
- Disable Webpack's built-in `minimize` feature. Embroider already applies Terser after the webpack build is complete. There is no need to double-minimize the output.
- Update ember-cli-progress-ci to print to stderr instead of stdout. For some reason, pups (used by discourse_docker) buffers the stdout of commands and only prints when they are finished. stderr does not have this same limitation, so switching will mean sysadmins can see the progress of the ember build in real-time.
Given the number of variables it's hard to promise exact numbers. But, in my tests on a DO droplet with 1GB RAM (+2GB swap), this reduced the `ember build` portion of a `./launcher rebuild app` from ~50 minutes to ~15 minutes.
This service-worker caching functionality was disabled by default in 1c58395bca, and the setting to re-enable was marked as experimental. Now we are dropping all the related logic.
Skipping babel for qunit is part of the default embroider blueprint. Adding these doesn't seem to have a measurable effect on build time, but it does stop this message from being printed in the build log:
```
The code generator has deoptimised the styling of /Users/david/discourse/discourse/node_modules/sinon/pkg/sinon-esm.js as it exceeds the max of 500KB
```
(extracted from #23678)
* Move Wizard back into main app, remove Wizard addon
* Remove Wizard-related resolver or build hacks
* Install and enable `@embroider/router`
* Add "wizard" to `splitAtRoutes`
In a fully optimized Embroider app, route-based code splitting more
or less Just Work™ – install `@embroider/router`, subclass from it,
configure which routes you want to split and that's about it.
However, our app is not "fully optimized", by which I mean we are
not able to turn on all the `static*` flags.
In Embroider, "static" means "statically analyzable". Specifically
it means that all inter-dependencies between modules (files) are
explicitly expressed as `import`s, as opposed to `{{i18n ...}}`
magically means "look for the default export in app/helpers/i18n.js"
or something even more dynamic with the resolver.
Without turning on those flags, Embroider behaves conservatively,
slurps up all `app` files eagerly into the primary bundle/chunks.
So, while you _could_ turn on route-based code splitting, there
won't be much to split.
The commits leading up to this involves a bunch of refactors and
cleanups that 1) works perfectly fine in the classic build, 2) are
good and useful in their own right, but also 3) re-arranged things
such that most dependencies are now explicit.
With those in place, I was able to move all the wizard code into
the "app/static" folder. Embroider does not eagerly pull things from
this folder into any bundle, unless something explicitly "asks" for
them via `imports`. Conversely, things from this folder are not
registered with the resolver and are not added to the `loader.js`
registry.
In conjunction with route-based code splitting, we now have the
ability to split out islands of on-demand functionalities from the
main app bundle.
When you split a route in Embroider, it automatically creates a
bundle/entrypoint with the relevant routes/templates/controllers
matching that route prefix. Anything they import will be added to
the bundle as well, assuming they are not already in the main app
bundle, which is where the "app/static" folder comes into play.
The "app/static" folder name is not special. It is configured in
ember-cli-build.js. Alternatively, we could have left everything
in their normal locations, and add more fine-grained paths to the
`staticAppPaths` array. I just thought it would be easy to manage
and scale, and less error-prone to do it this way.
Note that putting things in `app/static` does not guarantee that
it would not be part of the main app bundle. For example, if we
were to add an `import ... from "app/static/wizard/...";` in a
main bundle file (say, `app.js`), then that chunk of the module
graph would be pulled in. (Consider using `await import(...)`?)
Overtime, we can build better tooling (e.g. lint rules and babel
macros to make things less repetitive) as we expand the use of
this pattern, but this is a start.
Co-authored-by: Godfrey Chan <godfreykfc@gmail.com>
We funnel vendored javascript through ember-cli, but that's only used for the testem environment. Therefore, there's no need to minify it in production builds. In my tests, this reduces peak RSS of a production build from 3.53GB to 3.15GB.
This value is included when generating static asset URLs. Updating the value will allow site operators to invalidate all asset urls to recover from configuration issues which may have been cached by CDNs/browsers.
- Update optional-features to tie the `jquery-integration` flag to the current ember version
- Wrap ember-4-specific logic in ember-cli-build with a version check
- Update global-compat.js to add the jquery global if it doesn't exist (i.e. if we're on a modern ember version)
Extracted from https://github.com/discourse/discourse/pull/21720. This is a no-op under our current Ember 3.28 version.
- Skip rendering DModalLegacy when running Ember 5
- Move named outlet inside the DModalLegacy component file
- Exclude that DModalLegacy template from the build when running Ember 5
- Skip LegacySupport version of modal service when running Ember 5
- Add error popup for legacy modals when running Ember 5
Extracted from https://github.com/discourse/discourse/pull/21720. This is a no-op under our current Ember 3.28 version.
The default for webpack is to keep cached values indefinitely. In discourse, this unbound memory usage causes node to raise an OOM error after 50-100 rebuilds in development mode (with source maps enabled). Setting maxGenerations=1 means that the cache will be cleaned up regularly. With this change, I see no discernible increase in memory after 150+ rebuilds.
Previously, the discourse-hbr plugin took the entire app tree as its input, and the result would then be merged into the app. This is wasteful and more likely to cause problems in the build pipeline.
See also https://github.com/discourse/discourse/pull/24376
Ember-cli has built-in error pages when there is a build error. Previously these were not being used in Discourse because our custom proxy middleware was too early in the stack. This commit reorders things so that the "broccoli-watcher" middleware runs before our custom proxy. It also disables the `historySupportMiddleware`, which doesn't make sense in our 'always proxy' setup.
- Remove vendored copy
- Update Rails implementation to look for language definitions in node_modules
- Use webpack-based dynamic import for hljs core
- Use browser-native dynamic import for site-specific language bundle (and fallback to webpack-based dynamic import in tests)
- Simplify markdown implementation to allow all languages into the `lang-{blah}` className
- Now that all languages are passed through, resolve aliases at runtime to avoid the need for the pre-built `highlightjs-aliases` index
Previously, we were parsing webpack JS chunk filenames from the HTML files which ember-cli generates. This worked ok for simple entrypoints, but falls apart once we start using async imports(), which are not included in the HTML.
This commit uses the stats plugin to generate an assets.json file, and updates Rails to parse it instead of the HTML. Caching on the Rails side is also improved to avoid reading from the filesystem multiple times per request in develoment.
Co-authored-by: Godfrey Chan <godfreykfc@gmail.com>
With Embroider, we can rely on async `import()` to do the splitting
for us.
This commit extracts from `pretty-text` all the parts that are
meant to be loaded async into a new `discourse-markdown-it` package
that is also a V2 addon (meaning that all files are presumed unused
until they are imported, aka "static").
Mostly I tried to keep the very discourse specific stuff (accessing
site settings and loading plugin features) inside discourse proper,
while the new package aims to have some resembalance of a general
purpose library, a MarkdownIt++ if you will. It is far from perfect
because of how all the "options" stuff work but I think it's a good
start for more refactorings (clearing up the interfaces) to happen
later.
With this, pretty-text and app/lib/text are mostly a kitchen sink
of loosely related text processing utilities.
After the refactor, a lot more code related to setting up the
engine are now loaded lazily, which should be a pretty nice win. I
also noticed that we are currently pulling in the `xss` library at
initial load to power the "sanitize" stuff, but I suspect with a
similar refactoring effort those usages can be removed too. (See
also #23790).
This PR does not attempt to fix the sanitize issue, but I think it
sets things up on the right trajectory for that to happen later.
Co-authored-by: David Taylor <david@taylorhq.com>
Nowadays, themes/plugins have their templates compiled at build-time, so there is no need for us to carry the template compiler on the frontend during tests
- Remove the wildcard crawler. This was already excluding almost all file types, but the exclude list was missing '.gjs' which meant those files were unnecessarily being hoisted into the `public/` directory during precompile
- Automatically include all ember-cli-generated assets without needing them to be listed. The main motivation for this change is to allow us to start using async imports via Embroider/Webpack. The filenames for those new async bundles will not be known in advance.
- Skips sprockets fingerprinting on Embroider/Webpack chunk JS files. Their filenames already include a fingerprint, and having sprockets change the filenames will cause problems for the async import feature (where filenames are included deep inside js bundles)
This commit also updates our ember-cli build so that it skips building plugin tests in the production environment. This should provide a slight build speed improvement.
`@ember/jquery` was necessary to automate the `app.import()` but
that is no longer necessary with `ember-auto-import`. A secondary
thing it does is bringing back the `this.$` feature, but with a
deprecation. It is my understanding that the deprecation has long
be fully absorbed into both core and plugins so we shouldn't need
this package anymore.
Running addonPostprocessTree manually was causing ember-auto-import's postprocess hook to run and generate extra unnecessary chunks. The only reason called addonPostprocessTree directly was to allow the terser plugin to run on the extra public trees. We can do the terser postprocessing manually instead.
This commit is approximately the inverse of e1d27400f5.
This commit also removes ember-auto-import as dependencies of admin/wizard/discourse-plugins because they are not 'real' ember addons, and so it isn't serving any useful purpose. (see also https://github.com/discourse/discourse/pull/23974)
As of #23867 this is now a real package, so updating the imports to
use the real package name, rather than relying on the alias. The
name change in the package name is because `I18n` is not a valid
name as NPM packages must be all lowercase.
This commit also introduces an eslint rule to prevent importing from
the old I18n path.
For themes/plugins, the old 'i18n' name remains functional.
We'll probably have to keep the globals around for compatibility, but we should always import it ourselves. We'll followup with an updated eslint config to enforce this.
Currently, `window.I18n` is defined in an old school hand written
script, inlined into locale/*.js by the Rails asset pipeline, and
then the global variable is shimmed into a pseudo AMD module later
in `module-shims.js`.
This approach has some problems – for one thing, when we add a new
V2 addon (e.g. in #23859), Embroider/Webpack is stricter about its
dependencies and won't let you `import from "I18n";` when `"I18n"`
isn't listed as one of its `dependencies` or `peerDependencies`.
This moves `I18n` into a real package – `discourse-i18n`. (I was
originally planning to keep the `I18n` name since it's a private
package anyway, but NPM packages are supposed to have lower case
names and that may cause problems with other tools.)
This package defines and exports a regular class, but also defines
the default global instance for backwards compatibility. We should
use the exported class in tests to make one-off instances without
mutating the global instance and having to clean it up after the
test run. However, I did not attempt that refactor in this PR.
Since `discourse-i18n` is now included by the app, the locale
scripts needs to be loaded after the app chunks. Since no "real"
work happens until later on when we kick things off in the boot
script, the order in which the script tags appear shouldn't be a
problem. Alternatively, we can rework the locale bundles to be more
lazy like everything else, and require/import them into the app.
I avoided renaming the imports in this commit since that would be
quite noisy and drowns out the actual changes here. Instead, I used
a Webpack alias to redirect the current `"I18n"` import to the new
package for the time being. In a separate commit later on, I'll
rename all the imports in oneshot and remove the alias. As always,
plugins and the legacy bundles (admin/wizard) still relies on the
runtime AMD shims regardless.
For the most part, I avoided refactoring the actual I18n code too
much other than making it a class, and some light stuff like `var`
into `let`.
However, now that it is in a reasonable format to work with (no
longer inside the global script context!) it may also be a good
opportunity to refactor and make clear what is intended to be
public API vs internal implementation details.
Speaking of, I took the librety to make `PLACEHOLDER`, `SEPARATOR`
and `I18nMissingInterpolationArgument` actual constants since it
seemed pretty clear to me those were just previously stashed on to
the `I18n` global to avoid polluting the global namespace, rather
than something we expect the consumers to set/replace.
This reverts commit 42070d49da.
Overriding Error.stack like this seems to break the browser's own sourcemapping of stack-traces. Plus, it adds quite a significant performance overhead to tests (QUnit seems to rely on Error.stack even when tests pass). Reverting for now, but perhaps we can build a way to make this only apply to the UI-displayed stack traces in future 🤔
Discourse has a custom stylesheet pipeline which compiles things 'just in time'. The only place we were still running sass files through sprockets was for the `/tests` route in development mode. This use can be removed by compiling the relevant stylesheets through ember-cli instead (which we were already doing for testem runs)
This work was prompted by the incompatibility of dartsass-sprockets with the latest sass-embedded release (https://github.com/tablecheck/dartsass-sprockets/issues/13)
By default this is linked to the `tests` boolean, which we disabled for Embroider builds in 96674859. We want deprecation-workflow features to be available in production builds, so let's enable it unconditionally.
Until now, we have allowed testing themes in production environments via `/theme-qunit`. This was made possible by hacking the ember-cli build so that it would create the `tests.js` bundle in production. However, this is fundamentally problematic because a number of test-specific things are still optimized out of the Ember build in production mode. It also makes asset compilation significantly slower, and makes it more difficult for us to update our build pipeline (e.g. to introduce Embroider).
This commit removes the ability to run qunit tests in production builds of the JS app when the Embdroider flag is enabled. If a production instance of Discourse exists exclusively for the development of themes (e.g. discourse.theme-creator.io) then they can add `EMBER_ENV: development` to their `app.yml` file. This will build the entire app in development mode, and has a significant performance impact. This must not be used for real production sites.
This commit also refactors many of the request specs into system specs. This means that the tests are guaranteed to have Ember assets built, and is also a better end-to-end test than simply checking for the presence of certain `<script>` tags in the HTML.
Discourse core now builds and runs with Embroider! This commit adds
the Embroider-based build pipeline (`USE_EMBROIDER=1`) and start
testing it on CI.
The new pipeline uses Embroider's compat mode + webpack bundler to
build discourse code, and leave everything else (admin, wizard,
markdown-it, plugins, etc) exactly the same using the existing
Broccoli-based build as external bundles (<script> tags), passed
to the build as `extraPublicTress` (which just means they get
placed in the `/public` folder).
At runtime, these "external" bundles are glued back together with
`loader.js`. Specifically, the external bundles are compiled as
AMD modules (just as they were before) and registered with the
global `loader.js` instance. They expect their `import`s (outside
of whatever is included in the bundle) to be already available in
the `loader.js` runtime registry.
In the classic build, _every_ module gets compiled into AMD and
gets added to the `loader.js` runtime registry. In Embroider,
the goal is to do this as little as possible, to give the bundler
more flexibility to optimize modules, or omit them entirely if it
is confident that the module is unused (i.e. tree-shaking).
Even in the most compatible mode, there are cases where Embroider
is confident enough to omit modules in the runtime `loader.js`
registry (notably, "auto-imported" non-addon NPM packages). So we
have to be mindful of that an manage those dependencies ourselves,
as seen in #22703.
In the longer term, we will look into using modern features (such
as `import()`) to express these inter-dependencies.
This will only be behind a flag for a short period of time while we
perform some final testing. Within the next few weeks, we intend
to enable by default and remove the flag.
---------
Co-authored-by: David Taylor <david@taylorhq.com>
Previously we were patching ember-cli so that it would split the test bundle into two halves: the helpers, and the tests themselves. This was done so that we could use the helpers for `/theme-qunit` without needing to load all the core tests. This patch has proven problematic to maintain, and will become even harder under Embroider.
This commit removes the patch, so that ember-cli goes back to generating a single `tests.js` bundle. This means that core test definitions will now be included in the bundle when using `/theme-qunit`, and so this commit also updates our test module filter to exclude them from the run. This is the same way that we handle plugin tests on the regular `/tests` route, and is fully supported by qunit.
For now, this keeps `/theme-qunit` working in both development and production environments. However, we are very likely to drop support in production as part of the move to Embroider.
This resolves the issue in #23064.
This issue arises because we need to produce the trees for the
auxilary bundles in `ember-cli-build.js` to pass these trees as
argument to `app.toTree()`. In order to produce these trees, the
code internally need to set up babel, which deep-clones the addons'
babel configs.
When using `@embroider/macros`, the addon's babel config includes a
`MacrosConfig` object which is not supposed to be touched until the
configs are "finalized". In a classic build, the finalization step
happens when `app.toTree()` is called. In Embroider, this happens
somewhere deeper inside `CompatApp`.
We need to produce these auxilary bundle trees before we call
`app.toTree()` or before constructing `CompatApp` because they
need to be passed as arguments to these functions. So this poses a
tricky chicken-and-egg timing issue. It was difficult to find a
workaround for this that works for both the classic and Embroider
build pipeline.
Of all the internal addons that uses the auxilary bundle pattern,
this only affets `pretty-text` as it is (for now, at least) the
only addon that uses `@embroider/macros`.
Taking a step back, the only reason (for now, at least) it was
introduced was for the loader shim for the `xss` package. This
package is actually used inside the lazily loaded markdown-it
bundle. However, we didn't have a better way to include the dep
into the lazy bundle directly, so it ends up going into the main
addon tree, and, inturns, the discourse core bundle.
In core's main loader shim manifest, we already have an entry for
`xss`. This was perhaps a mistake at the time, but it doesn't make
a difference – as mentioned above, `xss` needs to be included into
the main bundle anyway.
So, for now, the simpliest solution is to avoid `@embroider/macros`
in these internal addons for the time being. Ideally we would soon
absorb these back into core as lazily loaded (`import()`-ed) code
managed by Webpack when we fully switch over to Embroider.
This adds a new `loaderShim()` function to ensure certain modules
are present in the `loader.js` registry and therefore runtime
`require()`-able.
Currently, the classic build pipeline puts a lot of things in the
runtime `loader.js` registry automatically. For example, all of
the ember-auto-import packages are in there.
Going forward, and especially as we switch to the Embroider build
pipeline, this will not be guarenteed. We need to keep an eye on
what modules (packages) our "external" bundles (admin, wizard,
markdown-it, plugins, etc) are expecting to be present and put
them into the registry proactively.
This has been proposed as the new default, and is currently in-use on many large ember apps without issue. It is already the default under Embroider. Testing locally, this seems to make incremental builds in development at least 2x faster.
https://github.com/ember-cli/ember-cli/issues/8681
pass the extra public trees to `app.toTree()` to match:
0e00f2bf15/packages/test-setup/src/index.ts (L24-L27)
The ember-cli-terser addon now takes care of minifying all additional trees, so we can remove our custom terser-related logic
This babel plugin is intended to supress the deprecation warnings
from building plugins, however, discourse-plugins does not actually
consume this plugin at all. Currently this happens to work due to
how the babel worker processes are shared and the timing/ordering
of the build, but it will stop working with the embroider build.
This commit extracts the plugin the a shared package so that it
can be properly consumed by discourse-plugins as well as core.
- Unify the silencing methods, use a WeakMap to remember the seen objects
- Export a proper plugin and use the absolute path in the config, instead
of the proprietary config from `broccoli-babel-transpiler`
The latter causes problems in Embroider which doesn't use the broccoli
based babel pipeline.
Generally follows the same pattern as #22520
There are some changes here, notably it uses the addon's babel
settings rather than the app's, and it goes through the same
treatment as the rest of the addon code (which may include more
than just babel).
However, this probably brings us closer to the normal expectations
you have around developing addon code, and in any case, does not
seem to have any effect on the final output:
```
$ diff dist/assets/markdown-it-bundle.js /tmp/dist-before/assets/markdown-it-bundle.js
```
