When redirecting to login, we store a destination_url cookie, which the user is then redirected to after login. We never want the user to be redirected to a JSON URL. Instead, we should return a 403 in these situations.
This should also be much less confusing for API consumers - a 403 is a better representation than a 302.
1b3d124a introduced a logic change which meant that we attempted to bootstrap, even on pages without any `preloadJson` (i.e. non-ember HTML pages from Discourse). This commit restores the original logic, making sure to avoid `?.`.
Old OnceOff job could perform pretty slowly on sites with millions of emails
New implementation operates in batches in a migration, minimizing locking.
Related to: 20f736aa11.
`auto_update` is true by default at the database level, but it doesn't make sense for `auto_update` to be true on themes that are not imported from a Git repository.
Under some conditions, these varied responses could lead to cache poisoning, hence the 'security' label.
Previously the Rails application would serve JSON data in place of HTML whenever Ember CLI requested an `application.html.erb`-rendered page. This commit removes that logic, and instead parses the HTML out of the standard response. This means that Rails doesn't need to customize its response for Ember CLI.
Under some conditions, these varied responses could lead to cache poisoning, hence the 'security' label.
For the stable branch, we are disabling the use of Ember CLI against production sites. A new implementation has been added to the tests-passed/beta branches
Part of overall strategy to remove jQuery file uploader
from the codebase. Also added some helper functionality to
the uppy mixin to allow for non-autostart uploads (all
previous upload changes have been for auto start components.)
The commit 20b2a42f49 broke
upload handlers, because previously we passed through the
native File object to the handler, not the uppy-wrapped
File object.
* REFACTOR: Improve support for consolidating notifications.
Before this commit, we didn't have a single way of consolidating notifications. For notifications like group summaries, we manually removed old ones before creating a new one. On the other hand, we used an after_create callback for likes and group membership requests, which caused unnecessary work, as we need to delete the record we created to replace it with a consolidated one.
We now have all the consolidation rules centralized in a single place: the consolidation planner class. Other parts of the app looking to create a consolidable notification can do so by calling Notification#consolidate_or_save!, instead of the default Notification#create! method.
Finally, we added two more rules: one for re-using existing group summaries and another for deleting duplicated dashboard problems PMs notifications when the user is tracking the moderator's inbox. Setting the threshold to one forces the planner to apply this rule every time.
I plan to add plugin support for adding custom rules in another PR to keep this one relatively small.
* DEV: Introduces a plugin API for consolidating notifications.
This commit removes the `Notification#filter_by_consolidation_data` scope since plugins could have to define their criteria. The Plan class now receives two blocks, one to query for an already consolidated notification, which we'll try to update, and another to query for existing ones to consolidate.
It also receives a consolidation window, which accepts an ActiveSupport::Duration object, and filter notifications created since that value.
Recently, the wrong new behavior appeared – we started to suggest to invited users usernames like "user1".
To reproduce:
1. Create an invitation with default settings, do not restrict it to email
2. Copy an invitation link and follow it in incognito mode
See username already filled, with eg “user1”. See screenshot. Should be empty.
This bug was very likely introduced by my recent changes to UserNameSuggester.
* Running the tests only in the ember cli env hid the fact that the pending posts feature wasn't working in the legacy environment
* Tests were using ember-cli-only APIs while there are widely used testing APIs in Discourse that support both ember envs
* `ember-test-selectors` was in both dependencies and devDependencies in discourse/package.json
* `qunit-dom` in package.json was not only unused but also defunct, as it wasn't pulled into the legacy env app
A followup to #14501, and #15128.
Some reports, like the Web Crawler User Agents report, have very long strings that need to be truncated when displayed. However, there is no way to see the full value without exporting the report or inspecting the elements using dev tools. This PR set a `title` attribute with the full value to the reports `<td>` elements so that the full value is shown on hover.
A post error validation would return a 422 status code. This status code was not accepted with the recent changes to bootstrap-json/index.js and would return a "Discourse Build Error" string, preventing any kind of bootbox popup error in the composer.
We have a couple of site setting, `slow_down_crawler_user_agents` and `slow_down_crawler_rate`, that are meant to allow site owners to signal to specific crawlers that they're crawling the site too aggressively and that they should slow down.
When a crawler is added to the `slow_down_crawler_user_agents` setting, Discourse currently adds a `Crawl-delay` directive for that crawler in `/robots.txt`. Unfortunately, many crawlers don't support the `Crawl-delay` directive in `/robots.txt` which leaves the site owners no options if a crawler is crawling the site too aggressively.
This PR replaces the `Crawl-delay` directive with proper rate limiting for crawlers added to the `slow_down_crawler_user_agents` list. On every request made by a non-logged in user, Discourse will check the User Agent string and if it contains one of the values of the `slow_down_crawler_user_agents` list, Discourse will only allow 1 request every N seconds for that User Agent (N is the value of the `slow_down_crawler_rate` setting) and the rest of requests made within the same interval will get a 429 response.
The `slow_down_crawler_user_agents` setting becomes quite dangerous with this PR since it could rate limit lots if not all of anonymous traffic if the setting is not used appropriately. So to protect against this scenario, we've added a couple of new validations to the setting when it's changed:
1) each value added to setting must 3 characters or longer
2) each value cannot be a substring of tokens found in popular browser User Agent. The current list of prohibited values is: apple, windows, linux, ubuntu, gecko, firefox, chrome, safari, applewebkit, webkit, mozilla, macintosh, khtml, intel, osx, os x, iphone, ipad and mac.
* DEV: Improve PresenceChannel state storage
Replaces some objects with Maps, and removes the redundant _presentChannels Set.
* DEV: Automatically leave PresenceChannels when in the background
If a tab has been in the background for 10s, or there has been no user activity for 60s, then the user will be removed from all PresenceChannels until activity resumes. Developers can opt-out of this by passing `{onlyWhileActive: false}` to the `enter` method.
The leak was introduced in #11722 and a test was added that relied on it in #14563
This PR fixes the leak (bookmarks-test), fixes the test that relied on it (fast-edit-test), and repleces some ad-hoc code with cloneJSON helper (other files)
The inProgressUploads is meant to be used to display these uploads
in a UI, and Ember will only update the array in the UI if pushObject
is used to notify it.
This is a big change to change over to using the uppy
upload mixin in the composer by default. This gets rid
of the temporary composer-editor-uppy component, as well
as removing the old ComposerUpload mixin and copying over
any missing functions that were not yet implemented by
ComposerUploadUppy. This has been working well on our
hosting for some time now and has led us to several
bug fixes.
This commit also deletes the old plugin API for adding
preprocessors for the uploads. The accepted method of doing
this now is via an uppy preprocessor plugin, which we have
several examples of in the core codebase.
Leaving the `enable_experimental_composer_uploader` site setting
intact for now because some plugins still rely on it, this
will be removed at a later date.
One step closer to ending the jQuery file uploader saga...
Widgets instances are ephemeral - they change on every re-render. We always want to notify the 'most recent' widget instance of events. This regressed in 1b9cf1b1 because the touchStart and drag hooks would persist the widget instance from the initial render. This commit switches TouchStart and Drag back to the pattern other events use, so that the most recent instance is always called. The performance benefits of per-element event listeners are retained.
Currently when a user creates posts that are moderated (for whatever
reason), a popup is displayed saying the post needs approval and the
total number of the user’s pending posts. But then this piece of
information is kind of lost and there is nowhere for the user to know
what are their pending posts or how many there are.
This patch solves this issue by adding a new “Pending” section to the
user’s activity page when there are some pending posts to display. When
there are none, then the “Pending” section isn’t displayed at all.
In jQuery file upload land, we were sending a single file through
at a time to matching upload handlers. This in turn required plugin
authors to marshal the files as they came through one by one if they
wanted to group them together to do something with them. Now that
we are using uppy, files come through in the groups they are added
in (for example dropping multiple, selecting multiple from the system
file dialogue).
This commit changes the matching upload handlers to send through
all matching files at once instead of piecemeal.
Error introduced in #14781
```
Error: Assertion Failed: You attempted to update <(unknown):ember3217>.bookmarks to "<(unknown):ember3846>", but it is being tracked by a tracking context, such as a template, computed property, or observer. In order to make sure the context updates properly, you must invalidate the property when updating it. You can mark the property as `@tracked`, or use `@ember/object#set` to do this.
```
In f6528afa01 I added parity support
for composer upload handlers to the uppy-ized composer. However the
way I assumed that it was only possible to handle a single file
upload at a time was false; it only appeared this way in the old
jQuery file upload composer because jQuery file upload sent through
files one at a time even if multiple were added at once. This caused
issues in certain plugins and themes by third parties.
This commit fixes the issue by making the uppy upload handler work
the same as the old one, by capturing all of the added files that
have matching handlers then going through them one by one and passing
them to the handler function.
For widget event handlers, we register a single listener on the `<body>`, and then notify the relavent widget (if any) when the event fires.
`touchstart` and `touchmove` events are particularly performance sensitive because they block scrolling on mobile. Therefore we want to avoid registering global non-passive listeners for these events.
This commit updates the WidgetTouchStartHook and WidgetDragHook implementations to automatically register listeners on the specific widget DOM elements when required.
This commit removes the last global scroll-blocking event handler from Discourse core. That means that mobile scrolling is now completely decoupled from our JS app. Even if the JS app is completely blocked (e.g. during rendering), scrolling will now continue to work. This should make things feel a lot smoother, especially on lower performance devices.
These were set to `passive: true` in ff72522f.
However, two consumers of this mixin (topic-navigation and site-header) do need to call `e.preventDefault()`, so we can't use passive listeners here.
That's ok, because this mixin only applies to a specific component's element, not the entire page. So having these non-passive listeners doesn't affect the vast majority of scrolling
This mixin calls the "scrolled" method of some object with no parameters, so there is no way that consumers would ever call `event.preventDefault()`. Therefore we can make the listeners passive, and improve scrolling performance on mobile.
This commit also updates the mixin to remove JQuery usage. The API is slightly modified to remove the need for an event 'name' for binding/unbinding.
The calls to `.bindScrolling` and `.unbindScrolling` in user-stream.js are removed because they are already called by the LoadMore mixin which is applied to the component.
The `bindScrolling` method claimed to offer debouncing-by-default. However, a bug in the `opts` parsing meant that debouncing was skipped if a 'name' was passed in. Therefore the only consumer actually being debounced was the LoadMore mixin. This commit fixes the opts parsing, so all consumers get the same behavior.
However, when scrolling, debounce is rarely what we want. The documentation of `bindScrolling` says "called every 100ms". In fact, debounce means that the functions were only called 'after the user **stops scrolling** for 100ms'. If you're scrolling very slowly (e.g. when using momentum-based scrolling on mobile), then this can be quite frustrating. This is why "Load more" is only triggered on topics/topic-lists when you completely stop scrolling.
Therefore, this commit also replaces the default 'debounce' with a 'throttle'. The 'throttle' is configured with `immediate = false`, so that it fires on the trailing edge, and therefore the final call will always be **after** we finish scrolling. (the default `immediate: true` would fire on the leading edge, and so the last call could be up to 100ms **before** we finish scrolling).
Registering non-passive listeners for the touchstart event can affect scroll performance on mobile devices, and now shows a warning in Chrome. Our current version of Ember unconditionally registers all event listeners, even if they're unused. It also doesn't support passive event listeners. Once we get to Ember 4.0, it lazily registers event listeners, and supports passive listeners via the `{{on` helper.
We already disable the ember `mousemove` and `touchmove` events for performance, so it makes sense to do the same for `touchstart`. We are not using `touchstart` anywhere in core, and I cannot find any official/unofficial plugins which use it. If a `touchstart` event is required, plugins/themes can always register their own listeners (preferably on a specific element, rather than the whole `document`)
We do call `event.preventDefault()` on these events. They're limited to a single element, so performance impact should be negligable. Adding `passive: false` prevents the chrome dev tools warning.
None of these places call `event.preventDefault()`. Therefore we can register the event listeners as 'passive', and improve scroll performance on mobile devices.
Skipping methods we don't use gives us mem/perf gains (minuscule but still), but more importantly fixes warnings about `Poll#open` (created by `enum :status`) conflicting with some internal AR method. 😃
Calling `setProperty("--header-offset", newValue)` will always cause a 'Recalculate Style' event, even if the value is unchanged. On my browser, these 'Recalculate Style' events take about 6-7ms each time the `dockCheck` function is run.
This commit stores the 'previous' value in an instance variable, and only calls setProperty if the value has changed. This brings the total runtime of `dockCheck` down to about 70µs on my machine.
`pending`, `approved`, `rejected`, `ignored`, and `deleted` scope method were accessible on all model classes… 😂
Fixes `Creating scope :pending. Overwriting existing method DiscoursePostEvent::EventDate.pending.` warnings.
The theme creator endpoints return JSON with a 201 CREATED
status code. With the recent changes to bootstrap-json/index.js
for the Ember CLI proxy, these endpoints were broken because
201 was not an accepted status code. This commit simply
adds 201 to the array, but prettier forced a reformat as well!
* FEATURE: Optionally send a 'noindex' header in non-canonical responses
This will be used in a SEO experiment.
Co-authored-by: David Taylor <david@taylorhq.com>
- Remove JQuery
- Remove legacy `document.webkitHidden` support. None of our currently supported browsers need this
- Use `passive` event listeners. These allows the browser to process the events first, before passing control to us
- Add a new `unseenTime` parameter. This allows consumers to request a delay before being notified about the browser going into the background
- Add a method for removing a callback
- Fire the callback when presence changes in either direction. Previously it would only fire when the user becomes present after a period of inactivity.
- Ensure callbacks are only called once for each state change. Previously they would be called every 60s, regardless of the value
- Listen to the `visibilitychanged` and `focus` events, treating them as equivalent to user action. This will make messagebus re-activate more quickly when switching back to a stale tab
- Add test helpers
- Delete the unused `discourse/lib/page-visible` module.
- Call message-bus's onVisibilityChange API directly, rather than dispatching a fake event on the `document`
This commit adds token_hash and scopes columns to email_tokens table.
token_hash is a replacement for the token column to avoid storing email
tokens in plaintext as it can pose a security risk. The new scope column
ensures that email tokens cannot be used to perform a different action
than the one intended.
To sum up, this commit:
* Adds token_hash and scope to email_tokens
* Reuses code that schedules critical_user_email
* Refactors EmailToken.confirm and EmailToken.atomic_confirm methods
* Periodically cleans old, unconfirmed or expired email tokens
When this setting is turned on, it will check that normalized emails
are unique. Normalized emails are emails without any dots or plus
aliases.
This setting can be used to block use of aliases of the same email
address.
Use @here to mention all users that were allowed to topic directly or
through group, who liked topics or read the topic. Only first 10 users
will be notified.
c401d641 introduced a new translation key for auth providers, and provided new strings for core providers. However, not all plugins have added this string. This commit makes the screenreader title fallback to the regular title in those cases.
In b8c8909a9d, we introduced a regression
where users may have had their `UserStat.first_unread_pm_at` set
incorrectly. This commit introduces a migration to reset `UserStat.first_unread_pm_at` back to
`User#created_at`.
Follow-up to b8c8909a9d.
This commit adds handlers for the composer uppy mixin to allow
for cancelling individual file uploads, not just all of them
at once. This is also combined with better tracking of in progress
uploads along with their progress percentage, for UI that needs
to be able to display the progress for individual files and
also cancel individual files.
To use this, a cancel button in the UI should call a function like this:
```javascript
cancelSingleUpload(fileId) {
this.appEvents.trigger(`${this.eventPrefix}:cancel-upload`, {
fileId,
});
},
```
Additionally, the `inProgressUploads` can be shown in the UI. It is an array of objects with the file name, ID, and the progress percentage. We can add more data to this if needed down the line.
This takes the uppy chunking algorithm and combines it with some
form submission from resumable.js for parity with the current
backup controller to make local backup uploads work with uppy.
We can then use this to replace the resumable-upload component
and the resumable.js library from our codebase, once stable.
This is disabled by default, so people using local backups will not
be affected. The enable_experimental_backup_uploader site setting
must be enabled for this to work.
The code that checked this permission was duplicated everytime a new
settings of this type was added. This commit changes the behavior of
some functionality because some feature checks were bypassed for staff
members.
Similar to site settings, adds support for `refresh` option to theme settings.
```yaml
super_feature_enabled:
type: bool
default: false
refresh: true
```
We are pushing /notification-alert/#{user_id} and /notification/#{user_id}
messages to MessageBus from both PostAlerter and User#publish_notification_state.
This can cause memory issues on large sites with many users. This commit
stems the bleeding by only sending these alert messages if the user
in question has been seen in the last 30 days, which eliminates a large
chunk of users on some sites.
When rendering the markdown code blocks we replace the
offending characters in the output string with spans highlighting a textual
representation of the character, along with a title attribute with
information about why the character was highlighted.
The list of characters stripped by this fix, which are the bidirectional
characters considered relevant, are:
U+202A
U+202B
U+202C
U+202D
U+202E
U+2066
U+2067
U+2068
U+2069
When rendering the markdown code blocks we replace the
offending characters in the output string with spans highlighting a textual
representation of the character, along with a title attribute with
information about why the character was highlighted.
The list of characters stripped by this fix, which are the bidirectional
characters considered relevant, are:
U+202A
U+202B
U+202C
U+202D
U+202E
U+2066
U+2067
U+2068
U+2069
In the topic lists, it's important that we apply `pointer-events: none;` to the links. 0e371d4 updated the selector used for this css.
In `templates/list/topic-list-item.hbs`, `.main-link` is applied to the same element as `.topic-list-data`, so the new selector applied correctly.
In `templates/mobile/list/topic-list-item.hbr`, `.main-link` is nested within `.topic-list-data`, so the new selector did not apply correctly.
This commit switches the selector back to simply `.main-link`, so that it works for both mobile and desktop.
Calling `window.getComputedStyle` during initialization causes the browser to pause and 'Recalculate Style'. On my machine, this adds about 7ms to boot time. Instead, we can check for the `rtl` class on the html element, which is added by the server, and doesn't require computing styles.
The inefficiency here is that we were previously fetching all the
records from `TopicAllowedUser` before filtering against a limited subset of
users based on `User#last_seen_at`.
This fixes an issue CvX found on PR #14666 where a previous fix
overwrote a computed property.
The better fix (as is often the case with Ember) is to remove an
observer and call methods when things change ourselves.
* FEATURE: Always show advanced invite options
The UI is more simple and more efficient than how it was when the
advanced options toggle was introduced. It does not make sense to keep
it anymore.
* UX: Minor copy edits
* UX: Merge expire invite controls
There were two controls in the create invite modal. One was a static
text that displayed how much time is left until the invite expires. The
other one was a datetime selector that set the time the invite expires.
This commit merges the two controls in a single one: staff users will
continue to see the datetime selector without the static text and
regular users will only see the static text because they cannot set
when the invite expires.
* UX: Remove invite link
It should only be visible after the invite was created.
We have two JS assets which are included in the `<body>` of responses. We were including the `<link rel='preload'` hint alongside the script tag in the body. Instead, we can move the preload hint to the `<head>` so that the browser discovers it earlier, and can start preloading the assets while the body is loading.
Time spent in the 'find module with suffix' portion of our `customResolve` function were adding up to around 100ms-150ms when booting the app. This time is spread over 150+ calls, so it's not immediately obvious in flamegraphs.
This commit implements a (reversed) [Trie](https://en.wikipedia.org/wiki/Trie) which enables fast suffix-based lookups on a list of strings.
In my tests, this requires < 5ms to initialize, and brings the cumulative 'find module with suffix' time down to `< 5ms`. This corresponds to a ~100ms improvement in LCP metrics in my browser.
The only behavior change is to remove support for module filenames which are **not** dasherized. I haven't found any core/theme/plugin modules which are not dasherized in their filenames.
Previously, incorrect reply counts are displayed in the "top categories" section of the user summary page since we included the `moderator_action` and `small_action` post types.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
This commit refactors the direct external upload routes (get presigned
put, complete external, create/abort/complete multipart) into a
helper which is then included in both BackupController and the
UploadController. This is done so UploadController doesn't need
strange backup logic added to it, and so each controller implementing
this helper can do their own validation/error handling nicely.
This is a follow up to e4350bb966
Currently, Discourse rate limits all incoming requests by the IP address they
originate from regardless of the user making the request. This can be
frustrating if there are multiple users using Discourse simultaneously while
sharing the same IP address (e.g. employees in an office).
This commit implements a new feature to make Discourse apply rate limits by
user id rather than IP address for users at or higher than the configured trust
level (1 is the default).
For example, let's say a Discourse instance is configured to allow 200 requests
per minute per IP address, and we have 10 users at trust level 4 using
Discourse simultaneously from the same IP address. Before this feature, the 10
users could only make a total of 200 requests per minute before they got rate
limited. But with the new feature, each user is allowed to make 200 requests
per minute because the rate limits are applied on user id rather than the IP
address.
The minimum trust level for applying user-id-based rate limits can be
configured by the `skip_per_ip_rate_limit_trust_level` global setting. The
default is 1, but it can be changed by either adding the
`DISCOURSE_SKIP_PER_IP_RATE_LIMIT_TRUST_LEVEL` environment variable with the
desired value to your `app.yml`, or changing the setting's value in the
`discourse.conf` file.
Requests made with API keys are still rate limited by IP address and the
relevant global settings that control API keys rate limits.
Before this commit, Discourse's auth cookie (`_t`) was simply a 32 characters
string that Discourse used to lookup the current user from the database and the
cookie contained no additional information about the user. However, we had to
change the cookie content in this commit so we could identify the user from the
cookie without making a database query before the rate limits logic and avoid
introducing a bottleneck on busy sites.
Besides the 32 characters auth token, the cookie now includes the user id,
trust level and the cookie's generation date, and we encrypt/sign the cookie to
prevent tampering.
Internal ticket number: t54739.
* FIX: register customOptions as select kit filter
We are allowing plugins to define custom filters which are added to CUSTOM_USER_SEARCH_OPTIONS const. However, we need to have static placeholder for custom filters, so those props will be passed, and we can use it later.
* fix
When 31035010af
was done it failed to take into account the case where the smtp_enabled
site setting was true, but the topic had no allowed groups / no
incoming email record, which caused errors for topics even with
nothing to do with group SMTP.
Uppy adds the file name as the "name" parameter in the
payload by default, which means that for things like the
emoji uploader which have a name param used by the controller,
that param will be passed as the file name. We already use
the existing file name if the name param is null, so this
commit just does further cleanup of the name param, removing
the extension if it is a filename so we don't end up with
emoji names like blah_png.
* FEATURE: display warning when sharing a topic in a restricted category
If a topic belongs to a category that is not readable by everyone, display a text warning of "Only visible to members of groups: [group_a], [group_b]"
* DEV: Adding a new category means we need to bump this value
* DEV: pass category to showModal
- ensures arrow up/down doesn’t also apply to textarea while autocomplete is opened
- ensures esc is closing autocomplete and also not closing composer while autocomplete is opened
- Removes jquery
- Removes a not unregistered listener and uses component event
- Removes external-url class as it was only valid in one case of the dropdown
- Uses @action
- Tagless
- Other minor changes
After permanently deleting the first post of a topic the user was
sometimes stuck on the page because of an infinite loop. This problem
happened more often in Firefox.
This patch takes the small component we had for sticky avatars and adds
it into our core code base.
A small refactor has been made to have a `StickyAvatars` dedicated class.
When uploading emoji with the new uppy upload mixin, we were
not sending the name of the emoji in the payload, or more
accurately uppy was already using the file name as the name
value and we were not overriding it from data. This commit
changes the behaviour for single files uploaded via the uppy
upload mixin, by merging the file's meta object with this.data
from the parent component.
When there are multiple groups on a topic, we were selecting
the first from the topic allowed groups to act as the sender
email address when sending group SMTP replies via PostAlerter.
However, this was not ordered, and since there is no created_at
column on TopicAllowedGroup we cannot order this nicely, which
caused just a random group to be used (based on whatever postgres
decided it felt like that morning).
This commit changes the group used for SMTP sending to be the
group using the email_username of the to address of the first
incoming email for the topic, if there are more than one allowed
groups on the topic. Otherwise it just uses the only SMTP enabled
group.
Sometimes, a user may have a malformed email such as
`test@test.com<mailto:test@test.com` their email address,
and as a topic participant will be included as a CC email
when sending a GroupSmtpEmail. This causes the CC parsing to
fail and further down the line in Email::Sender the code
to check the CC addresses expects an array but gets a string
instead because of the parse failure.
Instead, we can just check if the CC addresses are valid
and drop them if they are not in the GroupSmtpEmail job.
Also removes "appEventsCache". (and reduces the reported test memory usage by ~33%)
There's no longer any need to remove appEvent listeners in application-instance initializers' `teardown`, as app instances are recreated before each test (in both legacy and ember cli envs)
Uppy and Resumable slice up their chunks differently, which causes a difference
in this algorithm. Let's take a 131.6MB file (137951695 bytes) with a 5MB (5242880 bytes)
chunk size. For resumable, there are 26 chunks, and uppy there are 27. This is
controlled by forceChunkSize in resumable which is false by default. The final
chunk size is 6879695 (chunk size + remainder) whereas in uppy it is 1636815 (just remainder).
This means that the current condition of uploaded_file_size + current_chunk_size >= total_size
is hit twice by uppy, because it uses a more correct number of chunks. This
can be solved for both uppy and resumable by checking the _previous_ chunk
number * chunk_size as the uploaded_file_size.
An example of what is happening before that change, using the current
chunk number to calculate uploaded_file_size.
chunk 26: resumable: uploaded_file_size (26 * 5242880) + current_chunk_size (6879695) = 143194575 >= total_size (137951695) ? YES
chunk 26: uppy: uploaded_file_size (26 * 5242880) + current_chunk_size (5242880) = 141557760 >= total_size (137951695) ? YES
chunk 27: uppy: uploaded_file_size (27 * 5242880) + current_chunk_size (1636815) = 143194575 >= total_size (137951695) ? YES
An example of what this looks like after the change, using the previous
chunk number to calculate uploaded_file_size:
chunk 26: resumable: uploaded_file_size (25 * 5242880) + current_chunk_size (6879695) = 137951695 >= total_size (137951695) ? YES
chunk 26: uppy: uploaded_file_size (25 * 5242880) + current_chunk_size (5242880) = 136314880 >= total_size (137951695) ? NO
chunk 27: uppy: uploaded_file_size (26 * 5242880) + current_chunk_size (1636815) = 137951695 >= total_size (137951695) ? YES
Intersection observer callback can be called after the component gets destroyed:
```
Assertion Failed: calling set on destroyed object: <@ember/component:ember6019>.docked = false
at assert (ember:37774:17)
at _set2 (ember:17304:46)
at Class.set (ember:29529:29)
at Class._intersectionHandler (discourse/app/components/topic-progress:135:16)
at Backburner._run (ember:56389:25)
at Backburner._join (ember:56365:21)
at Backburner.join (ember:56082:19)
at join (ember:42874:28)
at IntersectionObserver.eval (ember:42978:19)
```
This api allows to add a dropdown at the bottom of a topic, note that this API is mobile only for now.
Also included in the commit:
- various doc fixes
- adding tests for both buttons and dropdowns APIs
- uses thrown instead of @ember/error to ensure execution is halted when incorrect parameters are given
This PR introduces a new `enable_experimental_backup_uploads` site setting (default false and hidden), which when enabled alongside `enable_direct_s3_uploads` will allow for direct S3 multipart uploads of backup .tar.gz files.
To make multipart external uploads work with both the S3BackupStore and the S3Store, I've had to move several methods out of S3Store and into S3Helper, including:
* presigned_url
* create_multipart
* abort_multipart
* complete_multipart
* presign_multipart_part
* list_multipart_parts
Then, S3Store and S3BackupStore either delegate directly to S3Helper or have their own special methods to call S3Helper for these methods. FileStore.temporary_upload_path has also removed its dependence on upload_path, and can now be used interchangeably between the stores. A similar change was made in the frontend as well, moving the multipart related JS code out of ComposerUppyUpload and into a mixin of its own, so it can also be used by UppyUploadMixin.
Some changes to ExternalUploadManager had to be made here as well. The backup direct uploads do not need an Upload record made for them in the database, so they can be moved to their final S3 resting place when completing the multipart upload.
This changeset is not perfect; it introduces some special cases in UploadController to handle backups that was previously in BackupController, because UploadController is where the multipart routes are located. A subsequent pull request will pull these routes into a module or some other sharing pattern, along with hooks, so the backup controller and the upload controller (and any future controllers that may need them) can include these routes in a nicer way.
PERF: Update like count in visible posts without an extra GET per like
Currently when a user is reading a topic and some post in it receive a
like from another user, the Ember app will be notified via MessageBus
and issue a GET to `/posts/{id}` to get the new like count. This worked
fine for us until today, but it can easily create a self-inflicted DDoS
when a topic with a large number of visitors gets a large number of
likes, since we will issue `visitors * likes` GET requests requests.
This patch optimizes this flow, by sending the new like count down in
the MessageBus notification, removing any need for the extra request.
It shouldn't cause any drift on the count because we send down the full
count instead of the difference too.
Possible follow-ups could include handling like removal.
`d-modal-body.js` was setting the text of a `modal-alert` element to `""`, but not removing any classes on that element. Changing this to call `_clearFlash()` ensures that a variety of styling classes are also removed from the element, which prevents empty alert elements being included on any subsequent modals that are displayed.
Several other controllers have also been modified to change the class of the error from `alert-error` to `error. The `alert-` is unnecessary, as it is added by `_flash(msg)` within `d-modal-body.js`.
We are only using list_multipart_parts right now in the
uploads controller for multipart uploads to check if the
upload exists; thus we don't need up to 1000 parts.
Also adding a note for future explorers that list_multipart_parts
only gets 1000 parts max, and adding params for max parts
and starting parts.
This commit introduces a new s3:ensure_cors_rules rake task
that is run as a prerequisite to s3:upload_assets. This rake
task calls out to the S3CorsRulesets class to ensure that
the 3 relevant sets of CORS rules are applied, depending on
site settings:
* assets
* direct S3 backups
* direct S3 uploads
This works for both Global S3 settings and Database S3 settings
(the latter set directly via SiteSetting).
As it is, only one rule can be applied, which is generally
the assets rule as it is called first. This commit changes
the ensure_cors! method to be able to apply new rules as
well as the existing ones.
This commit also slightly changes the existing rules to cover
direct S3 uploads via uppy, especially multipart, which requires
some more headers.
Since db7be947df the functionality
to not save a bookmark when pressing ESC on the modal has been
broken, because a new initiatedByESC event was introduced there
in d-modal instead of clicking on the modal close button. This
commit fixes the issue by adding an initiatedByESC property
to onClose inside application.js, and checking for that OR
initiatedByCloseButton in the bookmark modal.
This commit, while changing the watched word uploader to use
uppy, also fixes a minor bug with the UppyUploadMixin where
the file input's value was not cleared after reset, which
prevented subsequent file uploads. The composer mixin already
has this fix.
* FIX: ensure required_tag_group_name is null if no value present
If the array was present but empty `required_tag_group_name` would be set to undefined, which would then be removed from the payload of the remote request.
Addming the length check ensures the value is set to null, which is sent as an empty value (which the backend sees, and can remove it and persist the change on the Category object).
This commit changes the emoji uploader to use the UppyUploadMixin,
and makes some minor changes to the emoji uploader (tightening the
copy for drag and drop and adding a percentage for the upload).
Since no other uppy upload mixin components have needed to upload
multiple files so far, this necessitated adding a tracker for the
in progress uploads so we know when to reset the uploader once all
uploads are complete.
At the moment, the emoji uploader cannot be used for direct S3 uploads
because the admin emoji controller creates other records and does other
magic with the emojis. At some point we need to factor this kind of thing
into the ExternalUploadManager.transform! action to complete external
uploads.
Previously, suppressed category topics are included in the digest emails if the user visited that topic before and the `TopicUser` record is created with any notification level except 'muted'.
Trying to use a local test hostname other than localhost
(e.g. discourse.test )for discourse development was difficult due
the fact that localhost was hardcoded in a few places. This patch
uses existing environment variables to allow a developer to use a
different domain when developing.
Signed-off-by: Ryan Lerch <rlerch@redhat.com>
We are no longer able to display the image returned by Instagram directly within a Discourse site (either in the composer, or within a cooked post within a topic), so:
- Display an image placeholder in the composer preview
- A cooked post should use an iframe to display the Instagram 'embed' content
When sending a full search request to backend (i.e. when hitting Enter),
the debouncer needs to be cancelled, otherwise it will get invoked and
trigger a second search request to the backend.
When deleting a for_topic bookmark, we were calling
bookmark.attachedTo() for the bookmarks:changed event,
but the bookmark was not always a Bookmark model instance,
so sometimes that call would error. Now we make sure that
the bookmarks in the topic.bookmarks JS array are all
bookmark model instances, and added a test to cover this
deleting for_topic bookmark case as well.
* FIX: allowed_theme_ids should not be persisted in GlobalSettings
It was observed that the memoized value of `GlobalSetting.allowed_theme_ids` would be persisted across requests, which could lead to unpredictable/undesired behaviours in a multisite environment.
This change moves that logic out of GlobalSettings so that the returned theme IDs are correct for the current site.
Uses get_set_cache, which ultimately uses DistributedCache, which will take care of multisite issues for us.
A follow up PR should investigate why `proposal-logical-assignment-operators` is not getting used here (test file?) but this should be enough to get things running.
* DEV: Sanitize HTML admin inputs
This PR adds on-save HTML sanitization for:
Client site settings
translation overrides
badges descriptions
user fields descriptions
I used Rails's SafeListSanitizer, which [accepts the following HTML tags and attributes](018cf54073/lib/rails/html/sanitizer.rb (L108))
* Make sure that the sanitization logic doesn't corrupt settings with special characters
I was previously relying on `this.isDestroying` returning `true` during `willDestroyElement`. This was an incorrect assumption.
This commit refactors the logic into an explicit `cleanup` function, and also adds some cleanup for empty keys in the `subscribedProxy` array
The flow goes from:
- getting current user object
- creating a POJO using some of the current user keys
- passing this POJO around, which end up being used in message bus
- the processing fn associated ens up doing User.create on this object will both create a User object, but also inject store in it, store is holding a reference to currentUser Object and...
BOOM, we have an object holding a reference to the same object, which JSON.stringify used in prepareBody of pretender doesn't like.
This PR doesn't change any behavior, but just removes code that wasn't in use. This is a pretty dangerous place to change, since it gets called during user's registration. At the same time the refactoring is very straightforward, it's clear that this code wasn't doing any work (it still needs to be double-checked during review though). Also, the test coverage of UserNameSuggester is good.
* PERF: Remove JOIN on categories for PM search
JOIN on categories is not needed when searchin in private messages as
PMs are not categorized.
* DEV: Use == for string comparison
* PERF: Optimize query for allowed topic groups
There was a query that checked for all topics a user or their groups
were allowed to see. This used UNION between topic_allowed_users and
topic_allowed_groups which was very inefficient. That was replaced with
a OR condition that checks in either tables more efficiently.
This commit adds uploadHandler support to composer uploads using
uppy. The only things we have that are using this are discourse-brightcove and
discourse-video, which both pop modal windows to handle the file upload and
completely leave out all the composer-type flows. This implementation simply
follows the existing one, where if a single file is uploaded and there
is a matching upload handler we take control away from uppy and hand
it off to the upload handler.
Trying to get this kind of thing working within uppy would require a few
changes because they have no way to restrict uploaders to certain file types
and with the way their uploaders are run it doesn't look like it would be easy
to add this either, so I don't think this is worth the work unless at some
point in the future we plan to have more upload handler integrations.
I also fixed an issue with `cleanUpComposerUploadHandler` which is used
in tests to reset the state of `uploadHandlers` in the composer. This
was doing `uploadHandlers = []` to clear that array, but that creates
a brand new array so anything else referencing the original array will
lose that reference. Better to set `uploadHandlers.length = 0` to
clear it. This was breaking the tests I added to see if upload handlers
were working.
We were previously showing the "n new or updated topics" alert on
category routes like `/c/category-slug/ID/none` on every new/unread
topic update. This PR looks up the category by ID, which should be more
precise.
By default, Rails only includes the Vary:Accept header in responses when the Accept: header is included in the request. This means that proxies/browsers may cache a response to a request with a missing Accept header, and then later serve that cached version for a request which **does** supply the Accept header. This can lead to some very unexpected behavior in browsers.
This commit adds the Vary:Accept header for all requests, even if the Accept header is not present in the request. If a format parameter (e.g. `.json` suffix) is included in the path, then the Accept header is still omitted. (The format parameter takes precedence over any Accept: header, so the response is no longer varies based on the Accept header)
Previously, `loadLibs` was called inside the `optimize` function of
the media-optimization-worker, which meant that it could be hit
multiple times causing load errors (as seen in b69c2f7311)
This commit moves that call to a specific message handler (the `install` message)
for the service worker, and refactors the service for the media-optimization-worker
to wait for this installation to complete before continuing with processing
image optimizations.
This way, we know for sure based on promises and worker messages
that the worker is installed and has all required libraries
loaded before we continue on with attempting any processing. The
change made in b69c2f7311 is no
longer needed with this commit.
This new app event will fire whenever a bookmark is created,
edited, or deleted for a post or topic, and replaces these old
app events which had inconsistent APIs:
* page:bookmark-post-toggled
* topic:bookmark-toggled
When the event is triggered, the arguments are in this order:
1. bookmark - The bookmark record created or changed. Will be null
if the bookmark was deleted.
2. target - Object with target (post or topic) and targetId (post ID
or topic ID)
When inviting a group to a topic, there may be members of
the group already in the topic as topic allowed users. These
can be safely removed from the topic, because they are implicitly
allowed in the topic based on their group membership.
Also, this prevents issues with group SMTP emails, which rely
on the topic_allowed_users of the topic to send to and cc's
for emails, and if there are members of the group as topic_allowed_users
then that complicates things and causes odd behaviour.
We also ensure that the OP of the topic is not removed from
the topic_allowed_users when a group they belong to is added,
as it will make it harder to add them back later.
* DEV: allow composer option to skip jumping to a post on save
* DEV: refactor js safe access in jump logic
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
This happens because the state of `canLoadMore` is not cleared as the
refresh occurs, which is enough to make the page think a footer should
be displayed.
No tests here because it's tricky to test refreshing and none of our
existing acceptance tests seem to.
This removes all custom controllers and redis/messagebus logic from discourse-presence, and replaces it with core's new PresenceChannel system.
All functionality should be retained. This implementation should scale much better to large numbers of users, reduce the number of HTTP requests made by clients, and reduce the volume of messages on the MessageBus.
For more information on PresenceChannel, see 31db8352
We don't support any browser needing this for very long: https://caniuse.com/?search=selectionStart
I'm keeping some protection so It doesn’t crash but ultimately `element.selectionStart` should be enough.
Im not removing this in the commit, but the `caret_position.js` file seems barely used.
We had code to open the bookmark modal in two places -- the bookmark
list and also from within a topic. This caused the two code paths to
drift, as in the bookmark list we were not passing in the forTopic or
autoDeletePreferences data into the modal, and we were also not refreshing
the bookmark list when the bookmark was deleted from within the modal.
This commit moves the modal opening code into an importable
function from the controllers/bookmark module, and all callers
have to do is pass it an instance of Bookmark and also options
for what to do for the following:
* onAfterSave
* onAfterDelete
* onCloseWithoutSaving
An upstream validation bug in the aws-sdk-sns library could enable RCE under certain circumstances. This commit updates the upstream gem, and adds additional validation to provide defense-in-depth.
An upstream validation bug in the aws-sdk-sns library could enable RCE under certain circumstances. This commit updates the upstream gem, and adds additional validation to provide defense-in-depth.
The `generate`, `rotate` and `suspicious` auth token logs are now always logged regardless of the `verbose_auth_token_logging` setting because we rely no these to detect suspicious logins.
Before this fix, jumping to posts using the topic timeline scrollbar
will not update the counts since the topic scrollarea is not rerendered.
Follow-up to db337b10ee
Previously when clicking the Delete button for small action posts
there was no way to recover this post if the action was accidental.
Now if canRecover is true on the post, which it is just after it
is deleted and the post is fetched from the server again, we show
an undo button which calls the recover endpoint for the post.
We also now disallow the editing of the post if it is deleted, and
show the proper deleted red CSS on the small action post when deleted.
This is a follow-up to https://github.com/discourse/discourse/pull/14541. This adds a hidden setting for restoring the old behavior for those users who rely on it. We'll likely deprecate this setting at some point in the future.
This commit removes the recipient's username from the
respond to / participants list that is shown at the bottom
of user notification emails. For example if the recipient's
username was jsmith, and there were participants ljones and
bmiller, we currently show this:
> "reply to this email to respond to jsmith, ljones, bmiller"
or
> "Participants: jsmith, ljones, bmiller"
However this is a bit redundant, as you are not replying to
yourself here if you are the recipient user. So we omit the
recipient user's username from this list, which is only used
in the text of the email and not elsewhere.
Instead of using image-uploader, which relies on the old
UploadMixin, we can now use the uppy-image-uploader which
uses the new UppyUploadMixin which is stable enough and
supports both regular XHR uploads and direct S3 uploads,
controlled by a site setting (default to XHR).
At some point it may make sense to rename uppy-image-uploader
back to image-uploader, once we have gone through plugins
etc. and given a bit of deprecation time period.
This commit also fixes `for_private_message`, `for_site_setting`,
and `pasted` flags not being sent via uppy uploads onto the
UploadCreator, both via regular XHR uploads and also through
external/multipart uploads.
The uploaders changed are:
* site setting images
* badge images
* category logo
* category background
* group flair
* profile background
* profile card background
This commit makes the following change to the Edit Bookmark
modal window for clarity:
* If the user is editing an existing bookmark without a reminder set,
hide the "none needed" option. This will draw more attention to the
delete button.
* If the user is editing an existing bookmark with a reminder set for the
future, change the "none needed" option to say "remove reminder, keep bookmark"
To do this, I needed to provide an option to override the labels
for time shortcuts in certain cases, so I could keep the NONE shortcut
but have the different wording.
Two reasons for this change:
1. Better utilization of the screen space (i.e. displaying more than 5 entries on a 13" display)
2. Making user link elements smaller fixes user-card positioning (it no longer displays far to the right, away from the user name/avatar)
The method was only used for mega topics but it was redundant as the
first post can be determined from using the condition where
`Post#post_number` equal to one.
This commit bumps the following uppy modules:
* @uppy/aws-s3
* @uppy/aws-s3-multipart
* @uppy/core
* @uppy/drop-target
* @uppy/xhr-upload
This is done so we can use the new functionality for retrying
failed prepareUploadParts calls, introduced in
e435f4a917.
I also needed to make some changes to composer-upload-uppy to
support this retrying, while at the same time being able to
throw a bootbox with the error message if the number of retries
are exceeded.
To clarify, this problem is not about the topic posts stream, it's about posts streams like the user Activity one in the profile page (or in technical terms anything using the `{{user-stream}}` component).
Post decorations are currently applied inside a `didInsertElement` hook of the `{{user-stream}}` component. However, when the user scrolls the component will load more posts but these will be missing decorations because the `didInsertElement` is only fired once at the beginning of the component lifecycle.
This PR makes the component keep track of the last decorated post/DOM node, and when new posts are loaded the component fire an event for each new post and pass the post's DOM node with the event. Our plugin API
(I noticed this problem when I was working on https://github.com/discourse/discourse-follow/pull/37)
Co-authored-by: Robin Ward <robin.ward@gmail.com>
* FIX: do not display add to calendar for past dates
There is no value in saving past dates into calendar
* FIX: remove postId and move ICS to frontend
PostId is not necessary and will make the solution more generic for dates which doesn't belong to a specific post.
Also, ICS file can be generated in JavaScript to avoid calling backend.
In most cases, these links are handled in JavaScript, so the `href` and `target` are not used. However, when the `link-to-post` refers to a post which is not currently loaded in the DOM (e.g. it is the OP), then the href is used, and we need to add a `target` to prevent page navigation within the embed iframe.
The legacy testing environment will remove the User.current() value before disposing of controllers/components. Presence often involves making HTTP calls during disposal of components, so this can cause issues.
Production, and the modern Ember-CLI environment, do not require this hack, so it is behind an `isTesting() && isLegacyEmber()` check.
Sometimes administrators want to permanently delete posts and topics
from the database. To make sure that this is done for a good reasons,
administrators can do this only after one minute has passed since the
post was deleted or immediately if another administrator does it.
