Custom emoji, profile background, and card background were being set to secure, which we do not want as they are always in a public context and result in a 403 error from the ACL if linked directly.
### General Changes and Duplication
* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file.
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.
### Viewing Secure Media
* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`
### Removed
We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.
* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
Some specs use psql to test database restores and dropping the table after the test needs to happen outside of rspec because of transactions. The previous attempt lead to some changes to be stored in the test database.
FIX: raised a proper NotFound exception when filtering groups by username with invalid username.
FIX: properly filter the groups based on current user visibility when viewing another user's groups.
DEV: Guardian.can_see_group?(group) is now using Guardian.can_see_groups(groups) instead of duplicating the same code.
FIX: spec for groups_controller#index when group directory is disabled for logged in user.
FIX: groups_controller.sortable specs to actually test all sorting combinations.
DEV: s/response_body/body/g for slightly shorter spec code.
FIX: rewrote the "view another user's groups" specs to test all group_visibility and members_group_visibility combinations.
DEV: Various refactoring for cleaner and more consistent code.
An archive containing lots of small files could trigger an error even though the amount of decompressed data was way below the maximum allowed size. This happened because the decompression algorithm used the chunk size for calculating the remaining size instead of the actual size of the decompressed chunk.
The QUnit rake task starts a server in test mode. We need a tweak to allow dynamic CSP hostnames in test mode. This tweak is already present in development mode.
To allow CSP to work, the browser host/port must match what the server sees. Therefore we need to disable the enforce_hostname middleware in test mode. To keep rspec and production as similar as possible, we skip enforce_hostname using an environment variable.
Also move the qunit rake task to use unicorn, for consistency with development and production.
* Add a rake task to disable secure media. This sets all uploads to `secure: false`, changes the upload ACL to public, and rebakes all the posts using the uploads to make sure they point to the correct URLs. This is in a transaction for each upload with the upload being updated the last step, so if the task fails it can be resumed.
* Also allow viewing media via the secure url if secure media is disabled, redirecting to the normal CDN url, because otherwise media links will be broken while we go and rebake all the posts + update ACLs
Previously we had the ability to download a simple .gz file
new changes mean we have a a tar.gz file that needs some levels
of fiddling to get extracted correctly
MaxMind now requires an account with a license key to download files.
Discourse admins can register for such an account at:
https://www.maxmind.com/en/geolite2/signup
License key generation is available in the profile section.
Once registered you can set the license key using `DISCOURSE_MAXMIND_LICENSE_KEY`
This amends it so we unconditionally skip MaxMind DB downloads if no license key exists.
Many security scanners ship invalid mime types, this ensures we return
a very cheap response to the clients and do not log anything.
Previous attempt still re-dispatched the request to get proper error page
but in this specific case we want no error page.
Added a fix to gracefully error with a Webauthn::SecurityKeyError if somehow a user provides an unkown COSE algorithm when logging in with a security key.
If `COSE::Algorithm.find` returns nil we now fail gracefully and log the algorithm used along with the user ID and the security key params for debugging, as this will help us find other common algorithms to implement for webauthn
This used to work due to side effects.
`rake parallel:migrate` used to work very inconsistently and would only migrate
some of the databases.
This introduces the recommended change to db.yml so the correct database is
found based off TEST_ENV_NUMBER if for some reason we did not set it using
RAILS_DB
Also avoids a bunch of schema dumping which is not needed when migrating
parallel specs
DB number 1 is very odd cause for whatever reason parallel spec is not
setting it.
- Refactor source_url to avoid using eval in development
- Precompile handlebars in development
- Include template compilers when running qunit
- Remove unsafe-eval in development CSP
- Include unsafe-eval only for qunit routes in development
- Ensure that the 'notify_moderators' flag is always the last flag when using custom flags.
- Support passign a custom FlagSettings object when replacing flags to reuse existing ones.
People rarely want to have their avatars show up as the preview image on social media platforms. Instead, we should fall back to the site opengraph image.
It used to check how many quotes were inside a post, without taking
considering that some quotes can contain other quotes. This commit
selects only top level quotes.
I had to use XPath because I could not find an equivalent CSS
selector.
Meta thread: https://meta.discourse.org/t/sending-a-pm-with-the-following-title-causes-an-error/135654/3
We had an issue where if someone sent a PM with crazy
characters that are stripped and we end up with only
a number, the topic redirect errored because the slug was
a number. so instead we return the default as well if
the slug is a number after prettification
According to the [Rails
Source](https://github.com/rails/rails/blob/master/activerecord/lib/active_record/railties/databases.rake#L20)
the `ActiveRecord::Migrator.migrations_paths` are overwritten with the
value of `ActiveRecord::Tasks::DatabaseTasks.migrations_paths` every
time the config is loaded.
This caused a bug for Discourse development where if you ran:
`rake db:drop db:create db:migrate` in one line, you would not get our
post migrations, as those had a custom value for `migrations_paths`.
The fix is to use `ActiveRecord::Tasks::DatabaseTasks.migrations_paths`
to set up all our custom paths. Everything seems to work as expected.
Adds a custom bookmark-clock icon to discourse-additional.svg for use with the new bookmarks with reminder functionality.
Also add some code to correctly refresh the post-stream icon for bookmark to show the clock after save.
Plugins can add it via API if they need to use `eval`:
```
extend_content_security_policy(script_src: [:unsafe_eval])
```
See https://meta.discourse.org/t/104243
API keys are now only visible when first created. After that, only the first four characters are stored in the database for identification, along with an sha256 hash of the full key. This makes key usage easier to audit, and ensures attackers would not have access to the live site in the event of a database leak.
This makes the merge lower risk, because we have some time to revert if needed. Once the change is confirmed to be working, we will add a second commit to drop the `key` column.
The following methods have long been deprecated in ruby due to flaws in their implementation per http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-core/29293?29179-31097:
URI.escape
URI.unescape
URI.encode
URI.unencode
escape/encode are just aliases for one another. This PR uses the Addressable gem to replace these methods with its own encode, unencode, and encode_component methods where appropriate.
I have put all references to Addressable::URI here into the UrlHelper to keep them corralled in one place to make changes to this implementation easier.
Addressable is now also an explicit gem dependency.
Note: All of this functionality is hidden behind a hidden, default false, site setting called `enable_bookmarks_with_reminders`. Also, any feedback on Ember code would be greatly appreciated!
This is part 1 of the bookmark improvements. The next PR will address the backend logic to send reminder notifications for bookmarked posts to users. This PR adds the following functionality:
* We are adding a new `bookmarks` table and `Bookmark` model to make the bookmarks a first-class citizen and to allow attaching reminders to them.
* Posts now have a new button in their actions menu that has the icon of an actual book
* Clicking the button opens the new bookmark modal.
* Both name and the reminder type are optional.
* If you close the modal without doing anything, the bookmark is saved with no reminder.
* If you click the Cancel button, no bookmark is saved at all.
* All of the reminder type tiles are dynamic and the times they show will be based on your user timezone set in your profile (this should already be set for you).
* If for some reason a user does not have their timezone set they will not be able to set a reminder, but they will still be able to create a bookmark.
* A bookmark can be deleted by clicking on the book icon again which will be red if the post is bookmarked.
This PR does NOT do anything to migrate or change existing bookmarks in the form of `PostActions`, the two features live side-by-side here. Also this does nothing to the topic bookmarking.
This is required because bin/rake automatically loads plugins when migrating. In our continuous integration, we don't want plugins to break the core build. They should only be loaded for the plugin build.
We like to stay as close as possible to latest with rubocop cause the cops
get better.
This update required some code changes, specifically the default is to avoid
explicit returns where implicit is done
Also this renames a few rules
Non UTF-8 user_agent requests were bypassing logging due to PG always
wanting UTF-8 strings.
This adds some conversion to ensure we are always dealing with UTF-8
`available_disk_space` calls `df` which exits with an error if the `uploads` path doesn't exist. That's often the case when the `Discourse.store.external?` is true.
By doing the `external?` check first the `disable_if_low_on_disk_space` does less work and doesn't output any errors to the console.
This fixes the following issues:
* The link element on the lightbox which pops open the lightbox was linking to the S3 URL with a private ACL instead of the secure media URL for the image
* Change to use `@post.with_secure_media?` in `CookedPostProcessor` for URL cooking, as in some cases, like when a post is edited and an upload is added, `upload.secure?` can be false which resulted in `srcset` URLs not being cooked correctly to secure media upload urls.
This feature adds the ability to define synonyms for tags, and the ability to merge one tag into another while keeping it as a synonym. For example, tags named "js" and "java-script" can be synonyms of "javascript". When searching and creating topics using synonyms, they will be mapped to the base tag.
Along with this change is a new UI found on each tag's page (for example, `/tags/javascript`) where more information about the tag can be shown. It will list the synonyms, which categories it's restricted to (if any), and which tag groups it belongs to (if tag group names are public on the `/tags` page by enabling the "tags listed by group" setting). Staff users will be able to manage tags in this UI, merge tags, and add/remove synonyms.
This bug was causing some unusual behavior when the last post is filtered (e.g. from an ignored user). In some situations this would cause suggested topics to be omitted from the payload.
The next_page specs have been updated to remove most of the stubs
* Support for custom messages and redirects when creating posts
When a post/topic is created Discourse serializes a `NewPostResult`
object. Normally this contains a status like `created_post` or
errors describing why the post could not be created.
There are times when a plugin might want to take the inputted post
and do something in the background. In this case, the plugin
can return a custom `message` and `route_to` attribute in the
`NewPostResult`.
If present, the message will be displayed in an alert, and when "Ok" is
clicked the user will be routed to the new URL.
* Destroy the draft in parallel
Note:
```
def foo(bar: 1)
end
foo({bar: 2})
# raises a deprecation, instead use:
foo(**{bar: 2})
```
Additionally when matching regexes always use strings. It does not make
sense to match a non string to a regex.
We already cache failed onebox URL requests client-side, we now want to cache this on the server-side for extra protection. failed onebox previews will be cached for 1 hour, and any more requests for that URL will fail with a 404 status. Forcing a rebake via the Rebake HTML action will delete the failed URL cache (like how the oneboxer preview cache is deleted).
If a user has more than 60 active sessions, the oldest sessions will be terminated automatically. This protects performance when logging in and when loading the list of recently used devices.
This is a bottom up rewrite of Discourse cache to support faster performance
and a limited surface area.
ActiveSupport::Cache::Store accepts many options we do not use, this partial
implementation only picks the bits out that we do use and want to support.
Additionally params are named which avoids typos such as "expires_at" vs "expires_in"
This also moves a few spots in Discourse to use Discourse.cache over setex
Performance of setex and Discourse.cache.write is similar.
Discourse.cache is a more consistent method to use and offers clean fallback
if you are skipping redis
This is part of a larger change that both optimizes Discoruse.cache and omits
use of setex on $redis in favor of consistently using discourse cache
Bench does reveal that use of Rails.cache and Discourse.cache is 1.25x slower
than redis.setex / get so a re-implementation will follow prior to porting
This amends our API so we provide it with the draft key when saving a post
this means post creator can clean up the draft consistently even if we are
doing fancy stuff like replying to a new topic or new pm or whatever.
There will be some followup work to clean it up so client never calls destroy
on draft during normal operation and the #create/#update endpoints takes care of it
every time
In `post_creator`, the ACL update is only necessary when uploads need to be secured.
This should fix a regression with S3 clones that do not support updating ACLs.
* Add timezone to user_options table
* Also migrate existing timezone values from UserCustomField,
which is where the discourse-calendar plugin is storing them
* Allow user to change their core timezone from Profile
* Auto guess & set timezone on login & invite accept & signup
* Serialize user_options.timezone for group members. this is so discourse-group-timezones can access the core user timezone, as it is being removed in discourse-calendar.
* Annotate user_option with timezone
* Validate timezone values
There was an issue on dev where when uploading secure media, the href of the media was correctly being replaced in the CookedPostProcessor, but the srcset urls were not being replaced correctly. This is because UrlHelper.cook_url was returning the asset host URL for the media for secure media instead of returning early with the proxied secure proxy url.
Previously our custom exception handler was unable to handle situations
where an invalid mime type was sent, resulting in a warning log
This ensures we pretend a request is HTML for the purpose of rendering
the error page if an invalid mime type from a scanner is shipped to the app
* FEATURE: Normalize the service worker route
Update cache headers so they are not immutable outside of the rails app
Add the ability to purge the service worker cache from localhost
Rails -> nginx will pass immutable flags so the file is cached until reloaded.
In most cases, nginx will have its cache flushed on rebuild (new image)
For those needing dynamic re-caching (such as upgrading via the UI),
a rake task for flushing the service worker script is provided
through `assets:flush_sw`
This is only defined in a console environment. For example:
```
[1] pry(main)> SiteSetting.info(:title)
=> {:resolved_value=>"Globally Overridden Title",
:default_value=>"Discourse",
:global_override=>"Globally Overridden Title",
:database_value=>"Test Discourse",
:refresh?=>false,
:client?=>true,
:secret?=>false}
```
* Abort CensoredWordsValidator early if censored_words_regexp nil. Sometimes censored_words_regex can end up nil, erroring the validator. This handles the nil condition and also adds a spec for the validator
The secure media functionality relied on `SiteSetting.enable_s3_uploads?` which, as we found in dev, did not take into account global S3 settings via `GlobalSetting.use_s3?`. We now use `SiteSetting.Upload.enable_s3_uploads` instead to be more consistent.
Also, we now validate `enable_s3_uploads` changes, because if `GlobalSetting.use_s3?` is true users should NOT be enabling S3 uploads manually.
We were having issues in development mode where the JS code had errors due to a bad cache. When starting a server in development mode in bin/unicorn we now get the git sha of the discourse HEAD and get a git sha of all plugins, and store them in a file. If the sha has changed then we delete tmp/cache to refresh the assets cache.
* Fix an issue where if an edit was made to a post with a reason provided, and then another edit was made with no reason, the original edit reason got wiped out
* We now always make a post revision (even with ninja edits) if an edit reason has been provided and it is different from the current edit reason
Co-Authored-By: Sam <sam.saffron@gmail.com>
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access.
A few notes:
- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
For this to work we need to overwrite `db:rollback` in our Rakefile like
we do for migrate, so that it removes the load_config dependency. This
allows our custom migration paths to work.
This version hash is used for the filename, and so browsers/CDNs cache based on it. Previously the version hash was based only on the list of requested icons. This can cause issues in a couple of situations, most commonly when developing themes with custom icons:
- A requested icon does not exist, and then later is added to the theme. The bundle output changes, but the hash did not
- The SVG content of an icon changes, but the name of the icon does not. The bundle output changes, but the hash did not
* When viewing a tag, the search widget will now show a checkbox to scope the search by tag, which will limit search results to that tag on desktop and mobile
`FileUtils.cd` and `Dir.chdir` cause the working directory to change for the entire process. We run sidekiq jobs, hijacked requests and deferred jobs in threads, which can make working directory changes have unintended side-effects.
- Add a rubocop rule to warn about usage of Dir.chdir and FileUtils.cd
- Added rubocop:disable for scripts used outside the app
- Refactored code using cd to use alternative methods
- Temporarily skipped the rubocop check for lib/backup_restore. This will require more complex refactoring, so I will create a separate PR for review
This method had grown into a monster. Its query had bugs
that I couldn't fix, and new features would be hard to add.
Also I don't understand how it all works anymore...
Replace it with common table expressions that can be queried
to generate the results we need, instead of subtracting
results using lots of "NOT IN" clauses.
Fixed are bugs with tag schemas that use combinations of
tag groups, parent tags, and one-tag-per-topic restrictions.
For example: https://meta.discourse.org/t/130991/6
Previously we were always hard-coding expiry, this allows the secure session
to correctly handle custom expiry times
Also adds a ttl method for looking up time to live
Previous versions of the mail-receiver used query based api credentials,
if we detect this we will show a message in the admin panel to update
the mail receiver.
POSIX's `head` specification states: "The application shall ensure that the number option-argument is a positive decimal integer"
Negative values are supported on GNU `head`, so this works in the discourse docker image. However, in some environments (e.g. macOS), the system `head` version fails with a negative `n` parameter.
This commit does two things:
Checks the status at each stage of the pipe, so it cannot fail silently
Flip the `ls` command to list in descending time order, and use `tail -n +501` instead of `head -n -500`.
The visible result is that macOS users no longer see head: illegal line count -- -500 printed throughout the test suite.
When starting autospec, it says
> Press [ENTER] to stop the current run
However, [ENTER] does nothing unless a spec has failed. Sometimes I want to abort anyway, so that the run is restarted.
I want to use autospec while working on a single spec file. At the moment, it will start running all specs once it completes the file I'm working on. With parallel mode enabled, this causes CPU usage to spike dramatically, affecting IDE performance, battery life, and fan noise. I would prefer that it only runs all specs when I explicitly press [ENTER]
This commit adds a new ENV variable `AUTO_RUN_ALL`. To prevent auto-running all specs, set it to 0. The default behavior remains unchanged.
Instead of enabling `suppress_from_latest` setting on many categories now we can enable `mute_all_categories_by_default` site setting. Then users should opt-in to categories for them to appear in the latest and categories pages.
This change adds a message to the admin panel if it detects an api
requests that doesn't use the new header based authentication method.
The message is to warn people to switch to header based auth and links
to the api documention topic on meta for more info.
This makes it easy to run multiple commands with the same keyword arguments. The main use is for using `chdir` across multiple commands. The `Dir.chdir` method is not concurrency safe because it switches the working directory of the entire process.
Issue was mentioned in this [meta topic](https://meta.discourse.org/t/send-a-notification-to-watching-users-when-adding-tag/125314)
It is working well when category is changed because NotifyCategoryChange job already got that code:
```
if post&.topic&.visible?
post_alerter = PostAlerter.new
post_alerter.notify_post_users(post, User.where(id: args[:notified_user_ids]))
post_alerter.notify_first_post_watchers(post, post_alerter.category_watchers(post.topic))
end
```
For NotifyTagChange job notify post users were missing so it worked only when your notification was set to `watching first post`
- Allow revoking keys without deleting them
- Auto-revoke keys after a period of no use (default 6 months)
- Allow multiple keys per user
- Allow attaching a description to each key, for easier auditing
- Log changes to keys in the staff action log
- Move all key management to one place, and improve the UI
Previously theme translations were loaded along with other plugin API scripts. These run after pre-initializers and initializers when the app boots. This commit moves theme translation loading into pre-initializers, so their behaviour matches core translations more closely.
This is a follow-up to the new feature that allows a category to
require a certain number of tags from a tag group. The tag input will
shows results from the required group if none have been chosen yet.
Once a require tag is selected, the tag input will include other
results as usual. Staff users can ignore this restriction, so the input
behaviour is unchanged for them.
* use image alt as a fallback when there's no title
* update spec
we used to check that the overlay information is added when the image has a titie. This adds 2 more scenarios. One where an image has both a title and an alt, in which case the title should be used and alt ignored.
The other is when there's only an alt, it should then be used to generate the overlay
Also:
Move includes call higher which makes it possible to run all of the
intermediate queries for easier debugging.
Add tests for TagsController with categories in the path.
Meta thread: https://meta.discourse.org/t/cant-dismiss-unread-if-last-post-is-an-assign-or-whisper/131823/7
* when sending a whisper, the highest_staff_post_number is set
in the next_post_number method for a Topic, but the
highest_post_number is left alone. this leaves a situation
where highest_staff_post_number is > highest_post_number
* when TopicsBulkAction#dismiss_posts was run, it was only setting the topic_user
highest_seen_post_number using the highest_post_number from the topic, so if
the user was staff and the last post in a topic was a whisper
their highest seen number was not set, and the topic stayed unread
Found through testing that the bug wasn't to do with Assign/Unassign as they do not affect the post numbers, only whispering does.
In a category's settings, the Tags tab has two new fields to
specify the number of tags that must be added to a topic
from a tag group. When creating a new topic, an error will be
shown to the user if the requirement isn't met.
This was not causing any known issue, because the system user ID is always the same across all sites. However, we should cache this on a per-site basis to be safe.
This ensures we only update last_posted_at which is user facing for non messages
and non whispers.
We still update this date for secure categories, we do not revert it for
deleted posts.
Adds the settings:
raw_email_max_length, raw_rejected_email_max_length, delete_rejected_email_after_days.
These settings control retention of the "raw" emails logs.
raw_email_max_length ensures that if we get incoming email that is huge we will truncate it removing uploads from the raw log.
raw_rejected_email_max_length introduces an even more aggressive truncation for rejected incoming mail.
delete_rejected_email_after_days controls how many days we will keep rejected emails for (default 90)
* FEATURE: Site setting/ui to allow users to set their primary group
* prettier and remove logic from account template
* added 1 to 43 to make web_hook_user_serializer_spec pass
If we are searching for categories by their slugs, it doesn't make sense
to include subcategories since a slug, by itself, does not necessarily
uniquely identify a subcategory.
Similarly, the empty string as a slug is not a good category identifier.
The default locale is :en_US, which is just a thin layer over :en. In
other words, :en_US has the :en locale as a fallback. When "en.yml" is
edited, only the :en locale is refreshed and :en_US becomes stale.
This commit ensures that there is a dependency on the fallback locales
too.
Set `DEBUG_NODE=1` when running `rake smoke:test` and use your favorite tool to debug the smoke tests. See https://nodejs.org/en/docs/guides/debugging-getting-started/ for more information.
The debugger will break at the beginning of the smoke tests when the env variable is set.
That commit introduced a bug to the system: f69dacf979
Restore works fine for multisite, however, stopped working for non-multisite.
Reason for that was that `establish_connection` method got a check if the multisite instance is available:
```
def self.instance
@instance
end
def self.establish_connection(opts)
@instance.establish_connection(opts) if @instance
end
```
However, the reload method don't have that check
```
def self.reload
@instance = new(instance.config_filename)
end
```
To solve it, let's ensure we are in a multisite environment before call reload
While editing the first post it does't bumped the topic when the new post revision created. Because we wrongly assumed that the hidden tags are changed even when no tags are updated.
Doing .pluck(:column).first is a very common pattern in Discourse and in
most cases, a limit cause isn't being added. Instead of adding a limit
clause to all these callsites, this commit adds two new methods to
ActiveRecord::Relation:
pluck_first, equivalent to limit(1).pluck(*columns).first
and pluck_first! which, like other finder methods, raises an exception
when no record is found
- Increase size of the reviewable's conversation excerpt to prevent truncation of the new copy
- Remove the `domain` parameter from the `flag_linked_posts_as_spam` method in the user model since it is no longer needed
- Remove the `domain` interpolation variable from all translation files
- Add "All posts from this user that include links should be reviewed." to server.en.yml for added clarity on why the posts entered the queue
Trying to truncate encoded slugs will mean that we have to keep the URL
valid, which can be tricky as you have to be aware of multibyte
characters.
Since we already have upper bounds for the title, the slug won't grow
for more than title*6 in the worst case. The slug column in the topic
table can store that just fine.
Added a test to ensure that a generated slug is a valid URL too, so we
don't introduce regressions in the future.
Under exceptional situations the automatic draft feature can fail.
This new **hidden, default off** site setting
`backup_drafts_to_pm_length` will automatically backup any draft that is
saved by the system to a dedicated PM (originating from self)
The body of that PM will contain the text of the reply.
We can enable this feature strategically on sites exhibiting issues to
diagnose issues with the draft system and offer a recourse to users who
appear to lose drafts. We automatically checkpoint these drafts every 5
minutes forcing a new revision each 5 minutes so you can revert to old
content.
Longer term we are considering automatically enabling this kind of feature
for extremely long drafts where the risk is really high one could lose
days of writing.
This reverts commit ab74a50d85.
We really want to upgrade redis, but discovered some edge cases
around failover we need to test.
Holding off on the upgrade till a bit more testing happens
This feature amends it so instead of using one challenge and honeypot
statically per site we have a rotating honeypot and challenge value which
changes every hour.
This means you must grab a fresh copy of honeypot and challenge value once
an hour or account registration will be rejected.
We also now cycle the value of the challenge when after successful account
registration forcing an extra call to hp.json between account registrations
Client has been made aware of these changes.
Additionally this contains a JavaScript workaround for:
https://bugs.chromium.org/p/chromium/issues/detail?id=987293
This is client side code that is specific to Chrome user agent and swaps
a PASSWORD type honeypot with a TEXT type honeypot.
When an admin changes the site setting slug_generation_method to
encoded, we weren't really encoding the slug, but just allowing non-ascii
characters in the slug (unicode).
That brings problems when a user posts a link to topic without the slug, as
our topic controller tries to redirect the user to the correct URL that contains
the slug with unicode characters. Having unicode in the Location header in a
response is a RFC violation and some browsers end up in a redirection loop.
Bug report: https://meta.discourse.org/t/-/125371?u=falco
This commit also checks if a site uses encoded slugs and clear all saved slugs
in the db so they can be regenerated using an onceoff job.
Our instance used for template rendering needs a lock to ensure there is
no race condition where rendering happens on 2 threads at the same time.
This can lead to local poisoning which can cause unexpected results in
emails
* [WIP] - default turbo spec env to test
* FEATURE: support for --fast-fail in bin/turbo_rspec
* fast-fail -> fail_fast to match rspec
* Moved thread killing outside of fail-fast check
* Removed failure_count incrementation from fast_fail_met
If the setting is turned on, then the user will receive information
about the subject: if it was deleted or requires some special access to
a group (only if the group is public). Otherwise, the user will receive
a generic #404 error message. For now, this change affects only the
topics and categories controller.
This commit also tries to refactor some of the code related to error
handling. To make error pages more consistent (design-wise), the actual
error page will be rendered server-side.
Using popups is becoming increasingly rare. Full page redirects are already used on mobile, and for some providers. This commit removes all logic related to popup authentication, leaving only the full page redirect method.
For more info, see https://meta.discourse.org/t/do-we-need-popups-for-login/127988
Currently, if you try to run `./bin/turbo_rspec` you will got that error `There are pending migrations, run rake parallel:migrate`
Reason for that is that command is running in `development` mode which includes plugins migration files in ActiveRecord::Migrator.migrations_paths:
```
["db/migrate",
"/home/lis2/projects/discourse/plugins/discourse-details/db/migrate",
"/home/lis2/projects/discourse/plugins/discourse-details/db/post_migrate",
"/home/lis2/projects/discourse/plugins/discourse-local-dates/db/migrate",
"/home/lis2/projects/discourse/plugins/discourse-local-dates/db/post_migrate",
...
]
```
A workaround solution would be to run the command with the TEST environment like `RAILS_ENV=test ./bin/turbo_rspec`
I want to propose in this PR to override migration_paths to check only Discourse core migrations.
We preload to ensure as much memory as possible is reused from unicorn master
to various workers using copy-on-write (sidekiq, unicorn)
This migrates the preloading code into the Discourse module for easier
reuse and adds 3 notable preloading changes
1. We attempt to localize a string on each site, ensuring we warmup
the i18n
2. We preload all our templates (compiling .erb to class)
3. We warm-up our search tokenizer which uses cppjieba which is a large
memory consumer, this will only cause a warmup on CJK sites or sites with
the special site setting enabled.
Previous to this fix we were leaking methods on the internal action view
template class per render.
This caused email generation to be very low and a steady memory leak in the
application in sidekiq when sending out emails
The behavior change is new to Rails 6 so this fix does not need to be
backported into stable.
It was possible to add a category to more than one default group, e.g. "default categories muted" and "default categories watching first post".
The bug was caused by category validations inadvertently comparing strings and numbers.
Overwriting the same file with 'convert' is not always working as expected.
Adding a temporary file as the destination of the downsize makes this operation much more reliable.
Also switched to using (the more aggressive) 50% resize instead of halving the number of pixels.
* FEATURE: Adds an extra protection layer when decompressing files.
* Rename exporter/importer to zip importer. Update old locale
* Added a new composite class to decompress a file with multiple strategies
* Set max file size inside a site setting
* Ensure that file is deleted after compression
* Sanitize path and files before compressing/decompressing
Threadsafety
Since we use the same redis connection in multiple threads, a rogue
transaction in another thread can trample the connection state
(watched keys) that we need to acquire and release the lock properly.
This is fixed by preventing other threads from using the connection
when we are performing these actions.
Off-by-one error
A distributed mutex is now consistently determined to be expired if
the current time is strictly greater than the expire time.
Unwatch before transaction
Since the redis connection is used by so much of the code, it is
difficult to ensure that any watched keys have been cleared. In order
to defend against this rogue connection state, an unwatch has been
added before locking and unlocking.
Logging
Hopefully this log message is more clear.
I introduced DemonBase because I had got some conflict between `demon/base.rb` and `jobs/base.rb`, however, to not rename base class, it is possible to use regex on absolute path in Zeitwerk custom inflector.
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains.
We no longer need to use Rails "require_dependency" anywhere and instead can just use standard
Ruby patterns to require files.
This is a far reaching change and we expect some followups here.
Adds 2 factor authentication method via second factor security keys over [web authn](https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API).
Allows a user to authenticate a second factor on login, login-via-email, admin-login, and change password routes. Adds registration area within existing user second factor preferences to register multiple security keys. Supports both external (yubikey) and built-in (macOS/android fingerprint readers).
Currently, the topic is only validated for censored words and should be validated for blocked words as well.
Blocked word validation is now used by both Post and Topic. To avoid code duplication, I extracted blocked words validation code into separate Validator, and use it in both places.
The only downside is that even if the topic contains blocked words validation message is saying "Your post contains a word that's not allowed: tomato" but I think this is descriptive enough.
Forums without previously calculated scores would return the same values
for low/medium/high sensitivity. Now those are scaled based on the
default value.
The default value has also been changed from 10.0 to 12.5 based on
observing data from live discourse forums.
This means that TL0 users can message groups with "Who can message this
group?" set to "Everyone".
It also means that members of a group with "Who can message this
group?" set to "members, moderators and admins" can also message the
group, even when their trust level is below min_trust_to_send_messages.
In Rails 6 due to internal changes, the following sequence no longer works:
```
RAILS_ENV=test bin/rake db:migrate
RAILS_ENV=test bin/rake db:schema:dump
dropdb discourse_test
createdb discourse_test
RAILS_ENV=test bin/rake db:schema:load
RAILS_ENV=test bin/rake db:migrate
```
What appears to be happening is that our tracking of plugin migrations is
being missed on schema:dump or load.
A more comprehensive fix restoring schema:dump / load support will be
investigated.
Prior to the new review queue there were a couple special cases where
posts would be auto hidden:
* If a TL3 or above flagged a TL0 post as spam
* If a TL4 or above flagged a non-staff, non-TL4 post as spam, inappropriate or off
topic.
These cases are now removed in favour of the scoring system.
Prior to this change plugin migrations were not working and multisite
migrations not working.
Rails internals changed so we need to account for it.
Specifically semantics of `db:migrate` in rails changed so it is sort of
a "multisite:migrate".
This is a temporary workaround for the issue in https://github.com/rails/rails/pull/36949
Discussing a proper fix in Rails with the Rails team.
Prior to this fix we were spinning up a thread every time we closed a connection
to the db.
* Adjustments to pass specs on Rails 6.0.0
* Use classic autoloader instead of Zeitwerk
* Update Rails 6.0.0 deprecated methods
* Rails 6.0.0 not allowing column with integer name
* Drop freedom_patches/rails6.rb
* Default value for trigger_transactional_callbacks? is true
* Bump rspec-rails version to 4.0.0.beta2
* FIX: inline_uploads and subfolder
* if subfolder, also look for images with a path containing
cdn_url + relative_url_root
* FIX: migrate_to_s3 task and subfolder
* Extract QuickAccessPanel from UserNotifications.
* FEATURE: Quick access panels in user menu.
This feature adds quick access panels for bookmarks and personal
messages. It allows uses to browse recent items directly in the user
menu, without being redirected to the full pages.
* REFACTOR: Use QuickAccessItem for messages.
Reusing `DefaultNotificationItem` feels nice but it actually requires a
lot of extra work that is not needed for a quick access item.
Also, `DefaultNotificationItem` shows an incorrect tooptip ("unread
private message"), and it is not trivial to remove / override that.
* Use a plain JS object instead.
An Ember object was required when `DefaultNotificationItem` was used.
* Prefix instead suffix `_` for private helpers.
* Set to null instead of deleting object keys.
JavaScript engines can optimize object property access based on the
object’s shape. https://mathiasbynens.be/notes/shapes-ics
* Change trivial try/catch to one-liners.
* Return the promise in case needs to be waited on.
* Refactor showAll to a link with href
* Store `emptyStatePlaceholderItemText` in state.
* Store items in Session singleton instead.
We can drop `staleItems` (and `findStaleItems`) altogether. Because
`(old) items === staleItems` when switching back to a quick access
panel.
* Add `limit` parameter to the `user_actions` API.
* Explicitly import Session instead.
* FEATURE: Add tl2 threshold for editing new posts
* Adds a new setting and for tl2 editing posts (30 days same as old value)
* Sets the tl0/tl1 editing period as 1 day
* FIX: Spec uses wrong setting
* Fix site setting on guardian spec
* FIX: post editing period specs
* Avoid shared examples
* Use update_columns to avoid callbacks on user during tests
This commit introduces 2 features:
1. DISCOURSE_COMPRESS_ANON_CACHE (true|false, default false): this allows
you to optionally compress the anon cache body entries in Redis, can be
useful for high load sites with Redis that lives on a separate server to
to webs
2. DISCOURSE_ANON_CACHE_STORE_THRESHOLD (default 2), only pop entries into
redis if we observe them more than N times. This avoids situations where
a crawler can walk a big pile of topics and store them all in Redis never
to be used. Our default anon cache time for topics is only 60 seconds. Anon
cache is in place to avoid the "slashdot" effect where a single topic is
hit by 100s of people in one minute.
Start tracking the date an api key was last used. This has already been
the case for user_api_keys.
This information can provide us with the ability to automatically expire
unused api keys after N days.
This allows custom plugins such as prometheus exporter to log how many
requests are stored in the anon cache vs used by the anon cache.
This metric allows us to fine tune cache behaviors
* FIX: User should get notified when a post is deleted
* FEATURE: Notify posters when restoring flagged posts
* Fix typo
Co-Authored-By: Régis Hanol <regis@hanol.fr>
* Improve tests
This reverts commit e805d44965.
We now have mechanisms in place to ensure heartbeat will always
be scheduled even if the scheduler is overloaded per: 098f938b
* The read indicator now shows up when no member has read the last post of the topic (written by a non-member)
* The read indicator works on mobile and receives live updates from message bus
* The icon we display in the topic list was changed
* Added a title to the indicator to indicate its purpose when hovering over it
In some very specific cases (large sites) shared drafts can introduce a
performance hit due to the mechanism used to filter out topics
This avoids the entire process when shared drafts are not enabled
* Revert "Revert "FEATURE: Publish read state on group messages. (#7989) [Undo revert] (#8024)""
This reverts commit 36425eb9f0.
* Fix: Show who read only if the attribute is enabled
* PERF: Precalculate the last post readed by a group member
* Use book-reader icon instear of far-eye
* FIX: update topic groups correctly
* DEV: Tidy up read indicator update on write
This is a very long standing bug we had, if a plugin attempted to amend a
serializer core was not "correcting" the situation for all descendant classes
this often only showed up in production cause production eager loads serializers
prior to plugins amending them.
This is a critical fix for various plugins
* FIX: Heartbeat check per sidekiq process
* Rename method
* Remove heartbeat queues of previous bootups
* Regis feedback
* Refactor before_start
* Update lib/demon/sidekiq.rb
Co-Authored-By: Régis Hanol <regis@hanol.fr>
* Update lib/demon/sidekiq.rb
Co-Authored-By: Régis Hanol <regis@hanol.fr>
* Expire redis keys after 3600 seconds
* Don't use redis to store the list of queues
This reverts commit 39c31a3d76.
Sorry about this, we have decided againse supporting 0-RTT directly in
core, this can be supported with similar hacks to this commit in a
plugin.
That said, we recommend against using a 0-RTT proxy for the Discourse
app due to inherit risk of replay attacks.
This change allows themes and components access to theme assets.
This means that inside theme js you can now get the URL for an asset with:
```
settings.theme_uploads.name
```
* Reenable: "FEATURE: Publish read state on group messages. (#7989)"
This reverts commit 67f5cc1ce8.
* FIX: Read indicator only appears when the group setting is enabled
* Enable or disable read state based on group attribute
* When read state needs to be published, the minimum unread count is calculated in the topic query. This way, we can know if someone reads the last post
* The option can be enabled/disabled from the UI
* The read indicator will live-updated using message bus
* Show read indicator on every post
* The read indicator now shows read count and can be expanded to see user avatars
* Read count gets updated everytime someone reads a message
* Simplify topic-list read indicator logic
* Unsubscribe from message bus on willDestroyElement, removed unnecesarry values from post-menu, and added a comment to explain where does minimum_unread_count comes from
Previously we relied on side effects to set tracking state correctly
when inviting groups to messages
Also has a minor optimisation in that we use pluck instead of pulling in
full record
This adds support for a `<d-topics-list>` tag you can embed in your site
that will be rendered as a list of discourse topics. Any attributes on
the tag will be passed as filters. For example:
`<d-topics-list discourse-url="URL" category="1234">` will filter to category 1234.
To use this feature, enable the `embed topics list` site setting. Then
on the site you want to embed, include the following javascript:
`<script
src="http://URL/javascripts/embed-topics.js"></script>`
Where `URL` is your discourse forum's URL.
Then include the `<d-topics-list discourse-url="URL">` tag in your HTML document and it will
be replaced with the list of topics.
In some very rare cases CssParser could be loaded but CssParser::Parser not
this ensures we check for the actual constant we plan to call for concurrent
digest generations
There are 5 visibility levels (similar to group visibility)
public (default)
logged-in users
members only
staff
owners
Admins & group owners always have visibility to group members.
Running this inline makes more sense otherwise there is extreme risk in
saturating sidekiq queue.
This also reworks ordering and selection so we double check if a post needs
rebaking prior to rebaking, this unlocks the ability to run this rake task
from multiple consoles.
This column often breaks remaps because of some weird, long URLs. The data isn't that important, so truncating a couple of URLs doesn't hurt that much.
When a user liked, unliked and liked again the same post, the poster
would receive a notification such as "X and X liked ...". This happened
because PostActionNotifier.post_action_created was called twice.
This displays more useful messages for the most common issues we see:
- CSRF (when the user switches browser)
- Invalid IAT (when the server clock is wrong)
- OAuth::Unauthorized for OAuth1 providers, when the credentials are incorrect
This commit also stops earlier for disabled authenticators. Now we stop at the request phase, rather than the callback phase.
