This reverts commit ab74a50d85.
We really want to upgrade redis, but discovered some edge cases
around failover we need to test.
Holding off on the upgrade till a bit more testing happens
This feature amends it so instead of using one challenge and honeypot
statically per site we have a rotating honeypot and challenge value which
changes every hour.
This means you must grab a fresh copy of honeypot and challenge value once
an hour or account registration will be rejected.
We also now cycle the value of the challenge when after successful account
registration forcing an extra call to hp.json between account registrations
Client has been made aware of these changes.
Additionally this contains a JavaScript workaround for:
https://bugs.chromium.org/p/chromium/issues/detail?id=987293
This is client side code that is specific to Chrome user agent and swaps
a PASSWORD type honeypot with a TEXT type honeypot.
When an admin changes the site setting slug_generation_method to
encoded, we weren't really encoding the slug, but just allowing non-ascii
characters in the slug (unicode).
That brings problems when a user posts a link to topic without the slug, as
our topic controller tries to redirect the user to the correct URL that contains
the slug with unicode characters. Having unicode in the Location header in a
response is a RFC violation and some browsers end up in a redirection loop.
Bug report: https://meta.discourse.org/t/-/125371?u=falco
This commit also checks if a site uses encoded slugs and clear all saved slugs
in the db so they can be regenerated using an onceoff job.
Our instance used for template rendering needs a lock to ensure there is
no race condition where rendering happens on 2 threads at the same time.
This can lead to local poisoning which can cause unexpected results in
emails
* [WIP] - default turbo spec env to test
* FEATURE: support for --fast-fail in bin/turbo_rspec
* fast-fail -> fail_fast to match rspec
* Moved thread killing outside of fail-fast check
* Removed failure_count incrementation from fast_fail_met
If the setting is turned on, then the user will receive information
about the subject: if it was deleted or requires some special access to
a group (only if the group is public). Otherwise, the user will receive
a generic #404 error message. For now, this change affects only the
topics and categories controller.
This commit also tries to refactor some of the code related to error
handling. To make error pages more consistent (design-wise), the actual
error page will be rendered server-side.
Using popups is becoming increasingly rare. Full page redirects are already used on mobile, and for some providers. This commit removes all logic related to popup authentication, leaving only the full page redirect method.
For more info, see https://meta.discourse.org/t/do-we-need-popups-for-login/127988
Currently, if you try to run `./bin/turbo_rspec` you will got that error `There are pending migrations, run rake parallel:migrate`
Reason for that is that command is running in `development` mode which includes plugins migration files in ActiveRecord::Migrator.migrations_paths:
```
["db/migrate",
"/home/lis2/projects/discourse/plugins/discourse-details/db/migrate",
"/home/lis2/projects/discourse/plugins/discourse-details/db/post_migrate",
"/home/lis2/projects/discourse/plugins/discourse-local-dates/db/migrate",
"/home/lis2/projects/discourse/plugins/discourse-local-dates/db/post_migrate",
...
]
```
A workaround solution would be to run the command with the TEST environment like `RAILS_ENV=test ./bin/turbo_rspec`
I want to propose in this PR to override migration_paths to check only Discourse core migrations.
We preload to ensure as much memory as possible is reused from unicorn master
to various workers using copy-on-write (sidekiq, unicorn)
This migrates the preloading code into the Discourse module for easier
reuse and adds 3 notable preloading changes
1. We attempt to localize a string on each site, ensuring we warmup
the i18n
2. We preload all our templates (compiling .erb to class)
3. We warm-up our search tokenizer which uses cppjieba which is a large
memory consumer, this will only cause a warmup on CJK sites or sites with
the special site setting enabled.
Previous to this fix we were leaking methods on the internal action view
template class per render.
This caused email generation to be very low and a steady memory leak in the
application in sidekiq when sending out emails
The behavior change is new to Rails 6 so this fix does not need to be
backported into stable.
It was possible to add a category to more than one default group, e.g. "default categories muted" and "default categories watching first post".
The bug was caused by category validations inadvertently comparing strings and numbers.
Overwriting the same file with 'convert' is not always working as expected.
Adding a temporary file as the destination of the downsize makes this operation much more reliable.
Also switched to using (the more aggressive) 50% resize instead of halving the number of pixels.
* FEATURE: Adds an extra protection layer when decompressing files.
* Rename exporter/importer to zip importer. Update old locale
* Added a new composite class to decompress a file with multiple strategies
* Set max file size inside a site setting
* Ensure that file is deleted after compression
* Sanitize path and files before compressing/decompressing
Threadsafety
Since we use the same redis connection in multiple threads, a rogue
transaction in another thread can trample the connection state
(watched keys) that we need to acquire and release the lock properly.
This is fixed by preventing other threads from using the connection
when we are performing these actions.
Off-by-one error
A distributed mutex is now consistently determined to be expired if
the current time is strictly greater than the expire time.
Unwatch before transaction
Since the redis connection is used by so much of the code, it is
difficult to ensure that any watched keys have been cleared. In order
to defend against this rogue connection state, an unwatch has been
added before locking and unlocking.
Logging
Hopefully this log message is more clear.
I introduced DemonBase because I had got some conflict between `demon/base.rb` and `jobs/base.rb`, however, to not rename base class, it is possible to use regex on absolute path in Zeitwerk custom inflector.
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains.
We no longer need to use Rails "require_dependency" anywhere and instead can just use standard
Ruby patterns to require files.
This is a far reaching change and we expect some followups here.
Adds 2 factor authentication method via second factor security keys over [web authn](https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API).
Allows a user to authenticate a second factor on login, login-via-email, admin-login, and change password routes. Adds registration area within existing user second factor preferences to register multiple security keys. Supports both external (yubikey) and built-in (macOS/android fingerprint readers).
Currently, the topic is only validated for censored words and should be validated for blocked words as well.
Blocked word validation is now used by both Post and Topic. To avoid code duplication, I extracted blocked words validation code into separate Validator, and use it in both places.
The only downside is that even if the topic contains blocked words validation message is saying "Your post contains a word that's not allowed: tomato" but I think this is descriptive enough.
Forums without previously calculated scores would return the same values
for low/medium/high sensitivity. Now those are scaled based on the
default value.
The default value has also been changed from 10.0 to 12.5 based on
observing data from live discourse forums.
This means that TL0 users can message groups with "Who can message this
group?" set to "Everyone".
It also means that members of a group with "Who can message this
group?" set to "members, moderators and admins" can also message the
group, even when their trust level is below min_trust_to_send_messages.
In Rails 6 due to internal changes, the following sequence no longer works:
```
RAILS_ENV=test bin/rake db:migrate
RAILS_ENV=test bin/rake db:schema:dump
dropdb discourse_test
createdb discourse_test
RAILS_ENV=test bin/rake db:schema:load
RAILS_ENV=test bin/rake db:migrate
```
What appears to be happening is that our tracking of plugin migrations is
being missed on schema:dump or load.
A more comprehensive fix restoring schema:dump / load support will be
investigated.
Prior to the new review queue there were a couple special cases where
posts would be auto hidden:
* If a TL3 or above flagged a TL0 post as spam
* If a TL4 or above flagged a non-staff, non-TL4 post as spam, inappropriate or off
topic.
These cases are now removed in favour of the scoring system.
Prior to this change plugin migrations were not working and multisite
migrations not working.
Rails internals changed so we need to account for it.
Specifically semantics of `db:migrate` in rails changed so it is sort of
a "multisite:migrate".
This is a temporary workaround for the issue in https://github.com/rails/rails/pull/36949
Discussing a proper fix in Rails with the Rails team.
Prior to this fix we were spinning up a thread every time we closed a connection
to the db.
* Adjustments to pass specs on Rails 6.0.0
* Use classic autoloader instead of Zeitwerk
* Update Rails 6.0.0 deprecated methods
* Rails 6.0.0 not allowing column with integer name
* Drop freedom_patches/rails6.rb
* Default value for trigger_transactional_callbacks? is true
* Bump rspec-rails version to 4.0.0.beta2
* FIX: inline_uploads and subfolder
* if subfolder, also look for images with a path containing
cdn_url + relative_url_root
* FIX: migrate_to_s3 task and subfolder
* Extract QuickAccessPanel from UserNotifications.
* FEATURE: Quick access panels in user menu.
This feature adds quick access panels for bookmarks and personal
messages. It allows uses to browse recent items directly in the user
menu, without being redirected to the full pages.
* REFACTOR: Use QuickAccessItem for messages.
Reusing `DefaultNotificationItem` feels nice but it actually requires a
lot of extra work that is not needed for a quick access item.
Also, `DefaultNotificationItem` shows an incorrect tooptip ("unread
private message"), and it is not trivial to remove / override that.
* Use a plain JS object instead.
An Ember object was required when `DefaultNotificationItem` was used.
* Prefix instead suffix `_` for private helpers.
* Set to null instead of deleting object keys.
JavaScript engines can optimize object property access based on the
object’s shape. https://mathiasbynens.be/notes/shapes-ics
* Change trivial try/catch to one-liners.
* Return the promise in case needs to be waited on.
* Refactor showAll to a link with href
* Store `emptyStatePlaceholderItemText` in state.
* Store items in Session singleton instead.
We can drop `staleItems` (and `findStaleItems`) altogether. Because
`(old) items === staleItems` when switching back to a quick access
panel.
* Add `limit` parameter to the `user_actions` API.
* Explicitly import Session instead.
* FEATURE: Add tl2 threshold for editing new posts
* Adds a new setting and for tl2 editing posts (30 days same as old value)
* Sets the tl0/tl1 editing period as 1 day
* FIX: Spec uses wrong setting
* Fix site setting on guardian spec
* FIX: post editing period specs
* Avoid shared examples
* Use update_columns to avoid callbacks on user during tests
This commit introduces 2 features:
1. DISCOURSE_COMPRESS_ANON_CACHE (true|false, default false): this allows
you to optionally compress the anon cache body entries in Redis, can be
useful for high load sites with Redis that lives on a separate server to
to webs
2. DISCOURSE_ANON_CACHE_STORE_THRESHOLD (default 2), only pop entries into
redis if we observe them more than N times. This avoids situations where
a crawler can walk a big pile of topics and store them all in Redis never
to be used. Our default anon cache time for topics is only 60 seconds. Anon
cache is in place to avoid the "slashdot" effect where a single topic is
hit by 100s of people in one minute.
Start tracking the date an api key was last used. This has already been
the case for user_api_keys.
This information can provide us with the ability to automatically expire
unused api keys after N days.
This allows custom plugins such as prometheus exporter to log how many
requests are stored in the anon cache vs used by the anon cache.
This metric allows us to fine tune cache behaviors
* FIX: User should get notified when a post is deleted
* FEATURE: Notify posters when restoring flagged posts
* Fix typo
Co-Authored-By: Régis Hanol <regis@hanol.fr>
* Improve tests
This reverts commit e805d44965.
We now have mechanisms in place to ensure heartbeat will always
be scheduled even if the scheduler is overloaded per: 098f938b
* The read indicator now shows up when no member has read the last post of the topic (written by a non-member)
* The read indicator works on mobile and receives live updates from message bus
* The icon we display in the topic list was changed
* Added a title to the indicator to indicate its purpose when hovering over it
In some very specific cases (large sites) shared drafts can introduce a
performance hit due to the mechanism used to filter out topics
This avoids the entire process when shared drafts are not enabled
* Revert "Revert "FEATURE: Publish read state on group messages. (#7989) [Undo revert] (#8024)""
This reverts commit 36425eb9f0.
* Fix: Show who read only if the attribute is enabled
* PERF: Precalculate the last post readed by a group member
* Use book-reader icon instear of far-eye
* FIX: update topic groups correctly
* DEV: Tidy up read indicator update on write
This is a very long standing bug we had, if a plugin attempted to amend a
serializer core was not "correcting" the situation for all descendant classes
this often only showed up in production cause production eager loads serializers
prior to plugins amending them.
This is a critical fix for various plugins
* FIX: Heartbeat check per sidekiq process
* Rename method
* Remove heartbeat queues of previous bootups
* Regis feedback
* Refactor before_start
* Update lib/demon/sidekiq.rb
Co-Authored-By: Régis Hanol <regis@hanol.fr>
* Update lib/demon/sidekiq.rb
Co-Authored-By: Régis Hanol <regis@hanol.fr>
* Expire redis keys after 3600 seconds
* Don't use redis to store the list of queues
This reverts commit 39c31a3d76.
Sorry about this, we have decided againse supporting 0-RTT directly in
core, this can be supported with similar hacks to this commit in a
plugin.
That said, we recommend against using a 0-RTT proxy for the Discourse
app due to inherit risk of replay attacks.
