This updates a bunch of gems that have very low risk of breaking anything
or are mostly used in dev. Overall our goal is to be on latest gems with
latest bug fixes so this is us working in that direction.
Previously it was unclear why certain gems are being held back cause Gemfile
had no comment explaining it.
I tried to add some explanation from memory and remove some exceptions that
seemed to be superfluous.
This upgrades shoulda to latest, it appears to work once a couple of assertions
are removed
Also update http accept language used to auto detect language from http header
this is tested
Zeitwerk small update seems fine
This reverts commit ab74a50d85.
We really want to upgrade redis, but discovered some edge cases
around failover we need to test.
Holding off on the upgrade till a bit more testing happens
Previously some local micro-benchmarks revealed it was not giving any perf
benefits.
Now that we upgraded to 2.6.5 we are seeing some segfaults.
No need to carry this dependency around anymore.
We can re-evaluate in future if it improves perf and fix the segfaults.
Bump onebox version, and add new styling
Commit, PR and Issue oneboxes are updated with a new design. Timestamps are now localized using local-dates (if installed).
Bump onebox version to include new github rendering, and add relevant CSS
Avatars are reduced in size significantly, and icons are added to easily differentiate PRs and commits. The 'Issue:' prefix is removed from issue oneboxes, to make them consistent with commits and PRs.
We preload to ensure as much memory as possible is reused from unicorn master
to various workers using copy-on-write (sidekiq, unicorn)
This migrates the preloading code into the Discourse module for easier
reuse and adds 3 notable preloading changes
1. We attempt to localize a string on each site, ensuring we warmup
the i18n
2. We preload all our templates (compiling .erb to class)
3. We warm-up our search tokenizer which uses cppjieba which is a large
memory consumer, this will only cause a warmup on CJK sites or sites with
the special site setting enabled.
Adds 2 factor authentication method via second factor security keys over [web authn](https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API).
Allows a user to authenticate a second factor on login, login-via-email, admin-login, and change password routes. Adds registration area within existing user second factor preferences to register multiple security keys. Supports both external (yubikey) and built-in (macOS/android fingerprint readers).
This updates rubyzip library so that callers can trust entries when
extracting files avoiding situations where a rogues zip imported by a rogue
admin could cause a disk space issue.
This corrects an XSS in ?pp=help.
Also removes the jQuery dependency from rack-mini-profiler and restricts
memory sensitive profiling methods development only.
* Adjustments to pass specs on Rails 6.0.0
* Use classic autoloader instead of Zeitwerk
* Update Rails 6.0.0 deprecated methods
* Rails 6.0.0 not allowing column with integer name
* Drop freedom_patches/rails6.rb
* Default value for trigger_transactional_callbacks? is true
* Bump rspec-rails version to 4.0.0.beta2
New version of mini scheduler allows you to select the name of a schedule
in the history page in `/sidekiq/scheduler/history`.
This is handy for quickly looking up timing trends.
This commit introduces 2 features:
1. DISCOURSE_COMPRESS_ANON_CACHE (true|false, default false): this allows
you to optionally compress the anon cache body entries in Redis, can be
useful for high load sites with Redis that lives on a separate server to
to webs
2. DISCOURSE_ANON_CACHE_STORE_THRESHOLD (default 2), only pop entries into
redis if we observe them more than N times. This avoids situations where
a crawler can walk a big pile of topics and store them all in Redis never
to be used. Our default anon cache time for topics is only 60 seconds. Anon
cache is in place to avoid the "slashdot" effect where a single topic is
hit by 100s of people in one minute.
Prior to this commit if we ever checked in a per_host directive and scheduled
job was previously global, total chaos could ensue as a feedback loop would
start
We very rarely used per_host to date, but just started making use of it
for heartbeats
Under extreme load on large databases certain regular jobs can take quite
a while to run. We need to ensure we never starve a sidekiq from running
mini scheduler, cause without it we are unable to queue stuff such as
heartbeat jobs.
This adds a 1 minute rate limit to all JS error reporting per IP. Previously
we would only use the global rate limit.
This also introduces DISCOURSE_ENABLE_JS_ERROR_REPORTING, if it is set to
false then no JS error reporting will be allowed on the site.
The message_bus performs a fair amount of work prior to hijacking requests
this change ensures that if there is a situation where the server is flooded
message_bus will inform client to back off for 30 seconds + random(120 secs)
This back-off is ultra cheap and happens very early in the middleware.
It corrects a situation where a flood to message bus could cause the app
to become unresponsive
MessageBus update is here to ensure message_bus gem properly respects
Retry-After header and status 429.
Under normal state this code should never trigger, to disable raise the
value of DISCOURSE_REJECT_MESSAGE_BUS_QUEUE_SECONDS, default is to tell
message bus to go away if we are queueing for 100ms or longer
This feature adds the ability to customize the HTML part of all emails using a custom HTML template and optionally some CSS to style it. The CSS will be parsed and converted into inline styles because CSS is poorly supported by email clients. When writing the custom HTML and CSS, be aware of what email clients support. Keep customizations very simple.
Customizations can be added and edited in Admin > Customize > Email Style.
Since the summary email is already heavily styled, there is a setting to disable custom styles for summary emails called "apply custom styles to digest" found in Admin > Settings > Email.
As part of this work, RTL locales are now rendered correctly for all emails.
* Revert "Revert "FEATURE: admin/user exports are compressed using the zip format (#7784)""
This reverts commit f89bd55576.
* Replace .tar.zip with .zip
* FEATURE: admin/user exports are compressed using the zip format
* Update translations. Theme exporter now exports .zip file. Theme importer supports .zip and .gz files
* Fix controller test, updated locale and skip saving the csv export to disk
Overcommit uses prebuilt hooks and require global installation.
To avoid this issues replace it with Lefthook.
Lefthook will be installed with npm packages. New contributors
will have fully consistent git hooks.
* Updated test-prof
* Made rails_helper.rb use new test-prof APIs
Instead of the previous temporary hacks.
* Added environment option to disable prefabrication
It was removed mistakenly
We recently noticed a leak, this introduces a new method on MiniRacer::Context
```
context.write_heap_snapshot(path)
```
To dump current memory in v8 context to a file, this can then be analyzed
in chrome and other similar tools
Can be triggered in production using rbtrace
AWS is a big moving target, this fills gaps in the API. Technically we
use such a tiny surface area that it probably does not matter, but it is
good to be up to date here.
This updates another batch of gems that are lowish risk
Most of the gem changes are here for Rails 6 / Ruby 2.6.3 support
Excon did some stuff around better cipher ordering
- Update github_blob engine to support displaying stl files
- FEATURE: add `data-original-href` attribute to Vimeo iframes
- Add poster image for video oneboxes
Adds the parallel_tests gem, and redis/postgres configuration for running rspec tests in parallel. To use:
```
rake parallel:rake[db:create]
rake parallel:rake[db:migrate]
rake parallel:spec
```
This brings the test suite from 12m20s to 3m11s on my macOS machine
- Fixes deprecation regarding usage of BigDecimal in dev
- Handle edge case where query_hash would clear a non existent result
- Minor perf improvement to query_single
Most important thing though is that we are now on the latest gem
The only reason this was done was to remove this deprecation when running
tests.
```
/home/sam/.rbenv/versions/2.6.1/lib/ruby/gems/2.6.0/gems/mini_sql-0.1.10/lib/mini_sql/coders.rb:5: warning: BigDecimal.new is deprecated; use BigDecimal() method instead.
```
Ruby 2.5.3 has an upatched issue that crashes unicorn after fork:
https://bugs.ruby-lang.org/issues/14634
This will be patched in 2.5.4 however for now just warn people dev is slower
and disable async logging on the older rubies
Another group of gems updated, better_errors is fairly safe as its for dev
chunky_png fairly safe, used for getting image info from pngs, erubi update
is very safe. Sanitize is used by onebox and should always be on latest.
Long term we should think of making sanitize an optional dependency on onebox
cause we are happy to just provide methods from core to do this and it would
remove nokogumbo and sanitize deps.
This follows up on 03376224 which reverted the redis update which is not
supported, rest of the gems should be fine.(unicorn / nokogiri / sidekiq /
pg)
This updates a few more lower risk gems, the main goal here is to have
nothing outdated. Avoiding a giant commit is going to make it slightly
easier to partially roll back if something goes wrong
Two very low risk updates, message_bus has been released no need to depend
on pre-release.
mini_racer update is for a very minor change (shared isolates are not used
in discourse so it is not a fix we technically need)
Launching Discourse no longer should require foreman in dev. We can simply
use `bin/unicorn` which automatically launches a sidekiq worker.
The foreman gem depends on thor ~> 0.19.1 which is no longer supported in
rails 6. So this pre-emptively prepares us for it.
This temporary revert is here cause a certain immune system in logster
when a large number of events are groupes is no longer working in
production.
What this means is that if you have 10 thousand errors grouped it can
impose an very large amount of load on redis.
This reverts commit abee39ecd0.
Fixes linear-gradient CSS issue on iOS
Increases z-index for the actions menu to prevent the screen divider/splitter from obstructing the menu
e43e98dc61
This provides us with instrumentation missing after rails upgrade
Latest version of rails uses exec_params internally which is no longer
routed to intercepted methods in mini profiler 1.0.0
Message bus uses a key to keep track of the last id for each channel
this key was never expired even if channel data expired
This change ensures we also expire the tracking key, it means a lot to us
cause each topic has a channel, so if you have 1 million topics that is
1 million keys that may persist forever
This release contains security fixes to the underlying rack library
used by Discourse.
Impact is not too high as we do not use request.scheme in our templates
* FEATURE: Added MaxMindDb to resolve IP information.
* FEATURE: Added browser detection based on user agent.
* FEATURE: Added recently used devices in user preferences.
* DEV: Added acceptance test for recently used devices.
* UX: Do not show 'Show more' button if there aren't more tokens.
* DEV: Fix unit tests.
* DEV: Make changes after code review.
* Add more detailed unit tests.
* Improve logging messages.
* Minor coding style fixes.
* DEV: Use DropdownSelectBoxComponent and run Prettier.
* DEV: Fix unit tests.
Introduce new patterns for direct sql that are safe and fast.
MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API
- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder
See more at: https://github.com/discourse/mini_sql
This updates tests to use latest rails 5 practice
and updates ALL dependencies that could be updated
Performance testing shows that performance has not regressed
if anything it is marginally faster now.
This upgrades message bus to latest and corrects an issue where
it did not track the fact it was polling for the first 30 seconds
It corrected internal state after, but this caused some odd issues
* Feature: Push notifications for Android
Notification config for desktop and mobile are merged.
Desktop notifications stay as they are for desktop views.
If mobile mode, push notifications are enabled.
Added push notification subscriptions in their own table, rather than through
custom fields.
Notification banner prompts appear for both mobile and desktop when enabled.
* Use a EmailValidator.email_regexp for `Email.is_valid?`
check as we're seeing an increase in allocation when
parsing email addresses wih `Mail::Address`.
implemented review items.
Blocking previous codes - valid 2-factor auth tokens can only be authenticated once/30 seconds.
I played with updating the “last used” any time the token was attempted but that seemed to be overkill, and frustrating as to why a token would fail.
Translatable texts.
Move second factor logic to a helper class.
Move second factor specific controller endpoints to its own controller.
Move serialization logic for 2-factor details in admin user views.
Add a login ember component for de-duplication
Fix up code formatting
Change verbiage of google authenticator
add controller tests:
second factor controller tests
change email tests
change password tests
admin login tests
add qunit tests - password reset, preferences
fix: check for 2factor on change email controller
fix: email controller - only show second factor errors on attempt
fix: check against 'true' to enable second factor.
Add modal for explaining what 2fa with links to Google Authenticator/FreeOTP
add two factor to email signin link
rate limit if second factor token present
add rate limiter test for second factor attempts
oga gem is automatically required by the aws gem
the oga gem retains about 1mb of memory, aws now uses nokogiri
This also removes the html normalize from the pretty text specs that was
a fair bit buggy as the polls test shows.
