Previously we only allowed one image optimization per machine, this meant there
was cross talk between avatar resizing and Sidekiq. This could lead to large
amounts of starvation when optimized image version changed which in turn could
block the Sidekiq queue.
This increases amount of allowed load on machines but this is preferable to
having crosstalk between avatar resizing and Sidekiq.
Some cloud providers (Google Memorystore) do not support any CLIENT commands
By setting :id to nil in the redis config hash we can avoid these commands.
This adds a special global setting GCE users can enable:
`DISCOURSE_REDIS_SKIP_CLIENT_COMMANDS = true`
We have the periodical job that regularly will rebake old posts. This is
used to trickle in update to cooked markdown. The problem is that each rebake
can issue multiple background jobs (post process and pull hotlinked images)
Previously we had no per-cluster limit so cluster running 100s of sites could
flood the sidekiq queue with rebake related jobs.
New system introduces a hard limit of 300 rebakes per 15 minutes across a
cluster to ensure the sidekiq job is not dominated by this.
We also reduced `rebake_old_posts_count` to 80, which is a safer default.
This reverts commit 993f847a2c.
There is an edge case where the link click redirect fails when the URL has trailing slash. Need to figure out a better fix for this.
Previously we had no idea what algorithm generated thumbnails, this starts tracking the version.
We also bumped up the version to force all optimized images to be generated. This is important cause we recently introduced pngquant which results in much smaller images.
This feature ensures optimized images run via pngquant, this results extreme amounts of savings for resized images. Effectively the only impact is that the color palette on small resized images is reduced to 256.
To ensure safety we only apply this optimisation to images smaller than 500k.
This commit also makes a bunch of image specs less fragile.
Previously if upload had missing width and height we would calculate
on first use BUT we (me) forgot to save this to the database
This was particularly bad on home page cause category images (when old)
miss dimensions.
This generates a 10x10 PNG thumbnail for each lightboxed image.
If Image Lazy Loading is enabled (IntersectionObserver API) then
we'll load the low res version when offscreen. As the image scrolls
in we'll swap it for the high res version.
We use a WeakMap to track the old image attributes. It's much less
memory than storing them as `data-*` attributes and swapping them
back and forth all the time.
* Dashboard doesn't timeout anymore when Amazon S3 is used for backups
* Storage stats are now a proper report with the same caching rules
* Changing the backup_location, s3_backup_bucket or creating and deleting backups removes the report from the cache
* It shows the number of backups and the backup location
* It shows the used space for the correct backup location instead of always showing used space on local storage
* It shows the date of the last backup as relative date
`SiteSerializer#is_readonly` is cached for an anonymous user so we have
to clear the cache when disabling readonly mode. Otherwise, the site may
appear to be in readonly mode for an extended period of time.
Some URLs in browsers are non compliant and contain twos `#` this commit adds
special handling for this edge case by auto encoding any fragments containing `#`
Do not send an activation email to users invited via email. They
already confirmed their email address by clicking the invite link.
Users invited via link will need to confirm their email address before
they can login.
This was an indentation mistake introduced in 44eba0b. Pretty understandable, considering we are indented 8 levels deep in this method. Will follow-up with a refactor to improve this.
Previously we would notify on small actions if they were whispers
this inconsistently lead to all sorts of problems including
- collapsed "N replies" after assign
- empty push notifications
New behavior adds an api to explicitly send push notifications as well
if needed: create_notification_alert
UserStat has some special logic to keep adding time read if repeat calls
are made in intervals less than 100 seconds. This is called regularly
when we update read timings on a topic.
We only need to cache this key in redis for 100 seconds, however previously
we would keep it forever, 1 key per user. This has potential of bloating
a very large amount of keys for no longer active users in redis.
* FEATURE: allow plugins and themes to extend the default CSP
For plugins:
```
extend_content_security_policy(
script_src: ['https://domain.com/script.js', 'https://your-cdn.com/'],
style_src: ['https://domain.com/style.css']
)
```
For themes and components:
```
extend_content_security_policy:
type: list
default: "script_src:https://domain.com/|style_src:https://domain.com"
```
* clear CSP base url before each test
we have a test that stubs `Rails.env.development?` to true
* Only allow extending directives that core includes, for now
Changes to functionality
- Removed syncing of user metadata including gender, location etc.
These are no longer available to standard Facebook applications.
- Removed the remote 'revoke' functionality. No other providers have
it, and it does not appear to be standard practice in other apps.
- The 'facebook_no_email' event is no longer logged. The system can
cope fine with a missing email address.
Data is migrated to the new user_associated_accounts table.
facebook_user_infos can be dropped once we are confident the data has
been migrated successfully.
A generic implementation of Auth::Authenticator which stores data in the
new UserAssociatedAccount model. This should help significantly reduce the duplicated
logic across different auth providers.
* Add missing icons to set
* Revert FA5 revert
This reverts commit 42572ff
* use new SVG syntax in locales
* Noscript page changes (remove login button, center "powered by" footer text)
* Cast wider net for SVG icons in settings
- include any _icon setting for SVG registry (offers better support for plugin settings)
- let themes store multiple pipe-delimited icons in a setting
- also replaces broken onebox image icon with SVG reference in cooked post processor
* interpolate icons in locales
* Fix composer whisper icon alignment
* Add support for stacked icons
* SECURITY: enforce hostname to match discourse hostname
This ensures that the hostname rails uses for various helpers always matches
the Discourse hostname
* load SVG sprite with pre-initializers
* FIX: enable caching on SVG sprites
* PERF: use JSONP for SVG sprites so they are served from CDN
This avoids needing to deal with CORS for loading of the SVG
Note, added the svg- prefix to the filename so we can quickly tell in
dev tools what the file is
* Add missing SVG sprite JSONP script to CSP
* Upgrade to FA 5.5.0
* Add support for all FA4.7 icons
- adds complete frontend and backend for renamed FA4.7 icons
- improves performance of SvgSprite.bundle and SvgSprite.all_icons
* Fix group avatar flair preview
- adds an endpoint at /svg-sprites/search/:keyword
- adds frontend ajax call that pulls icon in avatar flair preview even when it is not in subset
* Remove FA 4.7 font files
* First take on subsetting svg icons
* FontAwesome 5 svg subset WIP
* Include icons from plugins/badges into svg sprite subset
* add svg icon support to themes
* Add spec for SvgSprite
* Misc. SVG icon fixes
* Use FA5 svgs in local-dates plugin
* CSS adjustments, fix SVG icons in group flair
* Use SVG icons in poll plugin
* Add SVG icons to /wizard
We regressed and optimized images no longer worked with svg
The following adds the correct logic to simply copy file for svgs
and bypasses resizing for svg avatars
previously we would ignore socket error, but this would mean that
there could be conditions where we would keep trying to download
gravatars forever (in an hourly job)
Also acquire a transaction per link instead of failing when
any of the links can't be processed.
This prevents ActiveRecord from rolling back the transaction
and the next SQL statement sent to PG will fail. This is
however hard to test as it only happens when there are
two competing process trying to process this method at the
same time.
* FEATURE: add branch option to remote theme import
* FIX: Add missing variable in params
* FIX: Add missing param for import_theme method
* SPEC: Add test methods for branch support in git import
* FIX: Add missing space to scss style
* Do not assume default branch as master
* Change branch field placeholder
* FIX: add missing div start tag
- By default, behaviour is not changed: tags are made lowercase upon creation and edit.
- If force_lowercase_tags is disabled, then mixed case tags are allowed.
- Tags must remain case-insensitively unique. This is enforced by ActiveRecord and Postgres.
- A migration is added to provide a `UNIQUE` index on `lower(name)`. Migration includes a safety to correct any current tags that do not meet the criteria.
- A `where_name` scope is added to `models/tag.rb`, to allow easy case-insensitive lookups. This is used instead of `Tag.where(name: "blah")`.
- URLs remain lowercase. Mixed case URLs are functional, but have the lowercase equivalent as the canonical.
If we detect redis is in readonly we can not correctly get a mutex
raise an exception to notify caller
When getting optimized images avoid the distributed mutex unless
for some reason it is the first call and we need to generate a thumb
In redis readonly no thumbnails will be generated
The intent from day one was to keep MAX_TO_KEEP stylesheets per target
however the DELETE statement did not perform target filtering
This meant we often deleted the wrong stylesheets from the cache
Previously we would raise a 500 error if a moderator tried to agree on a
flag another moderator deferred.
This can happen cause the UX for flags does not live refresh as flags
are handled
For a normal upload url
Before
```
Warming up --------------------------------------
264.000 i/100ms
Calculating -------------------------------------
2.754k (± 8.4%) i/s - 13.728k in 5.022066s
```
After
```
Warming up --------------------------------------
341.000 i/100ms
Calculating -------------------------------------
3.435k (±11.6%) i/s - 17.050k in 5.045676s
```
Use:
locale
locale_force_update
To force user locale on users where SiteSetting.allow_user_locale is enabled
Note: If an invalid locale is specified no action will occur
Previously we used width and height for thumbnails, new code ensures
1. We auto correct width and height
2. We added extra columns for thumbnail_width and height, this is determined
by actual upload and no longer passed in as a side effect
3. Optimized Image now stores filesize which can be used for analysis, decisions
Also
- fixes Android image manifest as a side effect
- fixes issue where a thumbnail generated that is smaller than the upload is no longer used
* FIX: don't allow inviting more than `max_allowed_message_recipients` setting allows
* add specs for guardian
* user preferences for auto track shouldn't be applicable to PMs (it auto watches on visit)
Execlude PMs from "Automatically track topics I enter..." and "When I post in a topic, set that topic to..." user preferences
* groups take only 1 slot in PM
* just return if topic is a PM
In the past the filename of the origin was used as the source
for the extension of the file when optimizing on upload.
We now use the actual calculated extension based on upload data.
At the moment core providers are hard-coded in Javascript, and plugin providers get added to the JS payload at compile time. This refactor means that we only ship enabled providers to the client.
* drafts in user profile: only show to user herself (not to admins), use avatar replying to (instead of topic OP), add keyboard shortcut for drafts, simplify display labels
* use JSON when testing Draft.stream
* add drafts.json endpoint, user profile tab with drafts stream
* improve drafts stream display in user profile
* truncate excerpts in drafts list, better handling for resume draft action
* improve draft stream SQL query, add rspec tests
* if composer is open, quietly close it when user opens another draft from drafts stream; load PM draft only when user is in /u/username/messages (instead of /u/username)
* cleanup
* linting fixes
* apply prettier styling to modified files
* add client tests for drafts, includes a fixture for drafts.json
* improvements to code following review
* refresh drafts route when user deletes a draft open in the composer while being in the drafts route; minor prettier scss fix
* added more spec tests, deleted an acceptance test for removing drafts that was too finicky, formatting and code style fixes, added appEvent for draft:destroyed
* prettier, eslint fixes
* use "username_lower" from users table, added error handling for rejected promises
* adds guardian spec for can_see_drafts, adds improvements following code review
* move DraftsController spec to its own file
* fix failing drafts qunit test, use getOwner instead of deprecated this.container
* limit test fixture for draft.json testing to new_topic request only
Checking `plugin.enabled?` while initializing plugins causes issues in two ways:
- An application restart is required for changes to take effect. A load-balanced multi-server environment could behave very weirdly if containers restart at different times.
- In a multisite environment, it takes the `enabled?` setting from the default site. Changes on that site affect all other sites in the cluster.
Instead, `plugin.enabled?` should be checked at runtime, in the context of a request. This commit removes `plugin.enabled?` from many `instance.rb` methods.
I have added a working `plugin.enabled?` implementation for methods that actually affect security/functionality:
- `post_custom_fields_whitelist`
- `whitelist_staff_user_custom_field`
- `add_permitted_post_create_param`
It was a dropdown to provide choices of color schemes,
and only one scheme could be shown.
With this commit, multiple color scheme previews can be displayed on
one page at the same time, making admins choose color schemes more
easily.
Theme preview windows are shrinked.
Imported default color schemes.
Co-Authored-By: Misaka 0x4e21 <misaka4e21@gmail.com>
- moderation tab
- sorting/pagination
- improved third party reports support
- trending charts
- better perf
- many fixes
- refactoring
- new reports
Co-Authored-By: Simon Cossar <scossar@users.noreply.github.com>
This was not picked up by tests because scheduled jobs are run immediately
and in the current thread (and therefore the current database transaction).
This particular case sometimes occurs inside multiple nested transactions,
so simply moving the offending line outside of the transaction is not enough.
Implemented TransactionHelper, which allows us to use `TransactionHelper.after_commit`
to define code to be run after the current transaction has been committed.
Use this event to filter the list of auto bumped topics.
EG:
on(:filter_auto_bump_topics) do |_category, filters|
filters.push(->(r) { r.where(<<~SQL)
NOT EXISTS(
SELECT 1 FROM topic_custom_fields
WHERE topic_id = topics.id
AND name = 'accepted_answer_post_id'
)
SQL
})
end
- We spread out bumping through the day, if you are bumping
4 topics then a topic will be bumped every 6 hours
- We add a small, bumping action at the bottom of the post to
denote a topic got bumped
### navigate_to_first_post_after_read setting for categories
When enabled on categories logged on users will return to OP after
reading the entire category. (useful for documentation categories)
### num_auto_bump_daily
Set a number of topics that will automatically bump daily on a category.
- Every 15 minutes we will check if any category has this setting
- Categories with the setting are shuffled
- We exclude pinned, closed, category description and archived topics
- Maximum of 1 topic for the list of categories is bumped till limit reached per category
- We always try to bump oldest first
- Limit is elastic using a RateLimiter that ensures that we only bump N per day
Also some minor organisation on category settings
Froze strings on category.rb
* Phase 0 for user-selectable theme components
- Drops `key` column from the `themes` table
- Drops `theme_key` column from the `user_options` table
- Adds `theme_ids` (array of ints default []) column to the `user_options` table and migrates data from `theme_key` to the new column.
- Removes the `default_theme_key` site setting and adds `default_theme_id` instead.
- Replaces `theme_key` cookie with a new one called `theme_ids`
- no longer need Theme.settings_for_client
* Add possibility to add hidden posts with PostCreator
* FEATURE: Create hidden posts for received spam emails
Spamchecker usually have 3 results: HAM, SPAM and PROBABLY_SPAM
SPAM gets usually directly rejected and needs no further handling.
HAM is good message and usually gets passed unmodified.
PROBABLY_SPAM gets an additional header to allow further processing.
This change addes processing capabilities for such headers and marks
new posts created as hidden when received via email.
I know that **Naming is CRITICAL** and that **Refactoring only NOT welcome**.
But since I spotted this (consistent) typo and the change does not affect any
functionality -- I checked the presence of "asscoiated" in the code base, I
guess the first rule trumps the second one.
It also gave me a false pretext to bypass my reluctance to use Google forms and
sign de CLA. Typos hurt the eye.
3 years is a very conservative limit that allows for a very wide buffer
for year-over-year analysis. The max is set to 5 years because that is
the policy listed for logging in hosted Discourse.
Introduce new patterns for direct sql that are safe and fast.
MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API
- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder
See more at: https://github.com/discourse/mini_sql
