Twitter removed OpenGraph tags from their pages. We can no longer
extract all the information (for example, the quoted tweet) we need
to render Oneboxes without using their API.
Dead and large images are replaced with a placeholder, either a broken
chain icon or a short text. This commit no longer applies this
transformation for images inside Oneboxes, but removes them instead.
This is a much better description of its function. It performs idempotent normalization of a URL. If consumers truly need to `encode` a URL (including double-encoding of existing encoded entities), they can use the existing `.encode` method.
normalized_encode in addressable has a number of issues, including https://github.com/sporkmonger/addressable/issues/472
To temporaily work around those issues for the majority of cases, we try parsing with `::URI`. If that fails (e.g. due to non-ascii characters) then we will fall back to addressable.
Hopefully we can simplify this back to `Addressable::URI.normalized_encode` in the future.
This commit also adds support for unicode domain names and emoji domain names with escape_uri.
This removes an unneeded hack checking for pre-signed urls, which are now handled by the general case due to starting off valid and only being minimally normalized. Previous test case continues to pass.
UrlHelper.s3_presigned_url? which was somewhat wide was removed.
Followup to d66115d918
* Makes sure the `actor_preferences` all initialize with an empty array instead of nil if there are no preferences e.g. the actor is not ignoring anyone
* If the actor has disabled all PMs make `actor_disallowing_pms?` always return true
* FIX: don't memoize site setting in guardian
Memoizing site settings can make tests more fragile and harder to debug
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
This commit introduces several fine-grained methods
to UserCommScreener which can be used to show the actor
who they are ignoring/muting/blocking DMs from in order
to prevent them initiating conversation with those users
or to display relevant information in the UI to the
actor.
This will be used in a companion PR in discourse-chat,
and is a follow up to 74584ff3ca
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Co-authored-by: Osama Sayegh <asooomaasoooma90@gmail.com>
* FEATURE: track stats around failing scheduled jobs
Discourse.job_exception_stats can now be used to gather stats around how
many regular scheduled jobs failed in the current process.
This will be consumed by the Prometheus plugin and potentially other
monitoring plugins.
* FEATURE: Add case-sensitivity flag to watched_words
Currently, all watched words are matched case-insensitively. This flag
allows a watched word to be flagged for case-sensitive matching.
To allow allow for backwards compatibility the flag is set to false by
default.
* FEATURE: Support case-sensitive creation of Watched Words via API
Extend admin creation and upload of Watched Words to support case
sensitive flag. This lays the ground work for supporting
case-insensitive matching of Watched Words.
Support for an extra column has also been introduced for the Watched
Words upload CSV file. The new column structure is as follows:
word,replacement,case_sentive
* FEATURE: Enable case-sensitive matching of Watched Words
WordWatcher's word_matcher_regexp now returns a list of regular
expressions instead of one case-insensitive regular expression.
With the ability to flag a Watched Word as case-sensitive, an action
can have words of both sensitivities.This makes the use of the global
Regexp::IGNORECASE flag added to all words problematic.
To get around platform limitations around the use of subexpression level
switches/flags, a list of regular expressions is returned instead, one for each
case sensitivity.
Word matching has also been updated to use this list of regular expressions
instead of one.
* FEATURE: Use case-sensitive regular expressions for Watched Words
Update Watched Words regular expressions matching and processing to handle
the extra metadata which comes along with the introduction of
case-sensitive Watched Words.
This allows case-sensitive Watched Words to matched as such.
* DEV: Simplify type casting of case-sensitive flag from uploads
Use builtin semantics instead of a custom method for converting
string case flags in uploaded Watched Words to boolean.
* UX: Add case-sensitivity details to Admin Watched Words UI
Update Watched Word form to include a toggle for case-sensitivity.
This also adds support for, case-sensitive testing and matching of Watched Word
in the admin UI.
* DEV: Code improvements from review feedback
- Extract watched word regex creation out to a utility function
- Make JS array presence check more explicit and readable
* DEV: Extract Watched Word regex creation to utility function
Clean-up work from review feedback. Reduce code duplication.
* DEV: Rename word_matcher_regexp to word_matcher_regexp_list
Since a list is returned now instead of a single regular expression,
change `word_matcher_regexp` to `word_matcher_regexp_list` to better communicate
this change.
* DEV: Incorporate WordWatcher updates from upstream
Resolve conflicts and ensure apply_to_text does not remove non-word characters in matches
that aren't at the beginning of the line.
Fixes edge case from fa5f3e228c.
In case the acting user is sent in with the target_user_ids,
we do not need to load those preferences, because even if the
acting user is preventing PMs or muting etc they need to always be able to
send themselves messages.
The "everyone" group is an automatic group and GroupUser records do not
exist for it. This commit allows all users if the group everyone is one
of the groups in the setting "pm_tags_allowed_for_groups".
* FIX: Rejected emails should not be cleaned up before their logs
If we delete the rejected emails before we delete their associated logs
we will receive 404 errors trying to inspect an email message for that
log.
* don't add a blank line
* test for max value as well
* pr cleanup and add migration
* Fix failing test
* FEATURE: revamped wizard
* UX: Wizard redesign (#17381)
* UX: Step 1-2
* swap out images
* UX: Finalize all steps
* UX: mobile
* UX: Fix test
* more test
* DEV: remove unneeded wizard components
* DEV: fix wizard tests
* DEV: update rails tests for new wizard
* Remove empty hbs files that were created because of rebase
* Fixes for rebase
* Fix wizard image link
* More rebase fixes
* Fix rails tests
* FIX: Update preview for new color schemes: (#17481)
* UX: make layout more responsive, update images
* fix typo
* DEV: move discourse logo svg to template only component
* DEV: formatting improvements
* Remove unneeded files
* Add tests for privacy step
* Fix banner image height for step "ready"
Co-authored-by: Jordan Vidrine <30537603+jordanvidrine@users.noreply.github.com>
Co-authored-by: awesomerobot <kris.aubuchon@discourse.org>
Currently when generating oneboxes if the connection timeouts and we’re
using the `FinalDestination#get` method, then it raises an exception.
We already catch this exception when using the
`FinalDestination#resolve` method so this patch just applies the same
logic to `FinalDestination#get`.
All other tests that are setting grade_period use either unitless `0`, `1.minute` or `5.minutes` so it wasn't clear if `5` was meant to be seconds (it was)
Our theme system is very complex and it can take a while to figure out how to invalidate the various types of caches that are used throughout the theme system. So, having a single helper method that invalidates everything can be useful in emergency situations where there is no time to read through the code and figure out how to clear the various caches.
Internal ticket: t64732.
When a topic was published from a shared draft and it had tags, the
users watching the tags were not notified. The problem was that the
topics are usually created in a secret category and publishing it just
moves an existent topic to the target category, without making any
changes to the tags.
Tag edit notifications are either created by PostActionNotifier or
PostRevisor. PostActionNotifier already checks if the site setting is
enabled. but PostRevisor scheduled a NotifyTagChange job without
checking disable_tags_edit_notifications.
This commit introduces a new plugin API to register
a group of stats that will be included in about.json
and also conditionally in the site about UI at /about.
The usage is like this:
```ruby
register_about_stat_group("chat_messages", show_in_ui: true) do
{
last_day: 1,
"7_days" => 10,
"30_days" => 100,
count: 1000,
previous_30_days: 120
}
end
```
In reality the stats will be generated any way the implementer
chooses within the plugin. The `last_day`, `7_days`, `30_days,` and `count`
keys must be present but apart from that additional stats may be added.
Only those core 4 stat keys will be shown in the UI, but everything will be shown
in about.json.
The stat group name is used to prefix the stats in about.json like so:
```json
"chat_messages_last_day": 2322,
"chat_messages_7_days": 2322,
"chat_messages_30_days": 2322,
"chat_messages_count": 2322,
```
The `show_in_ui` option (default false) is used to determine whether the
group of stats is shown on the site About page in the Site Statistics
table. Some stats may be needed purely for reporting purposes and thus
do not need to be shown in the UI to admins/users. An extension to the Site
serializer, `displayed_about_plugin_stat_groups`, has been added so this
can be inspected on the client-side.
It makes more sense to use user_ids for the UserCommScreener
introduced in fa5f3e228c since
in most cases the ID will be available, not the username. This
was discovered while starting work on a plugin that will
use this. In the cases where only usernames are available
the extra query is negligble.
The idea behind this refactor is to centralise all of the user ignoring / muting / disallow PM checks in a single place, so they can be used consistently in core as well as for plugins like chat, while improving the main bulk of the checks to run in a single fast non-AR query.
Also fixed up the invite error when someone is muting/ignoring the user that is trying to invite them to the topic.
Previously the spec was hardcoded to expect a certain compiled result. The precise result will change as we update to more recent Ember versions.
Instead, we can compile via the special theme compiler, and then compile our expected result via the normal compiler. If they match, then things are working as intended. This technique should be safe across template-compiler upgrades.
* FIX: min/max username length limits weren't validated
The custom validators introduced in e0d7cda made so we ignored the mix
and max values set on site_settings.yml. That change allowed admins to
set values outside of the range defined on the yaml file.
Related to https://meta.discourse.org/t/group-names-with-more-than-60-characters-broken/232115?u=falco
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Logging out failed when the current user was cached by an instance of `Auth::DefaultCurrentUserProvider` and `#log_off_user` was called on a different instance of that class.
Co-authored-by: Sam <sam.saffron@gmail.com>
This happened when a middleware accessed the `currentUser` before a controller had a chance to populate the `action_dispatch.request.path_parameters` env variable. In that case Discourse would always cache `nil` as `currentUser`.
Before, whispers were only available for staff members.
Config has been changed to allow to configure privileged groups with access to whispers. Post migration was added to move from the old setting into the new one.
I considered having a boolean column `whisperer` on user model similar to `admin/moderator` for performance reason. Finally, I decided to keep looking for groups as queries are only done for current user and didn't notice any N+1 queries.
Updates automatically data on the stats section of the topic.
It will update automatically the following information: likes, replies and last reply (timestamp and user)
This PR introduces a new hidden site setting that allows admins to display a splash screen while site assets load.
The splash screen can be enabled via the `splash_screen` hidden site setting.
This is what the splash screen currently looks like
5ceb72f085.mp4
Once site assets load, the splash screen is automatically removed.
To control the loading text that shows in the splash screen, you can change the preloader_text translation string in admin > customize > text
Future versions of redis will validate this port number causing the tests
relying on this to fail with:
```
Redis::CommandError:
ERR Invalid master port
```
Also change from an IPv4 address that might feasibly be in use to an IPv6
random ULA address that almost *certainly* won't be.
At some point in the past we decided to rename the 'regular' notification state of topics/categories to 'normal'. However, some UI copy was missed when the initial renaming was done so this commit changes the spots that were missed to the new name.
```
WARNING: Using `expect { }.not_to raise_error(SpecificErrorClass)` risks false positives, since literally any other error would cause the expectation to pass, including those raised by Ruby (e.g. `NoMethodError`, `NameError` and `ArgumentError`), meaning the code you are intending to test may not even get reached. Instead consider using `expect { }.not_to raise_error` or `expect { }.to raise_error(DifferentSpecificErrorClass)`. This message can be suppressed by setting: `RSpec::Expectations.configuration.on_potential_false_positives = :nothing`. Called from /var/www/discourse/spec/lib/retrieve_title_spec.rb:155:in `block (3 levels) in <main>'.
```
This doesn't cope well with gzipped, binary, or large responses. Ideally we would teach FinalDestination to safely retrieve and decode some of the response body. But for now, let's remove the broken implementation.
This reverts commit 94c3bbc2d1.
At this current point in time, we do not have enough data on whether
this centralisation is the trade-offs of coupling features into a single
channel.
When sending emails with delivery_method_options -> return_response
set to true, the SMTP sending code inside Mail will return the SMTP
response when calling deliver! for mail within the app. This commit
ensures that Email::Sender captures this response if it is returned
and stores it against the EmailLog created for the sent email.
A follow up PR will make this visible within the admin email UI.
* FIX: Fix a bug that is accessing the values in a hash wrongly and write tests
I decided to write tests in order to be confident in my refactor that's in the next commit.
Meanwhile I have discovered a potential bug. The `title_attr` key was accessed as a string,
but all the keys are actually symbols so it was never evaluated to be true.
irb(main):025:0> d = {key: 'value'}
=> {:key=>"value"}
irb(main):026:0> d['key']
=> nil
irb(main):027:0> d[:key]
=> "value"
* DEV: Extract methods for readability
I will be adding a new method following the conventions in place for adding a new normalizer. And this will make the readability of the `raw` block even more difficult; so I am extracting self contained private methods beforehand.
* FEATURE: Parse JSON-LD and introduce Movie object
JSON LD data is very easily transferable to Ruby objects because they contain types. If these types are mapped to Ruby objects, it is also better to make all the parsed data very explicit and easily extendable.
JSON-LD has many more standardized item types, with a full list here: https://schema.org/docs/full.html
However in order to decrease the scope, I only adapted the movie type.
* DEV: Change inheritance between normalizers
Normalizers are not supposed to have an inheritance relationships amongst each other. They are all normalizers, but all normalizing separate protocols. This is why I chose to extract a parent class and relieve Open Graph off that responsibility. Removing the parent class altogether could also a possibility, but I am keeping the scope limited to having a more accurate representation of the normalizers while making it easier to add a new one.
* Lint changes
* Bring back the Oembed OpenGraph inheritance
There is one test that caught that this inheritance was necessary. I still think modelling wise this inheritance shouldn't exist, but this can be tackled separately.
* Return empty hash if the json received is invalid
Before this change if there was a parsing error with JSON it would throw an exception. The goal of this commit is to rescue that exception and then log a warning. I chose to use Discourse's logger wrapper `warn_exception` to have the backtrace and not just used Rails logger. I considered raising an `InvalidParameters` error however if the JSON here is invalid it should not block showing of the Onebox, so logging is enough.
* Prep to support more JSONLD schema types with case
* Extract mustache template object created from JSONLD
This PR changes the rescue block to rescue only Net::TimeoutError exceptions and removes the log line to prevent clutter the logs with errors that are ignored. Other errors can bubble up because they're errors we probably want to know about
This table holds associations between uploads and other models. This can be used to prevent removing uploads that are still in use.
* DEV: Create upload_references
* DEV: Use UploadReference instead of PostUpload
* DEV: Use UploadReference for SiteSetting
* DEV: Use UploadReference for Badge
* DEV: Use UploadReference for Category
* DEV: Use UploadReference for CustomEmoji
* DEV: Use UploadReference for Group
* DEV: Use UploadReference for ThemeField
* DEV: Use UploadReference for ThemeSetting
* DEV: Use UploadReference for User
* DEV: Use UploadReference for UserAvatar
* DEV: Use UploadReference for UserExport
* DEV: Use UploadReference for UserProfile
* DEV: Add method to extract uploads from raw text
* DEV: Use UploadReference for Draft
* DEV: Use UploadReference for ReviewableQueuedPost
* DEV: Use UploadReference for UserProfile's bio_raw
* DEV: Do not copy user uploads to upload references
* DEV: Copy post uploads again after deploy
* DEV: Use created_at and updated_at from uploads table
* FIX: Check if upload site setting is empty
* DEV: Copy user uploads to upload references
* DEV: Make upload extraction less strict
Following the Rails 7 upgrade, the `DISCOURSE_SMTP_ENABLE_START_TLS`
setting doesn’t work anymore. This is because Rails upgraded the
`net-smtp` gem to the 0.3.1 version which enables `starttls` by default.
The `mail` gem doesn’t support this new behavior yet and doesn’t know
how to disable TLS. This should be fixed in an upcoming release.
Meanwhile applying this patch allows us to get back the previous
behavior which is expected by many.
* FIX: Email Send post has already been taken error
Adding a failing test first before coming up with a good solution.
Related: 357011eb3b
The above commit changed
```
PostReplyKey.find_or_create_by_safe!
```
to
```
PostReplyKey.create_or_find_by!
```
But I don't think it is working as a 1-1 replacement because of the
`Validation failed: Post has already been taken` error we are receiving
with this change. Also we need to make sure we don't re-introduce any
concurrency issues.
Reported: https://meta.discourse.org/t/224706/13
* Remove rails unique constraint and rely on db index
I believe this is what is causing `create_or_find_by!` to fail. Because
we have a unique constraint in the db I think we can remove this rails
unique constraint?
* clean up spec wording
Twitter does not allow SVGs to be used for twitter:image
metadata (see https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/markup)
so we should fall back to the site logo if the image option
provided to `crawlable_meta_data` or SiteSetting.site_twitter_summary_large_image_url
is an SVG, and do not add the meta tag for twitter:image at all
if the site logo is an SVG.
In 7328a2bfb0 we changed the
InlineOneboxer#onebox_for method to run the title of the
onebox through WatchedWord#censor_text. However, it is
allowable for the title to be nil, which was causing this
error in production:
> NoMethodError : undefined method gsub for nil:NilClass
We just need to check whether the title is nil before trying
to censor it.
Previously we hardcoded the DOWNLOAD_URL_EXPIRES_AFTER_SECONDS const
inside S3Helper to be 5 minutes (300 seconds). For various reasons,
some hosted sites may need this to be longer for other integrations.
The maximum expiry time for presigned URLs is 1 week (which is
604800 seconds), so that has been added as a validation on the
setting as well. The setting is hidden because 99% of the time
it should not be changed.
Censored watched words were not censored inside the title of an inline
oneboxes. Malicious users could exploit this behaviour to insert bad
words. The same issue has been fixed for regular Oneboxes in commit
d184fe59ca.
When searching for PMs or PMs in a group inbox, results in the header search were not being limited to 5 with a "More" link to the full page search. This PR fixes that.
It also simplifies the logic and updates the search API docs to include recently added `in:messages` and `group_messages:groupname` options.
When saving / creating bookmarks, we have code to save
the user's preference of bookmark_auto_delete_preference
to their user_options.
Unfortunately this can cause weirdness when plugins
have code using BookmarkManager to set the auto delete preference for
only a specific bookmark.
This commit introduces a save_user_preferences option (false
by default) so that this user preference is not saved unless
specified by the consumer of BookmarkManager, so plugins will
not have to worry about it.
Previously, with the default `editing_grace_period`, hotlinked images were pulled 5 minutes after a post is created. This delay was added to reduce the chance of automated edits clashing with user edits.
This commit refactors things so that we can pull hotlinked images immediately. URLs are immediately updated in the post's `cooked` HTML. The post's raw markdown is updated later, after the `editing_grace_period`.
This involves a number of behind-the-scenes changes including:
- Schedule Jobs::PullHotlinkedImages immediately after Jobs::ProcessPost. Move scheduling to after the `update_column` call to avoid race conditions
- Move raw changes into a separate job, which is delayed until after the ninja-edit window
- Move disable_if_low_on_disk_space logic into the `pull_hotlinked_images` job
- Move raw-parsing/replacing logic into `InlineUpload` so it can be easily be shared between `UpdateHotlinkedRaw` and `PullUserProfileHotlinkedImages`
Meta topic: https://meta.discourse.org/t/prevent-to-linkify-when-there-is-a-redirect/226964/2?u=osama.
This commit adds a new site setting `block_onebox_on_redirect` (default off) for blocking oneboxes (full and inline) of URLs that redirect. Note that an initial http → https redirect is still allowed if the redirect location is identical to the source (minus the scheme of course). For example, if a user includes a link to `http://example.com/page` and the link resolves to `https://example.com/page`, then the link will onebox (assuming it can be oneboxed) even if the setting is enabled. The reason for this is a user may type out a URL (i.e. the URL is short and memorizable) with http and since a lot of sites support TLS with http traffic automatically redirected to https, so we should still allow the URL to onebox.
This commit migrates all bookmarks to be polymorphic (using the
bookmarkable_id and bookmarkable_type) columns. It also deletes
all the old code guarded behind the use_polymorphic_bookmarks setting
and changes that setting to true for all sites and by default for
the sake of plugins.
No data is deleted in the migrations, the old post_id and for_topic
columns for bookmarks will be dropped later on.
Previously we were only applying the restriction to `a[href]` and `img[src]`. This commit ensures we apply the same logic to all allowlisted media src attributes.
22a7905f restructured how we load Ember CLI assets in production. Unfortunately, it also broke sourcemaps for those assets. This commit fixes that regression via a couple of changes:
- It adds the necessary `.map` paths to `config.assets.precompile`
- It swaps Sprockets' default `SourcemappingUrlProcessor` with an extended version which maintains relative URLs of maps
Currently the only way to allow tagging on pms is to use the `allow_staff_to_tag_pms` site setting. We are removing that site setting and replacing it with `pm_tags_allowed_for_groups` which will allow for non staff tagging. It will be group based permissions instead of requiring the user to be staff.
If the existing value of `allow_staff_to_tag_pms` is `true` then we include the `staff` groups as a default for `pm_tags_allowed_for_groups`.
We have not used anything related to bookmarks for PostAction
or UserAction records since 2020, bookmarks are their own thing
now. Deleting all this is just cleaning up old cruft.
Latest redis interoduces a block form of multi / pipelined, this was incorrectly
passed through and not namespaced.
Fix also updates logster, we held off on upgrading it due to missing functions
This fixes a corner case of the perf optimization in d4e35f5.
When you have the the same post showing in multiple tab/devices and like
said post in one place, we updated the like count but didn't flip the
`acted` bool in the front-end. This caused a small visual desync.
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
A bit of a mixed bag, this addresses several edge areas of bookmarks and makes them compatible with polymorphic bookmarks (hidden behind the `use_polymorphic_bookmarks` site setting). The main ones are:
* ExportUserArchive compatibility
* SyncTopicUserBookmarked job compatibility
* Sending different notifications for the bookmark reminders based on the bookmarkable type
* Import scripts compatibility
* BookmarkReminderNotificationHandler compatibility
This PR also refactors the `register_bookmarkable` API so it accepts a class descended from a `BaseBookmarkable` class instead. This was done because we kept having to add more and more lambdas/properties inline and it was very messy, so a factory pattern is cleaner. The classes can be tested independently as well.
Some later PRs will address some other areas like the discourse narrative bot, advanced search, reports, and the .ics endpoint for bookmarks.
Admins won't be able to disable strip_image_metadata if they don't
disable composer_media_optimization_image_enabled first since the later
will strip the same metadata on client during upload, making disabling
the former have no effect.
Bug report at https://meta.discourse.org/t/-/223350
