Commit Graph

3382 Commits

Author SHA1 Message Date
David Taylor
a3577435f7
FEATURE: Additional control of iframes in oneboxes (#10523)
This commit adds a new site setting "allowed_onebox_iframes". By default, all onebox iframes are allowed. When the list of domains is restricted, Onebox will automatically skip engines which require those domains, and use a fallback engine.
2020-08-27 20:12:13 +01:00
Guo Xiang Tan
40c6d90df3 PERF: Create a partial regular post_search_data index on large sites.
With the addition of `PostSearchData#private_message`, a partial
index consisting of only search data from regular posts can be created.
The partial index helps to speed up searches on large sites since PG
will not have to do an index scan on the entire search data index which
has shown to be a bottle neck.
2020-08-27 13:42:00 +08:00
Vinoth Kannan
618a7ecb35 FIX: default_tags_muted site setting won't have tag ids.
Instead it only have list of tag names separated by comma.
89fcb75af2
2020-08-26 23:05:29 +05:30
Faizaan Gagan
2100de449e Update spec/components/concern/has_custom_fields_spec.rb
Co-authored-by: Régis Hanol <regis@hanol.fr>
2020-08-25 09:52:18 -04:00
Faizaan Gagan
7e5290203f Update spec/components/concern/has_custom_fields_spec.rb
Co-authored-by: Régis Hanol <regis@hanol.fr>
2020-08-25 09:52:18 -04:00
fzngagan
c363189858 Accounted for the change while reading the fields added specs to confirm working 2020-08-25 09:52:18 -04:00
siriwatknp
80b92cf469 test: 💍 add test for thai tag 2020-08-25 16:12:26 +08:00
Guo Xiang Tan
05174df5c0
FIX: Restrict personal_messages: advanced search filter to admin.
The filter noops if an incorrect username is passed. This filter is not
exposed as part of the UI but is only used when an admin transitions
from a search within a user's personal messages to the full page search.

Follow-up to 4b30799054.
2020-08-24 13:53:48 +08:00
Guo Xiang Tan
4b30799054
FIX: Correct personal_messages:<username> advanced search filter.
Renamed from `private_messages` to `personal_messages` without
deprecation because the `private_messages` advanced search filter never
worked in the first place when it was implemented.
2020-08-24 11:54:30 +08:00
Guo Xiang Tan
106a2f58a2
DEV: Drop support for deprecated in:private search filter. 2020-08-21 17:18:39 +08:00
Guo Xiang Tan
ab5d738231
DEV: Improve search spec to test for actual posts.
Testing for count is a pitfall since a wrong post can be returned and
the tests will still pass.
2020-08-21 15:49:26 +08:00
Gerhard Schlager
11647b79f7 FIX: SiteSettings::LocalProcessProvider didn't work on multisite
It always used "test" as current site.
2020-08-20 11:15:20 +02:00
Vinoth Kannan
89fcb75af2 FIX: default_tags_muted setting should work for anonymous users too. 2020-08-20 10:40:03 +05:30
Blake Erickson
4395e4d165 FIX: Sending a PM through a flag on a deleted post
Because we allow all the other flag types on a deleted post we should be
able to send a pm to the user letting them know why we deleted their
post.

Bug report:

https://meta.discourse.org/t/-/161156
2020-08-19 17:36:52 -06:00
Vinoth Kannan
8348a41124
FEATURE: add regular_categories field in site setting & user option. (#10477)
Like "default watching" and "default tracking" categories option now the "regular" categories support is added. It will be useful for sites that are muted by default. The user option will be displayed only if `mute_all_categories_by_default` site setting is enabled.
2020-08-20 00:35:04 +05:30
Blake Erickson
ea2e58e622
DEV: Bump rotp gem to latest version (#10472)
The rotp gem is currently pinned to version 5.1.0 and this will bump it
up to version 6.0.1.

Follow up to: 85d4370f79

because this issue we were waiting on is now closed:

https://github.com/mdp/rotp/issues/98

Because version 6 is now encoding the params I needed to update the
tests as well.
2020-08-19 09:16:33 -06:00
Penar Musaraj
882b0aac19
DEV: Let themes extend color definitions (#10429)
Themes can now declare custom colors that get compiled in core's color definitions stylesheet, thus allowing themes to better support dark/light color schemes. 

For example, if you need your theme to use tertiary for an element in a light color scheme and quaternary in a dark scheme, you can add the following SCSS to your theme's `color_definitions.scss` file: 

```
:root {
  --mytheme-tertiary-or-quaternary: #{dark-light-choose($tertiary, $quaternary)};
}
```

And then use the `--mytheme-tertiary-or-quaternary` variable as the color property of that element. You can also use this file to add color variables that use SCSS color transformation functions (lighten, darken, saturate, etc.) without compromising your theme's compatibility with different color schemes.
2020-08-18 13:02:13 -04:00
jahan-ggn
65649eaef0
User card settings (#10302)
* settings implemented

* prettier

* settings updated

* rubocop

* prettier

* Revert "rubocop"

This reverts commit 7805145a7d.

* Revert "prettier"

This reverts commit 2c53f4fa12.

* settings updated and changed

* rubocop

* changes applied

* final changes done

* Server side feature added

* spec changed

* changed user_updater and profile file

* Fix user card specs

* web hook serializer solved

* site-setting changed

Co-authored-by: Mark VanLandingham <markvanlan@gmail.com>
2020-08-17 12:37:45 -04:00
David Taylor
0a5376084d
FIX: Ensure auto close notice is posted with system locale
Previously it was created with the locale of the user who created the final post in the topic
2020-08-17 15:40:47 +01:00
Guo Xiang Tan
c2605d4194
DEV: Fix search to be more intentional about what it is testing.
Asserting for the posts length is not accurate since an incorrect post
can be returned and the assertion will still pass.
2020-08-14 15:32:18 +08:00
Martin Brennan
ffb31b8d2b
FIX: Do not require tagging to be enabled for IMAP archive and delete (#10426)
Previously we did an early return if either SiteSetting.tagging_enabled or SiteSetting.allow_staff_to_tag_pms was false when updating the email on the IMAP server -- however this also stopped us from archiving or deleting emails if either of these were disabled.
2020-08-13 14:04:40 +10:00
David Taylor
bd0a7553c4
DEV: Detect when s3 inventory failure is caused by etag difference (#10427) 2020-08-13 09:30:28 +10:00
Penar Musaraj
6dd9f2eca2
FIX: color scheme selection with non-default theme
This fixes an issue where a non-default theme set to use the base color
scheme (i.e. the theme had an empty `color_scheme_id`) was loading the
default theme's color scheme instead.
2020-08-12 08:49:13 -04:00
Guo Xiang Tan
603a4d1794
DEV: Fix randomly failing spec. 2020-08-12 16:34:56 +08:00
Guo Xiang Tan
93f8396b4b
FIX: Limit PG headline based search blurb generation to 200 characters.
* Recovers omission characters '...' in blurb as well.
2020-08-12 15:34:27 +08:00
Martin Brennan
95b71b35d6
FEATURE: IMAP delete email sync for group inboxes (#10392)
Adds functionality to reflect topic delete in Discourse to IMAP inbox (Gmail only for now) and reflecting Gmail deletes in Discourse.

Adding lots of tests, various refactors and code improvements.

When Discourse topic is destroyed in PostDestroyer mark the topic incoming email as imap_sync: true, and do the opposite when post is recovered.
2020-08-12 10:16:26 +10:00
Penar Musaraj
c05aced094
FIX: Invalidate cache when updating color scheme colors (#10417) 2020-08-11 16:28:59 -04:00
Martin Brennan
b950b3fb3f
DEV: Add verified to uploads and fill in S3 inventory (#10406)
When we run the S3 inventory, mark uploads that exist as verified true, those that don't as verified false, and uploads not included in the check / not yet checked as verified nil.
2020-08-11 14:43:51 +10:00
Blake Erickson
2032c11f78
FIX: Return 422 when creating topics with tags w/out permission (#10400)
The UI prevents users from trying to create tags on topics when they
don't have permission, but if you are trying to add tags to a topic via
the API and you don't have permission before this change it would
silently succeed in creating the topic, but it wouldn't have any tags.

Now a 422 error will be returned with an error message when trying to
create a topic with tags when tagging is disabled or you don't have
enough trust level to add tags to a topic.

Bug report: https://meta.discourse.org/t/-/70525/14
2020-08-10 16:14:15 -06:00
David Taylor
4476680d4b
UX: Set silence_reason using the system locale
Previously we would use the user's locale, which can be confusing for moderators.
2020-08-10 18:51:54 +01:00
Penar Musaraj
f179510a68
FIX: include both name and id in color scheme stylesheet filename slugs (#10397) 2020-08-07 13:43:45 -04:00
jbrw
3593e582a3
FIX - limit number of embedded media items in a post (#10391)
* FIX - limit number of embedded media items in a post

* Add renamed settings to DeprecatedSettings
2020-08-07 12:08:59 -04:00
Guo Xiang Tan
053cbe3112
PERF: Limit characters used to generate headline for search blurb.
We determined using the following benchmark script that limiting to 2500 chars would mean a maximum of
25ms spent generating headlines.

```
require 'benchmark/ips'

string = <<~STRING
Far far away, behind the word mountains...
STRING

def sql_excerpt(string, l = 1000000)
  DB.query_single(<<~SQL)
  SELECT TS_HEADLINE('english', left('#{string}', #{l}), PLAINTO_TSQUERY('mountains'))
  SQL
end

def ruby_excerpt(string)
  output = DB.query_single("SELECT '#{string}'")[0]
  Search::GroupedSearchResults::TextHelper.excerpt(output, 'mountains', radius: 100)
end

puts "Ruby Excerpt: #{ruby_excerpt(string)}"
puts "SQL Excerpt: #{sql_excerpt(string)}"
puts

Benchmark.ips do |x|
  x.time = 10

  [1000, 2500, 5000, 10000, 20000, 50000].each do |l|
    short_string = string[0..l]

    x.report("ts_headline excerpt #{l}") do
      sql_excerpt(short_string, l)
    end

    x.report("actionview excerpt #{l}") do
      ruby_excerpt(short_string)
    end
  end

  x.compare!
end
```

```
actionview excerpt 1000:    20570.7 i/s
actionview excerpt 2500:    17863.1 i/s - 1.15x  (± 0.00) slower
actionview excerpt 5000:    14228.9 i/s - 1.45x  (± 0.00) slower
actionview excerpt 10000:    10906.2 i/s - 1.89x  (± 0.00) slower
actionview excerpt 20000:     6255.0 i/s - 3.29x  (± 0.00) slower
ts_headline excerpt 1000:     4337.5 i/s - 4.74x  (± 0.00) slower
actionview excerpt 50000:     3222.7 i/s - 6.38x  (± 0.00) slower
ts_headline excerpt 2500:     2240.4 i/s - 9.18x  (± 0.00) slower
ts_headline excerpt 5000:     1258.7 i/s - 16.34x  (± 0.00) slower
ts_headline excerpt 10000:      667.2 i/s - 30.83x  (± 0.00) slower
ts_headline excerpt 20000:      348.7 i/s - 58.98x  (± 0.00) slower
ts_headline excerpt 50000:      131.9 i/s - 155.91x  (± 0.00) slower
```
2020-08-07 14:36:52 +08:00
Guo Xiang Tan
e60c74d3c1
FEATURE: Use PG ts_headline for highlighting topic title in search. 2020-08-07 12:43:09 +08:00
Krzysztof Kotlarek
12a00d6dc5
FEATURE: add advanced order to search (#10385)
Similar to `advanced_filter` I introduced `advanced_order`.

I needed a new option because default orders are evaluated after advanced_filter so I couldn't use it.

Also, that part is a little bit more generic
```
elsif word =~ /order:\w+/
  @order = word.gsub('order:', '').to_sym
nil
```

After those changes, I can use them in plugins in this way:
```
Search.advanced_order(:votes) do |posts|
  posts.reorder("COALESCE((SELECT dvvc.counter FROM discourse_voting_vote_counters dvvc WHERE dvvc.topic_id = subquery.topic_id), 0) DESC")
end
```
2020-08-07 12:47:00 +10:00
Penar Musaraj
87e2c9de24
DEV: Plugins can extend color definitions (#10383) 2020-08-06 09:46:17 -04:00
Penar Musaraj
6fdc711b4a
FEATURE: Allow users to opt out of automatic dark mode (#10377) 2020-08-06 09:45:37 -04:00
Guo Xiang Tan
2193d02433
PERF: Use PG headlines for blurb generation and highlighting for search. 2020-08-06 14:56:29 +08:00
David Taylor
cb12a721c4
REFACTOR: Refactor pull_hotlinked_images job
This commit should cause no functional change
- Split into functions to avoid deep nesting
- Register custom field type, and remove manual json parse/serialize
- Recover from deleted upload records

Also adds a test to ensure pull_hotlinked_images redownloads secure images only once
2020-08-05 12:14:59 +01:00
Kane York
13feb300a8
FIX: Topic map was incorrectly counting assign actions (#10360)
The assign plugin is one of two situations where a post can be both a whisper and a small-action. Check the action_code field to filter out small-actions.
2020-08-05 11:51:28 +10:00
Penar Musaraj
266c0c50d7
FIX: Load base color scheme when default theme is not set
Followup to c937afc
2020-08-04 12:15:07 -04:00
Martin Brennan
5a3494b1e1
FIX: IMAP archive fix and group list mailbox code unification (#10355)
* Fixed an issue I introduced in the last PR where I am just archiving everything regardless of whether it is actually archived in Discourse man_facepalming
* Refactor group list_mailboxes IMAP code to use providers, add specs, and add provider code to get the correct prodivder
2020-08-04 14:19:57 +10:00
Penar Musaraj
c937afc75e
FEATURE: automatic dark mode (#10341)
A first step to adding automatic dark mode color scheme switching. Adds a new SCSS file at `color_definitions.scss` that serves to output all SCSS color variables as CSS custom properties. And replaces all SCSS color variables with the new CSS custom properties throughout the stylesheets. 

This is an alpha feature at this point, can only be enabled via console using the `default_dark_mode_color_scheme_id` site setting.
2020-08-03 22:57:10 -04:00
Guo Xiang Tan
105d560177
SECURITY: 413 for GET, HEAD or DELETE requests with payload. 2020-08-03 14:21:33 +08:00
Guo Xiang Tan
32af607b70
DEV: Refactor anonymouse cache spec.
Mainly to properly categorize `Middleware::AnonymousCache` vs `Middleware::AnonymousCache::Helper` specs.
2020-08-03 14:17:11 +08:00
Martin Brennan
2920988b3a
FIX: IMAP sync email update uniqueness across groups and minor improvements (#10332)
Adds a imap_group_id column to IncomingEmail to deal with an issue where we were trying to update emails in the mailbox, calling IncomingEmail.where(imap_sync: true). However UID and UIDVALIDITY could be the same across accounts. So if group A used IMAP details for Gmail account A, and group B used IMAP details for Gmail account B, and both tried to sync changes to an email with UID of 3 (e.g. changing Labels), one account could affect the other. This even applied to Archiving!

Also in this PR:

* Fix error occurring if we do a uid_fetch and no emails are returned
* Allow for creating labels within the target mailbox (previously we would not do this, only use existing labels)
* Improve consistency for log messages
* Add specs for generic IMAP provider (Gmail specs still to come)
* Add custom archiving support for Gmail
* Only use Message-ID for uniqueness of IncomingEmail if it was generated by us
* Various refactors and improvements
2020-08-03 13:10:17 +10:00
David Taylor
16c65a94f7
PERF: Preload S3 inventory data for multisite clusters 2020-07-29 10:31:55 +01:00
Martin Brennan
9e5b213089
FIX: Ensure topic user bookmarked synced on bookmark auto-delete (#10323)
For the following conditions, the TopicUser.bookmarked column was not updated correctly:

* When a bookmark was auto-deleted because the reminder was sent
* When a bookmark was auto-deleted because the owner of the bookmark replied to the topic

This adds another migration to fix the out-of-sync column and also some refactors to BookmarkManager to allow for more of these delete cases. BookmarkManager is used instead of directly destroying the bookmark in PostCreator and BookmarkReminderNotificationHandler.
2020-07-29 09:43:32 +10:00
Martin Brennan
2e5b2d20ba
FIX: Resolve issue where deleted spam topics marked as Not Spam were not being recovered (#10322)
If a user posted a topic and Akismet decided it was spam, the topic gets deleted and put into the review queue. If a category moderator for that category marked the post/topic as "Not Spam" the topic did not get recovered correctly because Guardian.new(@user).can_review_topic?(@post.topic) returned false incorrectly because the topic was deleted.
2020-07-28 12:06:15 +10:00
Robin Ward
25f6136b27 Upgrade fastimage and remove our freedom patch 2020-07-27 13:23:17 -04:00
Guo Xiang Tan
309e41d341
DEV: Add spec for searching for whisper posts. 2020-07-27 11:56:08 +08:00
Krzysztof Kotlarek
e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Guo Xiang Tan
c6202af005
Update rubocop to 2.3.1. 2020-07-24 17:19:21 +08:00
Guo Xiang Tan
b1cc7825c5
DEV: Remove unused fabrication. 2020-07-24 09:54:34 +08:00
jbrw
2aec92d0b4
FEATURE - allow Group Moderators to edit category description (#10292)
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2020-07-23 09:50:00 -04:00
Guo Xiang Tan
609ba50fe8
DEV: Add more granularity to SearchIndexer versions.
Sometimes, we just want to reindex a specific model and not all the
things.
2020-07-23 14:24:06 +08:00
Sam Saffron
88dcdf776b
FEATURE: add tracked filter to topic lists
This adds a special filter to topic lists that will filter to tracked and
watched categories.

To use it you can visit:

`https://sitename/?filter=tracked`
`https://sitename/unread?filter=tracked`

and so on

Note, we do not include explicitly tracked and watched topics **outside** of
the tracked categories and tags.

We can consider a `filter=all_tracked` to cover this edge case.
2020-07-23 10:30:23 +10:00
David Taylor
c09b5807f3
FIX: Include resolved locale in anonymous cache key (#10289)
This only applies when set_locale_from_accept_language_header is enabled
2020-07-22 18:00:07 +01:00
Roman Rizzi
9fc7bd5797
FIX: Reviews that are auto-hidden by a trusted spam flagger should always have enough weight. (#10284) 2020-07-22 11:42:15 -03:00
Martin Brennan
93a8e34f47
FIX: Allow user to recover/delete post if they can review the topic (#10273)
To reproduce the initial issue here:

1. A user makes a post, which discourse-akismet marks as spam (I cheated and called `DiscourseAkismet::PostsBouncer.new.send(:mark_as_spam, post)` for this)
2. The post lands in the review queue
3. The category the topic is in has a `reviewable_by_group_id`
4. A user in that group goes and looks at the Review queue, decides the post is not spam, and clicks Not Spam
5. Weird stuff happens because the `PostDestroyer#recover` method didn't handle this (the user who clicked Not Spam was not the owner of the post and was not a staff member, so the post didn't get un-destroyed and post counts didn't get updated)

Now users who belong to a group who can review a category now have the ability to recover/delete posts fully.
2020-07-22 11:57:16 +10:00
David Taylor
ec4024fe6d
FIX: Keep by_users check in S3 inventory
Partial revert of 8515d8fa - the by_users check is ensuring we don't raise errors for fixtures
2020-07-21 17:19:56 +01:00
Vinoth Kannan
ef37460c93 FIX: delete synonyms in topics if target tag is already added.
Currently, while adding a synonym tag if both target and synonym tags are already available in a topic then it's returning an error.
2020-07-21 21:02:01 +05:30
Joffrey JAFFEUX
936a40adcf
linting (#10276) 2020-07-21 08:39:14 +02:00
Guo Xiang Tan
94fced2133
FIX: Handle PG readonly mode in Auth::DefaultCurrentUserProvider.
Avoid writing to the DB when PG is in readonly mode.
2020-07-21 13:44:05 +08:00
Martin Brennan
41b43a2a25
FEATURE: Add "delete on owner reply" bookmark functionality (#10231)
This adds an option to "delete on owner reply" to bookmarks. If you select this option in the modal, then reply to the topic the bookmark is in, the bookmark will be deleted on reply.

This PR also changes the checkboxes for these additional bookmark options to an Integer column in the DB with a combobox to select the option you want.

The use cases are:

* Sometimes I will bookmark the topics to read it later. In this case we definitely don’t need to keep the bookmark after I replied to it.
* Sometimes I will read the topic in mobile and I will prefer to reply in PC later. Or I may have to do some research before reply. So I will bookmark it for reply later.
2020-07-21 10:00:39 +10:00
Blake Erickson
690f17bcbe
FEATURE: Allow List for PMs (#10270)
* FEATURE: Allow List for PMs

This feature adds a new user setting that is disabled by default that
allows them to specify a list of users that are allowed to send them
private messages. This way they don't have to maintain a large list of
users they don't want to here from and instead just list the people they
know they do want. Staff will still always be able to send messages to
the user.

* Update PR based on feedback
2020-07-20 15:23:49 -06:00
jbrw
7ab5658462
FEATURE: Allow group moderators to add/remove staff notes (#10252)
* FEATURE: Allow group moderators to add/remove staff notes
2020-07-20 15:53:47 -04:00
David Taylor
5f3dfce4eb
FIX: Listing topics with muted mixed-case tags (#10268)
When visiting a tag page directly, we should display all topics, even if that tag is muted. This was not working for mixed-case tags.
2020-07-20 11:01:29 +01:00
Daniel Waterworth
9313706649 DEV: Enable preserve_email_structure_when_styling by default
In 1bd8a075, a hidden site setting was added that causes Email::Styles
to treat its input as a complete document in all cases.

This commit enables that setting by default.

Some tests were removed that were broken by this change. They tested the
behaviour of applying email styles to empty strings. They weren't useful
because:

 * Sending empty email is not something we ever intend to do,
 * They were testing incidental behaviour - there are lots of
   valid ways to process the empty string,
 * Their intent wasn't clear from their descriptions,
2020-07-20 10:21:32 +01:00
Robin Ward
8e3f667d7c FIX: Show background images for both slug formats
It seems there was a discrepancy in that background images were attached
to the full slug category class: `category-:slug-:id` and our body class
only had `category-:slug`.

This fix adds support for both formats.
2020-07-17 13:42:30 -04:00
David Taylor
fab8b8649e
PERF: Combine avatar_lookup and primary_group_lookup into user_lookup (#10253)
These two classes were running very similar queries, which could be expensive on large topics
2020-07-17 10:48:08 +01:00
Guo Xiang Tan
ff7678e210
FIX: Reindex posts when Topic#title or Category#name changes. 2020-07-17 11:12:31 +08:00
Mark VanLandingham
62d5a9690f
FIX: Remove user_deleted when staff recovers post (#10245) 2020-07-16 09:15:01 -05:00
Vinoth Kannan
3252cb847c FIX: : trigger user_updated event only if email changed after user creation.
Follow-up to 1460d7957c
2020-07-16 18:21:30 +05:30
Guo Xiang Tan
af87911178
FIX: in:title search should only search through topic first posts. 2020-07-16 12:21:19 +08:00
Guo Xiang Tan
8ceb7f490f
DEV: Ignore order of categories in search spec. 2020-07-16 09:29:23 +08:00
Guo Xiang Tan
6385fbbfbf
FIX: Ignore document length in search when ranking by relevance.
Considering document length in search introduced too much variance in
our search results such that it makes certain searches better but at the
same time made certain searches worst. Instead, we want to have a more
determistic way of ranking search so that it is easier to reason about
why a post is rank higher in search than another.

The long term plan to tackle repeated terms is to restrict the number of
positions for a given lexeme in our search index.
2020-07-15 13:43:14 +08:00
Guo Xiang Tan
5bf0a0893b
FIX: Search by relevance may return incorrect post number.
Follow up to d8c796bc4.

Note that his change increases query time by around 40% in the following
benchmark against `dev.discourse.org` but this is a tradeoff that has to be taken so that relevance
search is accurate.

```
require 'benchmark/ips'

Benchmark.ips do |x|
  x.config(time: 10, warmup: 2)

  x.report("current aggregate search query") do
    DB.exec <<~SQL
    SELECT "posts"."id", "posts"."user_id", "posts"."topic_id", "posts"."post_number", "posts"."raw", "posts"."cooked", "posts"."created_at", "posts"."updated_at", "posts"."reply_to_post_number", "posts"."reply_count", "posts"."quote_count", "posts"."deleted_at", "posts"."off_topic_count", "posts"."like_count", "posts"."incoming_link_count", "posts"."bookmark_count", "posts"."score", "posts"."reads", "posts"."post_type", "posts"."sort_order", "posts"."last_editor_id", "posts"."hidden", "posts"."hidden_reason_id", "posts"."notify_moderators_count", "posts"."spam_count", "posts"."illegal_count", "posts"."inappropriate_count", "posts"."last_version_at", "posts"."user_deleted", "posts"."reply_to_user_id", "posts"."percent_rank", "posts"."notify_user_count", "posts"."like_score", "posts"."deleted_by_id", "posts"."edit_reason", "posts"."word_count", "posts"."version", "posts"."cook_method", "posts"."wiki", "posts"."baked_at", "posts"."baked_version", "posts"."hidden_at", "posts"."self_edits", "posts"."reply_quoted", "posts"."via_email", "posts"."raw_email", "posts"."public_version", "posts"."action_code", "posts"."locked_by_id", "posts"."image_upload_id" FROM "posts" JOIN (SELECT *, row_number() over() row_number FROM (SELECT topics.id, min(posts.post_number) post_number FROM "posts" INNER JOIN "post_search_data" ON "post_search_data"."post_id" = "posts"."id" INNER JOIN "topics" ON "topics"."id" = "posts"."topic_id" AND ("topics"."deleted_at" IS NULL) LEFT JOIN categories ON categories.id = topics.category_id WHERE ("posts"."deleted_at" IS NULL) AND "posts"."post_type" IN (1, 2, 3, 4) AND (topics.visible) AND (topics.archetype <> 'private_message') AND (post_search_data.search_data @@ TO_TSQUERY('english', '''postgres'':*ABCD')) AND (categories.id NOT IN (
      SELECT categories.id WHERE categories.search_priority = 1
    )
    ) AND ((categories.id IS NULL) OR (NOT categories.read_restricted)) GROUP BY topics.id ORDER BY MAX((
      TS_RANK_CD(
        post_search_data.search_data,
        TO_TSQUERY('english', '''postgres'':*ABCD'),
        1|32
      ) *
      (
        CASE categories.search_priority
        WHEN 2
        THEN 0.6
        WHEN 3
        THEN 0.8
        WHEN 4
        THEN 1.2
        WHEN 5
        THEN 1.4
        ELSE
          CASE WHEN topics.closed
          THEN 0.9
          ELSE 1
          END
        END
      )
    )
    ) DESC, topics.bumped_at DESC LIMIT 51 OFFSET 0) xxx) x ON x.id = posts.topic_id AND x.post_number = posts.post_number WHERE ("posts"."deleted_at" IS NULL) ORDER BY row_number;
    SQL
  end

  x.report("current aggregate search query with proper ranking") do
    DB.exec <<~SQL
    SELECT "posts"."id", "posts"."user_id", "posts"."topic_id", "posts"."post_number", "posts"."raw", "posts"."cooked", "posts"."created_at", "posts"."updated_at", "posts"."reply_to_post_number", "posts"."reply_count", "posts"."quote_count", "posts"."deleted_at", "posts"."off_topic_count", "posts"."like_count", "posts"."incoming_link_count", "posts"."bookmark_count", "posts"."score", "posts"."reads", "posts"."post_type", "posts"."sort_order", "posts"."last_editor_id", "posts"."hidden", "posts"."hidden_reason_id", "posts"."notify_moderators_count", "posts"."spam_count", "posts"."illegal_count", "posts"."inappropriate_count", "posts"."last_version_at", "posts"."user_deleted", "posts"."reply_to_user_id", "posts"."percent_rank", "posts"."notify_user_count", "posts"."like_score", "posts"."deleted_by_id", "posts"."edit_reason", "posts"."word_count", "posts"."version", "posts"."cook_method", "posts"."wiki", "posts"."baked_at", "posts"."baked_version", "posts"."hidden_at", "posts"."self_edits", "posts"."reply_quoted", "posts"."via_email", "posts"."raw_email", "posts"."public_version", "posts"."action_code", "posts"."locked_by_id", "posts"."image_upload_id" FROM "posts" JOIN (SELECT *, row_number() over() row_number FROM (SELECT subquery.topic_id id, (ARRAY_AGG(subquery.post_number ORDER BY rank DESC, bumped_at DESC))[1] post_number, MAX(subquery.rank) rank, MAX(subquery.bumped_at) bumped_at FROM (SELECT "posts"."id", "posts"."user_id", "posts"."topic_id", "posts"."post_number", "posts"."raw", "posts"."cooked", "posts"."created_at", "posts"."updated_at", "posts"."reply_to_post_number", "posts"."reply_count", "posts"."quote_count", "posts"."deleted_at", "posts"."off_topic_count", "posts"."like_count", "posts"."incoming_link_count", "posts"."bookmark_count", "posts"."score", "posts"."reads", "posts"."post_type", "posts"."sort_order", "posts"."last_editor_id", "posts"."hidden", "posts"."hidden_reason_id", "posts"."notify_moderators_count", "posts"."spam_count", "posts"."illegal_count", "posts"."inappropriate_count", "posts"."last_version_at", "posts"."user_deleted", "posts"."reply_to_user_id", "posts"."percent_rank", "posts"."notify_user_count", "posts"."like_score", "posts"."deleted_by_id", "posts"."edit_reason", "posts"."word_count", "posts"."version", "posts"."cook_method", "posts"."wiki", "posts"."baked_at", "posts"."baked_version", "posts"."hidden_at", "posts"."self_edits", "posts"."reply_quoted", "posts"."via_email", "posts"."raw_email", "posts"."public_version", "posts"."action_code", "posts"."locked_by_id", "posts"."image_upload_id", (
      TS_RANK_CD(
        post_search_data.search_data,
        TO_TSQUERY('english', '''postgres'':*ABCD'),
        1|32
      ) *
      (
        CASE categories.search_priority
        WHEN 2
        THEN 0.6
        WHEN 3
        THEN 0.8
        WHEN 4
        THEN 1.2
        WHEN 5
        THEN 1.4
        ELSE
          CASE WHEN topics.closed
          THEN 0.9
          ELSE 1
          END
        END
      )
    )
     rank, topics.bumped_at bumped_at FROM "posts" INNER JOIN "post_search_data" ON "post_search_data"."post_id" = "posts"."id" INNER JOIN "topics" ON "topics"."id" = "posts"."topic_id" AND ("topics"."deleted_at" IS NULL) LEFT JOIN categories ON categories.id = topics.category_id WHERE ("posts"."deleted_at" IS NULL) AND "posts"."post_type" IN (1, 2, 3, 4) AND (topics.visible) AND (topics.archetype <> 'private_message') AND (post_search_data.search_data @@ TO_TSQUERY('english', '''postgres'':*ABCD')) AND (categories.id NOT IN (
      SELECT categories.id WHERE categories.search_priority = 1
    )
    ) AND ((categories.id IS NULL) OR (NOT categories.read_restricted))) subquery GROUP BY subquery.topic_id ORDER BY rank DESC, bumped_at DESC LIMIT 51 OFFSET 0) xxx) x ON x.id = posts.topic_id AND x.post_number = posts.post_number WHERE ("posts"."deleted_at" IS NULL) ORDER BY row_number;
    SQL
  end

  x.compare!
end
```

```
Warming up --------------------------------------
current aggregate search query
                         1.000  i/100ms
current aggregate search query with proper ranking
                         1.000  i/100ms
Calculating -------------------------------------
current aggregate search query
                         18.040  (± 0.0%) i/s -    181.000  in  10.035241s
current aggregate search query with proper ranking
                         12.992  (± 0.0%) i/s -    130.000  in  10.007214s

Comparison:
current aggregate search query:       18.0 i/s
current aggregate search query with proper ranking:       13.0 i/s - 1.39x  (± 0.00) slower
```
2020-07-15 11:45:56 +08:00
jbrw
06073fe8c6
FEATURE: Allow group moderators to close/archive topics
* FEATURE: Allow group moderators to close/archive topics
2020-07-14 12:36:19 -04:00
Guo Xiang Tan
94a2a70462
DEV: Use a longer TTL for pg readonly mode. 2020-07-14 16:15:58 +08:00
Guo Xiang Tan
5c230266d3
FIX: Inject extra lexemes for host lexeme.
```
discourse_development=# SELECT alias, lexemes FROM TS_DEBUG('www.discourse.org');
 alias |       lexemes
-------+---------------------
 host  | {www.discourse.org}

discourse_development=# SELECT TO_TSVECTOR('www.discourse.org');
      to_tsvector
-----------------------
 'www.discourse.org':1
```

Given the above lexeme, we will inject additional lexeme by splitting
the host on `.`. The actual tsvector stored will look something like

```
               tsvector
---------------------------------------
 'discourse':1 'discourse.org':1 'org':1 'www':1 'www.discourse.org':1
```
2020-07-14 15:32:40 +08:00
Guo Xiang Tan
5c31216aea
FIX: Search for whole URLs wasn't working. 2020-07-14 15:31:48 +08:00
Guo Xiang Tan
d8c796bc44
FIX: Ensure that aggregating search shows the post with the higest rank.
Previously, we would only take either the `MIN` or `MAX` for
`post_number` during aggregation meaning that the ranking is not
considered.

```
require 'benchmark/ips'

Benchmark.ips do |x|
  x.config(time: 10, warmup: 2)

  x.report("current aggregate search query") do
    DB.exec <<~SQL
    SELECT "posts"."id", "posts"."user_id", "posts"."topic_id", "posts"."post_number", "posts"."raw", "posts"."cooked", "posts"."created_at", "posts"."updated_at", "posts"."reply_to_post_number", "posts"."reply_count", "posts"."quote_count", "posts"."deleted_at", "posts"."off_topic_count", "posts"."like_count", "posts"."incoming_link_count", "posts"."bookmark_count", "posts"."score", "posts"."reads", "posts"."post_type", "posts"."sort_order", "posts"."last_editor_id", "posts"."hidden", "posts"."hidden_reason_id", "posts"."notify_moderators_count", "posts"."spam_count", "posts"."illegal_count", "posts"."inappropriate_count", "posts"."last_version_at", "posts"."user_deleted", "posts"."reply_to_user_id", "posts"."percent_rank", "posts"."notify_user_count", "posts"."like_score", "posts"."deleted_by_id", "posts"."edit_reason", "posts"."word_count", "posts"."version", "posts"."cook_method", "posts"."wiki", "posts"."baked_at", "posts"."baked_version", "posts"."hidden_at", "posts"."self_edits", "posts"."reply_quoted", "posts"."via_email", "posts"."raw_email", "posts"."public_version", "posts"."action_code", "posts"."locked_by_id", "posts"."image_upload_id" FROM "posts" JOIN (SELECT *, row_number() over() row_number FROM (SELECT topics.id, min(posts.post_number) post_number FROM "posts" INNER JOIN "post_search_data" ON "post_search_data"."post_id" = "posts"."id" INNER JOIN "topics" ON "topics"."id" = "posts"."topic_id" AND ("topics"."deleted_at" IS NULL) LEFT JOIN categories ON categories.id = topics.category_id WHERE ("posts"."deleted_at" IS NULL) AND "posts"."post_type" IN (1, 2, 3, 4) AND (topics.visible) AND (topics.archetype <> 'private_message') AND (post_search_data.search_data @@ TO_TSQUERY('english', '''postgres'':*ABCD')) AND (categories.id NOT IN (
      SELECT categories.id WHERE categories.search_priority = 1
    )
    ) AND ((categories.id IS NULL) OR (NOT categories.read_restricted)) GROUP BY topics.id ORDER BY MAX((
      TS_RANK_CD(
        post_search_data.search_data,
        TO_TSQUERY('english', '''postgres'':*ABCD'),
        1|32
      ) *
      (
        CASE categories.search_priority
        WHEN 2
        THEN 0.6
        WHEN 3
        THEN 0.8
        WHEN 4
        THEN 1.2
        WHEN 5
        THEN 1.4
        ELSE
          CASE WHEN topics.closed
          THEN 0.9
          ELSE 1
          END
        END
      )
    )
    ) DESC, topics.bumped_at DESC LIMIT 51 OFFSET 0) xxx) x ON x.id = posts.topic_id AND x.post_number = posts.post_number WHERE ("posts"."deleted_at" IS NULL) ORDER BY row_number;
    SQL
  end

  x.report("current aggregate search query with proper ranking") do
    DB.exec <<~SQL
    SELECT "posts"."id", "posts"."user_id", "posts"."topic_id", "posts"."post_number", "posts"."raw", "posts"."cooked", "posts"."created_at", "posts"."updated_at", "posts"."reply_to_post_number", "posts"."reply_count", "posts"."quote_count", "posts"."deleted_at", "posts"."off_topic_count", "posts"."like_count", "posts"."incoming_link_count", "posts"."bookmark_count", "posts"."score", "posts"."reads", "posts"."post_type", "posts"."sort_order", "posts"."last_editor_id", "posts"."hidden", "posts"."hidden_reason_id", "posts"."notify_moderators_count", "posts"."spam_count", "posts"."illegal_count", "posts"."inappropriate_count", "posts"."last_version_at", "posts"."user_deleted", "posts"."reply_to_user_id", "posts"."percent_rank", "posts"."notify_user_count", "posts"."like_score", "posts"."deleted_by_id", "posts"."edit_reason", "posts"."word_count", "posts"."version", "posts"."cook_method", "posts"."wiki", "posts"."baked_at", "posts"."baked_version", "posts"."hidden_at", "posts"."self_edits", "posts"."reply_quoted", "posts"."via_email", "posts"."raw_email", "posts"."public_version", "posts"."action_code", "posts"."locked_by_id", "posts"."image_upload_id" FROM "posts" JOIN (SELECT *, row_number() over() row_number FROM (SELECT subquery.topic_id id, (ARRAY_AGG(subquery.post_number))[1] post_number, MAX(subquery.rank) rank, MAX(subquery.bumped_at) bumped_at FROM (SELECT "posts"."id", "posts"."user_id", "posts"."topic_id", "posts"."post_number", "posts"."raw", "posts"."cooked", "posts"."created_at", "posts"."updated_at", "posts"."reply_to_post_number", "posts"."reply_count", "posts"."quote_count", "posts"."deleted_at", "posts"."off_topic_count", "posts"."like_count", "posts"."incoming_link_count", "posts"."bookmark_count", "posts"."score", "posts"."reads", "posts"."post_type", "posts"."sort_order", "posts"."last_editor_id", "posts"."hidden", "posts"."hidden_reason_id", "posts"."notify_moderators_count", "posts"."spam_count", "posts"."illegal_count", "posts"."inappropriate_count", "posts"."last_version_at", "posts"."user_deleted", "posts"."reply_to_user_id", "posts"."percent_rank", "posts"."notify_user_count", "posts"."like_score", "posts"."deleted_by_id", "posts"."edit_reason", "posts"."word_count", "posts"."version", "posts"."cook_method", "posts"."wiki", "posts"."baked_at", "posts"."baked_version", "posts"."hidden_at", "posts"."self_edits", "posts"."reply_quoted", "posts"."via_email", "posts"."raw_email", "posts"."public_version", "posts"."action_code", "posts"."locked_by_id", "posts"."image_upload_id", (
      TS_RANK_CD(
        post_search_data.search_data,
        TO_TSQUERY('english', '''postgres'':*ABCD'),
        1|32
      ) *
      (
        CASE categories.search_priority
        WHEN 2
        THEN 0.6
        WHEN 3
        THEN 0.8
        WHEN 4
        THEN 1.2
        WHEN 5
        THEN 1.4
        ELSE
          CASE WHEN topics.closed
          THEN 0.9
          ELSE 1
          END
        END
      )
    )
     rank, topics.bumped_at bumped_at FROM "posts" INNER JOIN "post_search_data" ON "post_search_data"."post_id" = "posts"."id" INNER JOIN "topics" ON "topics"."id" = "posts"."topic_id" AND ("topics"."deleted_at" IS NULL) LEFT JOIN categories ON categories.id = topics.category_id WHERE ("posts"."deleted_at" IS NULL) AND "posts"."post_type" IN (1, 2, 3, 4) AND (topics.visible) AND (topics.archetype <> 'private_message') AND (post_search_data.search_data @@ TO_TSQUERY('english', '''postgres'':*ABCD')) AND (categories.id NOT IN (
      SELECT categories.id WHERE categories.search_priority = 1
    )
    ) AND ((categories.id IS NULL) OR (NOT categories.read_restricted))) subquery GROUP BY subquery.topic_id ORDER BY rank DESC, bumped_at DESC LIMIT 51 OFFSET 0) xxx) x ON x.id = posts.topic_id AND x.post_number = posts.post_number WHERE ("posts"."deleted_at" IS NULL) ORDER BY row_number;
    SQL
  end

  x.compare!
end
```

```
Warming up --------------------------------------
current aggregate search query
                         1.000  i/100ms
current aggregate search query with proper ranking
                         1.000  i/100ms
Calculating -------------------------------------
current aggregate search query
                         17.726  (± 0.0%) i/s -    178.000  in  10.045107s
current aggregate search query with proper ranking
                         17.802  (± 0.0%) i/s -    178.000  in  10.002230s

Comparison:
current aggregate search query with proper ranking:       17.8 i/s
current aggregate search query:       17.7 i/s - 1.00x  (± 0.00) slower
```
2020-07-14 13:39:13 +08:00
Guo Xiang Tan
4009c9f711
DEV: Fix search specs to take note of order in assertions.
`contain_exactly` does not care about the order which isn't what we
want.
2020-07-14 13:37:44 +08:00
Guo Xiang Tan
ce39733b1a
FIX: Incorrect search blurb when advanced search filters are used take2
Also remove include_blurbs attribute which isn't used.
2020-07-14 11:50:40 +08:00
Kane York
8ddd45d524
PERF: topic_view participant post count: don't send back ID list (#10210)
On large topics, the cost of sending the entire post ID list back over to the database is signficant. Just have the DB recalculate the list of visible posts instead.
2020-07-13 18:42:09 -07:00
Robin Ward
7045a2a87c FIX: Don't strip noopener from oneboxes 2020-07-13 16:54:42 -04:00
Dan Ungureanu
c72bc27888
FEATURE: Implement support for IMAP and SMTP email protocols. (#8301)
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2020-07-10 12:05:55 +03:00
Robin Ward
b1c6ff9e1c FIX: Test output related to Discourse::VERSION
It's a little awkward to test constants by re-assigning them so
I've added a new parameter to `Discourse.find_compatible_resource`
which can be used by tests.
2020-07-09 14:57:27 -04:00
Martin Brennan
e0713455ca
PERF: Load topic bookmarks for the user in user_post_bookmarks (#10197)
Instead of loading all of the user bookmarks using all the post IDs in a topic, load all the bookmarks for a user using the topic ID. This eliminates a costly WHERE ID IN query.
2020-07-09 15:46:52 +10:00
Bianca Nenciu
bd842cd2b0
FEATURE: Parse images in email signatures (#10137)
* FEATURE: Parse images in email signatures

* DEV: Fix tests

* Code review
2020-07-08 15:50:30 +10:00
Guo Xiang Tan
0c742dd022
DEV: Simple formatting fix. 2020-07-07 15:46:14 +08:00
Vinoth Kannan
f3f30edf3f SPEC: use post number to create canoncial path in mega topics.
6d17765924
2020-07-07 12:20:31 +05:30
Jeff Wong
339549d14a
Support plugin and Theme compatibility version manifests (#9995)
Adds a new rake task `plugin:checkout_compatible_all` and
`plugin:checkout_compatible[plugin-name]` that check out compatible plugin
versions.

Supports a .discourse-compatibility file in the root of plugins and themes that
list out a plugin's compatibility with certain discourse versions:

eg: .discourse-compatibility
```
2.5.0.beta6: some-git-hash
2.4.4.beta4: some-git-tag
2.2.0: git-reference
```

This ensures older Discourse installs are able to find and install older
versions of plugins without intervention, through the manifest only.

It iterates through the versions in descending order. If the current Discourse
version matches an item in the manifest, it checks out the listed plugin target.
If the Discourse version is greater than an item in the manifest, it checks out
the next highest version listed in the manifest.

If no versions match, it makes no change.
2020-07-06 14:48:00 -07:00
David Taylor
da0fc0a9d3
DEV: Cleanup PostActionType and ReviewableScore changes in tests
Followup to 2df388ffd7
2020-07-06 17:12:23 +01:00
David Taylor
7f2b5a446a
PERF: Remove post_upload recovery in daily EnsureS3UploadsExistence job (#10173)
This is a very expensive process, and it should only be required in exceptional circumstances. It is possible to run a similar recovery using `rake uploads:recover` (5284d41a8e/lib/upload_recovery.rb (L135-L184))
2020-07-06 16:26:40 +01:00
David Taylor
977766e7a8
FEATURE: sso_overrides_(email|username|name) for all auth methods
These settings previously applied only to discourse-sso. Now they work for all external authentication methods.
2020-07-06 10:18:45 +01:00
Vinoth Kannan
06d426bd87 FIX: skip hidden posts while generating canonical url.
Previously, while generating the topic page's canoncial url we used the current post number. It will create invalid canonical path if the topic has whsiper posts. Now we only taking the visible posts for current page index calculation.
2020-07-05 14:04:31 +05:30
Roman Rizzi
2df388ffd7
DEV: Plugins can extend ReviewableScore types. (#10156) 2020-07-02 11:47:43 -03:00
David Taylor
0edffcc47d
FIX: Correct version comparison logic when comparing stable to beta (#10135)
* FIX: Correct version comparison logic when comparing stable to beta

For example, version 1.3.0 should be considered higher than 1.3.0.beta3. So `Discourse.has_needed_version?('1.3.0', '1.3.0.beta3')` should return true

* Switch to use Gem::Version to compare versions
2020-06-29 17:52:33 +10:00
Régis Hanol
7109d94ee7 FIX: properly invalidate inline oneboxes when rebaking
When rebaking a post we were invalidating _regular_ oneboxes but not inline oneboxes.

DEV: also renamed 'InlineOneboxer.purge' to 'InlineOneboxer.invalidate' to keep
the API consistent with 'Oneboxer.invalidate'
2020-06-24 11:54:54 +02:00
Régis Hanol
91c89df68a FIX: onebox local topic when using slug-less URL
When linking to a topic in the same Discourse, we try to onebox the link to show the title
and other various information depending on whether it's a "standard" or "inline" onebox.

However, we were not properly detecting links to topics that had no slugs (eg. https://meta.discourse.org/t/1234).
2020-06-23 17:18:38 +02:00
Martin Brennan
e92909aa77
FIX: Use ActionDispatch::Http::ContentDisposition for uploads content-disposition (#10108)
See https://meta.discourse.org/t/broken-pipe-error-when-uploading-to-a-s3-clone-a-pdf-with-a-name-containing-e-i-etc/155414

When setting content-disposition for attachment, use the ContentDisposition class to format it. This handles filenames with weird characters and localization (accented characters) correctly.
2020-06-23 17:10:56 +10:00
Guo Xiang Tan
add2a9411e
DEV: Remove specs that are no longer relevant. 2020-06-23 12:09:04 +08:00
Guo Xiang Tan
25db91e351
Revert "FIX: These tests are broken"
This reverts commit b1114b9a20.
2020-06-23 08:41:10 +08:00
Robin Ward
b1114b9a20 FIX: These tests are broken 2020-06-22 16:30:46 -04:00
Bianca Nenciu
68f767a557
FEATURE: Check if selectable avatars exist before enabling them (#10032) 2020-06-22 16:58:26 +03:00
Daniel Waterworth
9cf77372a2 FIX: Guardian#can_remove_allowed_users? shouldn't break for ownerless topics
A topic can outlive its original author. TopicGuardian should still work
in this situation.
2020-06-19 10:35:52 +01:00
Bianca Nenciu
db1bebddce
FIX: Hide the post history for TL4 (#10065) 2020-06-18 13:27:51 +03:00
Dan Ungureanu
d21a08c284
DEV: Deprecate Category#url_with_id in favor of Category#url (#9972) 2020-06-18 11:32:14 +03:00
Dan Ungureanu
7ed7b1ef64
DEV: Add test (#10064)
Follow-up-to 84dfaad137
2020-06-17 21:41:16 +03:00
Robin Ward
e8756e1a95 FIX: Muted/Ignore should prevent PMs regardless of case sensitivity 2020-06-17 14:26:14 -04:00
Guo Xiang Tan
0ff86b00cb
DEV: Upgrade Redis to 4.2.1. 2020-06-15 10:05:22 +08:00
Joffrey JAFFEUX
4b793a1072
FIX: allows PM owner to remove any user if >= TL2 (#10036) 2020-06-12 12:54:28 +02:00
Guo Xiang Tan
3aab98781a
Fix tests. 2020-06-12 10:22:22 +08:00
Dan Ungureanu
b7e70850e4
FIX: Allow users to add emails which were deleted before 2020-06-11 14:54:11 +03:00
Dan Ungureanu
5bfe1ee4f1
FEATURE: Improve UX support for multiple email addresses (#9691) 2020-06-10 19:11:49 +03:00
Guo Xiang Tan
a3dfd553a1
Revert "Bump redis to 4.2.0."
This reverts commit 98bc28cea2.
2020-06-10 14:52:05 +08:00
Guo Xiang Tan
98bc28cea2
Bump redis to 4.2.0. 2020-06-10 14:28:56 +08:00
Jeff Wong
70a88111dd
FIX: prevent re-flagging when we have reviewed flags before (#10010)
FIX: prevent re-flagging when we have reviewed flags before

Fixes an edge case where a review can be reflagged when:
User flags as inappropriate.
Moderator rejects the flag.
Another user re-flags the post as spam.

Before, anyone was able to re-flag as inappropriate despite it being flagged
previously. With this, users are unable to re-flag for the same reason
regardless of reviewable status.
2020-06-09 15:26:10 -07:00
Penar Musaraj
2d880b42a3
UX: Add simple-list setting type (#9970) 2020-06-04 10:44:54 -04:00
Guo Xiang Tan
2188ccccd5 DEV: Remove initiating_user keyword arg from EmailUpdater.
The guardian contains the acting user.
2020-06-04 13:21:56 +08:00
Blake Erickson
a89574ccb9 FIX: Inline error when converting html to markdown
Looks like some html elements like `aside` and `section` will throw an error
when checking if they are inline or not. The commit simply handles

```
Job exception: undefined method `inline?' for nil:NilClass
```

and adds a test for it.
2020-06-03 15:59:19 -06:00
Guo Xiang Tan
04a291ceea
DEV: Fix race conditions due to directory removal for uploads spec. 2020-06-03 12:28:39 +08:00
Sam Saffron
57a3d4e0d2
FEATURE: whitelist theme repo mode (experimental)
In some restricted setups all JS payloads need tight control.

This setting bans admins from making changes to JS on the site and
requires all themes be whitelisted to be used.

There are edge cases we still need to work through in this mode
hence this is still not supported in production and experimental.

Use an example like this to enable:

`DISCOURSE_WHITELISTED_THEME_REPOS="https://repo.com/repo.git,https://repo.com/repo2.git"`

By default this feature is not enabled and no changes are made.

One exception is that default theme id was missing a security check
this was added for correctness.
2020-06-03 13:19:57 +10:00
Guo Xiang Tan
389bdcf5ab
DEV: Fix implementation for DiscourseRedis#exists. 2020-06-01 13:11:32 +08:00
Guo Xiang Tan
df62407f35
DEV: Implement multiple keys support for DiscourseRedis#exists. 2020-06-01 11:20:26 +08:00
David Taylor
e159fb06df
FEATURE: Download remote images even for old posts (#9925)
When a post is rebaked, the admins expect it to work the same regardless of how old the post is.
2020-05-29 17:13:55 +01:00
David Taylor
28f46c171c
FIX: Pull hotlinked images even when edited by system users (#9890)
Previously the pull hotlinked images job was skipped after system edits. This ensured that we never had an infinite loop of system-edit/pull-hotlinked/system-edit/pull-hotlinked etc.

A side effect was that edits made by system for any other reason (e.g. API, removing full quotes) would prevent pulling hotlinked images. This commit removes the system edit check, and replaces it with another method to avoid an infinite job scheduling loop.
2020-05-29 13:07:47 +01:00
Vinoth Kannan
ce1491e830
UX: remove in:unpinned filter from advanced search page. (#9911) 2020-05-29 00:47:28 +05:30
Joffrey JAFFEUX
77801aa9be
FIX: allows to have custom emoji translation without static file (#9893) 2020-05-27 20:11:52 +02:00
Penar Musaraj
b1c726be0d
Remove support for FontAwesome 4.7 icon names (#9871) 2020-05-26 14:53:32 -04:00
Gerhard Schlager
69ee94b526 FIX: XML files could be detected as SVG files 2020-05-26 18:18:20 +02:00
Vinoth Kannan
8e56197728
UX: use "icon-picker" & "image-uploader" fields to set group flair. (#9779) 2020-05-25 11:08:47 +05:30
Michael Brown
d9a02d1336
Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eee.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
Jeff Atwood
20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6f, reversing
changes made to 2660c2e21d.
2020-05-22 20:25:56 -07:00
Guo Xiang Tan
f4b82f1dc0
DEV: Fix randomly failing spec.
If a user is created with an id of 999, a `upload.user_id ==
user_avatar.user_id` will return true. This fix increases the id of the
upload to something that we will not hit in the foreseeable future.
2020-05-21 11:41:07 +08:00
Martin Brennan
df68d11c38
FEATURE: Add topic excerpt max length site setting (#9847)
Adds a new topic_excerpt_maxlength site setting.

* When topic excerpt is requested for a post, use the new topic_excerpt_maxlength site setting to limit the size of the excerpt
* Remove code for getting/setting Post.excerpt_size as it is not used anywhere
2020-05-21 13:19:48 +10:00
Osama Sayegh
02f44def56
FIX: Don't blow up when trying to parse invalid or non-ASCII URLs (#9838)
* FIX: Don't blow up when trying to parseinvalid or non-ASCII URLs

Follow-up to 72f139191e
2020-05-20 12:46:27 +03:00
Martin Brennan
72f139191e
FIX: S3 store has_been_uploaded? was not taking into account s3 bucket path (#9810)
In some cases, between Discourse forums the hostname of a URL could match if they are hosting S3 files on the same bucket but the S3 bucket path might not. So e.g. https://testbucket.somesite.com/testpath/some/file/url.png vs https://testbucket.somesite.com/prodpath/some/file/url.png. So has_been_uploaded? was returning true for the second URL, even though it may have been uploaded on a different Discourse forum.

This is a very rare case but must be accounted for, because this impacts UrlHelper.is_local which mistakenly thinks the file has already been downloaded and thus allows the URL to be cooked, where we want to return the full URL to be downloaded using PullHotlinkedImages.
2020-05-20 10:40:38 +10:00
Guo Xiang Tan
96c02caba7
DEV: Change use of Redis flushall to flushdb.
FLUSHALL removes all keys from all databases. Instead we only want to
remove keys from the current Redis database.
2020-05-19 10:20:00 +08:00
Guo Xiang Tan
d01c336899
DEV: Clean up some Redis leaks in test env. 2020-05-18 17:27:37 +08:00
David Taylor
461b4e5cc6
DEV: Add framework for filtered plugin registers (#9763)
* DEV: Add framework for filtered plugin registers

Plugins often need to add values to a list, and we need to filter those lists at runtime to ignore values from disabled plugins. This commit provides a re-usable way to do that, which should make it easier to add new registers in future, and also reduce repeated code.

Follow-up commits will migrate existing registers to use this new system

* DEV: Migrate user and group custom field APIs to plugin registry

This gives us a consistent system for checking plugin enabled state, so we are repeating less logic. API changes are backwards compatible
2020-05-15 14:04:38 +01:00
David Taylor
0495a748d0
FIX: Use CDN URLs for topic thumbnails 2020-05-15 13:35:20 +01:00
Guo Xiang Tan
9ab5801a1b
FIX: Allow post migrations using #change to carry out unsafe migration 2020-05-15 14:23:27 +08:00
Blake Erickson
1a2b9435b0
DEV: Standardize table sorting verbiage (#9757)
* DEV: Standardize table sorting verbiage

This commit creates a common component that tables can use to make their
headers sortable. This commit also standardizes on using `desc` as the
default and passing in the `asc=true` flag to adjust the sorting
direction.

* Add deprecation warnings

Adds deprecation warnings if using previous params and maintains
backwards compatibility. Set the default sort value for group members to
be asc.

* switch group requests to use common table-header-toggle

* update fixture
2020-05-14 20:10:59 -06:00
Kane York
869f9b20a2
PERF: Dematerialize topic_reply_count (#9769)
* PERF: Dematerialize topic_reply_count

It's only ever used for trust level promotions that run daily, or compared to 0. We don't need to track it on every post creation.

* UX: Add symbol in TL3 report if topic reply count is capped

* DEV: Drop user_stats.topic_reply_count column
2020-05-14 15:42:00 -07:00
Mark VanLandingham
a047004c9a
FIX: Specific email error for replies to digest emails (#9770) 2020-05-14 09:04:58 -05:00
Dan Ungureanu
3ed6a0e904
FIX: Detect Wayback Machine using user agent (#9777) 2020-05-14 21:10:07 +10:00
David Taylor
d4717f5d1e
DEV: Reduce repeated code in discourse_plugin_registry (#9705)
Use a helper method to simplify creating a new register. Previously this would require creating lots of different methods manually, and adding every register to the clear/reset functions
2020-05-13 12:25:34 +01:00
David Taylor
6230f5c554
FEATURE: Allow parameter authentication for UserApiKeys (#9742)
This refactors default_current_user_provider in a few ways:
- Introduce a generic `api_parameter_allowed?` method which checks for whitelisted routes/formats
- Only read the api_key parameter on allowed routes. It is now completely ignored on other routes (previously it would raise a 403)
- Start reading user_api_key parameter on allowed routes
- Refactor tests as end-end integration tests

A plugin API for PARAMETER_API_PATTERNS will be added soon
2020-05-12 13:35:36 +01:00
Guo Xiang Tan
3c423faafd
DEV: Avoid repeating draft key logic in multiple places. 2020-05-12 11:14:03 +08:00
David Taylor
5fc51ed49c
DEV: Remove unused DiscoursePlugin class (#9715) 2020-05-11 15:46:54 +01:00
Bianca Nenciu
8149bfbaf1
FEATURE: Filter settings by plugin (#9692) 2020-05-10 14:07:45 +03:00
Robin Ward
f9608c0af5 DEV: Remove INLINE_ONEBOX_* constants
There were two constants here, `INLINE_ONEBOX_LOADING_CSS_CLASS` and
`INLINE_ONEBOX_CSS_CLASS` that were both longer than the strings they
were DRYing up: `inline-onebox-loading` and `inline-onebox`

I normally appreciate constants, but in this case it meant that we had
a lot of JS imports resulting in many more lines of code (and CPU cycles
spent figuring them out.)

It also meant we had an `.erb` file and had to invoke Ruby to create the
JS file, which meant the app was harder to port to Ember CLI.

I removed the constants. It's less DRY but faster and simpler, and
arguably the loss of DRYness is not significant as you can still search
for the `inline-onebox-loading` and `inline-onebox` strings easily if
you are refactoring.
2020-05-07 16:14:38 -04:00
Vinoth Kannan
744bbf6904 FEATURE: exclude muted categories from the "top" topics list. 2020-05-08 00:34:53 +05:30
Gerhard Schlager
ec2f3169ff FIX: Using the default_locale in locale fallbacks caused problems
Locale files get precompiled after deployment and they contained translations from the `default_locale`. That's especially bad in multisites, because the initial `default_locale` is `en_US`. Sites where the `default_locale` isn't `en_US` could see missing translations. The same thing could happen when users are allowed to chose a different locale.

This change simplifies the logic by not using the `default_locale` in the locale chain. It always falls back to `en` in case of missing translations.
2020-05-06 22:59:07 +02:00
Sam Saffron
f8c18ac888
DEV: remove fragile spec
The failover spec is very fragile and tests specific implementation
vs actual behavior

We rely on a different script during the build process to test
failover operates correctly
2020-05-06 16:21:48 +10:00
Martin Brennan
fa572d3a7a
DEV: Remove all code referencing at_desktop bookmark reminders (#9650)
We have found no need for these reminder types, so we are removing the code for them.
2020-05-06 15:22:43 +10:00
David Taylor
03818e642a
FEATURE: Include optimized thumbnails for topics (#9215)
This introduces new APIs for obtaining optimized thumbnails for topics. There are a few building blocks required for this:

- Introduces new `image_upload_id` columns on the `posts` and `topics` table. This replaces the old `image_url` column, which means that thumbnails are now restricted to uploads. Hotlinked thumbnails are no longer possible. In normal use (with pull_hotlinked_images enabled), this has no noticeable impact

- A migration attempts to match existing urls to upload records. If a match cannot be found then the posts will be queued for rebake

- Optimized thumbnails are generated during post_process_cooked. If thumbnails are missing when serializing a topic list, then a sidekiq job is queued

- Topic lists and topics now include a `thumbnails` key, which includes all the available images:
   ```
   "thumbnails": [
   {
     "max_width": null,
     "max_height": null,
     "url": "//example.com/original-image.png",
     "width": 1380,
     "height": 1840
   },
   {
     "max_width": 1024,
     "max_height": 1024,
     "url": "//example.com/optimized-image.png",
     "width": 768,
     "height": 1024
   }
   ]
  ```

- Themes can request additional thumbnail sizes by using a modifier in their `about.json` file:
   ```
    "modifiers": {
      "topic_thumbnail_sizes": [
        [200, 200],
        [800, 800]
      ],
      ...
  ```
  Remember that these are generated asynchronously, so your theme should include logic to fallback to other available thumbnails if your requested size has not yet been generated

- Two new raw plugin outlets are introduced, to improve the customisability of the topic list. `topic-list-before-columns` and `topic-list-before-link`
2020-05-05 09:07:50 +01:00
Krzysztof Kotlarek
9bff0882c3
FEATURE: Nokogumbo (#9577)
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
Krzysztof Kotlarek
37e93914fc
FIX: the muted message should be sent after edit (#9593)
Recently, we added feature that we are sending `/muted` to users who muted specific topic just before `/latest` so the client knows to ignore those messages - https://github.com/discourse/discourse/pull/9482

Same `/muted` message should be included when the post is edited
2020-05-01 08:33:57 +10:00
Vinoth Kannan
71241a50f7 DEV: improve code readability & add tests for user guardian.
a511bea4cc
2020-04-30 20:59:33 +05:30
Régis Hanol
501b19b6e0
FIX: server-side HtmlToMarkdown improvements (#9586)
TLDR; this commit vastly improves how whitespaces are handled when converting from HTML to Markdown.
It also adds support for converting HTML <tables> to markdown tables.

The previous 'remove_whitespaces!' method was traversing the whole HTML tree and used a heuristic to remove
leading and trailing whitespaces whenever it was appropriate (ie. mostly before and after HTML block elements)

It was a good idea, but it was very limited and leaded to bad conversion when the html had leading whitespaces on several lines for example.
One such example can be found [here](https://meta.discourse.org/t/86782).

For various reasons, most of the whitespaces in a HTML file is ignored when the page is being displayed in a browser.
The rules that the browsers follow are the [CSS' White Space Processing Rules](https://www.w3.org/TR/css-text-3/#white-space-rules).
They can be quite complicated when you take into account RTL languages and other various tidbits but they boils down to the following:

- Collapse whitespaces down to one space (0x20) inside an inline context (ie. nodes/tags that are being displaying on the same line)
- Remove any leading/trailing whitespaces inside an inline context

One quick & dirty way of getting this 90% solved would be to do 'HTML.gsub!(/[[:space:]]+/, " ")'.
We would also need to hoist <pre> elements in order to not mess with their whitespaces.
Unfortunately, this solution let some whitespaces creep around HTML tags which leads to more '.strip!' calls than I can bear.

I decided to "emulate" the browser's handling of whitespaces and came up with a solution in 4 parts

1. remove_not_allowed!

The HtmlToMarkdown library is recursively "visiting" all the nodes in the HTML in order to convert them to Markdown.
All the nodes that aren't handled by the library (eg. <script>, <style> or any non-textual HTML tags) are "swallowed".
In order to reduce the number of nodes visited, the method 'remove_not_allowed!' will automatically delete all the nodes
that have no "visitor" (eg. a 'visit_<tag>' method) defined.

2. remove_hidden!

Similar purpose as the previous method (eg. reducing number of nodes visited), there's no point trying to convert something that is hidden.
The 'remove_hidden!' method removes any nodes that was hidden using the "hidden" HTML attribute, some CSS or with a width or height equal to 0.

3. hoist_line_breaks!

The 'hoist_line_breaks!' method is there to handle <br> tags. I know those tiny <br> don't do much but they can be quite annoying.
The <br> tags are inline elements but they visually work like a block element (ie. they create a new line).
If you have the following HTML "<i>Foo<br>Bar</i>", it ends up visually similar to "<i>Foo</i><br><i>Bar</i>".
The latter being much more easy to process than the former, so that's what this method is doing.
The "hoist_line_breaks" will hoist <br> tags out of inline tags until their parent is a block element.

4. remove_whitespaces!

The "remove_whitespaces!" is where all the whitespace removal is happening. It's broken down into 4 methods as well

- remove_whitespaces!
- is_inline?
- collapse_spaces!
- remove_trailing_space!

The 'remove_whitespace!' method is recursively walking the HTML tree (skipping <pre> tags).
If a node has any children, they will be chunked into groups of inline elements vs block elements.
For each chunks of inline elements, it will call the "collapse_space!" and "remove_trailing_space!" methods.
For each chunks of block elements, it will call "remote_whitespace!" to keep walking the HTML tree recursively.

The "is_inline?" method determines whether a node is part of a inline context.
A node is inline iif it's a text node or it's an inline tag, but not <br>, and all its children are also inline.

The "collapse_spaces!" method will collapse any kind of (white) space into a single space (" ") character, even accros tags.
For example, if we have "  Foo \n<i> Bar </i>\t42", it will return "Foo <i>Bar </i>42".

Finally, the "remove_trailing_space!" method is there to remove any trailing space that might creep in at the end of the inline chunk.

This solution is not 100% bullet-proof.
It does not support RTL languages at all and has some caveats that I felt were not worth the work to get properly fixed.

FIX: better detection of hidden elements when converting HTML to Markdown
FIX: take into account the 'allowed_href_schemes' site setting when converting HTML <a> to Markdown
FIX: added support for 'mailto:' scheme when converting <a> from HTML to Markdown
FIX: added support for <img> dimensions when converting from HTML to Markdown
FIX: added support for <dl>, <dd> and <dt> when converting from HTML to Markdown
FIX: added support for multilines emphases, strongs and strikes when converting from HTML to Markdown
FIX: added support for <acronym> when converting from HTML to Markdown
DEV: remove unused 'sanitize' gem

Wow, did you just read all that?! Congratz, here's a cookie: 🍪.
2020-04-30 12:21:25 +02:00
Sam Saffron
d0d5a138c3
DEV: stop freezing frozen strings
We have the `# frozen_string_literal: true` comment on all our
files. This means all string literals are frozen. There is no need
to call #freeze on any literals.

For files with `# frozen_string_literal: true`

```
puts %w{a b}[0].frozen?
=> true

puts "hi".frozen?
=> true

puts "a #{1} b".frozen?
=> true

puts ("a " + "b").frozen?
=> false

puts (-("a " + "b")).frozen?
=> true
```

For more details see: https://samsaffron.com/archive/2018/02/16/reducing-string-duplication-in-ruby
2020-04-30 16:48:53 +10:00
Roman Rizzi
394babcae3
FIX: Only show the review page to users that can see it. Do not publish the reviewable count update message to everyone. (#9556) 2020-04-27 14:51:25 -03:00
Sam Saffron
8c1e008c59
DEV: Skip erratic spec for now
Spec fails intermittently due to CDN state.
2020-04-25 13:20:04 +10:00
Penar Musaraj
585e7bcfe8 DEV: update specs followup to 67e96f6 2020-04-23 16:11:17 -04:00
Arpit Jalan
39be639c37 FIX: update GitHub screen_name on login via GitHub 2020-04-23 20:54:26 +05:30
Dan Ungureanu
4e5f9d4cd1
DEV: Drop 'key' column from user_api_keys (#9388) 2020-04-22 12:13:19 +03:00
Jarek Radosz
ab52bed014
DEV: Remove the return value of disable_if_low_on_disk_space (#9469)
It was used only in specs.
2020-04-21 03:48:33 +02:00
Jarek Radosz
5a81e3999c
DEV: Remove bypass_bump from CookedPostProcessor (#9468)
It was only passing it along to `PullHotlinkedImages` and that class have not used that arg since April 2014 (c52ee665b4)
2020-04-21 03:48:19 +02:00
Robin Ward
8f5314bf98 FIX: An opts hash was not, in fact, optional :) 2020-04-20 14:17:13 -04:00
Jeff Wong
e3590d4ead
FEATURE: add user_session_refreshed trigger (#9412)
Trigger an event for plugins to consume when a user session is refreshed.

This allows external auth to be notified about account activity, and be
able to take action such as use oauth refresh tokens to keep oauth
tokens valid.
2020-04-14 09:32:24 -07:00
Robin Ward
e1f8014acd
FEATURE: Support for publishing topics as pages (#9364)
If the feature is enabled, staff members can construct a URL and publish a
topic for others to browse without the regular Discourse chrome.

This is useful if you want to use Discourse like a CMS and publish
topics as articles, which can then be embedded into other systems.
2020-04-08 12:52:36 -04:00
Sam Saffron
0375a5ac0b
DEV: reduce logging when no external id is specified
Previously we were returning an unknown sso error and logging a message
when external id was blank. This noise is not needed.
2020-04-08 12:42:28 +10:00
Blake Erickson
d04ba4b3b2
DEPRECATION: Remove support for api creds in query params (#9106)
* DEPRECATION: Remove support for api creds in query params

This commit removes support for api credentials in query params except
for a few whitelisted routes like rss/json feeds and the handle_mail
route.

Several tests were written to valid these changes, but the bulk of the
spec changes are just switching them over to use header based auth so
that they will pass without changing what they were actually testing.

Original commit that notified admins this change was coming was created
over 3 months ago: 2db2003187

* fix tests

* Also allow iCalendar feeds

Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>
2020-04-06 16:55:44 -06:00
Joffrey JAFFEUX
b248c553c2
DEV: ensures CustomEmoji cache is cleared after spec (#9361) 2020-04-06 19:41:59 +02:00
Vinoth Kannan
fd39c85c1a FIX: add category hashtags support for sub-sub categories.
Hashtags will include last two levels only (ex: "parent:child").
2020-04-06 20:43:38 +05:30
Krzysztof Kotlarek
ce00da3bcd
FIX: guardian always got user but sometimes it is anonymous (#9342)
* FIX: guardian always got user but sometimes it is anonymous

```
  def initialize(user = nil, request = nil)
    @user = user.presence || AnonymousUser.new
    @request = request
  end
```

AnonymouseUser defines `blank?` method
```
  class AnonymousUser
    def blank?
      true
    end
    ...
  end
```
so if we would use @user.present? it would be correct, however, just @user is always true
2020-04-06 09:56:47 +10:00
Daniel Waterworth
76610acb6f FIX: Default to light theme in wizard so that previews are displayed
Previously, without a theme selection, the previews wouldn't show.
2020-04-02 18:37:45 +01:00
Kane York
cdaa60b56b FEATURE: Allow admins to disable self-service account deletion
https://meta.discourse.org/t/-/146276
2020-04-01 15:16:07 -07:00
Mark VanLandingham
689c61b462
DEV: Allow plugins to add wizard steps after specific steps (#9315) 2020-04-01 08:36:50 -05:00
Arpit Jalan
b2a0d34bb7
FEATURE: add setting auto_approve_email_domains to auto approve users (#9323)
* FEATURE: add setting `auto_approve_email_domains` to auto approve users

This commit adds a new site setting `auto_approve_email_domains` to
auto approve users based on their email address domain.

Note that if a domain already exists in `email_domains_whitelist` then
`auto_approve_email_domains` needs to be duplicated there as well,
since users won’t be able to register with email address that is
not allowed in `email_domains_whitelist`.

* Update config/locales/server.en.yml

Co-Authored-By: Robin Ward <robin.ward@gmail.com>
2020-03-31 23:59:15 +05:30
Joffrey JAFFEUX
0996c3b7b3
FEATURE: allows multiple custom emoji groups (#9308)
Note: DBHelper would fail with a sql syntax error on columns like "group".

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2020-03-30 20:16:10 +02:00
Bianca Nenciu
7952cbb9a2
FIX: Perform crop using user-specified image sizes (#9224)
* FIX: Perform crop using user-specified image sizes

It used to resize the images to max width and height first and then
perform the crop operation. This is wrong because it ignored the user
specified image sizes from the Markdown.

* DEV: Use real images in test
2020-03-26 16:40:00 +02:00
Sam Saffron
25f1f23288
FEATURE: Stricter rules for user presence
Previously we would consider a user "present" and "last seen" if the
browser window was visible.

This has many edge cases, you could be considered present and around for
days just by having a window open and no screensaver on.

Instead we now also check that you either clicked, transitioned around app
or scrolled the page in the last minute in combination with window
visibility

This will lead to more reliable notifications via email and reduce load of
message bus for cases where a user walks away from the terminal
2020-03-26 17:36:52 +11:00
Mark VanLandingham
c14f6d4ced
FEATURE: Allow plugins to exclude wizard steps (#9275) 2020-03-25 11:36:42 -05:00
Kane York
58ae0d4bd9
DEV: Add test case for /srv/status probers (#9259) 2020-03-24 16:28:07 +11:00
Sam Saffron
10b37e1e36
FIX: add support for sub-sub category slugs in search
Previous to this change slugs for leaves in 3 level nestings would not work

Our UX picks only the last two levels

This also makes the results consistent for slugs as it enforces order.
2020-03-20 15:36:50 +11:00
David Taylor
19814c5e81
FIX: Allow CSP to work correctly for non-default hostnames/schemes (#9180)
- Define the CSP based on the requested domain / scheme (respecting force_https)
- Update EnforceHostname middleware to allow secondary domains, add specs
- Add URL scheme to anon cache key so that CSP headers are cached correctly
2020-03-19 19:54:42 +00:00
Vinoth Kannan
f6d6f1701f FIX: use the new duration attribute in set_or_create_timer method.
New `duration` attribute is introduced for the `set_or_create_timer` method in the commit aad12822b7 for "based on last post" and "auto delete replies" topic timers.
2020-03-19 21:45:05 +05:30
Dan Ungureanu
1393950dbc
FIX: Improve HTML to Markdown conversion (#9231)
This commit ensures that whitespaces are preserved in <pre>, but removed
inside text paragraphs.
2020-03-18 19:31:10 +02:00
David Taylor
3723c64257
DEV: Correct references to theme flags
Followup to d1474e94
2020-03-13 16:45:55 +00:00
David Taylor
3d71b68195
DEV: Introduce plugin api for conditionally rendering assets (#9200) 2020-03-13 15:30:31 +00:00
Martin Brennan
2237ba8c9d
FIX: Add topic deleted check to email/sender (#9166)
It already had a deleted post check and log reason, add a topic one too to avoid errors
2020-03-13 10:04:15 +10:00
Daniel Waterworth
59578dfc5b FIX: Notification emails with attachments are incorrectly structured
Two behaviors in the mail gem collide:

 1. Attachments are added as extra parts at the top level,
 2. When there are both text and html parts, the content type is set to
    'multipart/alternative'.

Since attachments aren't alternative renderings, for emails that contain
attachments and both html and text parts, some coercing is necessary.
2020-03-12 15:42:24 +00:00
Martin Brennan
793f39139a
FEATURE: Send notifications for time-based and At Desktop bookmark reminders (#9071)
* This PR implements the scheduling and notification system for bookmark reminders. Every 5 minutes a schedule runs to check any reminders that need to be sent before now, limited to **300** reminders at a time. Any leftover reminders will be sent in the next run. This is to avoid having to deal with fickle sidekiq and reminders in the far-flung future, which would necessitate having a background job anyway to clean up any missing `enqueue_at` reminders.

* If a reminder is sent its `reminder_at` time is cleared and the `reminder_last_sent_at` time is filled in. Notifications are only user-level notifications for now.

* All JavaScript and frontend code related to displaying the bookmark reminder notification is contained here. The reminder functionality is now re-enabled in the bookmark modal as well.

* This PR also implements the "Remind me next time I am at my desktop" bookmark reminder functionality. When the user is on a mobile device they are able to select this option. When they choose this option we set a key in Redis saying they have a pending at desktop reminder. The next time they change devices we check if the new device is desktop, and if it is we send reminders using a DistributedMutex. There is also a job to ensure consistency of these reminders in Redis (in case Redis drops the ball) and the at desktop reminders expire after 20 days.

* Also in this PR is a fix to delete all Bookmarks for a user via `UserDestroyer`
2020-03-12 10:16:00 +10:00
David Taylor
d1474e94a1
FEATURE: Allow themes to specify modifiers in their about.json file (#9097)
There are three modifiers:
- serialize_topic_excerpts (boolean)
- csp_extensions (array of strings)
- svg_icons (array of strings)

When multiple themes are active, the values will be combined. The combination method varies based on the setting. CSP/SVG arrays will be combined. serialize_topic_excerpts will use `Enumerable#any`.
2020-03-11 13:30:45 +00:00
Jarek Radosz
29b35aa64c
DEV: Improve flaky time-sensitive specs (#9141) 2020-03-10 22:13:17 +01:00
Jarek Radosz
aec26ad2f0
FIX: Preserve TopicCreator's timestamp resolution (#9158)
Continuation of #9140 (e35bc8b). It's the last piece required for #9141.
2020-03-10 15:35:40 +01:00
Roman Rizzi
826b4793c0
FEATURE: Approve suspect users is now true by default. The suspect users list was removed (#9151) 2020-03-10 08:56:42 -03:00
Jarek Radosz
e35bc8bebd
FIX: Preserve PostCreator's created_at resolution (#9140)
PostMover passes to PostCreator a `created_at` that is a `ActiveSupport::WithTimeZone` instance (and also `is_a? Time`). Previously it was always being passed through `Time.zone.parse` so it would lose sub-second information. Now, it takes `Time` input as-is, while still parsing other types.
2020-03-09 17:38:13 +01:00
Joffrey JAFFEUX
60b47d622e
UX: adds support for a color setting type (#9016) 2020-03-09 10:07:03 +01:00
David Taylor
5b3630dba3
FIX: Do not raise an error when in:all search is performed by anon (#9113)
Also improve in:all specs to catch to catch similar failures
2020-03-05 17:50:29 +00:00
Sam Saffron
e23e247dff
DEV: spec suite fails on leap years
Something about year math fails here on leap years, just
set up the clock so it is consistent to avoid it.
2020-02-29 18:30:08 +11:00
Dan Ungureanu
8cbb6e35cb
DEV: Fix build
Follow up to 60184a290c.
2020-02-28 11:31:04 +02:00
Robin Ward
a47e0a3fda FIX: TOTP could not be used on sites with colons in their names
This is because the TOTP gem identifies as a colon as an addressable
protocol. The solution for now is to remove the colon in the issuer
name.

Changing the issuer changes the token values, but now it was completely
broken for colons so this should not be breaking anyone new.
2020-02-20 16:35:30 -05:00
Roman Rizzi
c7787464cd
FEATURE: Admins can configure the reflag cooldown window and if posts flagged as spam by TL3+ users get automatically hidden (#9010) 2020-02-20 14:43:33 -03:00
Penar Musaraj
7a09e2cce2 DEV: Improve video onebox stripping spec
Followup to 70819080
2020-02-20 11:45:12 -05:00
Martin Brennan
254b57c812
FIX: When admin changes staff email still enforce old email confirm (#9007)
A follow-up correction to this change https://github.com/discourse/discourse/pull/9001.

When admin changes staff email still enforce old email confirm. Only allow auto-confirm of a new email by admin IF the target user is not also an admin. If an admin gets locked out of their email the site admin can use the rails console to solve the issue in a pinch.
2020-02-20 13:42:57 +10:00
Martin Brennan
97d8f19387
FIX: When admin changes another user's email auto-confirm the change (#9001)
When admin changes a user's email from the preferences page of that user:

* The user will not be sent an email to confirm that their
  email is changing. They will be sent a reset password email
  so they can set the password for their account at the new
  email address.
* The user will still be sent an email to their old email to inform
  them that it was changed.
* Admin and staff users still need to follow the same old + new
  confirm process, as do users changing their own email.
2020-02-20 09:52:21 +10:00
Robin Ward
041168c9b6 FIX: Group mentions were not being cooked the same was as previewed
If a group mention could be notified on preview it was given an `<a>`
tag with the `.notify` class. When cooked it would display differently.
This patch makes the server side cooking match the client preview.
2020-02-18 15:45:02 -05:00
romanrizzi
330d1ae9bc DEV: Correct test names 2020-02-17 11:47:18 -03:00
Neil Lalonde
c31039d51f FIX: tag topic counts wrong after adding synonyms
After adding a tag as a synonym of another tag,
both tags will have the wrong topic counts. It's
corrected within 12 hours by the EnsureDbConsistency
job. This fix ensures the topic counts are updated
much sooner.
2020-02-14 12:15:29 -05:00
Roman Rizzi
fadb2b7157
FEATURE: Users cannot reflag recently handled items using the same reason unless the post was edited, or it was reviewed more than 24 hours ago. (#8969) 2020-02-14 13:43:48 -03:00
Robin Ward
c2e58b6b85 FIX: Don't remove the topic image if posts don't have them 2020-02-13 14:00:30 -05:00
Gerhard Schlager
453bec9394 FEATURE: Add small action post to indicate forwarded email
This happens only when the sender of the email didn't write anything in their email.
2020-02-12 16:23:17 +01:00
Dan Ungureanu
ec40242b5c
FIX: Make inline oneboxes work with secured topics in secured contexts (#8895) 2020-02-12 12:11:28 +02:00
David Taylor
5919618a87
DEV: Drop legacy OpenID 2.0 support (#8894)
This is not used in core or official plugins, and has been printing a deprecation notice since v2.3.0beta4. All OpenID 2.0 code and dependencies have been dropped. The user_open_ids table remains for now, in case anyone has missed the deprecation notice, and needs to migrate their data.

Context at https://meta.discourse.org/t/-/113249
2020-02-07 17:32:35 +00:00
Vinoth Kannan
a0e0b1ef4b FIX: reset edit_reason in posts when creating a new version. 2020-02-07 09:40:16 +05:30
Penar Musaraj
49843f327e
FIX: Strip audio/video content from excerpt (#8881) 2020-02-06 15:08:13 -05:00
Penar Musaraj
4b6a47be48 DEV: do not persist force_custom_user_agent_hosts setting
Followup to f029e2
2020-02-06 11:56:54 -05:00
Penar Musaraj
0fd39cc511 FIX: Remove post/topic image_url on post edits
- resets image_url when image is removed from first post on edit
- excludes onebox icons from being featured as topic/post images
2020-02-06 11:23:08 -05:00
Penar Musaraj
f029e2eaf6 FEATURE: Add site setting for specific hosts using custom user agent when oneboxing
Followup to #00c406
2020-02-06 10:32:42 -05:00
Blake Erickson
9207c370d9 FIX: Spoiler logic should live inside of spoiler plugin
This commit removes logic about spoilers because it should live inside
of the discourse-spoiler-alert plugin.

This PR:

https://github.com/discourse/discourse-spoiler-alert/pull/38

also completely removes spoilers from excerpts in order to keep them
from leaking in topic previews and notifications.
2020-02-06 07:46:46 -07:00
Dan Ungureanu
c5e3faac00
FIX: Recovered posts with no user will be taken over by system user (#8834) 2020-02-06 10:19:04 +02:00
David Taylor
88779d849f
UX: Ignore name parameter from IDP when it is equal to email (#8869)
Some auth providers (e.g. Auth0 with default configuration) send the email address in the name field. In Discourse, the name field is made public, so this commit adds a safeguard to prevent emails being made public.
2020-02-05 16:03:18 +00:00
Régis Hanol
13d4b05963 FIX: allow both meta_data and custom_fields when creating a topic
For some reasons, we have two ways of associating "custom fields" to a new topic:
using 'meta_data' and 'custom_fields'.

However, if we were to provide both arguments, the 'meta_data' would be overwritten
by any 'custom_fields' provided.

This commit ensures we can use both and merges the 'custom_fields' with the 'meta_data'.
2020-02-05 10:40:22 +01:00
Blake Erickson
241d8f6452 FIX: Edit title respects min trust to edit post
This fix ensures that the site setting `post_edit_time_limit` does not
bypass the limit of the site setting `min_trust_to_edit_post`. This
prevents a bug where users that did not meet the minimum trust level to
edit could edit the title of topics.
2020-02-04 16:31:16 -07:00
Gerhard Schlager
e84d88ddea FIX: Email attachments with a size of 0 bytes caused error 2020-02-04 12:30:43 +01:00
Jarek Radosz
73b04976e5
FIX: Use updated_at in the S3 inventory job (#8823)
When we change upload's sha1 (e.g. when resizing images) it won't match the data in the most recent S3 inventory index. With this change the uploads that have been updated since the inventory has been generated are ignored.
2020-01-31 11:02:44 +01:00
Martin Brennan
8d77e99827
FIX: Stop encoding presigned URLs with UrlHelper (#8818)
When FinalDestination is given a URL it encodes it before doing anything else. however S3 presigned URLs should not be messed with in any way otherwise we can end up with 400 errors when downloading the URL e.g.

<Error><Code>InvalidToken</Code><Message>The provided token is malformed or otherwise invalid.</Message>

The signature of presigned URLs is very important and is automatically generated and should be preserved.
2020-01-31 09:09:34 +10:00
Dan Ungureanu
bf50087d72
Revert "FIX: Create post notices only for public posts (#8708)"
This reverts commit c2d051315d.
2020-01-30 14:26:08 +02:00
Arpit Jalan
62c21ba649 Remove bounce_score_threshold_deactivate setting.
Removed `bounce_score_threshold_deactivate` setting as the deactivate
threshold is not getting reached.
2020-01-30 16:17:31 +05:30
Martin Brennan
1150cd4621
FIX: Stop secure media URLs being censored too liberally in emails (#8817)
For example /t/ URLs were being replaced if they contained secure-media-uploads so if you made a topic called "Secure Media Uploads Are Cool" the View Topic link in the user notifications would be stripped out.

Refactored code so this secure URL detection happens in one place.
2020-01-30 16:19:14 +10:00
Neil Lalonde
34f564acd6 FIX: customized email subjects was ignored for some notifications
Email templates for notifications about new posts and topics weren't
using customized values.
2020-01-29 14:49:08 -05:00
Sam Saffron
7f3a30d79f FIX: blank cooked markdown could raise an exception in logs
Previously if somehow a user created a blank markdown document using tag
tricks (eg `<p></p><p></p><p></p><p></p><p></p><p></p>`) and so on, we would
completely strip the document down to blank on post process due to onebox
hack.

Needs a followup cause I am still unclear about the reason for empty p stripping
and it can cause some unclear cases when we re-cook posts.
2020-01-29 11:37:25 +11:00
Martin Brennan
ab3bda6cd0
FIX: Mitigate issue where legacy pre-secure hotlinked media would not be redownloaded (#8802)
Basically, say you had already downloaded a certain image from a certain URL
using pull_hotlinked_images and the onebox. The upload would be stored
by its sha as an upload record. Whenever you linked to the same URL again
in a post (e.g. in our case an og:image on review.discourse) we would
would reuse the original upload record because of the sha1.

However when you turned on secure media this could cause problems as
the first post that uses that upload after secure media is enabled
will set the access control post for the upload to the new post.
Then if the post is deleted every single onebox/link to that same image
URL will fail forever with 403 as the secure-media-uploads URL fails
if the access control post has been deleted.

To fix this when cooking posts and pulling hotlinked images, we only
allow using an original upload by URL if its access control post
matches the current post, and if the original_sha1 is filled in,
meaning it was uploaded AFTER secure media was enabled. otherwise
we just redownload the media again to be safe, as the URL will always
be new then.
2020-01-29 10:11:38 +10:00
David Taylor
c344f43211 UX: Admins should only see their own PMs when searching in:all
Admins are technically allowed to access all PMs, but it can be confusing to include them all in search. Follow-up to e0605029dc
2020-01-28 11:26:42 +00:00
adam j hartz
e0605029dc FEATURE: allow searching public topics and personal messages simultaneously (#8784)
The new search modifier `in:all` can be used to include both public and personal messages in the same search.

Co-authored-by: adam j hartz <hz@mit.edu>
2020-01-28 10:11:33 +00:00
Mark VanLandingham
20e3c0f386
FIX: Post reviser picking up edits for hidden posts (#8792) 2020-01-27 10:05:48 -06:00
Krzysztof Kotlarek
8cc09fc668
Revert "FIX: Redis fallback handler refactoring (#8771)" (#8776)
This reverts commit 4f677854d3.
2020-01-24 09:20:17 +11:00
Krzysztof Kotlarek
4f677854d3
FIX: Redis fallback handler refactoring (#8771)
* DEV: Add a fake Mutex that for concurrency testing with Fibers

* DEV: Support running in sleep order in concurrency tests

* FIX: A separate FallbackHandler should be used for each redis pair

This commit refactors the FallbackHandler and Connector:

 * There were two different ways to determine whether the redis master
   was up. There is now one way and it is the responsibility of the
   new RedisStatus class.

 * A background thread would be created whenever `verify_master` was
   called unless the thread already existed. The thread would
   periodically check the status of the redis master. However, checking
   that a thread is `alive?` is an ineffective way of determining
   whether it will continue to check the redis master in the future
   since the thread may be in the process of winding down.

   Now, this thread is created when the recorded master status goes from
   up to down. Since this thread runs the only part of the code that is
   able to bring the recorded status up again, we ensure that only one
   thread is probing the redis master at a time and that there is always
   a thread probing redis master when it is recorded as being down.

 * Each time the status of the redis master was checked periodically, it
   would spawn a new thread and immediately join on it. I assume this
   happened to isolate the check from the current execution, but since
   the join rethrows exceptions in the parent thread, this was not
   effective.

 * The logic for falling back was spread over the FallbackHandler and
   the Connector. The connector is now a dumb object that delegates
   responsibility for determining the status of redis to the
   FallbackHandler.

 * Previously, failing to connect to a master redis instance when it was
   not recorded as down would raise an exception. Now, this exception is
   passed to `Discourse.warn_exception` and the connection is made to
   the slave.

This commit introduces the FallbackHandlers singleton:

 * It is responsible for holding the set of FallbackHandlers.

 * It adds callbacks to the fallback handlers for when a redis master
   comes up or goes down. Main redis and message bus redis may exist on
   different or the same redis hosts and so these callbacks may all
   exist on the same FallbackHandler or on separate ones.

These objects are tested using fake concurrency provided by the
Concurrency module:

 * An `around(:each)` hook is used to cause each test to run inside a
   Scenario so that the test body, mocking cleanup and `after(:each)`
   callbacks are run in a different Fiber.

 * Therefore, holting the execution of the Execution abruptly (so that
   the fibers aren't run to completion), prevents the mocking cleaning
   and `after(:each)` callbacks from running. I have tried to prevent
   this by recovering from all exceptions during an Execution.

* FIX: Create frozen copies of passed in config where possible

* FIX: extract start_reset method and remove method used by tests

Co-authored-by: Daniel Waterworth <me@danielwaterworth.com>
2020-01-23 13:39:29 +11:00
Martin Brennan
1b3b0708c0
FEATURE: Update upload security status on post move, topic conversion, category change (#8731)
Add TopicUploadSecurityManager to handle post moves. When a post moves around or a topic changes between categories and public/private message status the uploads connected to posts in the topic need to have their secure status updated, depending on the security context the topic now lives in.
2020-01-23 12:01:10 +10:00
Martin Brennan
4646a38ae6
FIX: Use presigned URL to avoid 403 when pulling hotlinked images for secure media (#8764)
When we were pulling hotlinked images for oneboxes in the CookedPostProcessor, we were using the direct S3 URL, which returned a 403 error and thus did not set widths and heights of the images. We now cook the URL first based on whether the upload is secure before handing off to FastImage.
2020-01-23 09:31:46 +10:00
Mark VanLandingham
c5eec19368
FIX: Featuring topic on other users profile shows their topics (#8769) 2020-01-22 14:16:17 -06:00
Bianca Nenciu
7b7e1717f2
FIX: Quoting a quote preserves the original post information (#8746)
Let's say post #2 quotes post number #1. If a user decides to quote the
quote in post #2, it should keep the information of post #1
("user_1, post: 1, topic: X"), instead of replacing with current post
info ("user_2, post: 2, topic: X").
2020-01-22 16:10:23 +02:00
Dan Ungureanu
89bd7ba45f
FIX: Use new tag routes (#8683)
Commit 1fb7a62 added unambiguous routes for tags. This commit ensures
that the new routes are used.
2020-01-21 19:23:08 +02:00
Leo McArdle
8883cca373 enqueue spam/dmarc failing emails instead of hiding (#8674)
* enqueue spam/dmarc failing emails instead of hiding

* add translations for dmarc/spam enqueued reasons

* unescape quote

* if email_in_authserv_id is blank return gray for all emails
2020-01-21 11:12:00 -05:00
Dan Ungureanu
788ddcc407
FIX: Make topic query include topics from sub-sub-categories (#8709) 2020-01-20 17:06:58 +02:00
Gerhard Schlager
ab07b945c2
Merge pull request #8736 from gschlager/rename_reply_id_column
REFACTOR: Rename `post_replies.reply_id` column to `post_replies.reply_post_id`
2020-01-17 17:24:49 +01:00
Gerhard Schlager
eeb2855a48 DEV: Add spec to find MF locale for en_US
Follow-up to aecadcb267
2020-01-16 14:40:53 +01:00
Joffrey JAFFEUX
948bd00340
FEATURE: line with only 1 to 3 emojis will now display as large emojis 2020-01-16 09:54:26 +01:00
Martin Brennan
7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
Dan Ungureanu
c2d051315d FIX: Create post notices only for public posts (#8708)
This also ensures only public posts are considered when creating post
notices for new and returning users.
2020-01-15 11:40:19 +01:00
Martin Brennan
66f2db4ea4 SECURITY: 2FA with U2F / TOTP 2020-01-15 11:27:12 +01:00
Sam Saffron
e89935127c DEV: correct flaky spec
This ensures that the user object is created fresh for each example.

This is required for this particular spec as we can not risk having a stale
object, which can lead to a flaky spec.
2020-01-06 10:54:18 +11:00
Vinoth Kannan
9a6606dd30 DEV: Add option to keep quoted content in post excerpt. 2020-01-04 18:56:52 +05:30
David Taylor
45c5f56ffc
PERF: Reduce DB queries when serializing ignore/mute information (#8629)
* PERF: Cache ignored and muted user ids in the current_user object
* PERF: Avoid DB queries when checking ignore/mute permission in guardian
2020-01-02 13:04:08 +00:00
David Taylor
61919ad39f
UX: Do not use avatars as fallback opengraph images for replies (#8605)
People rarely want to have their avatars show up as the preview image on social media platforms. Instead, we should fall back to the site opengraph image.
2019-12-20 13:17:14 +00:00
Bianca Nenciu
1bccd8eca9
FIX: Remove full nested quotes on direct reply (#8581)
It used to check how many quotes were inside a post, without taking
considering that some quotes can contain other quotes. This commit
selects only top level quotes.

I had to use XPath because I could not find an equivalent CSS
selector.
2019-12-20 10:24:34 +02:00
Vinoth Kannan
3b7f5db5ba
FIX: parallel spec system needs a dedicated upload folder for each worker. (#8547) 2019-12-18 11:21:57 +05:30
Joffrey JAFFEUX
81b4de39ee
FIX: prevents crash in discourse_tagging with empty term (#8548) 2019-12-17 10:55:06 +01:00
Martin Brennan
6ab12ed96b
FIX: If a prettified slug is a number, return defaultt (#8554)
Meta thread: https://meta.discourse.org/t/sending-a-pm-with-the-following-title-causes-an-error/135654/3

We had an issue where if someone sent a PM with crazy
characters that are stripped and we end up with only
a number, the topic redirect errored because the slug was
a number. so instead we return the default as well if
the slug is a number after prettification
2019-12-17 10:34:20 +10:00
Martin Brennan
beb91e7eff
FIX: require: false for rotp gem (#8540)
The ROTP gem is only used in a very small amount of places in the app, we don't need to globally require it.

Also set the Addressable gem to not have a specific version range, as it has not been a problem yet.

Some slight refactoring of UserSecondFactor here too to use SecondFactorManager to avoid code repetition
2019-12-17 10:33:51 +10:00
Mark VanLandingham
8c4ffaea1b
FEATURE: Modal for profile featured topic & admin wrench refactor (#8545) 2019-12-16 08:41:34 -08:00
Bianca Nenciu
3ec2081059
UX: Include public groups in mentionable groups set (#8516) 2019-12-12 13:13:40 +02:00
Sam Saffron
9d6a9bba96 DEV: correct flaky test
This is a slight workaround which helps somewhat now but is pending a larger
fix.

When this spec ran in parallel mode uploads could start cross talking and
an upload you expect to be there may vanish.

This works around the issue by making the upload unique every time it is
created

It also folds up an expensive test into the main one.
2019-12-12 16:54:22 +11:00
Mark VanLandingham
09d9baa6d7
FIX: Update S3 stubs for more aws-sdk API changes (#8534) 2019-12-11 11:26:52 -08:00
dependabot-preview[bot]
b90a592146 DEV: Bump aws-sdk-sns from 1.13.0 to 1.21.0 (#8490)
Bumps [aws-sdk-sns](https://github.com/aws/aws-sdk-ruby) from 1.13.0 to 1.21.0.
- [Release notes](https://github.com/aws/aws-sdk-ruby/releases)
- [Changelog](https://github.com/aws/aws-sdk-ruby/blob/master/gems/aws-sdk-sns/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-ruby/compare/1.13.0...1.21.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-12-11 06:13:17 -08:00
Mark VanLandingham
14cb386f1e
FEATURE: Featured topic for user profile & card (#8461) 2019-12-09 11:15:47 -08:00
Dan Ungureanu
aa24be1a9a
DEV: Extend plugin API for uploads (#8440)
* DEV: Add API to alter uploads Markdown

* DEV: Extract data attributes from image / download Markdown

For example '[test|attachment|hello=world]' will generate an 'a' element
with a data attribute: 'data-hello=world'.

This commit also makes MarkdownIt to transform '|attachment' into
'class="attachment"'. This transformation used to be a part of the
process which resolves short URLs (i.e. upload://).

* DEV: Export imageNameFromFileName
2019-12-09 16:20:03 +02:00
Dan Ungureanu
ebe6fa95be
FIX: Optimize images in Onebox (#8471)
This commit ensures that images in Onebox are being optimized, but not
converted to lightbox too.
2019-12-09 15:39:25 +02:00
Sam Saffron
494fe335d3 DEV: allow handling crawler reqs with no user agent
Followup to e440ec25 we treat no user agent as crawler reqs.
2019-12-09 18:40:10 +11:00
Sam Saffron
e440ec2519 FIX: crawler requests not tracked for non UTF-8 user agents
Non UTF-8 user_agent requests were bypassing logging due to PG always
wanting UTF-8 strings.

This adds some conversion to ensure we are always dealing with UTF-8
2019-12-09 17:43:51 +11:00
Jarek Radosz
067c4a7c10 DEV: Fix CookedPostProcessor specs
Broken in #8480
2019-12-09 03:16:25 +01:00
Gerhard Schlager
c88797bf0e FIX: Ignore DMARC for emails sent to mailing list mirror 2019-12-06 13:29:39 +01:00
Jarek Radosz
d07f039468 FIX: Secure Upload URLs in lightbox (#8451)
This fixes the following issues:

* The link element on the lightbox which pops open the lightbox was linking to the S3 URL with a private ACL instead of the secure media URL for the image
* Change to use `@post.with_secure_media?` in `CookedPostProcessor` for URL cooking, as in some cases, like when a post is edited and an upload is added, `upload.secure?` can be false which resulted in `srcset` URLs not being cooked correctly to secure media upload urls.
2019-12-05 09:13:09 +10:00
Neil Lalonde
875f0d8fd8
FEATURE: Tag synonyms
This feature adds the ability to define synonyms for tags, and the ability to merge one tag into another while keeping it as a synonym. For example, tags named "js" and "java-script" can be synonyms of "javascript". When searching and creating topics using synonyms, they will be mapped to the base tag.

Along with this change is a new UI found on each tag's page (for example, `/tags/javascript`) where more information about the tag can be shown. It will list the synonyms, which categories it's restricted to (if any), and which tag groups it belongs to (if tag group names are public on the `/tags` page by enabling the "tags listed by group" setting). Staff users will be able to manage tags in this UI, merge tags, and add/remove synonyms.
2019-12-04 13:33:51 -05:00
Robin Ward
888d56774a
DEV: HTML Builders should respect if a plugin is enabled or not (#8454)
Previously they would return the HTML regardless of whether the plugin
was enabled or not.
2019-12-04 12:26:23 -05:00
David Taylor
be1510b09d FIX: Use filtered posts when determining the next page
This bug was causing some unusual behavior when the last post is filtered (e.g. from an ignored user). In some situations this would cause suggested topics to be omitted from the payload.

The next_page specs have been updated to remove most of the stubs
2019-12-04 12:52:24 +00:00
Joffrey JAFFEUX
0d3d2c43a0
DEV: s/\$redis/Discourse\.redis (#8431)
This commit also adds a rubocop rule to prevent global variables.
2019-12-03 10:05:53 +01:00
Arpit Jalan
016732cced FEATURE: improve suspect user discovery 2019-12-02 16:39:28 +05:30
David Taylor
a6aada16bd DEV: Refactor API key specs to avoid hard-coding keys
By hard-coding keys, we are not testing the API key system end to end. This change also makes the specs more resilient to upcoming API key changes
2019-11-29 15:16:22 +00:00
Dan Ungureanu
1e0c2235a3
FIX: Optimize quoted images (#8427)
Only images that were part of a lightbox used to be optimized. This
patch ensures that quoted images are also optimized.
2019-11-29 15:18:42 +02:00
Krzysztof Kotlarek
b120728999
FEATURE: Ability to add components to all themes (#8404)
* FEATURE: Ability to add components to all themes

This is the first and functional step from that topic https://dev.discourse.org/t/adding-a-theme-component-is-too-much-work/15398/16

The idea here is that when a new component is added, the user can easily assign it to all themes (parents).

To achieve that, I needed to change a site-setting component to accept `setDefaultValues` action and `setDefaultValuesLabel` translated label.
Also, I needed to add `allowAny` option to disable that for theme selector.

I also refactored backend to accept both parent and child ids with one method to avoid duplication (Renamed `add_child_theme!` to more general `add_relative_theme!`)

* FIX: Improvement after code review

* FIX: Improvement after code review2

* FIX: use mapBy and filterBy directly
2019-11-28 16:19:01 +11:00
Martin Brennan
901054fd75
FIX: Cache failed onebox URL request server-side (#8421)
We already cache failed onebox URL requests client-side, we now want to cache this on the server-side for extra protection. failed onebox previews will be cached for 1 hour, and any more requests for that URL will fail with a 404 status. Forcing a rebake via the Rebake HTML action will delete the failed URL cache (like how the oneboxer preview cache is deleted).
2019-11-28 07:48:29 +10:00
Gerhard Schlager
d12f2580de FIX: Serve crawler view to Google PageSpeed 2019-11-27 22:15:34 +01:00
Roman Rizzi
1c9d18f094
FIX: Correctly ignore/approve replies when acting on a flagged post (#8425) 2019-11-27 17:19:44 -03:00
David Taylor
a227083c1c
FEATURE: Limit the number of active sessions for a user (#8411)
If a user has more than 60 active sessions, the oldest sessions will be terminated automatically. This protects performance when logging in and when loading the list of recently used devices.
2019-11-27 12:39:31 +00:00
Sam Saffron
88ecb650a9 DEV: Implement a faster Discourse.cache
This is a bottom up rewrite of Discourse cache to support faster performance
and a limited surface area.

ActiveSupport::Cache::Store accepts many options we do not use, this partial
implementation only picks the bits out that we do use and want to support.

Additionally params are named which avoids typos such as "expires_at" vs "expires_in"

This also moves a few spots in Discourse to use Discourse.cache over setex
Performance of setex and Discourse.cache.write is similar.
2019-11-27 16:11:49 +11:00
Sam Saffron
0fb497eb23 DEV: use Discourse.cache over Rails.cache
Discourse.cache is a more consistent method to use and offers clean fallback
if you are skipping redis

This is part of a larger change that both optimizes Discoruse.cache and omits
use of setex on $redis in favor of consistently using discourse cache

Bench does reveal that use of Rails.cache and Discourse.cache is 1.25x slower
than redis.setex / get so a re-implementation will follow prior to porting
2019-11-27 12:36:19 +11:00
Leo McArdle
2714149fd2 FEATURE: hide posts from incoming email based on dmarc verdict (#8333) 2019-11-26 15:55:22 +01:00
Sam Saffron
ebc82eb63e DEV: correct spec failures in PG 12
PG 12 changes internals in a subtle way, time jitter is noticed in a few new
spots (which is normal) and default ordering is a bit different which is meant
to be random anyway.
2019-11-26 16:39:14 +11:00
Dan Ungureanu
a992caf741
DEV: Replace magic values (#8398)
Follow-up to 35942f7c7c.
2019-11-25 14:32:19 +02:00
Martin Brennan
23714e77c4
FIX: Always return secure_proxy_without_cdn url for secure media (#8394)
There was an issue on dev where when uploading secure media, the href of the media was correctly being replaced in the CookedPostProcessor, but the srcset urls were not being replaced correctly. This is because UrlHelper.cook_url was returning the asset host URL for the media for secure media instead of returning early with the proxied secure proxy url.
2019-11-22 15:29:31 +10:00
Penar Musaraj
11d22293fb FIX: Allow private media uploads to be reused in login_required sites
In non-login-required sites, we prevent secure uploads already used in PMs from being used in public topics.

In login_required sites, secure uploads should be reusable in any topic, PM or not.
2019-11-21 09:14:06 -05:00
Sam Saffron
423ad5f0a4 FIX: do not log if an invalid mime type is passed to app
Previously our custom exception handler was unable to handle situations
where an invalid mime type was sent, resulting in a warning log

This ensures we pretend a request is HTML for the purpose of rendering
the error page if an invalid mime type from a scanner is shipped to the app
2019-11-21 15:51:34 +11:00
Sam Saffron
dd89a13e22 DEV: disable flaky spec
This is erratically failing in our production CI
2019-11-20 08:56:07 +11:00