Commit Graph

6975 Commits

Author SHA1 Message Date
Vinoth Kannan
e4fe864c0b FIX: use upload's cdn url in composer preview if available. 2019-10-14 08:47:15 +05:30
Roman Rizzi
01bc465db8
DEV: Split max decompressed setting for themes and backups (#8179) 2019-10-11 14:38:10 -03:00
Rafael dos Santos Silva
76ab0350f1
FIX: Properly encoded slugs when configured to (#8158)
When an admin changes the site setting slug_generation_method to
encoded, we weren't really encoding the slug, but just allowing non-ascii
characters in the slug (unicode).

That brings problems when a user posts a link to topic without the slug, as
our topic controller tries to redirect the user to the correct URL that contains
the slug with unicode characters. Having unicode in the Location header in a
response is a RFC violation and some browsers end up in a redirection loop.

Bug report: https://meta.discourse.org/t/-/125371?u=falco

This commit also checks if a site uses encoded slugs and clear all saved slugs
in the db so they can be regenerated using an onceoff job.
2019-10-11 12:38:16 -03:00
Régis Hanol
9a81cb9e55 FIX: ensure we remove tempfiles from disk when creating an upload
Follow-up to 46d12c5ad3
2019-10-11 11:13:10 +02:00
David Taylor
67787799bb DEV: Allow plugins to mark user custom fields as editable only by staff
This adds a staff_only parameter to the register_editable_user_custom_field API. The default is false, to maintain backwards compatibility.
2019-10-11 09:57:55 +01:00
Nacho Caballero
7d2f5240d9 FIX: Show a correct diff when editing consecutive paragraphs (#8177) 2019-10-11 03:50:37 -04:00
Neil Lalonde
09021c35a3 Version bump to v2.4.0.beta6 2019-10-10 11:46:28 -04:00
romanrizzi
fa66290b61 FIX: Don't fail when there's no directory to strip 2019-10-10 11:21:36 -03:00
Joffrey JAFFEUX
c5326682d6 DEV: improves speed of live reload css in core and plugins (#8161)
This corrects live refresh of CSS when working on plugins (and core) in many cases live refresh was not working.
2019-10-10 16:10:23 +11:00
Sam Saffron
5aaf7e3316 FIX: during concurrent emails generation renderer should not be reused
Our instance used for template rendering needs a lock to ensure there is
no race condition where rendering happens on 2 threads at the same time.

This can lead to local poisoning which can cause unexpected results in
emails
2019-10-10 08:50:48 +11:00
Roman Rizzi
5357ab3324
SECURITY: Safely decompress backups when restoring. (#8166)
* SECURITY: Safely decompress backups when restoring.

* Fix tests and update theme_controller_spec to work with zip files instead of .tar.gz
2019-10-09 11:41:16 -03:00
Mark VanLandingham
9b4aba0d39
DEV: support --fail-fast in bin/turbo_rspec (#8170)
* [WIP] - default turbo spec env to test

* FEATURE: support for --fast-fail in bin/turbo_rspec

* fast-fail -> fail_fast to match rspec

* Moved thread killing outside of fail-fast check

* Removed failure_count incrementation from fast_fail_met
2019-10-09 09:40:06 -05:00
Roman Rizzi
10478cb395
DEV: Bump uglifyjs (#7834)
* Rewrite uglifyjs command to work with 3.x

* Use ES5 syntax in plain JS files

* Use the older command if uglifyJS V2.x is installed
2019-10-09 10:02:49 -03:00
Régis Hanol
349c1cd085 FIX: remove site setting 'shadowed-by-global' option (#8061) 2019-10-08 12:43:26 -04:00
Dan Ungureanu
fdb1d3404c
FEATURE: Add site setting to show more detailed 404 errors. (#8014)
If the setting is turned on, then the user will receive information
about the subject: if it was deleted or requires some special access to
a group (only if the group is public). Otherwise, the user will receive
a generic #404 error message. For now, this change affects only the
topics and categories controller.

This commit also tries to refactor some of the code related to error
handling. To make error pages more consistent (design-wise), the actual
error page will be rendered server-side.
2019-10-08 14:15:08 +03:00
David Taylor
d2bceff133
FEATURE: Use full page redirection for all external auth methods (#8092)
Using popups is becoming increasingly rare. Full page redirects are already used on mobile, and for some providers. This commit removes all logic related to popup authentication, leaving only the full page redirect method.

For more info, see https://meta.discourse.org/t/do-we-need-popups-for-login/127988
2019-10-08 12:10:43 +01:00
Krzysztof Kotlarek
3f9673f23c FIX: solution for pending migrations for bin/turbo_rspec
Currently, if you try to run `./bin/turbo_rspec` you will got that error `There are pending migrations, run rake parallel:migrate`

Reason for that is that command is running in `development` mode which includes plugins migration files in ActiveRecord::Migrator.migrations_paths:
```
["db/migrate",
 "/home/lis2/projects/discourse/plugins/discourse-details/db/migrate",
 "/home/lis2/projects/discourse/plugins/discourse-details/db/post_migrate",
 "/home/lis2/projects/discourse/plugins/discourse-local-dates/db/migrate",
 "/home/lis2/projects/discourse/plugins/discourse-local-dates/db/post_migrate",
...
]
```

A workaround solution would be to run the command with the TEST environment like `RAILS_ENV=test ./bin/turbo_rspec`

I want to propose in this PR to override migration_paths to check only Discourse core migrations.
2019-10-08 10:32:40 +00:00
Sam Saffron
586dfcc795 PERF: avoid regex for hot path
Camelize is called quite a lot in zeitwerk, avoid using a regex here which
is far slower than using ends_with?
2019-10-08 13:43:54 +11:00
Sam Saffron
8d5f47dded PREF: optimise preloading application
We preload to ensure as much memory as possible is reused from unicorn master
to various workers using copy-on-write (sidekiq, unicorn)

This migrates the preloading code into the Discourse module for easier
reuse and adds 3 notable preloading changes

1. We attempt to localize a string on each site, ensuring we warmup
the i18n

2. We preload all our templates (compiling .erb to class)

3. We warm-up our search tokenizer which uses cppjieba which is a large
memory consumer, this will only cause a warmup on CJK sites or sites with
the special site setting enabled.
2019-10-07 00:33:37 -04:00
Sam Saffron
71ea4ad7fc PERF: reuse renderer when rendering email templates
Previous to this fix we were leaking methods on the internal action view
template class per render.

This caused email generation to be very low and a steady memory leak in the
application in sidekiq when sending out emails

The behavior change is new to Rails 6 so this fix does not need to be
backported into stable.
2019-10-06 23:57:03 -04:00
Jarek Radosz
bfe0bc0cbd
FIX: Check for category conflicts in SiteSetting validations (#8137)
It was possible to add a category to more than one default group, e.g. "default categories muted" and "default categories watching first post".

The bug was caused by category validations inadvertently comparing strings and numbers.
2019-10-06 20:50:07 +02:00
Jarek Radosz
87be6fe98c
DEV: Add a plugin incompatibility message (#8151)
* DEV: Add a plugin incompatibility message

* Extract the plugin_initialization_guard
2019-10-06 20:47:33 +02:00
Joffrey JAFFEUX
e23b9051ff
DEV: restart watcher after css change notifier crashed (#8157) 2019-10-04 13:20:51 -04:00
Régis Hanol
46d12c5ad3 FIX: properly downsize image on upload
Overwriting the same file with 'convert' is not always working as expected.
Adding a temporary file as the destination of the downsize makes this operation much more reliable.

Also switched to using (the more aggressive) 50% resize instead of halving the number of pixels.
2019-10-04 16:22:57 +02:00
Sam Saffron
6471fbd453 FIX: maxminddb:get task no longer working
Following our zeitwerk upgrade we needed to be more explicit about the
require
2019-10-04 09:47:33 +10:00
Robin Ward
a404cab509 The generated plugin should also freeze strings 2019-10-03 11:43:36 -04:00
Robin Ward
32a11a93ac Update the template for plugin generator to be rubocop compliant 2019-10-03 11:41:53 -04:00
Roman Rizzi
10565e4623
SECURITY: Safely decompress files. (#8124)
* FEATURE: Adds an extra protection layer when decompressing files.

* Rename exporter/importer to zip importer. Update old locale

* Added a new composite class to decompress a file with multiple strategies

* Set max file size inside a site setting

* Ensure that file is deleted after compression

* Sanitize path and files before compressing/decompressing
2019-10-03 10:19:35 -03:00
Justin DiRose
63fabdb6f2 FIX: first post true if user creates topic first (#8139)
Reported here: https://meta.discourse.org/t/user-was-banned-for-posting-a-reply-within-3-seconds/128823/12
The problem here is the user could create a topic, then reply and get
silenced on the second time (though technically their first post) for being below the
min_first_post_typing_time threshold.
2019-10-02 15:51:40 -04:00
Daniel Waterworth
1fdba2c5b2 FIX: Harden DistributedMutex
Threadsafety

  Since we use the same redis connection in multiple threads, a rogue
  transaction in another thread can trample the connection state
  (watched keys) that we need to acquire and release the lock properly.

  This is fixed by preventing other threads from using the connection
  when we are performing these actions.

Off-by-one error

  A distributed mutex is now consistently determined to be expired if
  the current time is strictly greater than the expire time.

Unwatch before transaction

  Since the redis connection is used by so much of the code, it is
  difficult to ensure that any watched keys have been cleared. In order
  to defend against this rogue connection state, an unwatch has been
  added before locking and unlocking.

Logging

  Hopefully this log message is more clear.
2019-10-02 13:00:41 +00:00
Michael Brown
dbe0111822 FEATURE: allow UploadRecovery to be run on a single post (#8094) 2019-10-02 14:57:36 +10:00
Krzysztof Kotlarek
35b1185a08 FIX: Revert Demon::DemonBase back to Demon::Base (#8132)
I introduced DemonBase because I had got some conflict between `demon/base.rb` and `jobs/base.rb`, however, to not rename base class, it is possible to use regex on absolute path in Zeitwerk custom inflector.
2019-10-02 14:54:08 +10:00
Sam Saffron
55ee9abecb DEV: clean up dependencies in spec
Follow up on zeitwork we needed to be a bit more explicit about a few
dependencies internally.

On certain orders the test suite could fail.
2019-10-02 14:50:54 +10:00
Sam Saffron
3f6af54fe1 DEV: minor fixes related to zeitwerk
A couple of specs stopped working post zeitwerk. Adding missing require
corrected inheritance to avoid a circular ref.
2019-10-02 14:28:18 +10:00
Krzysztof Kotlarek
427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Martin Brennan
68d35b14f4 FEATURE: Webauthn authenticator management with 2FA login (Security Keys) (#8099)
Adds 2 factor authentication method via second factor security keys over [web authn](https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API).

Allows a user to authenticate a second factor on login, login-via-email, admin-login, and change password routes. Adds registration area within existing user second factor preferences to register multiple security keys. Supports both external (yubikey) and built-in (macOS/android fingerprint readers).
2019-10-01 19:08:41 -07:00
Joffrey JAFFEUX
e5af03be01 DEV: prevents attemps to redefine THEME_TARGETS constant (#8121) 2019-10-02 10:54:31 +10:00
Krzysztof Kotlarek
f331b5eab2 FEATURE: topic title is validated for blocked words (#8127)
Currently, the topic is only validated for censored words and should be validated for blocked words as well.

Blocked word validation is now used by both Post and Topic. To avoid code duplication, I extracted blocked words validation code into separate Validator, and use it in both places.

The only downside is that even if the topic contains blocked words validation message is saying "Your post contains a word that's not allowed: tomato" but I think this is descriptive enough.
2019-10-02 10:38:34 +10:00
Neil Lalonde
279a94da90 Version bump to v2.4.0.beta5 2019-10-01 16:51:36 -04:00
Gerhard Schlager
9e4fb262cf FIX: Respect unicode whitelist when suggesting username 2019-10-01 20:33:09 +02:00
Joffrey JAFFEUX
cb8fa46970
DEV: prevents populate task to crash (#8111)
Generated emails/names/sentences were crashing with the following error:

```
Can not transliterate strings with ASCII-8BIT encoding
```
2019-09-22 05:18:48 -07:00
Daniel Waterworth
563253e9ed FIX: Fix options given to per-minute rate limiter
Previously the options for the per-minute and per-10-second rate
limiters were the same.
2019-09-20 10:48:59 +01:00
Robin Ward
3c6a5836c2 FIX: Sensitivity did not work by default
Forums without previously calculated scores would return the same values
for low/medium/high sensitivity. Now those are scaled based on the
default value.

The default value has also been changed from 10.0 to 12.5 based on
observing data from live discourse forums.
2019-09-19 13:26:17 -04:00
Penar Musaraj
c6cfbebf1f
FIX: ignore min_trust_to_send_messages when messaging groups (#8104)
This means that TL0 users can message groups with "Who can message this
group?" set to "Everyone".

It also means that members of a group with "Who can message this
group?" set to "members, moderators and admins" can also message the
group, even when their trust level is below min_trust_to_send_messages.
2019-09-18 15:23:13 -04:00
Robin Ward
7ae071282a FIX: Only apply post hide logic to flag actions 2019-09-18 09:39:09 -04:00
David Taylor
479fdaaea1
DEV: Allow specifying button class in reviewable action definitions (#8093)
This avoids the need for using `@extend` in SCSS, which can be problematic in plugins

For context, see https://review.discourse.org/t/fix-make-compatible-with-debundled-plugin-css-assets-feature/5297/7
2019-09-18 11:28:59 +01:00
Sam Saffron
1ca257be79 DEV: db:migrate no longer works after db:schema:load
In Rails 6 due to internal changes, the following sequence no longer works:

```
RAILS_ENV=test bin/rake db:migrate
RAILS_ENV=test bin/rake db:schema:dump
dropdb discourse_test
createdb discourse_test
RAILS_ENV=test bin/rake db:schema:load
RAILS_ENV=test bin/rake db:migrate
```

What appears to be happening is that our tracking of plugin migrations is
being missed on schema:dump or load.

A more comprehensive fix restoring schema:dump / load support will be
investigated.
2019-09-18 13:17:49 +10:00
Penar Musaraj
3debdc8131 SECURITY: XSS when oneboxing user profile location field
The XSS here is only possible if CSP is disabled. Low impact since CSP is enabled by default in SiteSettings.
2019-09-17 16:12:50 -04:00
Robin Ward
c3bbf643b1 FIX: Put back the TL3 -> TL0 spam thing
We talked about it and decided it's still relevant in the score world.
2019-09-17 16:04:46 -04:00
Robin Ward
4cd620e36e Remove special cases for flagging
Prior to the new review queue there were a couple special cases where
posts would be auto hidden:

* If a TL3 or above flagged a TL0 post as spam
* If a TL4 or above flagged a non-staff, non-TL4 post as spam, inappropriate or off
topic.

These cases are now removed in favour of the scoring system.
2019-09-17 13:44:15 -04:00
David Taylor
e1e8cac58f FIX: Correct theme SCSS error handling 2019-09-17 10:20:32 +01:00
David Taylor
e74f851728 FIX: Live reload plugin stylesheets when editing in development 2019-09-17 09:54:59 +01:00
David Taylor
3da9b99dbf FIX: Live reload plugin stylesheets when the color scheme changes 2019-09-17 09:54:55 +01:00
David Taylor
081c36a459 FIX: Do not include theme variables in plugin SCSS, and fix register_css 2019-09-17 09:54:52 +01:00
Sam Saffron
445d305154 DEV: initial migration can fail
db:migrate can issue translations due to module loading localizing
end user messages

This allows db:migrate to work even when db is blank
2019-09-17 13:38:01 +10:00
Sam Saffron
025d4ee91f FIX: Rails 6 multisite migrations and plugin migrations
Prior to this change plugin migrations were not working and multisite
migrations not working.

Rails internals changed so we need to account for it.

Specifically semantics of `db:migrate` in rails changed so it is sort of
a "multisite:migrate".
2019-09-13 09:17:54 +10:00
Roman Rizzi
568232052e
DEV: Remove FlagQuery class and old code (#8064) 2019-09-12 13:21:33 -03:00
David Taylor
67a98946b8 FIX: Do not log 'pull_hotlinked_images' edits in the staff action log 2019-09-12 15:55:45 +01:00
Sam Saffron
082f59842d DEV: bind for thread local vars should yield block
followup on #015051ec without this improvement monkey patch does not work.

bind should yield the block it is passed for it to work.
2019-09-12 17:57:57 +10:00
Sam Saffron
015051ecaf PERF: avoid spinning a thread each time we close a connection
This is a temporary workaround for the issue in https://github.com/rails/rails/pull/36949

Discussing a proper fix in Rails with the Rails team.

Prior to this fix we were spinning up a thread every time we closed a connection
to the db.
2019-09-12 17:34:04 +10:00
Krzysztof Kotlarek
32b8a2ccff DEV: Upgrade Discourse to Rails 6 (#8083)
* Adjustments to pass specs on Rails 6.0.0
* Use classic autoloader instead of Zeitwerk
* Update Rails 6.0.0 deprecated methods
* Rails 6.0.0 not allowing column with integer name
* Drop freedom_patches/rails6.rb
* Default value for trigger_transactional_callbacks? is true
* Bump rspec-rails version to 4.0.0.beta2
2019-09-12 10:41:50 +10:00
Gerhard Schlager
56c6e8e961 FIX: POP3 doesn't work with TLS 1.3 2019-09-11 18:43:19 +02:00
Michael Brown
503a11cc88 FIX: inline_uploads and subfolder (#8076)
* FIX: inline_uploads and subfolder

* if subfolder, also look for images with a path containing
  cdn_url + relative_url_root

* FIX: migrate_to_s3 task and subfolder
2019-09-11 11:50:48 +10:00
Bianca Nenciu
0d22beb81d
FIX: Improve Onebox detection (#8019)
Follow-up to 7c83d2eeb2.
2019-09-10 13:59:48 +03:00
Kyle Zhao
9b10a78d82 FEATURE: Quick access panels in user menu (#8073)
* Extract QuickAccessPanel from UserNotifications.

* FEATURE: Quick access panels in user menu.

This feature adds quick access panels for bookmarks and personal
messages. It allows uses to browse recent items directly in the user
menu, without being redirected to the full pages.

* REFACTOR: Use QuickAccessItem for messages.

Reusing `DefaultNotificationItem` feels nice but it actually requires a
lot of extra work that is not needed for a quick access item.

Also, `DefaultNotificationItem` shows an incorrect tooptip ("unread
private message"), and it is not trivial to remove / override that.

* Use a plain JS object instead.

An Ember object was required when `DefaultNotificationItem` was used.

* Prefix instead suffix `_` for private helpers.

* Set to null instead of deleting object keys.

JavaScript engines can optimize object property access based on the
object’s shape. https://mathiasbynens.be/notes/shapes-ics

* Change trivial try/catch to one-liners.

* Return the promise in case needs to be waited on.

* Refactor showAll to a link with href

* Store `emptyStatePlaceholderItemText` in state.

* Store items in Session singleton instead.

We can drop `staleItems` (and `findStaleItems`) altogether. Because
`(old) items === staleItems` when switching back to a quick access
panel.

* Add `limit` parameter to the `user_actions` API.

* Explicitly import Session instead.
2019-09-09 11:03:57 -04:00
Robin Ward
800e49f16e Version bump to 2.4.0.beta4 2019-09-06 16:08:03 -04:00
Gerhard Schlager
631315624d FIX: Topics with muted tag didn't show up when filtering by category and tag
It also removes the redundant `filter` parameter. Previously URLs looked like this:

```
http://example.com/tags/c/some-category/muted-tag/l/latest.json?filter=tags/c/some-category/muted-tag/l/latest
```

But it looks like the `filter` parameter was only used to find out if topics with a muted tag should be removed or not. But the same thing can be accomplished by using the first tag ID. The following URL looks a lot cleaner.

```
http://example.com/tags/c/some-category/muted-tag/l/latest.json
```
2019-09-06 20:38:03 +02:00
Rimian Perkins
6bbd83067d FEATURE: New post editing period for >= tl2 users (#8070)
* FEATURE: Add tl2 threshold for editing new posts

* Adds a new setting and for tl2 editing posts (30 days same as old value)
* Sets the tl0/tl1 editing period as 1 day

* FIX: Spec uses wrong setting

* Fix site setting on guardian spec

* FIX: post editing period specs

* Avoid shared examples
* Use update_columns to avoid callbacks on user during tests
2019-09-06 07:44:12 -04:00
Neil Lalonde
e90636eadc Version bump to v2.4.0.beta3 2019-09-04 11:16:01 -04:00
Gerhard Schlager
874985db71 DEV: Ignore duplicate prefixes when generating release notes 2019-09-04 15:45:48 +02:00
Sam Saffron
ed00f35306 FEATURE: improve performance of anonymous cache
This commit introduces 2 features:

1. DISCOURSE_COMPRESS_ANON_CACHE (true|false, default false): this allows
you to optionally compress the anon cache body entries in Redis, can be
useful for high load sites with Redis that lives on a separate server to
to webs

2. DISCOURSE_ANON_CACHE_STORE_THRESHOLD (default 2), only pop entries into
redis if we observe them more than N times. This avoids situations where
a crawler can walk a big pile of topics and store them all in Redis never
to be used. Our default anon cache time for topics is only 60 seconds. Anon
cache is in place to avoid the "slashdot" effect where a single topic is
hit by 100s of people in one minute.
2019-09-04 17:18:32 +10:00
Sam Saffron
dc9110cc43 FEATURE: track date api key was last used
Start tracking the date an api key was last used. This has already been
the case for user_api_keys.

This information can provide us with the ability to automatically expire
unused api keys after N days.
2019-09-03 18:11:30 +10:00
Sam Saffron
b9954b53bb FIX: report cached controller and action to loggers
Previously we would treat all cached hits in anon cache as "other"

This hinders analysis of cache performance and makes logging inaccurate
2019-09-03 10:55:16 +10:00
Sam Saffron
08743e8ac0 FEATURE: anon cache reports data to loggers
This allows custom plugins such as prometheus exporter to log how many
requests are stored in the anon cache vs used by the anon cache.

This metric allows us to fine tune cache behaviors
2019-09-02 18:45:35 +10:00
Daniel Waterworth
afeb7e4b55 DEV: Use recorded runtimes in turbo_rspec when running the whole test suite 2019-09-02 07:25:41 +01:00
Régis Hanol
75eebc904e FEATURE: new 'Discourse-Render' HTTP header 2019-08-30 20:45:18 +02:00
Roman Rizzi
e515324afa
Feature/Fix: Flagged posts user notifications (#8041)
* FIX: User should get notified when a post is deleted

* FEATURE: Notify posters when restoring flagged posts

* Fix typo

Co-Authored-By: Régis Hanol <regis@hanol.fr>

* Improve tests
2019-08-30 10:27:52 -03:00
David Taylor
e2449f9f23 Revert "Revert "Revert "FIX: Heartbeat check per sidekiq process (#7873)"""
This reverts commit c3497559be.
2019-08-30 11:26:16 +01:00
David Taylor
be96c4478e
FEATURE: Login with Discord (#8053)
This migrates the functionality of discourse-plugin-discord-auth into core. 

The plugin will automatically disable itself when core is updated: fd0867844d

For setup instructions, visit https://meta.discourse.org/t/configuring-discord-login-for-discourse/127129
2019-08-30 10:54:19 +01:00
Sam Saffron
e9cf0de588 FEATURE: do not replace &harr; with an emoji
&harr; technically can be an emoji, but usually will not be used in that way
2019-08-30 15:06:23 +10:00
Sam Saffron
c3497559be Revert "Revert "FIX: Heartbeat check per sidekiq process (#7873)""
This reverts commit e805d44965.
We now have mechanisms in place to ensure heartbeat will always
be scheduled even if the scheduler is overloaded per: 098f938b
2019-08-30 10:12:10 +10:00
Roman Rizzi
ebb389ef8a
UX: Read indicator improvements. (#8049)
* The read indicator now shows up when no member has read the last post of the topic (written by a non-member)
* The read indicator works on mobile and receives live updates from message bus
* The icon we display in the topic list was changed
* Added a title to the indicator to indicate its purpose when hovering over it
2019-08-29 12:03:43 -03:00
romanrizzi
1e89939383 DEV: Added wrapper to signal that NewPostManager#add_plugin_payload_attribute is part of the public API 2019-08-29 11:58:19 -03:00
Daniel Waterworth
6b9784cf8a FIX: Made turbo_rspec display errors in shared groups correctly 2019-08-29 12:41:14 +01:00
Daniel Waterworth
15c02c03c7 DEV: Split out multisite tests in bin/turbo_rspec
* A new process is started that just runs the multisite tests
 * The other processes are instructed to exclude the multisite tests
2019-08-29 11:47:58 +01:00
Sam Saffron
0d5d478146 PERF: avoid filtering shared drafts when not used
In some very specific cases (large sites) shared drafts can introduce a
performance hit due to the mechanism used to filter out topics

This avoids the entire process when shared drafts are not enabled
2019-08-29 11:37:20 +10:00
Régis Hanol
00dbc260d3 FIX: ensure avatar sizes are integers 2019-08-27 17:03:20 +02:00
Roman Rizzi
7c741fa0d6
FEATURE: Publish read state on group messages. (Originally introduced in #7989) (#8025)
* Revert "Revert "FEATURE: Publish read state on group messages. (#7989) [Undo revert] (#8024)""

This reverts commit 36425eb9f0.

* Fix: Show who read only if the attribute is enabled

* PERF: Precalculate the last post  readed by a group member

* Use book-reader icon instear of far-eye

* FIX: update topic groups correctly

* DEV: Tidy up read indicator update on write
2019-08-27 09:09:00 -03:00
OsamaSayegh
e805d44965 Revert "FIX: Heartbeat check per sidekiq process (#7873)"
This reverts commit 340855da55.
2019-08-27 11:56:23 +00:00
Gerhard Schlager
aecadcb267 FIX: Overriding _MF texts didn't work for en_US 2019-08-27 13:17:07 +02:00
Sam Saffron
bd5fa1737d FIX: add_to_serializer not correctly accounting for inheritance chains
This is a very long standing bug we had, if a plugin attempted to amend a
serializer core was not "correcting" the situation for all descendant classes
this often only showed up in production cause production eager loads serializers
prior to plugins amending them.

This is a critical fix for various plugins
2019-08-27 18:21:53 +10:00
Osama Sayegh
340855da55
FIX: Heartbeat check per sidekiq process (#7873)
* FIX: Heartbeat check per sidekiq process

* Rename method

* Remove heartbeat queues of previous bootups

* Regis feedback

* Refactor before_start

* Update lib/demon/sidekiq.rb

Co-Authored-By: Régis Hanol <regis@hanol.fr>

* Update lib/demon/sidekiq.rb

Co-Authored-By: Régis Hanol <regis@hanol.fr>

* Expire redis keys after 3600 seconds

* Don't use redis to store the list of queues
2019-08-26 09:33:49 +03:00
Sam Saffron
8cea78c833 Revert "FEATURE: Protect against replay attacks when using TLS 1.3 0-RTT (#8020)"
This reverts commit 39c31a3d76.

Sorry about this, we have decided againse supporting 0-RTT directly in
core, this can be supported with similar hacks to this commit in a
plugin.

That said, we recommend against using a 0-RTT proxy for the Discourse
app due to inherit risk of replay attacks.
2019-08-26 08:56:49 +10:00
Rafael dos Santos Silva
39c31a3d76
FEATURE: Protect against replay attacks when using TLS 1.3 0-RTT (#8020) 2019-08-23 11:52:47 -03:00
Osama Sayegh
68708db721 DEV: S3Inventory#unsorted_files should always return an array (#8034) 2019-08-23 17:59:31 +10:00
Gerhard Schlager
1f118b1309 FEATURE: Allow plugins to manipulate site settings during backup restore 2019-08-22 22:41:26 +02:00
Vinoth Kannan
41f22946c3 FIX: properly load desktop and mobile only plugin css assets.
5bd6b70d98
2019-08-22 08:39:10 +05:30
Daniel Waterworth
6924f1ab15 FIX: Race-condition in fallback handlers (#8005)
Calling `verify_master` in multiple threads simultaneously would cause
multiple threads to be spawned.
2019-08-21 15:47:44 +02:00
Arpit Jalan
75f37ac16a Rename lazyYT plugin directory name to lazy-yt 2019-08-21 14:35:14 +05:30
Sam Saffron
719a93c312 FEATURE: treat theme_uploads as settings in JavaScript
This change allows themes and components access to theme assets.

This means that inside theme js you can now get the URL for an asset with:

```
settings.theme_uploads.name
```
2019-08-21 16:51:10 +10:00
Vinoth Kannan
5bd6b70d98
DEV: debundle plugin css assets and don't load if disabled (#7646) 2019-08-20 22:09:52 +05:30
romanrizzi
36425eb9f0 Revert "FEATURE: Publish read state on group messages. (#7989) [Undo revert] (#8024)"
This reverts commit 5dda5c2f7c.
2019-08-20 13:29:22 -03:00
Roman Rizzi
5dda5c2f7c
FEATURE: Publish read state on group messages. (#7989) [Undo revert] (#8024)
* Reenable: "FEATURE: Publish read state on group messages. (#7989)"

This reverts commit 67f5cc1ce8.

* FIX: Read indicator only appears when the group setting is enabled
2019-08-20 11:57:25 -03:00
romanrizzi
67f5cc1ce8 Revert "FEATURE: Publish read state on group messages. (#7989)"
This reverts commit 1630dae2db.
2019-08-20 10:24:34 -03:00
Roman Rizzi
1630dae2db
FEATURE: Publish read state on group messages. (#7989)
* Enable or disable read state based on group attribute

* When read state needs to be published, the minimum unread count is calculated in the topic query. This way, we can know if someone reads the last post

* The option can be enabled/disabled from the UI

* The read indicator will live-updated using message bus

* Show read indicator on every post

* The read indicator now shows read count and can be expanded to see user avatars

* Read count gets updated everytime someone reads a message

* Simplify topic-list read indicator logic

* Unsubscribe from message bus on willDestroyElement, removed unnecesarry values from post-menu, and added a comment to explain where does minimum_unread_count comes from
2019-08-20 09:46:57 -03:00
Blake Erickson
893d30fb92 REVERT: External auth when redeeming invites
Reverting this commit:

87a0a6664e

because I'm extracting all of this logic into a plugin instead.
2019-08-16 12:44:14 -06:00
Sam Saffron
9a9e31f927 FIX: properly set notification levels on group invite
Previously we relied on side effects to set tracking state correctly
when inviting groups to messages

Also has a minor optimisation in that we use pluck instead of pulling in
full record
2019-08-16 18:23:51 +10:00
Robin Ward
23367e79ea
FEATURE: Embed topics list on remote sites via Javascript API. (#8008)
This adds support for a `<d-topics-list>` tag you can embed in your site
that will be rendered as a list of discourse topics. Any attributes on
the tag will be passed as filters. For example:

`<d-topics-list discourse-url="URL" category="1234">` will filter to category 1234.

To use this feature, enable the `embed topics list` site setting. Then
on the site you want to embed, include the following javascript:

`<script
src="http://URL/javascripts/embed-topics.js"></script>`

Where `URL` is your discourse forum's URL.

Then include the `<d-topics-list discourse-url="URL">` tag in your HTML document and it will
be replaced with the list of topics.
2019-08-15 13:41:06 -04:00
David Taylor
fafc6bcde0 FIX: Clear banner topic cache after remapping 2019-08-15 11:24:20 +01:00
Sam Saffron
d05b724c4e FIX: correct race condition loading library
In some very rare cases CssParser could be loaded but CssParser::Parser not
this ensures we check for the actual constant we plan to call for concurrent
digest generations
2019-08-15 17:16:41 +10:00
Sam Saffron
e53a171916 FIX: hold s3 related distributed locks longer
These operations are pretty expensive and can take multiple minutes due to
networking.

Hold distributed mutex for much longer.
2019-08-15 11:48:44 +10:00
Roman Rizzi
79957706b1
FIX: Store custom attributes that are needed by plugins in queuedpost payload (#8009) 2019-08-14 15:02:59 -03:00
Vinoth Kannan
88359b0f16
FEATURE: add support for group members visibility level (#8004)
There are 5 visibility levels (similar to group visibility)

public (default)
logged-in users
members only
staff
owners

Admins & group owners always have visibility to group members.
2019-08-14 19:00:04 +05:30
David Taylor
f4aa6096ab FIX: Convert omniauth authenticator names to symbols before comparing
This is necessary because some auth plugins define their name as a string
2019-08-14 12:57:11 +01:00
David Taylor
5755d3886f FIX: Do not raise exception if the authenticator email is missing
Followup to 296cdc53ee
2019-08-14 12:08:59 +01:00
Blake Erickson
296cdc53ee FIX: Downcase email coming back from auth-provider 2019-08-13 10:53:50 -06:00
Vinoth Kannan
9919ee1900 FIX: remove the tmp inventory files after the s3 uploads check. 2019-08-13 11:52:57 +05:30
Sam Saffron
7632fe0b58 PERF: rake posts:rebake_uncooked_posts runs inline
Running this inline makes more sense otherwise there is extreme risk in
saturating sidekiq queue.

This also reworks ordering and selection so we double check if a post needs
rebaking prior to rebaking, this unlocks the ability to run this rake task
from multiple consoles.
2019-08-13 10:28:42 +10:00
David Taylor
923c46f5df DEV: Provide method for auth plugins to generate a CSRF token 2019-08-13 01:13:08 +01:00
Gerhard Schlager
d686318133 FIX: Prevent failed remaps during restores
Additional changes:
* Verbose logging of remaps during restores
* Exclude the backup_metadata table from restores
2019-08-12 17:15:01 +02:00
Gerhard Schlager
c05739c3c8 FIX: Truncate topic_links.url to 500 chars during remap
This column often breaks remaps because of some weird, long URLs. The data isn't that important, so truncating a couple of URLs doesn't hurt that much.
2019-08-12 17:15:00 +02:00
Gerhard Schlager
8b6341669a REFACTOR: Less duplicate code in DbHelper.remap 2019-08-12 17:15:00 +02:00
Dan Ungureanu
2a98becfde
FIX: Do not create a double like notification. (#7999)
When a user liked, unliked and liked again the same post, the poster
would receive a notification such as "X and X liked ...". This happened
because PostActionNotifier.post_action_created was called twice.
2019-08-12 16:22:46 +03:00
David Taylor
1a8fee11a0 DEV: If only one auth provider is enabled allow GET request
In this case, the auth provider is acting as a SSO provider, and can be trusted to maintain its own CSRF protections.
2019-08-12 11:03:05 +01:00
David Taylor
750802bf56
UX: Improve error handling for common OmniAuth exceptions (#7991)
This displays more useful messages for the most common issues we see:
- CSRF (when the user switches browser)
- Invalid IAT (when the server clock is wrong)
- OAuth::Unauthorized for OAuth1 providers, when the credentials are incorrect

This commit also stops earlier for disabled authenticators. Now we stop at the request phase, rather than the callback phase.
2019-08-12 10:55:02 +01:00
Blake Erickson
87a0a6664e FEATURE: External auth when redeeming invites
This feature (when enabled) will allow for invite_only sites to require
external authentication before they can redeem an invite.

- Created hidden site setting to toggle this
- Enables sending invites with local logins disabled
- OAuth button added to invite form
- Requires OAuth email address to match invite email address
- Prevents redeeming invite if OAuth authentication fails
2019-08-11 12:20:02 -06:00
Gerhard Schlager
e4f14ca3d7 FIX: Disallow user self-delete when user posted in PMs
All posts created by the user are counted unless they are deleted,
belong to a PM sent between a non-human user and the user or belong
to a PM created by the user which doesn't have any other recipients.

It also makes the guardian prevent self-deletes when SSO is enabled.
2019-08-10 12:30:16 +02:00
Gerhard Schlager
74d78e3636 FIX: Drop readonly function when dropping table 2019-08-09 11:39:46 +02:00
Gerhard Schlager
7cb51d0e40 FIX: Create readonly functions during backup
Temporarily recreate already dropped functions in the discourse_functions schema in order to allow restoring of backups which still reference dropped functions.
2019-08-09 11:39:46 +02:00
Dan Ungureanu
8aa5df69f0
FIX: Detect DiscourseHub user agent.
It used to show up as an unknown device and browser in user preferences
(Recently Used Devices).
2019-08-09 11:58:15 +03:00
Sam Saffron
1f47ed1ea3 PERF: message_bus will be deferred by server when flooded
The message_bus performs a fair amount of work prior to hijacking requests
this change ensures that if there is a situation where the server is flooded
message_bus will inform client to back off for 30 seconds + random(120 secs)

This back-off is ultra cheap and happens very early in the middleware.

It corrects a situation where a flood to message bus could cause the app
to become unresponsive

MessageBus update is here to ensure message_bus gem properly respects
Retry-After header and status 429.

Under normal state this code should never trigger, to disable raise the
value of DISCOURSE_REJECT_MESSAGE_BUS_QUEUE_SECONDS, default is to tell
message bus to go away if we are queueing for 100ms or longer
2019-08-09 17:48:01 +10:00
Bianca Nenciu
7c83d2eeb2 FIX: Award 'First Onebox' badge just for Oneboxed URLs. (#7974) 2019-08-08 18:45:18 +02:00
David Taylor
3b8c468832 SECURITY: Require POST with CSRF token for OmniAuth request phase 2019-08-08 11:58:00 +01:00
David Taylor
7bd54eaceb
UX: Add login button icon to no_ember page (#7982) 2019-08-08 10:44:04 +01:00
Neil Lalonde
5f87089b67 FIX: remove dependency on present? in distributed_mutex lib 2019-08-07 15:39:51 -04:00
Gerhard Schlager
24877a7b8c FIX: Correctly encode non-ASCII filenames in HTTP header
Backport of fix from Rails 6: 890485cfce
2019-08-07 19:10:50 +02:00
Arpit Jalan
e90aac11cb fix the build 2019-08-07 16:39:58 +05:30
Arpit Jalan
b0e781e2d4 FIX: do not follow redirect on same host with path /login or /session 2019-08-07 16:26:55 +05:30
Leo McArdle
6296ae3d31 FEATURE: add setting to show content of forwarded emails in topics (#7935) 2019-08-07 12:32:19 +02:00
Joffrey JAFFEUX
5d72561f4f
FIX: allows replacement of digits and symbols emojis (#7978) 2019-08-07 11:38:58 +02:00
Rafael dos Santos Silva
606c0ed14d
FIX: S3 uploads were missing a cache-control header (#7902)
Admins still need to run the rake task to fix the files who where uploaded previously.
2019-08-06 14:55:17 -03:00
Bianca Nenciu
b60b2a342f
FIX: Show membership requests link just for group owners. (#7543) 2019-08-06 13:28:22 +03:00
Joffrey JAFFEUX
dbb410697e
FIX: generated controller should be in an underscored folder (#7975) 2019-08-06 10:40:38 +02:00
Sam Saffron
67f5ad5ac0 FEATURE: allow post process mutex to be held longer
Previously we would only hold the post process mutex for 1 minute, that is
not enough when processing a post with lots of images. This raises the bar
to 10 minutes.

It also cleans up error reporting around distributed mutexes expiring. We
used to double report.
2019-08-05 11:57:35 +10:00
David Taylor
39e0442de9 FIX: Various watched words improvements
- Client-side censoring fixed for non-chrome browsers. (Regular expression rewritten to avoid lookback)
- Regex generation is now done on the server, to reduce repeated logic, and make it easier to extend in plugins
- Censor tests are moved to ruby, to ensure everything works end-to-end
- If "watched words regular expressions" is enabled, warn the admin when the generated regex is invalid
2019-08-02 15:29:12 +01:00
Sam Saffron
4c6a0313f2 FEATURE: improve stats provided by rake db:stats
New stats include index size and sort by index + table size

Eg:

```
table_name                  | row_estimate | table_size | index_size | total_size
--------------------------------------------------------------------------
topic_views                 | 18260776     | 796 MB     | 2015 MB    | 2811 MB
post_timings                | 15813837     | 668 MB     | 1153 MB    | 1821 MB
posts                       | 386890       | 1532 MB    | 215 MB     | 1747 MB
post_search_data            | 317575       | 580 MB     | 162 MB     | 742 MB
incoming_emails             | 34412        | 734 MB     | 7160 kB    | 741 MB
incoming_links              | 5169302      | 306 MB     | 377 MB     | 683 MB
email_logs                  | 5            | 32 kB      | 418 MB     | 418 MB
topic_users                 | 2410166      | 217 MB     | 155 MB     | 372 MB
user_actions                | 1361253      | 101 MB     | 266 MB     | 366 MB
...

```
2019-08-02 17:25:42 +10:00
Vinoth Kannan
3e456d5c0b FIX: don't include multisite upload path to source URL if already exist. 2019-08-02 07:57:27 +05:30
Vinoth Kannan
a1fc35fdf7 FIX: make uploads recovery compatible with multisite. 2019-08-02 07:08:21 +05:30
Vinoth Kannan
e44d56e4d2 DEV: raise error only when 'STOP_ON_ERROR' env variable is available. 2019-08-01 23:54:06 +05:30
Guo Xiang Tan
7bd93eba3e FIX: Gravatar uploads being dependent on authorized_extensions. 2019-08-01 16:24:09 +08:00
Daniel Waterworth
20bc4a38a5
FIX: DistributedMutex (#7953) 2019-08-01 09:12:05 +01:00
Guo Xiang Tan
a1a61fcd2d DEV: Use File#file? in posts:missing_uploads rake task. 2019-08-01 10:59:34 +08:00
Régis Hanol
19dda59932 FIX: add back verbose option to DbHelper.remap 2019-07-31 17:30:08 +02:00
Rishabh
dcb47d902b
REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications (#7958)
* REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications

- The older name could cause some confusion because the setting does not disable all edit notifications, only system ones.

* FIX: Add frozen_string_literal: true in the migration

* DEV: Deprecate 'disable_edit_notifications'
2019-07-31 20:20:41 +05:30
Neil Lalonde
43365a2bf1 Fix some broken styles 2019-07-30 16:46:20 -04:00
Neil Lalonde
9656a21fdb
FEATURE: customization of html emails (#7934)
This feature adds the ability to customize the HTML part of all emails using a custom HTML template and optionally some CSS to style it. The CSS will be parsed and converted into inline styles because CSS is poorly supported by email clients. When writing the custom HTML and CSS, be aware of what email clients support. Keep customizations very simple.

Customizations can be added and edited in Admin > Customize > Email Style.

Since the summary email is already heavily styled, there is a setting to disable custom styles for summary emails called "apply custom styles to digest" found in Admin > Settings > Email.

As part of this work, RTL locales are now rendered correctly for all emails.
2019-07-30 15:05:08 -04:00
Penar Musaraj
a1fb718340 DEV: Tweak plugin rake task console message
Show "Allowing write to all repos!" message only once.
2019-07-30 09:53:56 -04:00
Penar Musaraj
b5705348b3 FIX: Correctly identify Chromium-based Edge
Chromium-based Edge browsers has only "Edg" in user agent string.
2019-07-29 16:51:08 -04:00
Régis Hanol
89fce2ce71 DEV: remove duplicate Remap class and use DbHelper.remap instead
Follow-up to 9cd3f96dee
2019-07-29 18:43:40 +02:00
David Taylor
933d279811
FIX: Support multi-file stylesheets in theme components (#7950) 2019-07-29 10:28:24 +01:00
Guo Xiang Tan
8a64b0c8e8 Revert "DEV: Remove unused kwarg and properly check for local missing uploads."
This reverts commit 97769f3d02.

The code is confusing but this change is quite risky. Defer for now
until we can look at it properly.
2019-07-29 14:35:34 +08:00
Guo Xiang Tan
97769f3d02 DEV: Remove unused kwarg and properly check for local missing uploads. 2019-07-29 14:21:06 +08:00
Guo Xiang Tan
e9dc48ccae DEV: Fix multisite support for posts:missing_uploads. 2019-07-29 10:56:11 +08:00
Joffrey JAFFEUX
d83f99fc2e
DEV: improves rails plugin generator (#7949)
Fixes bugs, simplifies code, more default files. General idea, more is more here as it's easier to just delete things than reading and passing all the options.
2019-07-27 18:21:44 +02:00
Osama Sayegh
525920a979
FIX: Better error when SSO fails due to blank secret (#7946)
* FIX: Better error when SSO fails due to blank secret

* Update spec/requests/session_controller_spec.rb

Co-Authored-By: Robin Ward <robin.ward@gmail.com>
2019-07-26 17:37:23 +03:00
Robin Ward
a727968112 FIX: Provide an error message if no valid tags were selected 2019-07-25 12:46:16 -04:00
Gerhard Schlager
fd12c414e7 DEV: Refactor helper methods for upload markdown
Follow-up to a61ff167
2019-07-25 16:36:35 +02:00
Gerhard Schlager
7e0eeed292 FEATURE: Add attachments to outgoing emails
This feature is off by default and can can be configured with the `email_total_attachment_size_limit_kb` site setting.

Co-authored-by: Maja Komel <maja.komel@gmail.com>
2019-07-25 15:57:45 +02:00
Gerhard Schlager
a61ff16740 DEV: Make attachment markdown reusable 2019-07-25 14:04:18 +02:00
Vinoth Kannan
47deb8b3da FIX: use same id for both original & optimized inventories in multisite setup. 2019-07-25 14:16:47 +05:30
Joffrey JAFFEUX
cc46de8f46
s/discourse-staff-notes/discourse-user-notes (#7936) 2019-07-24 20:04:27 +02:00
Régis Hanol
ae05245b00 DEV: plugin API to register User custom field types 2019-07-24 18:38:44 +02:00
David Taylor
b3e5f7a8c6 SECURITY: Sanitize email id for use as mutex key 2019-07-24 13:45:02 +01:00
David Taylor
b084d6c8df DEV: Add missing parenthesis for 47ad2a4d 2019-07-24 11:29:18 +01:00
David Taylor
0a6cae654b SECURITY: Add confirmation screen when connecting associated accounts 2019-07-24 10:28:15 +01:00
David Taylor
da4c1c5afc DEV: Remove trailing whitespace from 47ad2a4d 2019-07-24 10:27:44 +01:00
Angus McLeod
47ad2a4d7a DEV: Handle both name formats in managed authenticator (#7925) 2019-07-24 10:22:26 +01:00
Michael Brown
31f583855a DEV: pull static check out of loop
* followup to 08b28680
* as per https://review.discourse.org/t/4713/2
2019-07-23 17:18:16 -04:00
Gerhard Schlager
68b082e1a4 FIX: Ensure that jobs don't run immediately after migrate_to_s3 2019-07-23 17:42:12 +02:00
romanrizzi
c4be8541f6 FIX: use uniq instead of uniq! when checking for uncompressed root path. Use rails naming convention for ZipUtils 2019-07-23 07:57:50 -03:00
OsamaSayegh
f364317625 PERF: Improve query speed when looking up direct PMs
Follow up to 5fc5a7f5ae
2019-07-23 03:52:52 +00:00
Osama Sayegh
5fc5a7f5ae FEATURE: Add search operator to see all direct messages from a user (#7913)
* FEATURE: Add search operator to see all direct messages from a user

* Only show message if related messages >= 5

* Make "all messages" the hyperlink

* Review
2019-07-22 10:55:49 -04:00
Osama Sayegh
f14c6d81f4
FEATURE: Watched words improvements (#7899)
This commit contains 3 features:

- FEATURE: Allow downloading watched words
This introduces a button that allows admins to download watched words per action in a `.txt` file.

- FEATURE: Allow clearing watched words in bulk
This adds a "Clear All" button that clears all deleted words per action (e.g. block, flag etc.)

- FEATURE: List all blocked words contained in the post when it's blocked
When a post is rejected because it contains one or more blocked words, the error message now lists all the blocked words contained in the post.

-------

This also changes the format of the file for importing watched words from `.csv` to `.txt` so it becomes inconsistent with the extension of the file when watched words are exported.
2019-07-22 14:59:56 +03:00
Roman Rizzi
eb26bee046
DEV: group_list site settings should store IDs instead of group names (#7860)
* DEV: group_list site settings should store IDs instead of group names

* Ship site setting to know when we should migrate group_list settings

* Migrate existing group_list site settings

* Bump migration timestamp and don't set null when migrating is not possible.
2019-07-19 15:17:58 -03:00
Robin Ward
e47e0af123
FEATURE: Allow viewing of raw emails for reviewable queued posts (#7910)
If a post arrives via email but must be reviewed, we now show an
icon that can be clicked to view the raw contents of the email.

This is useful if Discourse's email parser is acting odd and the user
reviewing the post wants to know what the original contents were before
approving/rejecting the post.
2019-07-19 11:56:14 -04:00
Bianca Nenciu
9ba2c7cd8b
FIX: Set a minimum reading time per post. (#7842)
Topics containing only images could generate a reading time of zero minutes.
2019-07-19 18:15:38 +03:00
Gerhard Schlager
b73bd7fc1b FIX: Always backup local uploads in addition to files stored on S3 2019-07-19 15:13:05 +02:00
Blake Erickson
d26aa6e71e REFACTOR: Cleanup rake tasks based on feedback
Follow up to: [FEATURE: Create a rake task for destroying categories][1]

- `Discourse.system_user` is my friend
- Remove puts statements from rake tasks that don't return anything
- `for_each` is also my friend
- Use `human_users` to also exclude discobot
- Sort/format categories:list

[1]: 092eeb5ca3
2019-07-18 19:15:01 -06:00
Blake Erickson
b0c92bb0b9 REFACTOR: Clean up parameterized title
Follow up to [FIX: Empty backup names with unicode site titles][1]

- Use .presence - "It's cleaner"
- Update spec to use System.system_user so it is more readable

[1]: c8661674d4
2019-07-18 15:49:16 -06:00
Vinoth Kannan
ad04ce9f43 FIX: remove post upload record creation inside 'find_missing_uploads' method. 2019-07-19 01:44:08 +05:30
Roman Rizzi
f5c707c97a
FEATURE: Gz to zip for exports (#7889)
* Revert "Revert "FEATURE: admin/user exports are compressed using the zip format (#7784)""

This reverts commit f89bd55576.

* Replace .tar.zip with .zip
2019-07-18 09:34:48 -03:00
Gerhard Schlager
7e69c5cc36 Revert "FEATURE: Use configured quotation marks in fancy topic title"
This reverts most of commit ce8e099639.

The rake task to update fancy topic titles is still there, because that's useful even without this feature.
2019-07-18 11:55:49 +02:00
Blake Erickson
c8661674d4 FIX: Empty backup names with unicode site titles
If a site title contains unicode it may end up with an empty backup
filename because of the rails `parameterize` method we are calling.

This fix ensures that the backup filenames default to "discourse" if the
parameterized site title is empty.

Bug reported [here][1].

[1]: https://meta.discourse.org/t/backup-checksum-and-backup-name-missing-when-unicode-site-name/123192?u=blake
2019-07-17 17:07:10 -06:00
Blake Erickson
092eeb5ca3 FEATURE: Create a rake task for destroying categories
Created a rake task for destroying multiple categories along with any
subcategories and topics the belong to those categories.

Also created a rake task for listing all of your categories.

Refactored existing destroy rake tasks to use new logging method, that
allows for puts output in the console but prevents it from showing in
the specs.
2019-07-17 12:44:14 -06:00
Joffrey JAFFEUX
b3eb67976d
DEV: Upgrades to Ember 3.10 (#7871)
Co-Authored-By: majakomel <maja.komel@gmail.com>
2019-07-16 12:45:15 +02:00
Michael Brown
08b286808a
FIX: backups taken by pg_dump >= 11 are nonportable (#7893) 2019-07-15 18:07:44 -04:00
Rafael dos Santos Silva
1221d34284
FEATURE: Make Discourse work offline with WorkboxJS (#7870) 2019-07-15 13:05:55 -03:00
Vinoth Kannan
839916aa49
DEV: Debundle plugin javascript assets and don't load if disabled (#7566)
And don't load javascript assets if plugin is disabled.

* precompile auto generated plugin js assets

* SPEC: remove spec test functions

* remove plugin js from test_helper

Co-Authored-By: Régis Hanol <regis@hanol.fr>

* DEV: using equality is slightly easier to read than inequality

Co-Authored-By: Régis Hanol <regis@hanol.fr>

* DEV: use `select` method instead of `find_all` for readability

Co-Authored-By: Régis Hanol <regis@hanol.fr>
2019-07-15 20:22:54 +05:30
Neil Lalonde
6a2f30c277 Version bump to v2.4.0.beta2 2019-07-15 10:12:45 -04:00
Gerhard Schlager
ce8e099639 FEATURE: Use configured quotation marks in fancy topic title 2019-07-12 21:10:10 +02:00
Gerhard Schlager
9a11a8b33b FEATURE: Site setting for typographic quotation marks
Adds locale defaults for German and French
2019-07-11 23:19:28 +02:00
Robin Ward
1d38040579 SECURITY: SQL injection with default categories
This is a low severity security fix because it requires a logged in
admin user to update a site setting via the API directly to an invalid
value.

The fix adds validation for the affected site settings, as well as a
secondary fix to prevent injection in the event of bad data somehow
already exists.
2019-07-11 13:41:51 -04:00
Joffrey JAFFEUX
550e811652
DEV: allows lodash to be updated with rake javascript:update (#7881) 2019-07-11 16:57:03 +02:00
Blake Erickson
c76732722a FIX: Turn off search logging when read-only (#7877)
If `SiteSetting.log_search_queries` is enabled 500 errors will occur
when searching if the master db is down. This fix allows searching to
still work under these conditions.
2019-07-10 17:05:31 -07:00
romanrizzi
f89bd55576 Revert "FEATURE: admin/user exports are compressed using the zip format (#7784)"
This reverts commit 8b2580e20f.
2019-07-10 11:38:51 -03:00
Roman Rizzi
8b2580e20f
FEATURE: admin/user exports are compressed using the zip format (#7784)
* FEATURE: admin/user exports are compressed using the zip format

* Update translations. Theme exporter now exports .zip file. Theme importer supports .zip and .gz files

* Fix controller test, updated locale and skip saving the csv export to disk
2019-07-10 11:13:03 -03:00
Gerhard Schlager
5f0d38341e FIX: Remapping during restore was wrong for CDN URLs 2019-07-09 17:34:41 +02:00
Gerhard Schlager
4c1b8c7559 FIX: Remap differently when backup comes from multisite 2019-07-09 16:11:32 +02:00
Gerhard Schlager
a65a9a85d5 FEATURE: Remap uploads during restore when S3 or CDN changes
In order for this to work the Backuper stores a couple of site settings
in the new backup_metadata table, because the old setting values might
not be available on restore anymore.
2019-07-09 14:04:16 +02:00
Gerhard Schlager
5ffb722999 DEV: Less verbose remapping
It's hard to see which columns have been remapped when remapping prints
lots of "0 rows affected" lines. This changes it to output the row count
only for affected columns.
2019-07-09 14:04:16 +02:00
Gerhard Schlager
f2dc59d61f FEATURE: Add hidden setting to include S3 uploads in backups 2019-07-09 14:04:16 +02:00
Dan Ungureanu
9f5cfa192e
FEATURE: Allow Markdown in post notices. (#7864) 2019-07-09 14:42:02 +03:00
Daniel Waterworth
c3db5925a8 FIX: Turbo tests exit codes 2019-07-09 08:51:23 +01:00
Neil Lalonde
9cd3f96dee FIX: Remap shouldn't try to change read-only columns
Read-only columns are obsolete and not used in the code anymore.
Previously, remap would fail when trying to update a read-only column.
2019-07-08 16:52:52 -04:00
Penar Musaraj
b690fc3d98
FEATURE: Add new group visibility option for "logged on users" (#7814)
Groups can now be marked as visible to "logged on users". All automatic groups (except `everyone`) are now visible to "logged on users", previously they were marked as public but suppressed in the group page for non-staff.
2019-07-08 15:09:50 -04:00
Penar Musaraj
befcf67c90 DEV: run db:create and db:migrate on turbo specs
This ensures multisite specs run correctly when running ./bin/turbo_rspec
2019-07-08 14:27:51 -04:00
Robin Ward
c2c169f5b7 FIX: Fail if none of our tags could be updated
For example, if a category has a tag restriction and the API tries to
attempt to update it but cannot.

See:
https://meta.discourse.org/t/unallowed-tag-in-conversation-returns-200/122170
2019-07-05 11:40:18 -04:00
Osama Sayegh
65a6f3080e FIX: don't disable download_remote_images_to_local if site uses S3 (#7861) 2019-07-05 13:36:03 +10:00
Sam Saffron
b9a2a4d904 DEV: Avoid emoji for symbols
Rake task needs to work across a wide variety of terminal emulators.
Requiring an emoji font is superflous.
2019-07-05 09:41:15 +10:00
Ralph Rooding
1318e0b288 FEATURE: Rake themes installer (#7848)
* Delete remote_theme when deleting the theme

* Install themes and theme components through rake

* Removed unnecessary test
2019-07-04 14:33:05 -04:00
Penar Musaraj
03805e5a76
FIX: Ensure lightbox image download has correct content disposition in S3 (#7845) 2019-07-04 11:32:51 -04:00
Joffrey JAFFEUX
e9bb13c630
DEV: updates jquery-color to prevent it to pull old jquery version (#7857) 2019-07-04 15:03:21 +02:00
Gerhard Schlager
a5e80079d6 FEATURE: Add Belarusian language 2019-07-04 11:37:37 +02:00
Joffrey JAFFEUX
71bf9ec1b2
FEATURE: opt-in guidance on topics for users without access (#7852)
Co-Authored-By: majakomel <maja.komel@gmail.com>
Co-Authored-By: Robin Ward <robin.ward@gmail.com>
2019-07-04 10:12:39 +02:00
Joffrey JAFFEUX
9ee2c121c1
FIX: ensures emoji helper is working with custom emojis (#7843) 2019-07-03 09:23:40 +02:00
Sam Saffron
4dcc5f16f1 FEATURE: when under extreme load disable search
The global setting disable_search_queue_threshold
(DISCOURSE_DISABLE_SEARCH_QUEUE_THRESHOLD) which default to 1 second was
added.

This protection ensures that when the application is unable to keep up with
requests it will simply turn off search till it is not backed up.

To disable this protection set this to 0.
2019-07-02 11:22:01 +10:00
Bianca Nenciu
4f97f85178 DEV: Fix lint. (#7824) 2019-06-28 20:24:09 +02:00
Josh Moore
6c5689984f FEATURE: in:tagged search (srv side) (#7822)
* FEATURE: in:tagged and in:untagged advanced search filters

Similar to in:solved or in:unsolved, the filters check for an
existence of the topic_id in the topic_tags table.

see: https://meta.discourse.org/t/how-to-search-filter-untagged-topics/119641/2
2019-06-28 18:19:57 +10:00
Penar Musaraj
5cf5e73536 FIX: copy local theme changes to correct temp folder when diffing updates to remote theme
Previously, local changes were being copied to a subfolder, resulting in lots of false positive diff changes.
2019-06-27 13:01:34 -04:00
Daniel Waterworth
23c5da4617 DEV: Check for pending migrations before starting the turbo tests 2019-06-27 16:41:19 +01:00
Daniel Waterworth
d6aa92e98e DEV: Add a verbose option to ./bin/turbo_rspec 2019-06-27 15:49:21 +01:00
Daniel Waterworth
8a0be71b3c FIX: FakeExceptions should have the original class name 2019-06-27 11:43:53 +01:00
Daniel Waterworth
e58f67a0c0 FIX: an exception cause is itself an exception 2019-06-27 11:43:53 +01:00
Sam Saffron
8f7a387aa7 FEATURE: add support for tag group search
The behaviour of #TERM in search has been amended

1. We try category or subcategory slugs
2. We try tags
3. We try tag-groups

The term `hello #my-group` will search for all posts tagged with any of
the tags in the tag group `My Group`

Future work may be introducing a slug cache here or caching it in the table
but the assumption is that the number of tag groups will not be huge
2019-06-27 17:53:26 +10:00
Sam Saffron
5bc5c02af6 FIX: mark topics in sub categories as unread when dismissing parent
Previously we would only dismiss the parent category and leave the
child categories unread
2019-06-27 13:26:48 +10:00
Régis Hanol
b2a033e92b REFACTOR: use Ruby's sum 2019-06-27 01:54:40 +02:00
Régis Hanol
134a4c66b7 PERF: limit time spent diffing large blobs of text
REFACTOR: s/.length/.size/g
2019-06-27 01:45:52 +02:00
Daniel Waterworth
b3ec3fac93 FIX: turbo_rspec doesn't accept these options 2019-06-26 17:05:27 +01:00
Daniel Waterworth
68d54d818a DEV: Add USE_TURBO option to docker rake task (#7811) 2019-06-26 11:54:39 -04:00
Arpit Jalan
3c64dc3e7d
FIX: do not allow creation of topic if there is no category available for posting (#7786) 2019-06-26 16:32:53 +05:30
Sam Saffron
39b856dce7 DEV: correct edge case introduced in 333b5a19
We need to allow users to edit uncategorized topics out of uncategorized
when for some reason admin just turns it off.
2019-06-26 17:53:29 +10:00
Bianca Nenciu
557805249d
FIX: Ensure topic exists before making a banner. (#7781) 2019-06-25 13:49:29 +03:00
Guo Xiang Tan
d21594f4f7 Revert changes added by mistake in 2b19e2acc8. 2019-06-25 15:25:12 +08:00
Guo Xiang Tan
2b19e2acc8 Fix typo in a0aeabbb94. 2019-06-25 15:18:57 +08:00
Guo Xiang Tan
a0aeabbb94 DEV: Safer replacement for old scheme uploads for posts:missing_uploads. 2019-06-25 11:34:26 +08:00
David Taylor
afb5ec811d FIX: Don't use DistributedCache to store redis readonly state
This can cause unbound CPU usage in some cases, and excessive logging in other cases. This commit moves redis readonly information into the local process, but maintains the DistributedCache for postgres readonly state.
2019-06-25 11:20:34 +08:00
Joffrey JAFFEUX
f4e354afff FIX: ensures url to full reviewable conversation works on subfolder 2019-06-24 11:31:07 -04:00
Guo Xiang Tan
f3c13d80c9 DEV: Don't copy file when trying to recover missing uploads. 2019-06-24 16:32:59 +08:00
Guo Xiang Tan
2cc5561504 DEV: Correct description in posts:missing_uploads rake task. 2019-06-24 14:57:43 +08:00
Penar Musaraj
f51f37eddf FEATURE: apply a small penalty to closed topics when searching (#7782) 2019-06-21 12:03:45 +10:00
Sam Saffron
5bc92296be DEV: lint a bunch of files we missed 2019-06-21 11:33:41 +10:00
Sam Saffron
fd5c2fbac7 DEV: make parallel spec optional with autospec
At least temporarily while we refine the system keep parallel as optional
2019-06-21 11:00:28 +10:00
Daniel Waterworth
e18ce56f4b DEV: Add a new way to run specs in parallel with better output (#7778)
* DEV: Add a new way to run specs in parallel with better output

This commit:

 1. adds a new executable, `bin/interleaved_rspec` which works much like
    `rspec`, but runs the tests in parallel.

 2. adds a rake task, `rake interleaved:spec` which runs the whole test
    suite.

 3. makes autospec use this new wrapper by default. You can disable this
    by running `PARALLEL_SPEC=0 rake autospec`.

It works much like the `parallel_tests` gem (and relies on it), but
makes each subprocess use a machine-readable formatter and parses this
output in order to provide a better overall summary.

(It's called interleaved, because parallel was taken and naming is
hard).

* Make popen3 invocation safer

* Use FileUtils instead of shelling out

* DRY up reporter

* Moved summary logic into Reporter

* s/interleaved/turbo/g

* Move Reporter into its own file

* Moved run into its own class

* Moved Runner into its own file

* Move JsonRowsFormatter under TurboTests

* Join on threads at the end

* Acted on feedback from eviltrout
2019-06-21 10:59:01 +10:00
Robin Ward
0d84c5b894 FIX: If a user deletes a hidden post, it should not lose history 2019-06-20 12:38:16 -04:00
Joffrey JAFFEUX
cd6a130cd0
FEATURE: adds early support for new emojis (#7785)
🟫 🟪 🟦 🟩 🟨 🟧 🟥 🟤 🟣 🟢 🟡 🟠 🪒 🪑 🩺 🩹 🩸 🦯 🪓 🪔 🪕 🩰 🩳 :briefs: 🩱 🥻 🦺 🤿 🛺 🦼 🦽 🛕 :maté: 🧃 🦪 🧈 🧆 🧅 🧄 🦨 🦮 🧑‍🤝‍🧑 👩‍🦽 👨‍🦽 👩‍🦼 👨‍🦼 👩‍🦯 👨‍🦯 :woman_kneeling: :man_kneeling: :man_standing: :woman_standing: 🧏‍♀️ 🧏‍♂️ :hear_with_hearing_aid: 🦿 🦾 🤍 🤎
2019-06-20 11:30:09 +02:00
Robin Ward
e8b9f38374 FIX: Don't allow users to edit topic information when the OP is locked
see:
https://meta.discourse.org/t/user-able-to-edit-title-of-locked-post/104826
2019-06-18 14:22:38 -04:00
Neil Lalonde
b2af1101d9 Version bump to v2.4.0.beta1 2019-06-17 20:49:28 -04:00
Penar Musaraj
4bbb43cb2b DEV: pull plugin repos before linting them 2019-06-14 13:50:15 -04:00
Sam Saffron
ecb2fd8222 DEV: improve db:create to catch all cases
- No RAILS_ENV - create multisite / dev / test
- `test` RAILS_ENV - create multisite / test
2019-06-14 15:47:05 +10:00
Sam Saffron
a01488ae67 DEV: improve on rake db:create
Followup on 3af00a65 which broke build
2019-06-14 15:06:07 +10:00
Sam Saffron
3af00a65e6 FIX: site settings loading default values when no db
This fixes a condition where an intermittent db connection could cause
invalid site settings to be stored

It also removes a catch all we had.

Somewhere around Rails 5 `db:create` started wanting full environment
this is a problem for Discourse since it needs to boot up data from the
db.

This removes the catch all and surgically adds a db / redis bypass to
db:create task.
2019-06-14 14:21:07 +10:00
Guo Xiang Tan
eaa7527933 DEV: Switch posts:inline_uploads scope to be more generic. 2019-06-14 11:53:34 +08:00
Penar Musaraj
4182d8638c Remove duplicate entry for discourse-yearly-review 2019-06-13 13:56:38 -04:00
Vinoth Kannan
35d6fff69e PERF: use url instead of file key in temporary inventory table. 2019-06-13 22:03:58 +05:30
David Taylor
ed21128ee6 FIX: Do not change directory when decompressing S3 inventory
In sidekiq, jobs are run in multiple threads within the same process. `cd` affects the entire process, so can cause unexpected issues in other running jobs.
2019-06-13 17:13:50 +01:00
Penar Musaraj
66b15b9d87 DEV: support installing plugin gems in test images
- adds plugin:install_all_gems rake task

- adds UPDATE_ALL_PLUGINS env variable in docker:test
2019-06-13 09:58:57 -04:00
Guo Xiang Tan
9bce3d5e65 DEV: Multisite support for rake posts:inline_uploads. 2019-06-13 16:13:45 +08:00
Sam Saffron
215216df48 FIX: stop re-downloading backup maxmind
We omitted updating the mmdb_time which changed causing a constant
re-download of the maxmind db
2019-06-13 17:42:26 +10:00
Sam
fa2a5f6f56
FEATURE: SKIP_DB_AND_REDIS env var (#7756)
Sometimes we would like to create a base image without any DB access, this
assists in creating custom base images with custom plugins that already
includes `public/assets`

Following this change set you can run:

```
SPROCKETS_CONCURRENT=1 DONT_PRECOMPILE_CSS=1 SKIP_DB_AND_REDIS=1 RAILS_ENV=production bin/rake assets:precompile
```

Then it is straight forward to create a base image without needing a DB or
Redis.
2019-06-13 12:58:27 +10:00
Penar Musaraj
e47ca6633b DEV: Fix plugin:update_all task
Reenable subtask so it loops through all plugins.

Only run for plugins that are git repositories.
2019-06-12 17:21:56 -04:00
Guo Xiang Tan
d49c193289 FIX: posts:inline_uploads rake task needs to bypass bump. 2019-06-12 09:13:31 +08:00
Bianca Nenciu
934adb14d2
FIX: On tag change notify only users watching the tag. (#7707) 2019-06-11 18:06:54 +03:00
Guo Xiang Tan
864222372f Remove incorrectly committed byebug.
Follow up to 49b921147a.
2019-06-11 16:33:49 +08:00
Guo Xiang Tan
49b921147a FIX: Undefined method in posts:inline_uploads. 2019-06-11 16:28:41 +08:00
Guo Xiang Tan
40e67971f9 DEV: Add spec for Email::Sender for upload links in plain text emails. 2019-06-11 16:02:24 +08:00
Guo Xiang Tan
42ab016856 FIX: Use markdown for images and attachments in Email::Receiver. 2019-06-11 14:49:46 +08:00
Dan Ungureanu
a046f6ced5 FEATURE: Trigger Discourse events from authenticators. (#7724) 2019-06-11 11:28:42 +10:00
Guo Xiang Tan
06d974d55c FEATURE: Add base62 sha1 to cooked data attribute
* FEATURE: Add base62 sha1 to data attribute in `Post#cooked`.

* FIX: Use `Upload#short_url` when quoting an image.
2019-06-11 11:15:45 +10:00
Neil Lalonde
62345a7f8d Version bump to v2.3.0.beta11 2019-06-10 13:09:47 -04:00
Guo Xiang Tan
1cf3bf016c Make rubocop happy. 2019-06-10 09:49:02 +08:00
Guo Xiang Tan
29259b46ae DEV: Verbose mode for posts:inline_uploads rake task. 2019-06-10 08:59:11 +08:00
Vinoth Kannan
d74ee9dbce DEV: skip S3 inventory records without correct multisite prefix. 2019-06-08 18:36:06 +05:30
Sam Saffron
1d76b7b7b5 PERF: fix N+A+lot query
reviewable_counts could be `{}` which is technically blank.

We wanted to check for nil here
2019-06-08 12:30:21 +10:00
Gerhard Schlager
19edc4abb8 FIX: English locale must not fall back to any other locale 2019-06-07 21:53:01 +02:00
Neil Lalonde
a08b2589d4 FIX: removing hidden tag bumps topic when all tags are removed
JS sends empty string to remove all tags.
2019-06-07 14:25:46 -04:00
Joffrey JAFFEUX
8cd13b22a1
DEV: allows smoke test to run on macOS (#7735) 2019-06-07 18:20:48 +02:00
David Taylor
e3a9a2d2dd FIX: Avoid infinite loop if disk space is low
We now continue to enqueue the pull_hotlinked_images job for optimized images, even if disk space is low
2019-06-07 14:24:22 +01:00
David Taylor
65b0cafc03 FIX: Always schedule pull_hotlinked_images in cooked_post_processor
The job is now used to pull optimized images, and images from other sites on the same CDN. This needs to run even if download_remote_images is false
2019-06-07 13:08:23 +01:00
Sam Saffron
f88dced0b7 PERF: optimize lookup of reviewable info in post stream
This previously was a hot path in topic view. Avoids an expensive active
record operation and instead perform SQL directly which is far more
targeted and efficient
2019-06-07 18:12:30 +10:00
Dan Ungureanu
8bd815dab2 FIX: Permit new tags when allow_global_tags true. (#7722) 2019-06-07 15:45:16 +10:00
Régis Hanol
b1c5ea4289 FIX: use URI.regexp to find URLs in plain text 2019-06-07 01:26:06 +02:00
Vinoth Kannan
b7830680b6 DEV: use cdn url to download the external uploads to local. 2019-06-06 19:17:19 +05:30
Bianca Nenciu
f63b8bb79d FIX: Periodically ensure consistency of categories. (#7663) 2019-06-06 11:30:52 +02:00
Penar Musaraj
f00275ded3 FEATURE: Support private attachments when using S3 storage (#7677)
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
2019-06-06 13:27:24 +10:00
Roman Rizzi
c3a38d2304 DEV: Make groups/new extensible by plugins (#7642)
* Expose a new plugin outlet. Pass group model to the group-member-dropdown so it can be accessed by plugins

* Added controller tests for group custom fields. update custom fields when updating a group
2019-06-06 12:05:33 +10:00
Robin Ward
f1d547c301 FEATURE: Show "in reply to" on the review queue
We now show if a queued or flagged post is a reply to another when in
the review queue. It's especially helpful for queued posts where
normally they are linked to the topic where they are created, and you
have no context about the reply.

Note that this will only apply to new queued posts going forward.
Previously queued posts will not show the "in reply to"
2019-06-05 12:34:41 -04:00
Gerhard Schlager
a77d8bee68 FIX: Use correct locale when translating without cache
follow-up to 6d8eb9c1
2019-06-05 14:19:56 +02:00
Sam Saffron
19e3b3b1bc PERF: speed up topic poster lookups
During profiling looking up topic users popped up as a hot path, this
change more than halved the amount of work it does

It reduces object allocations and method calls and avoids repeate translation
of common terms
2019-06-05 18:28:36 +10:00
Sam Saffron
62141b6316 FEATURE: enable_performance_http_headers for performance diagnostics
This adds support for DISCOURSE_ENABLE_PERFORMANCE_HTTP_HEADERS
when set to `true` this will turn on performance related headers

```text
X-Redis-Calls: 10     # number of redis calls
X-Redis-Time: 1.02    # redis time in seconds
X-Sql-Commands: 102   # number of SQL commands
X-Sql-Time: 1.02      # duration in SQL in seconds
X-Queue-Time: 1.01    # time the request sat in queue (depends on NGINX)
```

To get queue time NGINX must provide: HTTP_X_REQUEST_START

We do not recommend you enable this without thinking, it exposes information
about what your page is doing, usually you would only enable this if you
intend to strip off the headers further down the stream in a proxy
2019-06-05 16:08:11 +10:00
Sam Saffron
6d8eb9c144 PERF: bypass slow locale lookups in various cases
Previously as soon as any override was defined we would regress to the slow
path for locale lookups. Additionally if `raise: true` was specified which
rails likes to add in views we would bypass the cache

The new design manages to use the fast path for many more cases
2019-06-05 14:30:25 +10:00
Daniel Waterworth
3407445831 FIX: Prevent deadlock (#7691)
Before the locking here was added, replying to a post and liking a post
at roughly the same time could cause a deadlock.

Liking a post grabs an update lock on the post and then on the topic (to
update like counts).

We now lock the replied to post before getting the topic lock so that we
can update the replied to post later without causing a deadlock.
2019-06-05 11:29:27 +10:00
Neil Lalonde
1cf0b549ab FIX: don't send post edit notification when hidden tags are changed
Create a hidden revision so staff can see the changed, but don't send
notifications to non-staff.
2019-06-04 15:48:15 -04:00
Neil Lalonde
dd1a034e7b Revert "FIX: don't send post edit notification when hidden tags are changed" 2019-06-04 15:48:15 -04:00
Gerhard Schlager
98a3e61a2d FIX: Remapping URLs didn't affect upload site settings 2019-06-04 18:37:10 +02:00
Gerhard Schlager
f7a2648694 FEATURE: Migrate uploads to S3 during restore 2019-06-04 15:47:36 +02:00
Gerhard Schlager
fb121815a5 DEV: Allow rebaking oncooked posts of single site 2019-06-04 15:47:36 +02:00
Régis Hanol
9756e35956 REVERT: FIX: handle clicks counters in quotes
Not quite a full revert of 7696b92c8c that isn't
actually required.
2019-06-04 11:59:44 +02:00
David Taylor
0508546fd2 FIX: Accept github theme urls with a trailing slash 2019-06-04 10:28:36 +01:00
Maja Komel
87d3b86484 FIX: better error message when user without permissions replies via email 2019-06-04 16:39:41 +08:00
Guo Xiang Tan
1991af2abb DEV: Switch InlineUploads to a regexp based implementation. 2019-06-04 15:54:25 +08:00
Guo Xiang Tan
d93e5fb00d DEV: Class that converts MD with old attachment links to new MD. 2019-06-04 15:54:25 +08:00
Arpit Jalan
af20d616de FEATURE: new rake task to invalidate broken images in post 2019-06-04 12:40:51 +05:30
Sam Saffron
6300d978e2 FIX: correctly escape regex url
duplicate link detection could fail in certain cases
2019-06-04 16:21:03 +10:00
David Taylor
81dcadf788
DEV: Stop compiling themes during DB migration. Recompile on cdn change. (#7676)
This is an improved implementation for bc8b7b13
2019-06-03 16:38:02 +01:00
Penar Musaraj
16982d2a69
Update UI for wizard themes further reading step (#7669) 2019-06-03 10:47:17 -04:00
Robin Ward
f86a5bd5a9 FIX: Regression with finding reviewable counts on topics
This was a fairly serious regression on sites with large (mega) topics,
however it was limited to staff.

The issue here is the query was using filtered_post_ids which I'd
assumed was already windowed to the current page, when in fact it was
all the ids in the topic. This fix corrects it by using the correct
windowed collection.
2019-06-03 09:08:39 -04:00
David Taylor
5e61893cb2 FIX: Display generic descriptor for associated account with no info 2019-06-03 12:14:02 +01:00
David Taylor
7500eed4c0
FEATURE: Multi-file javascript support for themes (#7526)
You can now add javascript files under `/javascripts/*` in a theme, and they will be loaded as if they were included in core, or a plugin. If you give something the same name as a core/plugin file, it will be overridden. Support file extensions are `.js.es6`, `.hbs` and `.raw.hbs`.
2019-06-03 10:41:00 +01:00
Dan Ungureanu
c1e7a1b292 UX: Merge settings related to muted tags. (#7656) 2019-06-03 12:23:23 +10:00
Maja Komel
42809f4d69 FIX: use crawler layout when saving url in Wayback Machine (#7667) 2019-06-03 12:13:32 +10:00
Rafael dos Santos Silva
725588f835 FIX: migrate_to_s3 wasn't IAM profile aware 2019-06-01 12:09:46 -03:00
Andrew Schleifer
d9abfafd72 north_korea: swap name/alias
because it needs to match the filename
2019-05-31 23:05:07 -05:00
Robin Ward
743ce093a9 FIX: Use the full URL for reviewable content to make the API easier 2019-05-31 15:29:53 -04:00
David Taylor
03363d03e8 FEATURE: Load theme setting descriptions from theme locale files
Previously theme setting descriptions were defined in the `settings.yml` file like this:
```
setting_name:
  default: "My Default Value"
  description:
    en: "English description"
    fr: "French description"
```

This commit allows developers to store the localised descriptions in the theme locale files instead:
```
en:
  theme_metadata:
    description: Theme Description
    settings:
      setting_name: "The localised description for setting_name"
```
2019-05-31 14:49:59 +01:00
Arpit Jalan
388433ec06 DEV: remove unused method (rake task deleted in 6a274fd) 2019-05-31 11:16:10 +05:30
Arpit Jalan
5c526e5abb FEATURE: new rake task to watch all replied topics 2019-05-31 11:08:00 +05:30
Guo Xiang Tan
6a274fd2cd DEV: Remove redundant rake task.
Onceoff job created in 028121b95b
2019-05-31 10:05:13 +08:00
Andrew Schleifer
cbaa9c4fca harmonize korean aliases 2019-05-30 15:48:12 -05:00
Robin Ward
2e0a40007b FIX: Category topics should not be deletable via review queue 2019-05-30 16:43:23 -04:00
Neil Lalonde
b4d1416076 Version bump to v2.3.0.beta10 2019-05-30 13:45:10 -04:00
Bianca Nenciu
8196af165d FIX: Add Chromebook detection. 2019-05-30 16:29:51 +03:00
Joffrey JAFFEUX
2548d5b0e3
DEV: autospec chrome check was failing on macOS (#7649) 2019-05-30 12:09:01 +02:00
Sam Saffron
0ba4ea9718 DEV: bin/rake autospec not working when saving a js file
Frozen string issue
2019-05-30 16:09:45 +10:00
romanrizzi
2fa8df7cd2 New: can_see_groups? method for better perf 2019-05-30 08:45:20 +08:00
Dan Ungureanu
8728850452 FEATURE: Mute topics tagged with both muted and unmuted tags. 2019-05-30 07:58:17 +08:00
Arpit Jalan
23fdaf07f2 FIX: create system message in user selected locale
DEV: refactor `create_from_system_user` to use `create` method
2019-05-29 21:43:43 +05:30
Bianca Nenciu
227c45107d FEATURE: Implement Onebox for posts including polls. (#7539) 2019-05-29 17:05:52 +02:00
Joffrey JAFFEUX
630e9814bc
datetime is not available at this point (#7630) 2019-05-29 14:06:32 +02:00
Joffrey JAFFEUX
6439004161
DEV: do not use STDERR to print tests timestamps (#7629) 2019-05-29 13:28:02 +02:00
Guo Xiang Tan
a3938f98f8 Revert changes to FileStore::S3Store#path_for in f0620e7118.
There are some places in the code base that assumes the method should
return nil.
2019-05-29 18:39:07 +08:00
Joffrey JAFFEUX
5d47d1833b
DEV: prints timestamp at every docker:test commands (#7628) 2019-05-29 12:20:33 +02:00
Vinoth Kannan
e12ae453e9 FIX: verify the exitence of s3_object properly without db name 2019-05-29 15:10:36 +05:30
Vinoth Kannan
9a9a06e34b DEV: add option to skip etag verification on 'migrate_to_s3' rake task 2019-05-29 14:16:36 +05:30
Guo Xiang Tan
a206da8e18 DEV: Mark discourse-encrypt plugin as official. 2019-05-29 10:20:46 +08:00
Guo Xiang Tan
f0620e7118 FEATURE: Support [description|attachment](upload://<short-sha>) in MD take 2.
Previous attempt was missing `post_uploads` records.
2019-05-29 09:26:32 +08:00
Penar Musaraj
a93aa5bbce Add "further reading for themes" step to wizard 2019-05-29 08:28:03 +08:00
Penar Musaraj
7c9fb95c15 Temporarily revert "FEATURE: Support [description|attachment](upload://<short-sha>) in MD. (#7603)"
This reverts commit b1d3c678ca.

We need to make sure post_upload records are correctly stored.
2019-05-28 16:37:01 -04:00
Vinoth Kannan
b3779dc377 DEV: rename 'uploads:missing' rake task into 'uploads:missing_files'.
To improve the readability.
2019-05-28 23:30:43 +05:30
Guo Xiang Tan
b1d3c678ca FEATURE: Support [description|attachment](upload://<short-sha>) in MD. (#7603) 2019-05-28 11:18:21 -04:00
Penar Musaraj
42818b810e Better handling of custom SVG sprites in themes when using S3 2019-05-28 10:58:19 -04:00
Ralph Rooding
04abf18194 Fixed typo in S3 rake task 2019-05-28 16:57:55 +08:00
Sam Saffron
3d2c3bd478 FIX: username suggester incorrectly skipping over whitelisted username
SSO uses a special param to username suggester that whitelists a username
due to previous work we amended our lookup logic and started ignoring this
whitelist.

The fix ensures we always respect it, and also improves on the original
implementation that forgot to normalize the username.
2019-05-28 16:48:46 +10:00
Sam Saffron
9a232e1a0a FEATURE: use GIVE_UP=1 to inform rake posts:missing_uploads you are done
We need this give up for cases where uploads can not be recovered

This also improves the recovery routines
2019-05-28 14:44:59 +10:00
Sam Saffron
c780d36ddc DEV: Correct posts:recover_uploads_from_index
Was creating a blank index file by mistake
2019-05-28 12:39:39 +10:00
Sam Saffron
3de8c58134 DEV: clean up backup maxmind path
This follows up on 6580025a

reuses a list and removes a map that was erronuously there.
2019-05-28 11:47:58 +10:00
Sam Saffron
769501c63a FIX: exception which was meant to be ignored and logged was failing
Thanks to Darix!
2019-05-28 11:45:12 +10:00
Sam Saffron
7429700389 FIX: ensure we can download maxmind without redis or db config
This also corrects FileHelper.download so it supports "follow_redirect"
correctly (it used to always follow 1 redirect) and adds a `validate_url`
param that will bypass all uri validation if set to false (default is true)
2019-05-28 10:28:57 +10:00
Sam Saffron
e4e2acf148 FIX: rake maxminddb:get broke cause it required environment 2019-05-28 08:59:28 +10:00
Robin Ward
d95a68b837 FEATURE: When suspending a user, allow the Delete + Replies action
Previously you could only delete the post
2019-05-27 12:27:16 -04:00
Dan Ungureanu
6bd082feab
FIX: Update mapping between locales and Postgres dictionaries. (#7606) 2019-05-27 16:52:09 +03:00
Blake Erickson
373b8a2139 FIX: Consistently handle category param
See https://meta.discourse.org/t/api-post-to-posts-json-inconsistent-between-users/118571
for more info.

This commit removes a 5 year old temporary fix that is no longer needed.

bc1824a6ed (diff-d8c648926664f849aec050757bfcb6f9R72)

The web interface uses category_id when creating a topic so I think we
should unify on category_id when using the api.
2019-05-27 16:39:56 +08:00
Sam Saffron
6580025af9 FEATURE: add backup directory for mmdb files
This new `DISCOURSE_MAXMIND_BACKUP_PATH` can be used a secondary location
for maxmind db. That way a build machine, for example can cache it on the
host and reuse between builds.

Also per 5bfeef77 added proper error raising for download fails from
dedicated rake task

This also moves "refresh_maxmind_db_during_precompile_days" to a global
setting, it did not make sense in a site setting
2019-05-27 16:51:24 +10:00
Guo Xiang Tan
f54e4b71b1 DEV: Make CookedPostProcessor#post_process_images method private. 2019-05-27 11:28:37 +08:00
Dan Ungureanu
da39d66e83
DEV: Improve support for plugins. (#7332)
* DEV: Allow NewPostManager handlers handle PMs.
* DEV: Add custom fields topic option to PostCreator.
* DEV: Add topic_id to serializer data.
* DEV: Wrap topic title from notification item in a span.
2019-05-25 16:53:03 +03:00
Régis Hanol
d6aacadc4e make 🤖 happy 2019-05-25 00:32:29 +02:00
Régis Hanol
4e5305cb67
FIX: catch proper exception 2019-05-25 00:14:54 +02:00
Michael Brown
5bfeef7705 FIX: warn instead of excepting if MaxMind DBs can't be downloaded (#7600) 2019-05-25 00:08:00 +02:00
Régis Hanol
3bc99e5372
FIX: gunzip the downloaded file 2019-05-25 00:02:23 +02:00
Michael Brown
14c7aa8829 FIX: gz_file might be nil here 2019-05-24 16:11:10 -04:00
Robin Ward
89b84651c3 Migrate score settings to use sensitivities
We hide scores so these settings no longer made sense.
2019-05-24 15:44:24 -04:00
Régis Hanol
82f162962f DEV: use different MaxMindDB download endpoint 2019-05-24 20:11:24 +02:00
Régis Hanol
2f5938c3bf DEV: use FileHelper to download MaxMindDB 2019-05-24 18:11:39 +02:00
Régis Hanol
8c9ffee3af TEMP: log more information when failing to download db 2019-05-24 17:33:52 +02:00
Régis Hanol
9ed2c54e27 TEMP: log information when failing to download db 2019-05-24 17:26:04 +02:00
Osama Sayegh
e20c30987c
FEATURE: detect theme errors and catch them (#7589)
* FEATURE: detect theme errors and catch them

* Bump COMPILER_VERSION

* Feedback

* Override eslint no console for one line

* Can't use our ajax method

* remove emoji from translation file
2019-05-24 17:25:55 +03:00
Régis Hanol
a51e2271af DEV: remove temporary rescue when downloading MaxMindDB
follow-up 8c8c925d1b
2019-05-24 15:13:19 +02:00
Gerhard Schlager
c1e9a70d59 FIX: Fallback locale was not available for extra translations
Translations from fallback locales were not sent to the client
for admin_js and wizard_js.
2019-05-24 11:38:26 +02:00
Régis Hanol
0e24cb0f78
DEV: don't skip discourse-calendar plugin specs 2019-05-24 11:35:22 +02:00
Sam Saffron
9ce06d53d1 DEV: improve upload recovery rake task
- Better error handling
- Automatically generates index file if missing
- Improved output
2019-05-24 13:36:03 +10:00
Guo Xiang Tan
791241e0b2 DEV: Fix rake task to delete old export topics. 2019-05-24 11:20:53 +08:00
Sam Saffron
76f7e90e23 FEATURE: rake posts:recover_uploads_from_index improvements
We now recover more smartly based on failed db remaps as well.
2019-05-24 12:38:38 +10:00
Régis Hanol
f9bb8dde3c TEMP: output the error to STDERR 2019-05-23 16:55:04 +02:00
Régis Hanol
8c8c925d1b TEMP: catch exception when failing to download MaxMindDB 2019-05-23 16:30:21 +02:00
Régis Hanol
cb3ddeca40 DEV: remove unnecessary requires 2019-05-23 16:01:40 +02:00
Sam Saffron
1059aafc55 FEATURE: rake task for recovery of uploads from an index file
`rake posts:recover_uploads_from_index`

Searches through all missing uploads in the cluster, if it finds one it
tries to find it in the "upload index file" and creates a new upload for
it.
2019-05-23 17:08:22 +10:00
Sam Saffron
7ce58df7bf lint the file
somehow I did not notice this on save
2019-05-23 15:28:41 +10:00
Sam Saffron
a5ce9cb470 FEATURE: fix_relative_upload_links now multisite safety
This also finds `<img src="/uploads/xyz` HTML images in raw and corrects
them. Also handles some cross multisite recovery and provides better output
2019-05-23 15:09:16 +10:00
Sam Saffron
96e58125a7 FEATURE: rename task to posts:destroy_old_data_exports
Previously we were only catching one type of data export, the new job will
catch every csv export we have

Job is pretty safe as it filters on system user id / pm with a particular
slug
2019-05-23 15:02:56 +10:00
Sam Saffron
954293655f FEATURE: rake posts:destroy_old_user_data_exports
Historically we would keep the user data export posts around but delete
the uploads.

This leaves a lot of broken uploads in the system.

This rake task allows us to clean up old mess.
2019-05-23 11:11:37 +10:00
Bianca Nenciu
631b16ef86 FIX: Fix build.
Follow-up to 25453eb.
2019-05-22 17:39:44 +03:00
Bianca Nenciu
25453eb922 FIX: Avoid logging a warn for 404s. 2019-05-22 15:18:38 +03:00
Gerhard Schlager
e224100023 FIX: Don't convert :) into Emoji when emojis or emoji shurtcuts are disabled 2019-05-22 11:44:39 +02:00
Sam Saffron
e8799f0ba4 DEV: improve uploads:recover job so it stores a map of old to new sha
Previous attempt created broken images
2019-05-22 15:51:09 +10:00
Sam Saffron
ebcb571de7 FIX: allow upload recovery to recover uploads with sha mismatch
Filename on disk may mismatch sha of file in some old 1X setups. This will
attempt to recover file even if sha1 mismatches. We had an old bug that
caused this.

This also adds `uploads:fix_relative_upload_links` which attempts to replace
urls of the format `/upload/default/...` with `upload://`
2019-05-22 15:24:36 +10:00
Sam Saffron
f772ecc597 DEV: Correct missing output detailing failure
uploads:s3_migration_status was failing but not returning proper output
2019-05-22 12:58:54 +10:00
Guo Xiang Tan
5429c9b5e9 DEV: Fix incorrectly hardcoded value in rake task. 2019-05-22 09:01:25 +08:00
Sam Saffron
a9af77d650 PERF: ensure priority for rake rebake tasks is ultra low
Flooding the default queue with lots of jobs is never a good thing. This
reduces the risk of sidekiq going into a bad state post rebake
2019-05-22 10:31:49 +10:00
Sam Saffron
d6daa60d0a FEATURE: multisite support for posts:rebake_uncooked_posts 2019-05-22 10:04:54 +10:00
Sam Saffron
a2428bd862 FEATURE: warn about sidekiq overload prior to migrating
Also makes pre-flight check ensure there is no giant backlog of posts that
need to be cooked
2019-05-22 10:04:33 +10:00
Sam Saffron
5fdc7b7ca2 Correct 59012fc0
Logic was flipped here by mistake, oops
2019-05-22 09:48:03 +10:00