Commit Graph

7412 Commits

Author SHA1 Message Date
Daniel Waterworth
4a9ee25c56 DEV: Add logging for stack level too deep exception in HtmlToMarkdown 2020-07-09 12:25:00 +01:00
Martin Brennan
e0713455ca
PERF: Load topic bookmarks for the user in user_post_bookmarks (#10197)
Instead of loading all of the user bookmarks using all the post IDs in a topic, load all the bookmarks for a user using the topic ID. This eliminates a costly WHERE ID IN query.
2020-07-09 15:46:52 +10:00
Martin Brennan
31e31ef449
SECURITY: Add content-disposition: attachment for SVG uploads
* strip out the href and xlink:href attributes from use element that
  are _not_ anchors in svgs which can be used for XSS
* adding the content-disposition: attachment ensures that
  uploaded SVGs cannot be opened and executed using the XSS exploit.
  svgs embedded using an img tag do not suffer from the same exploit
2020-07-09 13:31:48 +10:00
Bianca Nenciu
bd842cd2b0
FEATURE: Parse images in email signatures (#10137)
* FEATURE: Parse images in email signatures

* DEV: Fix tests

* Code review
2020-07-08 15:50:30 +10:00
Guo Xiang Tan
2e1eafae06
FIX: Incorrect search blurb when advanced search filters are used. 2020-07-08 11:59:49 +08:00
Gerhard Schlager
ab4c0a4970 FEATURE: Create SQL-only backup if there are no uploads
It doesn't make sense to compress the database dump twice if the backup doesn't contain any uploaded files.
2020-07-07 16:23:47 +02:00
Joffrey JAFFEUX
9b7000dbf1
FIX: ensures category order keeps consistent (#10165)
Before this change:
- first full page load would get category defaults defined un cateory settings
- a navigation to a topic and then back to categories list would reset defaut to the ones defined in discovery/topics
2020-07-07 09:56:38 +02:00
Jeff Wong
339549d14a
Support plugin and Theme compatibility version manifests (#9995)
Adds a new rake task `plugin:checkout_compatible_all` and
`plugin:checkout_compatible[plugin-name]` that check out compatible plugin
versions.

Supports a .discourse-compatibility file in the root of plugins and themes that
list out a plugin's compatibility with certain discourse versions:

eg: .discourse-compatibility
```
2.5.0.beta6: some-git-hash
2.4.4.beta4: some-git-tag
2.2.0: git-reference
```

This ensures older Discourse installs are able to find and install older
versions of plugins without intervention, through the manifest only.

It iterates through the versions in descending order. If the current Discourse
version matches an item in the manifest, it checks out the listed plugin target.
If the Discourse version is greater than an item in the manifest, it checks out
the next highest version listed in the manifest.

If no versions match, it makes no change.
2020-07-06 14:48:00 -07:00
Jarek Radosz
cb048d284d FIX: Handle the case where upload goes missing during downsizing 2020-07-06 18:51:38 +02:00
David Taylor
7f2b5a446a
PERF: Remove post_upload recovery in daily EnsureS3UploadsExistence job (#10173)
This is a very expensive process, and it should only be required in exceptional circumstances. It is possible to run a similar recovery using `rake uploads:recover` (5284d41a8e/lib/upload_recovery.rb (L135-L184))
2020-07-06 16:26:40 +01:00
Jarek Radosz
64ce12a758
FIX: OptimizedImage#filesize (#10095)
`OptimizedImage#filesize` calls `Discourse.store.download` with an OptimizedImage as an argument. It would in turn attempt to call `#original_filename` and `#secure?` on that object. Both would fail as these methods do not exist on OptimizedImage, only on Upload. We didn't know about these issues because:
1. `#calculate_filesize` is not called often, because the filesize is saved on OptimizedImage creation, so it's used mostly for manual filesize recalculation
2. we were using `rescue nil` which swallows all errors
2020-07-06 17:01:29 +02:00
Daniel Waterworth
1bd8a075d8 FIX: Make Email::Styles operate on html documents instead of fragments
`Nokogiri::HTML.fragment` is a huge hack (a comment in the source code
admits this). The current behavior of `Email::Styles` is to try to
emulate `fragment` using nokogumbo, but it misses some edge cases. In
particular, meta tags in a email template don't make it through to the
final email.

Instead of treating the provided HTML as an indeterminate fragment, this
commit makes `Email::Styles` treat the HTML as a complete document. This
means that the generated HTML for an email will now always contain top
level structure (a doctype, html, head and body tags).

This new behavior is behind a hidden site setting for now and defaults
off.
2020-07-06 11:45:39 +01:00
David Taylor
977766e7a8
FEATURE: sso_overrides_(email|username|name) for all auth methods
These settings previously applied only to discourse-sso. Now they work for all external authentication methods.
2020-07-06 10:18:45 +01:00
David Taylor
ec448a1516
DEV: Refactor Auth::Result for readability, recreate during signup flow 2020-07-06 10:18:41 +01:00
Vinoth Kannan
6d17765924 PERF: use post number to create canoncial path in mega topics.
We don't need page number accuracy in mega topics since it can be expensive.

06d426bd87
2020-07-06 10:31:19 +05:30
Sam Saffron
38a30a6e96
DEV: correct regression and correct tests
etag change in 31976ecf was incorrect, revert it

Also correct regression in test suite.
2020-07-06 10:56:19 +10:00
Sam Saffron
31976ecfeb
PERF: only update etag when it changes
Previously when synchronizing upload etags we would update every single one
regardless of change.
2020-07-06 10:40:04 +10:00
Vinoth Kannan
06d426bd87 FIX: skip hidden posts while generating canonical url.
Previously, while generating the topic page's canoncial url we used the current post number. It will create invalid canonical path if the topic has whsiper posts. Now we only taking the visible posts for current page index calculation.
2020-07-05 14:04:31 +05:30
romanrizzi
6b4cebed3e DEV: Instance#replace_flags block should be optional 2020-07-03 16:21:06 -03:00
Régis Hanol
48b4ed41f5 FIX: uploading an existing image as a site setting
The previous fix (f43c0a5d85) wasn't working for images that were already uploaded.
The "metadata" (eg. 'for_*' and 'secure' attributes) were not added to existing uploads.

Also used 'Upload.get_from_url' is the admin/site_setting controller to properly retrieve
an upload from its URL.

Fixed the Upload::URL_REGEX to use the \h (hexadecimal) for the SHA

Follow-up-to: f43c0a5d85
2020-07-03 19:16:54 +02:00
Martin Brennan
8ef782bdbd
FIX: Increase time of DOWNLOAD_URL_EXPIRES_AFTER_SECONDS to 5 minutes (#10160)
* Change S3Helper::DOWNLOAD_URL_EXPIRES_AFTER_SECONDS to 5 minutes, which controls presigned URL expiry and secure-media route cache time.
* This is done because of the composer preview refreshing while typing causes a lot of requests sent to our server because of the short URL expiry. If this ends up being not enough we can always increase the time or explore other avenues (e.g. GitHub has a 7 day validity for secure URLs)
2020-07-03 13:42:36 +10:00
Roman Rizzi
2df388ffd7
DEV: Plugins can extend ReviewableScore types. (#10156) 2020-07-02 11:47:43 -03:00
Guo Xiang Tan
6bab2acc9f
Fix typo.
Follow up to af52df2d
2020-07-02 14:23:10 +08:00
Guo Xiang Tan
af52df2d96
DEV: Add hidden site setting for PG search ranking normalization. 2020-07-02 14:11:18 +08:00
Gerhard Schlager
fc8e842773 FIX: Sometimes not all output of psql was logged during restores
There was a race condition which could prevent Discourse from logging the last couple of lines of output from psql.
2020-06-30 16:52:50 +02:00
Dan Ungureanu
fe284ffd06
Revert "DEV: Remove useless code (#10130)"
Some oneboxes still generate empty P tags (video oneboxes).

This reverts commit c299d02287.
2020-06-29 13:56:28 +03:00
Régis Hanol
860deeb072 FIX: identify slug-less topic urls everywhere
In 91c89df6, I fixed the onebox to support local topics with a slug-less URL.
This commit fixes all the other spots (search, topic links and user badges) where we look up for a local topic.

Follow-up-to: 91c89df6
2020-06-29 12:31:20 +02:00
David Taylor
0edffcc47d
FIX: Correct version comparison logic when comparing stable to beta (#10135)
* FIX: Correct version comparison logic when comparing stable to beta

For example, version 1.3.0 should be considered higher than 1.3.0.beta3. So `Discourse.has_needed_version?('1.3.0', '1.3.0.beta3')` should return true

* Switch to use Gem::Version to compare versions
2020-06-29 17:52:33 +10:00
Dan Ungureanu
c299d02287
DEV: Remove useless code (#10130)
protection is not needed and can easily be bypassed with empty divs anyway.
2020-06-29 17:49:30 +10:00
Blake Erickson
69803599a9 DEV: Refactor seed data filter
Added a small helper class to for seed data because we need to add the
same filter to multisite:migrate as we have in db:migrate. Having this
filter in both places means we can get rid of the SKIP_SEED flag.
2020-06-26 14:36:50 -06:00
Guo Xiang Tan
c16ad39f8e
DEV: Run seeds irregardless of post deploy migration flag.
Follow up to 01937b2d
2020-06-26 11:04:34 +08:00
Guo Xiang Tan
01937b2de2
Revert "FIX: Seed needs to run before optimizing site icons."
This reverts commit 715ddf3861.
2020-06-26 11:03:47 +08:00
Guo Xiang Tan
715ddf3861
FIX: Seed needs to run before optimizing site icons. 2020-06-26 08:58:53 +08:00
Justin DiRose
768bb406ee
DEV: Add knowledge-explorer to official plugin list (#10127) 2020-06-25 14:52:14 -05:00
Blake Erickson
01b6349a67
DEV: Add skip seed flag (#10116)
* add a flag to skip seed

* only seed when running post deployment migrations
2020-06-25 10:14:58 -06:00
Sam Saffron
689568c216
FIX: invalid urls should not break store.has_been_uploaded?
Breaking this method has wide ramification including breaking
search indexing.
2020-06-25 15:00:15 +10:00
Sam Saffron
3cb41d5429
PERF: stop adding more topics to search when not needed
The logic of adding additional search results does not seem to be
needed anymore.

It appears to be a relic of an old implementation.

This saves an entire search query for every search made.
2020-06-25 12:31:12 +10:00
Neil Lalonde
b9174c8e8a
Version bump to v2.6.0.beta1 2020-06-24 14:00:19 -04:00
Régis Hanol
7109d94ee7 FIX: properly invalidate inline oneboxes when rebaking
When rebaking a post we were invalidating _regular_ oneboxes but not inline oneboxes.

DEV: also renamed 'InlineOneboxer.purge' to 'InlineOneboxer.invalidate' to keep
the API consistent with 'Oneboxer.invalidate'
2020-06-24 11:54:54 +02:00
Guo Xiang Tan
b28d97b64a
FIX: Bump onebox for twitch video and clips embedding fix. 2020-06-24 11:00:30 +08:00
Régis Hanol
91c89df68a FIX: onebox local topic when using slug-less URL
When linking to a topic in the same Discourse, we try to onebox the link to show the title
and other various information depending on whether it's a "standard" or "inline" onebox.

However, we were not properly detecting links to topics that had no slugs (eg. https://meta.discourse.org/t/1234).
2020-06-23 17:18:38 +02:00
Daniel Waterworth
368af327fa DEV: Reduce size of begin-rescue region
Follow-up-to: e3e7905d9e
2020-06-23 10:14:09 +01:00
Guo Xiang Tan
84c12d8f3d
DEV: Allow multisite apps to boot with readonly db. 2020-06-23 16:34:25 +08:00
Martin Brennan
e92909aa77
FIX: Use ActionDispatch::Http::ContentDisposition for uploads content-disposition (#10108)
See https://meta.discourse.org/t/broken-pipe-error-when-uploading-to-a-s3-clone-a-pdf-with-a-name-containing-e-i-etc/155414

When setting content-disposition for attachment, use the ContentDisposition class to format it. This handles filenames with weird characters and localization (accented characters) correctly.
2020-06-23 17:10:56 +10:00
Guo Xiang Tan
1157d2a0ff
DEV: Print proper summary when errors have been reporter in turbo_rspec 2020-06-23 13:34:58 +08:00
Guo Xiang Tan
e3e7905d9e
FIX: TurboTests::Runner not failing on errors. 2020-06-23 11:45:18 +08:00
Guo Xiang Tan
0384b6d910
FIX: multisite:migrate failing to properly seed data. 2020-06-23 09:10:02 +08:00
Robin Ward
ac9577bcc7 FIX: Don't raise an exception if we can't update the user on demotion
This is causing issues when purging old users, if they are set up in the
exact condition where they will be demoted into another group, but also
do not have a primary email.
2020-06-22 15:43:59 -04:00
Dan Ungureanu
2e1efbde52
FIX: Pass local scope variable to inner function 2020-06-22 19:23:59 +03:00
Bianca Nenciu
68f767a557
FEATURE: Check if selectable avatars exist before enabling them (#10032) 2020-06-22 16:58:26 +03:00
Guo Xiang Tan
3370ef188e
FEATURE: Remove deprecated uploads url site settings.
The site settings have been replaced with direct image upload since
Discourse 2.3.
2020-06-22 14:32:29 +08:00
Aman Gupta Karmani
8a86705e51
FIX: handle heroku style HTTP_X_REQUEST_START (#10087) 2020-06-19 10:17:24 -04:00
Daniel Waterworth
9cf77372a2 FIX: Guardian#can_remove_allowed_users? shouldn't break for ownerless topics
A topic can outlive its original author. TopicGuardian should still work
in this situation.
2020-06-19 10:35:52 +01:00
Gerhard Schlager
2da6faf281 FEATURE: Add before_email_send event
Plugins can use it to customize the message (e.g. add header) before the email is sent.
2020-06-18 18:58:51 +02:00
Bernhard Suttner
e31471585a
DEV: allow to have duplicate topic titles if categegory is different (#10034)
Co-authored-by: Robin Ward <robin.ward@gmail.com>

Co-authored-by: Robin Ward <robin.ward@gmail.com>
2020-06-18 11:19:47 -04:00
Bianca Nenciu
db1bebddce
FIX: Hide the post history for TL4 (#10065) 2020-06-18 13:27:51 +03:00
Dan Ungureanu
d21a08c284
DEV: Deprecate Category#url_with_id in favor of Category#url (#9972) 2020-06-18 11:32:14 +03:00
Robin Ward
e8756e1a95 FIX: Muted/Ignore should prevent PMs regardless of case sensitivity 2020-06-17 14:26:14 -04:00
David Taylor
e29afa200a
FIX: Cleanup migrations with timestamps in the future
A future-dated migration was accidently introduced by me in 45c399f0. This was removed in b9762afc, but other migrations had already been generated based on its incorrect date. This commit removes the offending data in the schema_migrations table, and corrects the version in the published_pages migration.

This commit also adds a check to db:migrate which raises an error when invalid migration timestamps are used.
2020-06-17 15:58:22 +01:00
Guo Xiang Tan
45eb97c202
FIX: Thread safety issues with multisite:migrate and SeedFu. 2020-06-17 16:15:43 +08:00
Guo Xiang Tan
828ceab64b
DEV: Make rubocop happy. 2020-06-17 15:47:05 +08:00
Martin Brennan
e5da2d24e5
FIX: Add attachment content-disposition for all non-image files (#10058)
This will make it so the original filename is used when downloading all non-image files, bringing S3Store into line with the to_s3 migration and local storage. Video and audio files will still stream correctly in HTML players as well.

See https://meta.discourse.org/t/cannot-download-non-image-media-files-original-filenames-lost-when-uploaded-to-s3/152797 for a lot of extra context.
2020-06-17 11:16:37 +10:00
Krzysztof Kotlarek
dcb816b548
FIX: add table name to topic view query (#10052)
When plugin is hooking into TopicView joining other tables, it may fail because `created_at` is potentially available on 2 tables. Therefore we should explicitly define which `created_at` we want.
2020-06-17 10:40:01 +10:00
Robin Ward
7f8c5cf70b FIX: Allow plugins to provide test directories with transpiled .js 2020-06-16 14:31:01 -04:00
Jarek Radosz
669c940ec3 Revert "DEV: Remove the remaining ENV["TRAVIS"] usage (#10041)"
This reverts commit 78aff841e3.

See https://review.discourse.org/t/dev-remove-the-remaining-env-travis-usage-10041/12737/4?u=cvx
2020-06-16 19:42:00 +02:00
Jarek Radosz
78aff841e3
DEV: Remove the remaining ENV["TRAVIS"] usage (#10041) 2020-06-16 17:41:15 +02:00
Gerhard Schlager
859d9b75a7 FIX: Restoring backup from PG12 could fail on PG10
The `EXECUTE FUNCTION` syntax for `CREATE TRIGGER` statements was introduced in PostgreSQL 11. We need to replace `EXECUTE FUNCTION` with `EXECUTE PROCEDURE` in order to be able to restore backups created with PG12 on PG10.
2020-06-16 16:04:14 +02:00
Gerhard Schlager
4cff4892e8 Locale checks were disabled by mistake 2020-06-16 16:04:14 +02:00
Dan Ungureanu
84dfaad137
FIX: Fill acting_user field instead of target_user in history 2020-06-16 13:30:58 +03:00
Blake Erickson
a2713578dd DEV: Allow plugins to exclude seed data
This allows plugins to specify if they would like to filter out any seed
data files from running during migrations.
2020-06-15 15:30:25 -06:00
Guo Xiang Tan
58e52c0e4f
DEV: Use rails_failover gem for ActiveRecord and Redis failover handling 2020-06-15 15:47:07 +08:00
Guo Xiang Tan
d8cd912769
DEV: Switch to db config to disable advisory locks. 2020-06-15 14:33:41 +08:00
Guo Xiang Tan
0ff86b00cb
DEV: Upgrade Redis to 4.2.1. 2020-06-15 10:05:22 +08:00
Gerhard Schlager
36a3675e0a Update translations 2020-06-14 23:39:33 +02:00
Joffrey JAFFEUX
4b793a1072
FIX: allows PM owner to remove any user if >= TL2 (#10036) 2020-06-12 12:54:28 +02:00
Andrew Schleifer
b2c94cc8ea FIX: do not migrate backups in the new prefix 2020-06-12 02:56:07 +00:00
Guo Xiang Tan
78b5ab746c
DEV: No longer need to clear anon cache when toggling readonly mode. 2020-06-12 09:58:17 +08:00
Roman Rizzi
a4bfa35420
DEV: Pass the is_image flag when triggering the before_upload_creation event (#10031) 2020-06-11 11:06:48 -03:00
Jarek Radosz
3d55f2e3b7
FIX: Improvements and fixes to the image downsizing script (#9950)
Fixed bugs, added specs, extracted the upload downsizing code to a class, added support for non-S3 setups, changed it so that images aren't downloaded twice.

This code has been tested on production and successfully resized ~180k uploads.

Includes:

* DEV: Extract upload downsizing logic
* DEV: Add support for non-S3 uploads
* DEV: Process only images uploaded by users
* FIX: Incorrect usage of `count` and `exist?` typo
* DEV: Spec S3 image downsizing
* DEV: Avoid downloading images twice
* DEV: Update filesizes earlier in the process
* DEV: Return false on invalid upload
* FIX: Download images that currently above the limit (If the image size limit is decreased, then there was no way to resize those images that now fall outside the allowed size range)
* Update script/downsize_uploads.rb (Co-authored-by: Régis Hanol <regis@hanol.fr>)
2020-06-11 14:47:59 +02:00
Dan Ungureanu
b7e70850e4
FIX: Allow users to add emails which were deleted before 2020-06-11 14:54:11 +03:00
Guo Xiang Tan
2e8075bac3
FIX: Ignore Redis readonly errors in RateLimiter#rollback!.
This is similar to what we're doing in `RateLimiter#performed!`.
2020-06-11 15:13:11 +08:00
Guo Xiang Tan
90dc6c8ce5
DEV: Fix deprecation warning when using Redis#close with redis namespace. 2020-06-11 14:09:19 +08:00
Guo Xiang Tan
531eca985e
DEV: Log error backtrace in Discourse.warn_exception without logster. 2020-06-11 10:49:46 +08:00
Kane York
5f3fd23fa8
DEV: Add stub flush_sw task to prevent rebuild errors (#10021) 2020-06-10 13:07:37 -07:00
romanrizzi
ac555610d9 DEV: Trigger an event before creating an upload 2020-06-10 16:30:53 -03:00
Neil Lalonde
81a265cecd
Version bump to v2.5.0.beta7 2020-06-10 13:38:18 -04:00
Dan Ungureanu
5bfe1ee4f1
FEATURE: Improve UX support for multiple email addresses (#9691) 2020-06-10 19:11:49 +03:00
Guo Xiang Tan
a3dfd553a1
Revert "Bump redis to 4.2.0."
This reverts commit 98bc28cea2.
2020-06-10 14:52:05 +08:00
Guo Xiang Tan
98bc28cea2
Bump redis to 4.2.0. 2020-06-10 14:28:56 +08:00
Jeff Wong
70a88111dd
FIX: prevent re-flagging when we have reviewed flags before (#10010)
FIX: prevent re-flagging when we have reviewed flags before

Fixes an edge case where a review can be reflagged when:
User flags as inappropriate.
Moderator rejects the flag.
Another user re-flags the post as spam.

Before, anyone was able to re-flag as inappropriate despite it being flagged
previously. With this, users are unable to re-flag for the same reason
regardless of reviewable status.
2020-06-09 15:26:10 -07:00
Arpit Jalan
3094459cd9
FEATURE: multiple use invite links (#9813) 2020-06-09 20:49:32 +05:30
Guo Xiang Tan
c35649ccf1
DEV: Remove Redis noop logging.
When Redis drops into readonly mode, it floods the log. The log isn't
actionable and we have other means of knowing that an app is in readonly
mode.
2020-06-09 16:45:34 +08:00
Guo Xiang Tan
a1c13eb3c6
DEV: Redis failover should only clear redis recently readonly. 2020-06-09 16:36:31 +08:00
Guo Xiang Tan
4065cd198a
DEV: Print backtrace of error when plugin fails to initialize. 2020-06-09 10:25:43 +08:00
Joffrey JAFFEUX
d4caf69ed7
DEV: makes SKIP_INSTALL_PLUGINS called last and global (#9990) 2020-06-05 17:59:23 +02:00
Sam Saffron
1c48853ede
DEV: allow skipping of a single plugin when installing
SKIP_INSTALL_PLUGINS can be used to tell plugin:install_all_official
to skip a plugin.

Comma seperated list
2020-06-05 18:26:25 +10:00
Guo Xiang Tan
e82d4d8a75
DEV: Update rails_failover to avoid monkey patching Rails config. 2020-06-05 09:05:19 +08:00
Martin Brennan
36841210fb Move UploadSecurity public types into PUBLIC_TYPES constant
* this is so plugins can extend the public types if required
2020-06-05 10:56:57 +10:00
Penar Musaraj
2d880b42a3
UX: Add simple-list setting type (#9970) 2020-06-04 10:44:54 -04:00
Joshua Rosenfeld
76af25f753
FEATURE: Support append when bulk tagging via rake (#9978)
New argument option for the `tags:bulk_tag_category` rake task.
Backwards compatible - defaults to false (no append) if no argument given.
2020-06-04 09:33:48 -04:00