Commit Graph

283 Commits

Author SHA1 Message Date
Joffrey JAFFEUX
5eb6e9281a
FIX: manually adds frowning_face_with_open_mouth for apple (#13528) 2021-07-21 23:27:20 +02:00
Michael Brown
76a11e6dc9 DEV: fix test (missed a reference to master) 2021-07-19 12:47:45 -04:00
Michael Brown
aa12d12c0b discourse/discourse change from 'master' to 'main': update fixture data 2021-07-19 11:46:15 -04:00
David Taylor
8b89787426
SECURITY: Sanitize YouTube Onebox data (#13748)
CVE-2021-32764
2021-07-15 19:31:50 +01:00
jbrw
a64aea38b7
FIX: Don’t use user_generated images as avatar images in Oneboxed Twitter content (#13712)
By default, Twitter will return the URL for the avatar image of the tweet poster as the `og:image` value.

However, if the `user_generated` attribute is true, we should not use this as the avatar URL as this will be an URL of an image in the tweet itself (e.g., an image belonging to a tweeted news story).
2021-07-13 14:54:28 -04:00
Jarek Radosz
48b92d8897
DEV: Isolate multisite specs (#13634)
Mixing multisite and standard specs can lead to issues (e.g. when using `fab!`)
Disabled the (upcoming https://github.com/discourse/rubocop-discourse/pull/11) rubocop rule for two files that have thoroughly tangled both types of specs.
2021-07-07 18:57:42 +02:00
Penar Musaraj
35110f6681
FIX: Set CSP base-uri to self (#13654) 2021-07-07 09:43:48 -04:00
Arpit Jalan
05bdbd9f97
SECURITY: Onebox canonical links bypassing FinalDestination checks (#13605) 2021-07-01 20:09:29 +05:30
Arpit Jalan
b63c9febe8
FIX: ignore canonical link to localhost (#13577) 2021-06-30 13:55:17 +05:30
Krzysztof Kotlarek
a6363170e9 FIX: flaky search-spec
More precise expectations for search spec
2021-06-29 10:06:44 +08:00
Jarek Radosz
046a875222
DEV: Improve script/downsize_uploads.rb (#13508)
* Only shrink images that are used in Posts and no other models
* Don't save the upload if the size is the same
2021-06-24 00:09:40 +02:00
David Taylor
d2c5165052 FIX: Check all migrations for dropped columns/tables during restore
Previously only post-deploy migrations were being checked for DROPPED_(COLUMNS|TABLES) constants
2021-06-23 17:43:38 +01:00
Alan Guo Xiang Tan
44aa46ca05 Code review comments. 2021-06-21 11:06:58 +08:00
Alan Guo Xiang Tan
8e3691d537 PERF: Eager load Theme associations in Stylesheet Manager.
Before this change, calling `StyleSheet::Manager.stylesheet_details`
for the first time resulted in multiple queries to the database. This is
because the code was modelled in a way where each `Theme` was loaded
from the database one at a time.

This PR restructures the code such that it allows us to load all the
theme records in a single query. It also allows us to eager load the
required associations upfront. In order to achieve this, I removed the
support of loading multiple themes per request. It was initially added
to support user selectable theme components but the feature was never
completed and abandoned because it wasn't a feature that we thought was
worth building.
2021-06-21 11:06:58 +08:00
jbrw
fbfd1fd80b
FIX: Allow SVG uploads if dimensions are a fraction of a unit (#13409)
* FIX: Allow SVG uploads if dimensions are a fraction of a unit

`UploadCreator` counts the number of pixels in an file to determine if it is valid. `pixels` is calculated by multiplying the width and height of the image, as determined by FastImage.

SVG files can have their width/height expressed in a variety of different units of measurement. For example, ‘px’, ‘in’, ‘cm’, ‘mm’, ‘pt’, ‘pc’, etc are all valid within SVG files. If an image has a width of `0.5in`, FastImage may interpret this as being a width of `0`, meaning it will report the `size` as being `0`.

However, we don’t need to concern ourselves with the number of ‘pixels’ in a SVG files, as that is irrelevant for this file format, so we can skip over the check for `pixels == 0` when processing this file type.

* DEV: Speed up getting SVG dimensions

The `-ping` flag prevents the entire image from being rasterized before a result is returned. See:

https://imagemagick.org/script/command-line-options.php#ping
2021-06-17 15:56:11 -04:00
Jarek Radosz
e36377d9ab
DEV: Don't user before(:all)/after(:all) (#13389)
Leaking state and non-obvious order (before :all runs *before* RailsHelper.test_setup) are not worth it.
A replacement PR for #13370. Fixes some flaky specs, e.g.
```
bin/rspec './spec/components/freedom_patches/translate_accelerator_spec.rb[1:3]' './spec/jobs/clean_up_user_export_topics_spec.rb[1:1]' --tag ~type:multisite --seed 35994
```

Also included:
* DEV: No need for locale reset (we do it anyway in rails_helper in `test_setup`)
2021-06-15 17:25:06 +02:00
Roman Rizzi
fa57316a4e
FIX: Validate upload is still valid after calling the "before_upload_creation" event (#13091)
Since we use the event to perform additional validations on the file, we should check if it added any errors to the upload before saving it. This change makes the UploadCreator more consistent since we no longer have to rely on exceptions.
2021-06-15 10:10:03 -03:00
Penar Musaraj
6f76479054
FEATURE: Add upgrade-insecure-requests to CSP when force_https is enabled (#13348)
If force_https is enabled all resource (including markdown preview and so on) will be accessed using HTTPS

If for any reason you attempt to link to non HTTPS reachable content content may appear broken
2021-06-10 10:53:10 +10:00
Penar Musaraj
8336e732d3
DEV: Add manifest-src to CSP (#13319)
Defaults to `manifest-src: 'self'` and allows plugins/themes to extend it.
2021-06-08 09:32:31 -04:00
Penar Musaraj
f90c4bd6a1
DEV: Allow plugins to extend frame-ancestors (#13316) 2021-06-07 14:59:15 -04:00
jbrw
09bc95d46b
FIX: Quoting Oneboxed content should exclude formatting (#13296)
* FIX: Quoting Oneboxed content should exclude formatting

When a post is quoted that includes Oneboxed content, we should not include the formatting generated by the Onebox. Rather, we should attempt to collapse the link referenced by the Onebox to a single line text link.

* DEV: fix tests
2021-06-07 13:03:53 -04:00
Sam
435c4817cb
FEATURE: enable tagging by default (#13175)
Over the years we have found that a few communities never discovered tags.

Instead of having them default off we now have them default on, ensuring
that everyone finds out about them.

Co-authored-by: Dan Ungureanu <dan@ungureanu.me>
2021-06-07 18:07:46 +03:00
Arpit Jalan
2e4f07678e
FIX: IMDb links were being oneboxed as posters (#13310)
IMDb movie links were being rendered as posters. This was because
IMDb was sending `og:type` as `image` randomly in some cases. To
fix this we'll now default all IMDb links as article type. This will
ensure that the IMDb onebox link includes all the information instead
of showing just a poster without any context.
2021-06-07 18:45:59 +05:30
jbrw
461a2c334b
FIX: return an empty result if response from Amazon is missing expected attributes (#13173)
* FIX: return an empty result if response from Amazon is missing attributes

Check we have the basic attributes requires to construct a Onebox for Amazon.

This is an attempt to handle scenarios where we receive a valid 200-status response from an Amazon request that does not include the data we’re expecting.

* Update lib/onebox/engine/amazon_onebox.rb

Co-authored-by: Régis Hanol <regis@hanol.fr>

Co-authored-by: Régis Hanol <regis@hanol.fr>
2021-06-01 16:23:18 -04:00
Gerhard Schlager
3df928d609
DEV: Fix flaky specs (#13226)
Some specs failed when `LOAD_PLUGINS=1` was set while migrating the test DB and the narrative-bot plugin disabled the `send_welcome_message` site setting.
2021-06-01 14:38:55 +02:00
Penar Musaraj
06e1af2b1d
FIX: Giphy oneboxing when the response is an image (#13199) 2021-05-28 15:10:32 -04:00
Penar Musaraj
47e09700fe
FIX: Support pausing GIFs for giphy/tenor oneboxes (#13194) 2021-05-28 08:40:30 -04:00
Martin Brennan
501de809da
FIX: Do not mark badge image uploads as secure (#13193)
* FIX: Do not mark badge image uploads as secure

We do not need badge_image upload types to be marked as secure.
Post migration is the same as
https://github.com/discourse/discourse/pull/12081.

See
https://meta.discourse.org/t/secure-media-uploads/140017/122?u=martin
2021-05-28 12:35:52 +10:00
Martin Brennan
964da21817
FEATURE: Improve group email settings UI (#13083)
This overhauls the user interface for the group email settings management, aiming to make it a lot easier to test the settings entered and confirm they are correct before proceeding. We do this by forcing the user to test the settings before they can be saved to the database. It also includes some quality of life improvements around setting up IMAP and SMTP for our first supported provider, GMail. This PR does not remove the old group email config, that will come in a subsequent PR. This is related to https://meta.discourse.org/t/imap-support-for-group-inboxes/160588 so read that if you would like more backstory.

### UI

Both site settings of `enable_imap` and `enable_smtp` must be true to test this. You must enable SMTP first to enable IMAP.

You can prefill the SMTP settings with GMail configuration. To proceed with saving these settings you must test them, which is handled by the EmailSettingsValidator.

If there is an issue with the configuration or credentials a meaningful error message should be shown.

IMAP settings must also be validated when IMAP is enabled, before saving.

When saving IMAP, we fetch the mailboxes for that account and populate them. This mailbox must be selected and saved for IMAP to work (the feature acts as though it is disabled until the mailbox is selected and saved):

### Database & Backend

This adds several columns to the Groups table. The purpose of this change is to make it much more explicit that SMTP/IMAP is enabled for a group, rather than relying on settings not being null. Also included is an UPDATE query to backfill these columns. These columns are automatically filled when updating the group.

For GMail, we now filter the mailboxes returned. This is so users cannot use a mailbox like Sent or Trash for syncing, which would generally be disastrous.

There is a new group endpoint for testing email settings. This may be useful in the future for other places in our UI, at which point it can be extracted to a more generic endpoint or module to be included.
2021-05-28 09:28:18 +10:00
Dan Ungureanu
723d7de18c
Various GitHub Onebox improvements (#13163)
* FIX: Improve GitHub folder regexp in Onebox

It used to match any GitHub URL that was not matched by the other GitHub
Oneboxes and it did not do a good job at handling those. With this
change, the generic Onebox will handle the remaining URLs.

* FEATURE: Add Onebox for GitHub Actions

* FEATURE: Add Onebox for PR check runs

* FIX: Remove image from GitHub folder Oneboxes

It is a generic, auto-generated image which does not provide any value.

* DEV: Add tests

* FIX: Strip HTML comments from PR body
2021-05-27 12:38:42 +03:00
Arpit Jalan
283b08d45f
DEV: Absorb onebox gem into core (#12979)
* Move onebox gem in core library

* Update template file path

* Remove warning for onebox gem caching

* Remove onebox version file

* Remove onebox gem

* Add sanitize gem

* Require onebox library in lazy-yt plugin

* Remove onebox web specific code

This code was used in standalone onebox Sinatra application

* Merge Discourse specific AllowlistedGenericOnebox engine in core

* Fix onebox engine filenames to match class name casing

* Move onebox specs from gem into core

* DEV: Rename `response` helper to `onebox_response`

Fixes a naming collision.

* Require rails_helper

* Don't use `before/after(:all)`

* Whitespace

* Remove fakeweb

* Remove poor unit tests

* DEV: Re-add fakeweb, plugins are using it

* Move onebox helpers

* Stub Instagram API

* FIX: Follow additional redirect status codes (#476)

Don’t throw errors if we encounter 303, 307 or 308 HTTP status codes in responses

* Remove an empty file

* DEV: Update the license file

Using the copy from https://choosealicense.com/licenses/gpl-2.0/#

Hopefully this will enable GitHub to show the license UI?

* DEV: Update embedded copyrights

* DEV: Add Onebox copyright notice

* DEV: Add MIT license, convert COPYRIGHT.txt to md

* DEV: Remove an incorrect copyright claim

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Co-authored-by: jbrw <jamie@goatforce5.org>
2021-05-26 15:11:35 +05:30
Penar Musaraj
6e9ee3db03
FIX: Sort user bookmarks by reminder date (#13145) 2021-05-25 15:23:46 -04:00
Josh Soref
59097b207f
DEV: Correct typos and spelling mistakes (#12812)
Over the years we accrued many spelling mistakes in the code base. 

This PR attempts to fix spelling mistakes and typos in all areas of the code that are extremely safe to change 

- comments
- test descriptions
- other low risk areas
2021-05-21 11:43:47 +10:00
Arpit Jalan
130160537c
FEATURE: add support for "skip_validations" option in UploadCreator (#13094)
FIX: do not validate uploads when running `uploads:fix_missing_s3` task
2021-05-19 20:54:52 +05:30
Blake Erickson
23de5e959e
DEV: Remove duplicate test (#12989)
I think this test was duplicated on accident. We test for the
exact thing a few lines up.
2021-05-07 15:34:14 -06:00
Krzysztof Kotlarek
e29605b79f
FEATURE: the ability to search users by custom fields (#12762)
When the admin creates a new custom field they can specify if that field should be searchable or not.

That setting is taken into consideration for quick search results.
2021-04-27 15:52:45 +10:00
David Taylor
3436fef5e3
FIX: Update database_restorer to avoid shell use (#12731)
Follow-up to 0ec5fd5262
2021-04-16 13:39:45 +01:00
Osama Sayegh
cd24eff5d9
FEATURE: Introduce theme/component QUnit tests (take 2) (#12661)
This commit allows themes and theme components to have QUnit tests. To add tests to your theme/component, create a top-level directory in your theme and name it `test`, and Discourse will save all the files in that directory (and its sub-directories) as "tests files" in the database. While tests files/directories are not required to be organized in a specific way, we recommend that you follow Discourse core's tests [structure](https://github.com/discourse/discourse/tree/master/app/assets/javascripts/discourse/tests).

Writing theme tests should be identical to writing plugins or core tests; all the `import` statements and APIs that you see in core (or plugins) to define/setup tests should just work in themes.

You do need a working Discourse install to run theme tests, and you have 2 ways to run theme tests:

* In the browser at the `/qunit` route. `/qunit` will run tests of all active themes/components as well as core and plugins. The `/qunit` now accepts a `theme_name` or `theme_url` params that you can use to run tests of a specific theme/component like so: `/qunit?theme_name=<your_theme_name>`.

* In the command line using the `themes:qunit` rake task. This take is meant to run tests of a single theme/component so you need to provide it with a theme name or URL like so: `bundle exec rake themes:qunit[name=<theme_name>]` or `bundle exec rake themes:qunit[url=<theme_url>]`.

There are some refactors to how Discourse processes JavaScript that comes with themes/components, and these refactors may break your JS customizations; see https://meta.discourse.org/t/upcoming-core-changes-that-may-break-some-themes-components-april-12/186252?u=osama for details on how you can check if your themes/components are affected and what you need to do to fix them.

This commit also improves theme error handling in Discourse. We will now be able to catch errors that occur when theme initializers are run and prevent them from breaking the site and other themes/components.
2021-04-12 15:02:58 +03:00
Osama Sayegh
2b9ab3a0d9
Revert "FEATURE: Introduce theme/component QUnit tests (#12517)" (#12632)
This reverts commit a53d8d3e61 and 105634435f.

Reverted because the change broke some components. Will be added back in a few days.
2021-04-07 17:45:49 +03:00
Osama Sayegh
105634435f
FIX: Prevent double slashes in Ember templates paths (#12630)
Follow-up to https://github.com/discourse/discourse/pull/12517
2021-04-07 14:08:29 +03:00
Penar Musaraj
5096920500
FEATURE: Implement nonces for Google Tag Manager integration (#12531) 2021-03-26 11:19:31 -04:00
Sam
5a1318e3c3
DEV: upgrade mini_sql (#12465)
* DEV: upgrade mini_sql

Even though we are not planning on using this quite yet, mini_sql now supports
prepared statements.

Would like this upgrade merged so we can do some benchmarking.

Note, this will not work with pg_bouncer, but sites that are not using it
may benefit from the feature.

* implement multisite friendly prepared statements
2021-03-24 08:48:04 +11:00
Rafael dos Santos Silva
fb4486d5f1
FEATURE: Add CSP frame-ancestors support (#12404) 2021-03-22 16:00:25 -03:00
Martin Brennan
49f4c548ef
FEATURE: Bookmark pinning (#12431)
Users can now pin bookmarks from their bookmark list. This will anchor the bookmark to the top of the list, and show a pin icon next to it. This also applies in the nav bookmarks panel. If there are multiple pinned bookmarks they sort by last updated order.
2021-03-22 09:50:22 +10:00
Blake Erickson
44153cde18
FIX: Be able to handle long file extensions (#12375)
* FIX: Be able to handle long file extensions

Some applications have really long file extensions, but if we truncate
them weird behavior ensues.

This commit changes the file extension size from 10 characters to 255
characters instead.

See:

https://meta.discourse.org/t/182824

* Keep truncation at 10, but allow uppercase and dashes
2021-03-17 12:01:29 -06:00
Arpit Jalan
64d2f260a9
FIX: do not convert format for site setting uploads (#12410) 2021-03-16 20:14:41 +05:30
Gerhard Schlager
58c218a4bf
FIX: Remap old S3 endpoints during backup restore (#12276)
It also starts outputting exceptions on the console.
2021-03-03 21:10:09 +01:00
jbrw
a9b6f4d829
FIX - use ImageMagick to determine size of svg images (#12230)
SVG files can have dimensions expressed in inches, centimeters, etc., which may lead to the dimensions being misinterpreted (e.g. “8in” ends up as 8 pixels).

If the file type is `svg`, ask ImageMagick to work out what size the SVG file should be rendered on screen.

NOTE: The `pencil.svg` file was obtained from https://freesvg.org/1534028868, which has placed the file in to the public domain.
2021-03-01 11:44:00 -05:00
Martin Brennan
9f0f801ae3
FIX: Do not mark group_flair images as secure on upload (#12081)
See https://meta.discourse.org/t/secure-media-uploads-breaks-group-flair-image/173671/4

Group flair image uploads definitely do not need to be secure.
2021-02-16 12:34:03 +10:00
jbrw
70050a8ba3
FIX: should_alter_quality should respect png_to_jpg_quality (#12055)
`convert_to_jpeg!` is only called if `convert_png_to_jpeg?` and/or `should_alter_quality?` is true.

`convert_png_to_jpeg?` can be disabled by setting `SiteSetting.png_to_jpg_quality` to 100.

However, `should_alter_quality?` could be true if `SiteSetting.recompress_original_jpg_quality` was lower than the quality of the uploaded file, regardless of file type.

This commits changes `should_alter_quality?` so that uploaded png files will use the `SiteSetting.png_to_jpg_quality` value, rather than ``SiteSetting.recompress_original_jpg_quality` value.
2021-02-12 13:37:35 -05:00