Commit Graph

48 Commits

Author SHA1 Message Date
Alan Guo Xiang Tan
a6624af66e
DEV: Add isValidUrl helper function to theme migrations ()
This commit adds a `isValidUrl` helper function to the context in
which theme migrations are ran in. This helper function is to make it
easier for theme developers to check if a string is a valid URL or path
when writing theme migrations. This can be helpful in cases when
migrating a string based setting to `type: objects` which contain `type:
string` properties with URL validations enabled.

This commit also introduces the `UrlHelper.is_valid_url?` method
which actually checks that the URL string is of the valid format instead of
only checking if the URL string is parseable which is what `UrlHelper.relaxed_parse` does
and is not sufficient for our needs.
2024-04-30 16:45:07 +08:00
Martin Brennan
4e7929abb6
FIX: Wrong argument error being thrown in UrlHelper ()
We were throwing ArgumentError in UrlHelper.normalised_encode,
but it was incorrect -- we were passing ArgumentError.new
2 arguments which is not supported. Fix this and have a hint
of which URL is causing the issue for debugging.
2023-11-22 15:19:40 +10:00
Ted Johansson
95a82d608d SECURITY: Prevent Onebox cache overflow by limiting downloads and URL lengths 2023-11-09 13:39:18 +11:00
Ted Johansson
47b2667099
DEV: Remove deprecated UrlHelper#escape_uri helper ()
The UrlHelper#escape_uri helper has been deprecated and replaced by UrlHelper#normalized_encode, and was marked for removal in 3.0. This PR removes the method.
2023-10-19 10:37:14 +08:00
David Taylor
b81767c1b1
SECURITY: Limit URL length for theme remote () 2023-03-23 12:01:04 +00:00
Daniel Waterworth
666536cbd1
DEV: Prefer \A and \z over ^ and $ in regexes () 2023-01-20 12:52:49 -06:00
David Taylor
6417173082
DEV: Apply syntax_tree formatting to lib/* 2023-01-09 12:10:19 +00:00
Martin Brennan
8ebd5edd1e
DEV: Rename secure_media to secure_uploads ()
This commit renames all secure_media related settings to secure_uploads_* along with the associated functionality.

This is being done because "media" does not really cover it, we aren't just doing this for images and videos etc. but for all uploads in the site.

Additionally, in future we want to secure more types of uploads, and enable a kind of "mixed mode" where some uploads are secure and some are not, so keeping media in the name is just confusing.

This also keeps compatibility with the `secure-media-uploads` path, and changes new
secure URLs to be `secure-uploads`.

Deprecated settings:

* secure_media -> secure_uploads
* secure_media_allow_embed_images_in_emails -> secure_uploads_allow_embed_images_in_emails
* secure_media_max_email_embed_image_size_kb -> secure_uploads_max_email_embed_image_size_kb
2022-09-29 09:24:33 +10:00
David Taylor
3c81683955 DEV: Rename UriHelper.escape_uri to .normalized_encode
This is a much better description of its function. It performs idempotent normalization of a URL. If consumers truly need to `encode` a URL (including double-encoding of existing encoded entities), they can use the existing `.encode` method.
2022-08-09 11:55:25 +01:00
Sam Saffron
f0a0252526 FIX: broken onebox images due to url normalization bugs
normalized_encode in addressable has a number of issues, including https://github.com/sporkmonger/addressable/issues/472

To temporaily work around those issues for the majority of cases, we try parsing with `::URI`. If that fails (e.g. due to non-ascii characters) then we will fall back to addressable.

Hopefully we can simplify this back to `Addressable::URI.normalized_encode` in the future.

This commit also adds support for unicode domain names and emoji domain names with escape_uri.

This removes an unneeded hack checking for pre-signed urls, which are now handled by the general case due to starting off valid and only being minimally normalized. Previous test case continues to pass.

UrlHelper.s3_presigned_url? which was somewhat wide was removed.
2022-08-09 11:55:25 +01:00
Andrei Prigorshnev
09ad3ed41d
FEATURE: revert disallowing putting URLs in titles for TL0 users ()
This reverts a part of changes introduced by https://github.com/discourse/discourse/pull/13947

In that PR I:
1. Disallowed topic feature links for TL-0 users
2. Additionally, disallowed just putting any URL in topic titles for TL-0 users

Actually, we don't need the second part. It introduced unnecessary complexity for no good reason. In fact, it tries to do the job that anti-spam plugins (like Akismet plugin) should be doing.

This PR reverts this second change.
2021-08-06 20:07:42 +04:00
Andrei Prigorshnev
0c0a11b66a
FEATURE: Disallow putting urls in the title for TL-0 users ()
This disallows putting URLs in topic titles for TL0 users, which means that:

If a TL-0 user puts a link into the title, a topic featured link won't be generated (as if it was disabled in the site settings)
Server methods for creating and updating topics will be refusing featured links when they are called by TL-0 users
TL-0 users won't be able to put any link into the topic title. For example, the title "Hey, take a look at https://my-site.com" will be rejected.

Also, it improves a bit server behavior when creating or updating feature links on topics in the categories with disabled featured links. Before the server just silently ignored a featured link field that was passed to him, now it will be returning 422 response.
2021-08-05 13:38:39 +04:00
Neil Lalonde
c2c647b990
FIX: errors loading secure uploads when secure uploads is disabled () 2021-06-08 13:25:51 -04:00
Josh Soref
59097b207f
DEV: Correct typos and spelling mistakes ()
Over the years we accrued many spelling mistakes in the code base. 

This PR attempts to fix spelling mistakes and typos in all areas of the code that are extremely safe to change 

- comments
- test descriptions
- other low risk areas
2021-05-21 11:43:47 +10:00
David Taylor
13e39d8b9f
PERF: Improve cook_url performance for topic thumbnails ()
- Only initialize the S3Helper when needed
- Skip initializing the S3Helper for S3Store#cdn_url
- Allow cook_url to be passed a `local` hint to skip unnecessary checks
2020-12-30 18:13:13 +00:00
Sam
1cd6ff15dc
PERF: optimize homepage and topic performance ()
These are a few small tweaks that slightly improve performance.

- we omitted 1 query from the post guardian which could cause an N+1
- cook_url has been sped up a bit
- url helper avoids re-creating sets for no reason
2020-12-30 13:08:02 +00:00
jbrw
da9b837da0
DEV: More robust processing of URLs ()
* DEV: More robust processing of URLs

The previous `UrlHelper.encode_component(CGI.unescapeHTML(UrlHelper.unencode(uri))` method would naively process URLs, which could result in a badly formed response.

`Addressable::URI.normalized_encode(uri)` appears to deal with these edge-cases in a more robust way.

* DEV: onebox should use UrlHelper

* DEV: fix spec

* DEV: Escape output when rendering local links
2020-12-03 17:16:01 -05:00
Daniel Waterworth
721ee36425
Replace base_uri with base_path ()
DEV: Replace instances of Discourse.base_uri with Discourse.base_path

This is clearer because the base_uri is actually just a path prefix. This continues the work started in 555f467.
2020-10-09 12:51:24 +01:00
Robin Ward
00afd308c1 FIX: Escaped mailto URLs would raise an exception
This prevents exceptions from being raised if a URL has an invalid
component.
2020-10-05 14:42:44 -04:00
Martin Brennan
39b2fb8649
FIX: Invalid URLs could raise exceptions when calling UrlHelper.rails_route_from_url ()
Upload.secure_media_url? raised an exceptions when the URL was invalid,
which was a issue in some situations where secure media URLs must be
removed.

For example, sending digests used PrettyText.strip_secure_media,
which used Upload.secure_media_url? to replace secure media with
placeholders. If the URL was invalid, then an exception would be raised
and left unhandled.

Now instead in UrlHelper.rails_route_from_url we return nil if there is something wrong with the URL.

Co-authored-by: Bianca Nenciu <nenciu.bianca@gmail.com>
2020-09-30 15:20:00 +10:00
jbrw
e1896f7e71
FIX - rails_route_from_url would fail if path contained unicode characters 2020-08-28 15:10:10 -04:00
Martin Brennan
2352f4bfc7
DEV: Replace SECURE_MEDIA_ROUTE const with other methods ()
This is so if the route changes this const won't be around to bite us, use the Rails route methods instead.
2020-08-28 11:28:11 +10:00
Martin Brennan
5f1749a1c0 Minor change to case-insensitive regex for s3_presigned_url? 2020-02-03 14:22:35 +10:00
Martin Brennan
8d77e99827
FIX: Stop encoding presigned URLs with UrlHelper ()
When FinalDestination is given a URL it encodes it before doing anything else. however S3 presigned URLs should not be messed with in any way otherwise we can end up with 400 errors when downloading the URL e.g.

<Error><Code>InvalidToken</Code><Message>The provided token is malformed or otherwise invalid.</Message>

The signature of presigned URLs is very important and is automatically generated and should be preserved.
2020-01-31 09:09:34 +10:00
Martin Brennan
45b37a8bd1
FIX: Resolve pull hotlinked image and broken link issues for secure media URLs ()
When pull_hotlinked_images tried to run on posts with secure media (which had already been downloaded from external sources) we were getting a 404 when trying to download the image because the secure endpoint doesn't allow anon downloads.

Also, we were getting into an infinite loop of pull_hotlinked_images because the job didn't consider the secure media URLs as "downloaded" already so it kept trying to download them over and over.

In this PR I have also refactored secure-media-upload URL checks and mutations into single source of truth in Upload, adding a SECURE_MEDIA_ROUTE constant to check URLs against too.
2020-01-24 11:59:30 +10:00
Martin Brennan
edbc356593
FIX: Replace deprecated URI.encode, URI.escape, URI.unescape and URI.unencode ()
The following methods have long been deprecated in ruby due to flaws in their implementation per http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-core/29293?29179-31097:

URI.escape
URI.unescape
URI.encode
URI.unencode
escape/encode are just aliases for one another. This PR uses the Addressable gem to replace these methods with its own encode, unencode, and encode_component methods where appropriate.

I have put all references to Addressable::URI here into the UrlHelper to keep them corralled in one place to make changes to this implementation easier.

Addressable is now also an explicit gem dependency.
2019-12-12 12:49:21 +10:00
Martin Brennan
23714e77c4
FIX: Always return secure_proxy_without_cdn url for secure media ()
There was an issue on dev where when uploading secure media, the href of the media was correctly being replaced in the CookedPostProcessor, but the srcset urls were not being replaced correctly. This is because UrlHelper.cook_url was returning the asset host URL for the media for secure media instead of returning early with the proxied secure proxy url.
2019-11-22 15:29:31 +10:00
Penar Musaraj
102909edb3 FEATURE: Add support for secure media ()
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Vinoth Kannan
f92a6f7ac5 FIX: 'local_cdn_url' method should work for local relative urls too. 2019-10-14 11:39:16 +05:30
Sam Saffron
30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Jeff Wong
2863abefdd FIX: detect local assets for subfolder installs 2019-04-13 10:35:23 -07:00
Régis Hanol
664e90bd17 FIX: ensure local images use local CDN when uploads are stored on S3
When the S3 store was enabled, we were only applying the S3 CDN.
So all images stored locally, like the emojis, were never put on the local CDN.

Fixed a bunch of CookedPostProcessor test by adding a call to 'optimize_urls'
in order to get final URLs.

I also removed the unnecessary PrettyText.add_s3_cdn method since this is already
handled in the CookedPostProcessor.
2019-02-20 19:24:38 +01:00
Sam
671469bcc7 FIX: URLs containing two # would fail to work
Some URLs in browsers are non compliant and contain twos `#` this commit adds
special handling for this edge case by auto encoding any fragments containing `#`
2018-12-11 18:03:13 +11:00
Guo Xiang Tan
e1b16e445e Rename FileHelper.is_image? -> FileHelper.is_supported_image?. 2018-09-12 09:22:28 +08:00
Régis Hanol
de92913bf4 FIX: store the topic links using the cooked upload url 2018-08-14 12:23:32 +02:00
Gerhard Schlager
e30851e45a Move escape_uri method to a more suitable place 2017-12-12 20:17:46 +01:00
Sam
59640bae3b FIX: absolute URL for CDN should always be rooted with a protocol 2016-09-05 15:57:46 +10:00
Régis Hanol
caa1aea995 FIX: ensure emojis have absolute URLs and uses CDN 2016-07-15 18:37:51 +02:00
Régis Hanol
5169bcdb6e FIX: httpshttps ultra secure URLs 2016-06-30 16:55:01 +02:00
Sam
7a5a195dc0 FIX: properly support HTTPS CDN on HTTP site
Previously we changed all CDN links to schemaless.

This is desirable for non HTTPS sites, to ease migration to HTTPS.
It is not desirable for secure sites.

Once site is secure or CDN is secure a rebake should be required
to move it back to non-secure.
2015-07-24 14:08:32 +10:00
Régis Hanol
189cb3ff12 FEATURE: move migrate_to_new_scheme into a background job
- new hidden site setting 'migrate_to_new_scheme' (defaults to false)
- new rake tasks to toggle migration to new scheme
- FIX: migrate_to_new_scheme also works with CDN
- PERF: improve perf of the DbHelper.remap method
- REFACTOR: UrlHelper is now a class
2015-06-12 12:07:57 +02:00
Régis Hanol
20c9a312c7 FIX: clicks counter on attachments wasn't always working 2015-02-22 20:47:18 +01:00
Régis Hanol
de76b512c1 fix most deprecations in the specs (still some left) 2014-09-25 17:44:48 +02:00
Régis Hanol
c7330ed73f BUGFIX: errors when post-processing 'data images' 2014-07-18 17:54:18 +02:00
Régis Hanol
b024bebbe2 BUGFIX: emoji's url were relatives (was breaking the wordpress plugin) 2014-01-14 23:51:36 +01:00
Régis Hanol
2ce75a8523 FIX: canonical url should not use the CDN 2013-12-17 00:35:34 +01:00
Régis Hanol
418856a789 make sure assets always use schemaless absolute uris 2013-12-06 11:16:13 +01:00
Régis Hanol
31fd5b85bc FIX markdown hotlinked images were not properly pulled 2013-11-20 13:10:08 +01:00