Commit Graph

253 Commits

Author SHA1 Message Date
Martin Brennan
cd1c7d7560
FIX: Copying image markdown for secure media loading full image ()
* When copying the markdown for an image between posts, we were not adding the srcset and data-small-image attributes which are done by calling optimize_image! in cooked post processor
* Refactored the code which was confusing in its current state (the consider_for_reuse method was super confusing) and fixed the issue
2020-04-24 10:29:02 +10:00
Martin Brennan
0388653a4d
DEV: Upload and secure media retroactive rake task improvements ()
* Add uploads:sync_s3_acls rake task to ensure the ACLs in S3 are the correct (public-read or private) setting based on upload security

* Improved uploads:disable_secure_media to be more efficient and provide better messages to the user.

* Rename uploads:ensure_correct_acl task to uploads:secure_upload_analyse_and_update as it does more than check the ACL

* Many improvements to uploads:secure_upload_analyse_and_update

* Make sure that upload.access_control_post is unscoped so deleted posts are still fetched, because they still affect the security of the upload.

* Add escape hatch for capture_stdout in the form of RAILS_ENABLE_TEST_STDOUT. If provided the capture_stdout code will be ignored, so you can see the output if you need.
2020-03-03 10:03:58 +11:00
Martin Brennan
56b16bc68e
FIX: Never allow custom emoji to be marked secure ()
* Because custom emoji count as post "uploads" we were
marking them as secure when updating the secure status for post uploads.
* We were also giving them an access control post id, which meant
broken image previews from 403 errors in the admin custom emoji list.
* We now check if an upload is used as a custom emoji and do not
assign the access control post + never mark as secure.
2020-02-14 11:17:09 +10:00
Martin Brennan
1150cd4621
FIX: Stop secure media URLs being censored too liberally in emails ()
For example /t/ URLs were being replaced if they contained secure-media-uploads so if you made a topic called "Secure Media Uploads Are Cool" the View Topic link in the user notifications would be stripped out.

Refactored code so this secure URL detection happens in one place.
2020-01-30 16:19:14 +10:00
Martin Brennan
ab3bda6cd0
FIX: Mitigate issue where legacy pre-secure hotlinked media would not be redownloaded ()
Basically, say you had already downloaded a certain image from a certain URL
using pull_hotlinked_images and the onebox. The upload would be stored
by its sha as an upload record. Whenever you linked to the same URL again
in a post (e.g. in our case an og:image on review.discourse) we would
would reuse the original upload record because of the sha1.

However when you turned on secure media this could cause problems as
the first post that uses that upload after secure media is enabled
will set the access control post for the upload to the new post.
Then if the post is deleted every single onebox/link to that same image
URL will fail forever with 403 as the secure-media-uploads URL fails
if the access control post has been deleted.

To fix this when cooking posts and pulling hotlinked images, we only
allow using an original upload by URL if its access control post
matches the current post, and if the original_sha1 is filled in,
meaning it was uploaded AFTER secure media was enabled. otherwise
we just redownload the media again to be safe, as the URL will always
be new then.
2020-01-29 10:11:38 +10:00
Martin Brennan
45b37a8bd1
FIX: Resolve pull hotlinked image and broken link issues for secure media URLs ()
When pull_hotlinked_images tried to run on posts with secure media (which had already been downloaded from external sources) we were getting a 404 when trying to download the image because the secure endpoint doesn't allow anon downloads.

Also, we were getting into an infinite loop of pull_hotlinked_images because the job didn't consider the secure media URLs as "downloaded" already so it kept trying to download them over and over.

In this PR I have also refactored secure-media-upload URL checks and mutations into single source of truth in Upload, adding a SECURE_MEDIA_ROUTE constant to check URLs against too.
2020-01-24 11:59:30 +10:00
Martin Brennan
1b3b0708c0
FEATURE: Update upload security status on post move, topic conversion, category change ()
Add TopicUploadSecurityManager to handle post moves. When a post moves around or a topic changes between categories and public/private message status the uploads connected to posts in the topic need to have their secure status updated, depending on the security context the topic now lives in.
2020-01-23 12:01:10 +10:00
Martin Brennan
7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules ()
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
Martin Brennan
e7c7a05097
FIX: Mark secure media upload insecure automatically if used for theme component ()
When uploading a file to a theme component, and that file is existing and has already been marked as secure, we now automatically mark the file as secure: false, change the ACL, and log the action as the user (also rebake the posts for the upload)
2019-11-28 07:32:17 +10:00
Penar Musaraj
102909edb3 FEATURE: Add support for secure media ()
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Krzysztof Kotlarek
427d54b2b0 DEV: Upgrading Discourse to Zeitwerk ()
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Gerhard Schlager
cc7b24b88b FIX: Regularly reset unknown extension of uploads
Discourse will try to detect the extension the next time the upload is needed. Maybe there was just a transient error the last time.
2019-08-21 10:23:20 +02:00
Guo Xiang Tan
7bd93eba3e FIX: Gravatar uploads being dependent on authorized_extensions. 2019-08-01 16:24:09 +08:00
Guo Xiang Tan
73a45048a0 FIX: Upload#short_url generates incorrect URL when extension is nil. 2019-06-19 09:10:50 +08:00
Penar Musaraj
f00275ded3 FEATURE: Support private attachments when using S3 storage ()
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
2019-06-06 13:27:24 +10:00
Guo Xiang Tan
bb98785569 DEV: Fix broken Upload#base62_sha1. 2019-06-04 14:10:46 +08:00
Guo Xiang Tan
f0620e7118 FEATURE: Support [description|attachment](upload://<short-sha>) in MD take 2.
Previous attempt was missing `post_uploads` records.
2019-05-29 09:26:32 +08:00
Penar Musaraj
7c9fb95c15 Temporarily revert "FEATURE: Support [description|attachment](upload://<short-sha>) in MD. ()"
This reverts commit b1d3c678ca.

We need to make sure post_upload records are correctly stored.
2019-05-28 16:37:01 -04:00
Guo Xiang Tan
b1d3c678ca FEATURE: Support [description|attachment](upload://<short-sha>) in MD. () 2019-05-28 11:18:21 -04:00
Sam Saffron
30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Guo Xiang Tan
761e1e25f7 Remove tables incorrectly excluded in Upload.migrate_to_new_scheme.
Follow up to 4a03b21f80.
2019-04-25 20:27:52 +08:00
Vinoth Kannan
84cfe30c1f Use update_all instead of save to select and update particular columns
9ec6ef85b6
2019-04-25 11:56:57 +05:30
Guo Xiang Tan
4a03b21f80 Exclude large tables when remapping in Upload.migrate_to_new_scheme.
Those tables do not carry any information about uploads.
2019-04-25 11:55:48 +08:00
Guo Xiang Tan
be88ac8b7e Additional old Upload#url format for Upload.migrate_to_new_scheme. 2019-04-24 18:19:25 +08:00
Guo Xiang Tan
9ec6ef85b6 Don't validate post when saving in Upload.migrate_to_new_scheme. 2019-04-24 17:17:31 +08:00
Guo Xiang Tan
3c9495b989 DEV: Put a mutex around Upload.migrate_to_new_scheme.
This ensures that only one migration is running at any given point in
time across the instances.
2019-04-24 17:07:10 +08:00
Guo Xiang Tan
55f406bb79 Follow up to 845b68f9ab. 2019-04-24 16:57:45 +08:00
Guo Xiang Tan
845b68f9ab Update possible upload url format for Upload.migrate_to_new_scheme. 2019-04-24 16:26:54 +08:00
Guo Xiang Tan
c811e59898 Avoid hardcoding db name in Upload.migrate_to_new_scheme. 2019-04-24 15:25:24 +08:00
Guo Xiang Tan
573ddcbbd0 Disable migrate_to_new_scheme once there is nothing to migrate. 2019-04-24 14:00:00 +08:00
Guo Xiang Tan
3094a603b7 Follow up to 149411ec90. 2019-04-24 12:20:53 +08:00
Guo Xiang Tan
149411ec90 PERF: Speed up Upload.migrate_to_new_scheme by limiting remap scope.
Doing a `LIKE` on `Post#raw` and `Post#cooked` takes forever on large
sites.
2019-04-24 11:56:48 +08:00
Vinoth Kannan
914ada1c74 DEV: convert scheduled job EnsurePostUploadsExistence into a rake task 2019-04-09 02:07:35 +05:30
Guo Xiang Tan
52c4711e80 DEV: Fix method that was incorrectly made private. 2019-04-03 12:37:50 +08:00
Guo Xiang Tan
4037a2fd0b FIX: Restrict scope of old scheme upload migration.
Some sites have external URLs that don't even match `%/uploads/%' and
some sites surprise me with URLs that contains the default path when it
is a site in a multisite cluster. We can't do anything about those.
2019-04-03 11:56:41 +08:00
Guo Xiang Tan
feb731bffd FIX: Regenerate optimized images instead of migrating from old scheme.
`OptimizedImage.migrate_to_new_scheme` was optimizing optimized images
which we don't need to do. Regnerating the optimized image is way easier.
2019-04-03 09:45:02 +08:00
Guo Xiang Tan
d0a1922ad9 FIX: Upload.migrate_to_new_scheme undefined error when external image fails to download. 2019-03-28 16:00:13 +08:00
Guo Xiang Tan
161a3c3870 DEV: Don't optimize image when migrating to new scheme.
The image has already been uploaded, the migrate to new scheme job's is
just to correct the scheme and not the content of the upload.
2019-03-26 15:12:10 +08:00
Guo Xiang Tan
0ac6a58b09 FIX: Skip validation when saving in Upload#migrate_to_new_scheme. 2019-03-26 14:28:39 +08:00
Guo Xiang Tan
40b03e717b FIX: Upload#migrate_to_new_scheme should not migrate system uploads. 2019-03-14 12:45:03 +08:00
Guo Xiang Tan
b0c8fdd7da FIX: Properly support defaults for upload site settings. 2019-03-13 16:36:57 +08:00
Guo Xiang Tan
58b0e945bd
UX: Lightbox support for image uploader. () 2019-02-21 10:13:37 +08:00
Robin Ward
95f263995d FIX: Previous annotations were broken 2019-01-11 14:30:19 -05:00
Robin Ward
a3839495e0 Update annotations 2019-01-11 12:19:43 -05:00
Vinoth Kannan
75dbb98cca FEATURE: Add S3 etag value to uploads table () 2019-01-04 14:16:22 +08:00
Sam Saffron
303a535dba PERF: automatic upload size calculation not persisted
Previously if upload had missing width and height we would calculate
on first use BUT we (me) forgot to save this to the database

This was particularly bad on home page cause category images (when old)
miss dimensions.
2018-12-26 17:19:29 +02:00
Robin Ward
662cfc416b FEATURE: Show a blurry preview when lazy loading images
This generates a 10x10 PNG thumbnail for each lightboxed image.
If Image Lazy Loading is enabled (IntersectionObserver API) then
we'll load the low res version when offscreen. As the image scrolls
in we'll swap it for the high res version.

We use a WeakMap to track the old image attributes. It's much less
memory than storing them as `data-*` attributes and swapping them
back and forth all the time.
2018-12-19 01:57:30 +08:00
Robin Ward
e593d68beb Use an options hash instead of boolean parameters 2018-12-19 01:57:30 +08:00
Penar Musaraj
f8e6a37858 FIX: raise exception when getting dimensions of missing image
- follow-up on 0eacd45ab1
2018-12-03 10:19:49 -05:00
Penar Musaraj
0eacd45ab1 FIX: refactor ImageSizer.resize
reverts 140d9c2
2018-11-29 15:28:45 -05:00
Penar Musaraj
140d9c2910 FIX: call ImageSizer only if width/height are available 2018-11-29 15:03:02 -05:00
Guo Xiang Tan
44391ee8ab
FEATURE: Upload Site Settings. () 2018-11-14 15:03:02 +08:00
Arpit Jalan
3e17ef0507
Merge pull request from techAPJ/user-field-unique-key
FEATURE: add external details to user fields
2018-09-20 22:29:39 +05:30
Sam
df45e82377 SECURITY: only allow picking of avatars created by self ()
* SECURITY: only allow picking of avatars created by self

Also adds origin tracking to all uploads including de-duplicated uploads
2018-09-19 22:33:10 -07:00
Arpit Jalan
72be638728 FEATURE: add external details to user fields 2018-09-20 08:10:51 +05:30
Guo Xiang Tan
4a92c5b2d6 UploadRecovery should recover attachments too. 2018-09-19 10:44:36 +08:00
Guo Xiang Tan
3884e99e88 Add extra protection in Upload#get_from_url.
In case the extension goes missing from the URL.
2018-09-12 00:12:14 -07:00
Guo Xiang Tan
b3469bea2d FIX: Uploads not being linked correctly to posts.
Regression due to 1f636c445b.
2018-09-11 23:50:23 -07:00
Guo Xiang Tan
e1b16e445e Rename FileHelper.is_image? -> FileHelper.is_supported_image?. 2018-09-12 09:22:28 +08:00
Guo Xiang Tan
04d26c65e2 Refactor Upload.get_from_url to check length of sha1. 2018-09-10 10:10:39 +08:00
Guo Xiang Tan
1f636c445b PERF: Add fast path to find uploads before resorting to LIKE query.
For a normal upload url

Before

```
Warming up --------------------------------------
                       264.000  i/100ms
Calculating -------------------------------------
                          2.754k (± 8.4%) i/s -     13.728k in   5.022066s
```

After

```
Warming up --------------------------------------
                       341.000  i/100ms
Calculating -------------------------------------
                          3.435k (±11.6%) i/s -     17.050k in   5.045676s
```
2018-09-06 14:44:24 +08:00
Sam
e1975e293f FIX: when uploads are destroyed clear up avatar refs in user table
This also auto corrects twice daily when we ensure consistency
2018-08-31 14:46:42 +10:00
Sam
9ab1fb7dfc FEATURE: correctly store width and height on uploads
Previously we used width and height for thumbnails, new code ensures

1. We auto correct width and height
2. We added extra columns for thumbnail_width and height, this is determined
 by actual upload and no longer passed in as a side effect
3. Optimized Image now stores filesize which can be used for analysis, decisions

Also

- fixes Android image manifest as a side effect
- fixes issue where a thumbnail generated that is smaller than the upload is no longer used
2018-08-28 12:59:22 +10:00
Sam
9628c3cf97 FEATURE: automatically correct extension for bad uploads
This fixes with post thumbnails on the fly
2018-08-17 14:00:27 +10:00
Sam
796164b58c FIX: automatically correct bad avatars on access
Also start relying on upload extension for optimized images
2018-08-16 16:32:56 +10:00
Régis Hanol
de92913bf4 FIX: store the topic links using the cooked upload url 2018-08-14 12:23:32 +02:00
Régis Hanol
0cd9e2acb9 fix build 2018-08-04 01:56:26 +02:00
Régis Hanol
bf4d98e89d FIX: always store topic links using the upload url 2018-08-04 01:29:32 +02:00
Guo Xiang Tan
875008522d FIX: Discourse.S3BaseUrl did not account for subfolder bucket names. 2018-07-06 15:53:57 +08:00
Guo Xiang Tan
7163bf9323 FIX: Upload.get_from_url not respective subfolder in s3 bucket names. 2018-07-06 11:37:11 +08:00
Maja Komel
6c9190357c fix get from url for external uploads without CDN 2018-06-13 16:21:28 +10:00
Arpit Jalan
290ee312e6 FIX: handle invalid mailto links 2018-05-18 18:02:52 +05:30
Arpit Jalan
539cf32f87 FIX: handle encoded mailto links when looking for upload record 2018-05-17 12:44:53 +05:30
Arpit Jalan
91bf10bd12 FIX: create upload record for exported csv files 2018-04-20 00:27:49 +05:30
Guo Xiang Tan
142571bba0 Remove use of rescue nil.
* `rescue nil` is a really bad pattern to use in our code base.
  We should rescue errors that we expect the code to throw and
  not rescue everything because we're unsure of what errors the
  code would throw. This would reduce the amount of pain we face
  when debugging why something isn't working as expexted. I've
  been bitten countless of times by errors being swallowed as a
  result during debugging sessions.
2018-04-02 13:52:51 +08:00
Guo Xiang Tan
bfe38b2118 FIX: limit wasn't correctly applied to Upload.migrate_to_new_scheme. 2018-03-22 10:56:06 +08:00
Guo Xiang Tan
226ace1643 Update annotations. 2018-02-20 14:28:58 +08:00
Rafael dos Santos Silva
b9a343afe7 FIX: Allow shared CDN for s3 and assets 2018-01-12 01:08:15 -02:00
Arpit Jalan
daeb7694bc update annotations 2017-12-05 21:03:20 +05:30
Sam
4ea87b5ab8 Merge branch 's3_refactor' 2017-10-09 10:27:52 +11:00
Sam
70bb2aa426 FEATURE: allow specifying s3 config via globals
This refactors handling of s3 so it can be specified via GlobalSetting

This means that in a multisite environment you can configure s3 uploads
without actual sites knowing credentials in s3

It is a critical setting for situations where assets are mirrored to s3.
2017-10-06 16:20:01 +11:00
Guo Xiang Tan
d67f0b39ae Update annotations. 2017-10-06 11:13:01 +08:00
Sam
f766ea4257 Correct short url decoding for sha1s leading with zero
This was picked up cause we had a flaky test!
2017-08-23 11:08:18 -04:00
Sam
bcf7dc38c2 FEATURE: server side support for upload:// markdown
This allows uploads to be specified using short sha1 hash instead of full
URL

Client side change is pending
2017-08-22 11:46:23 -04:00
Sam
c68999e128 annotate models
WARNING annotators out there, be to run bin/annotate on RAILS_ENV=test on a clean db
2017-08-16 10:38:11 -04:00
Neil Lalonde
5d528f0d15 Merge pull request from dmacjam/search_posts_by_filetype
FEATURE: Search posts by filetype
2017-07-31 11:55:34 -04:00
Guo Xiang Tan
5012d46cbd Add rubocop to our build. () 2017-07-28 10:20:09 +09:00
Neil Lalonde
d8c27e3871 Merge branch 'master' into search_posts_by_filetype 2017-07-25 14:41:20 -04:00
Régis Hanol
c7c93e7159 FEATURE: new 'strip image metadata' site setting 2017-07-25 11:48:39 +02:00
Jakub Macina
8c445e9f17 Fix backend code for searching by a filetype as a combination of uploads and topic links. Add rspec test for extracting file extension in upload. 2017-07-06 19:19:31 +02:00
Régis Hanol
54e8fb0d89 FEATURE: new 'allow_staff_to_upload_any_file_in_pm' site setting 2017-06-12 22:41:29 +02:00
Robin Ward
cdbe027c1c Refactor FileHelper to use keyword arguments. 2017-05-24 13:54:26 -04:00
Régis Hanol
13e489b4ca replace the upload type whitelist with a sanitizer 2017-05-18 12:13:13 +02:00
Arpit Jalan
8e5b0c79ae FIX: allow images to be uploaded in wizard 2017-05-18 13:53:23 +05:30
Arpit Jalan
8c337ecf82 FIX: allow uploading of category logo and background 2017-05-11 12:48:54 +05:30
Régis Hanol
214939bb87 freeze array constants 2017-05-11 09:08:59 +02:00
Régis Hanol
9641d2413d REFACTOR: upload workflow creation into UploadCreator
- Automatically convert large-ish PNG/BMP to JPEG
- Updated fast_image to latest version
2017-05-11 00:16:57 +02:00
Sam
bc0b9af576 FEATURE: support uploads for themes
This allows themes to bundle various assets
2017-05-10 15:47:11 -04:00
Guo Xiang Tan
2af1b9e93c Add time out when optimizing images. 2017-04-20 15:21:43 +08:00
Guo Xiang Tan
60f1169077 REFACTOR: Reduce repetition in code. 2017-04-18 17:03:49 +08:00