Commit Graph

55 Commits

Author SHA1 Message Date
Dan Ungureanu
49395ec577
FIX: Fix image optimization pipeline (#9257)
* FIX: Do not use original filename to extract the original filename

Prefer extracting filename from the destination path, which is build
using extracted image information.

* UX: Show better error images
2020-03-25 12:59:16 +02:00
Martin Brennan
7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
Dan Ungureanu
4e130f1e03
SECURITY: Remove event handlers from SVG files 2019-12-11 17:04:50 +02:00
Penar Musaraj
102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Régis Hanol
9a81cb9e55 FIX: ensure we remove tempfiles from disk when creating an upload
Follow-up to 46d12c5ad3
2019-10-11 11:13:10 +02:00
Régis Hanol
46d12c5ad3 FIX: properly downsize image on upload
Overwriting the same file with 'convert' is not always working as expected.
Adding a temporary file as the destination of the downsize makes this operation much more reliable.

Also switched to using (the more aggressive) 50% resize instead of halving the number of pixels.
2019-10-04 16:22:57 +02:00
Krzysztof Kotlarek
427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Guo Xiang Tan
7bd93eba3e FIX: Gravatar uploads being dependent on authorized_extensions. 2019-08-01 16:24:09 +08:00
Sam Saffron
30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Guo Xiang Tan
ebca588fd0 DEV: Remove unused line of code. 2019-05-02 16:54:10 +08:00
Régis Hanol
2b721dff57 FIX: whitelist 'feGaussianBlur' and 'filter' svg elements 2019-04-01 12:56:47 +02:00
Penar Musaraj
d6d4a5ba4a FEATURE: support custom icons in themes (#7155)
* First take

* Add support for sprites in themes

Automatically register any custom icons added via themes or plugins

* Fix theme sprite caching

* Simplify test

* Update lib/svg_sprite/svg_sprite.rb

Co-Authored-By: pmusaraj <pmusaraj@gmail.com>

* Fix /svg-sprite/search request
2019-03-15 17:16:15 +11:00
Penar Musaraj
c50db76f5d FIX: do not treat TIFF, BMP, WEBP as images
Treating TIFF and BMP as images cause us to add them to IMG tags, this is very inconsistent across browsers.

You can still upload these files they will simply not be displayed in IMG tags.
2019-02-11 16:28:43 +11:00
Régis Hanol
c256121833 FIX: add support for style element in SVGs 2019-02-07 12:09:06 +01:00
Régis Hanol
788719d271 DEV: speed up posts base imports 2019-01-04 15:30:17 +01:00
Vinoth Kannan
75dbb98cca FEATURE: Add S3 etag value to uploads table (#6795) 2019-01-04 14:16:22 +08:00
Sam
86255faa08 FEATURE: do not switch to JPEG unless you meet 75k byte savings
This also adjusts the algorithm to expect

- 30% saving for JPEG conversion

AND

- Minimum of 75K bytes saved

The reasoning for increase of saving requirements is cause PNG may have been
uploaded unoptimized, 30% saving on PNG is very possible
2018-11-21 11:01:08 +11:00
Guo Xiang Tan
44391ee8ab
FEATURE: Upload Site Settings. (#6573) 2018-11-14 15:03:02 +08:00
Sam
0a442e319c FIX: correct svg handling for images
We regressed and optimized images no longer worked with svg

The following adds the correct logic to simply copy file for svgs
and bypasses resizing for svg avatars
2018-11-07 15:29:26 +11:00
Sam
df45e82377 SECURITY: only allow picking of avatars created by self (#6417)
* SECURITY: only allow picking of avatars created by self

Also adds origin tracking to all uploads including de-duplicated uploads
2018-09-19 22:33:10 -07:00
Guo Xiang Tan
e1b16e445e Rename FileHelper.is_image? -> FileHelper.is_supported_image?. 2018-09-12 09:22:28 +08:00
Sam
9ab1fb7dfc FEATURE: correctly store width and height on uploads
Previously we used width and height for thumbnails, new code ensures

1. We auto correct width and height
2. We added extra columns for thumbnail_width and height, this is determined
 by actual upload and no longer passed in as a side effect
3. Optimized Image now stores filesize which can be used for analysis, decisions

Also

- fixes Android image manifest as a side effect
- fixes issue where a thumbnail generated that is smaller than the upload is no longer used
2018-08-28 12:59:22 +10:00
Régis Hanol
d1607a387a FIX: only allow printable characters in uploads filename 2018-08-21 18:11:01 +02:00
Sam
f5fe58384f correct regression around file renaming 2018-08-20 16:08:05 +10:00
Sam
d7b1919ead correct specs 2018-08-20 12:46:14 +10:00
Sam
8b5e42ea16 FIX: always test and coerce to image on upload
In the past the filename of the origin was used as the source
for the extension of the file when optimizing on upload.

We now use the actual calculated extension based on upload data.
2018-08-20 12:18:59 +10:00
Guo Xiang Tan
a9e502936f FIX: Converting PNG to JPEG does not set the correct extension. 2018-08-17 13:09:48 +08:00
Régis Hanol
98898ff4b3 FIX: raise an exception when 'downsize'git st fails 2018-08-16 19:20:07 +02:00
Guo Xiang Tan
d10c9d7d75 FIX: Missing extensions for non-image uploads due to 2b57239389. 2018-08-13 10:58:55 +08:00
Guo Xiang Tan
2b57239389 FIX: Upload's content is the only source of truth for the file type. 2018-08-07 13:15:00 +08:00
Régis Hanol
f94aeaf6cf SECURITY: force IM decoder based on file extension - part 3 2018-07-25 23:55:06 +02:00
Régis Hanol
800c57c6ab SECURITY: force IM decoder based on file extension - part 2 2018-07-25 23:08:02 +02:00
Régis Hanol
4bf3bf6786 SECURITY: force IM decoder based on file extension 2018-07-25 22:00:04 +02:00
Guo Xiang Tan
1d74ccaaf8 Add compatibility for ImageMagick7. 2018-07-17 15:50:58 +08:00
Guo Xiang Tan
72a3457379 Bump discourse_image_optim which uses a global timeout.
Our previous solution has the timeout set at the worker level
which means the total timeout would be X timeout secs * N number of
workers.
2018-07-09 10:30:18 +08:00
Arpit Jalan
91bf10bd12 FIX: create upload record for exported csv files 2018-04-20 00:27:49 +05:30
Guo Xiang Tan
142571bba0 Remove use of rescue nil.
* `rescue nil` is a really bad pattern to use in our code base.
  We should rescue errors that we expect the code to throw and
  not rescue everything because we're unsure of what errors the
  code would throw. This would reduce the amount of pain we face
  when debugging why something isn't working as expexted. I've
  been bitten countless of times by errors being swallowed as a
  result during debugging sessions.
2018-04-02 13:52:51 +08:00
Régis Hanol
f5e170c6b5 FIX: catch all server-side error when uploading a file
UX: always show a message to the user whenever an error happens on the server when uploading a file
2017-12-27 16:33:25 +01:00
Gerhard Schlager
b47b378cb6 Retry PNG to JPG conversion with debug enabled on failure 2017-12-19 11:49:04 +01:00
Gerhard Schlager
f525d83b53 FIX: empty uploads and blank filenames caused errors during validation 2017-12-13 22:03:31 +01:00
Neil Lalonde
d8c27e3871 Merge branch 'master' into search_posts_by_filetype 2017-07-25 14:41:20 -04:00
Régis Hanol
c7c93e7159 FEATURE: new 'strip image metadata' site setting 2017-07-25 11:48:39 +02:00
Régis Hanol
1c12dd529f FIX: fix orientation even for small JPEGs 2017-07-10 16:35:23 +02:00
Jakub Macina
f0a674d620 Add extraction of upload extension. Add rspec test for search of post
with upload by extension.
2017-07-06 19:16:07 +02:00
Régis Hanol
aa69f78fed let's not use progressing rendering on images 2017-06-26 23:19:48 +02:00
Régis Hanol
ecec23fae4 FIX: automatic PNG to JPG conversion wasn't working 2017-06-26 14:21:47 +02:00
Régis Hanol
a9c0df0b58 FIX: always try to convert PNG to JPG when pasting an image 2017-06-23 12:13:48 +02:00
Régis Hanol
6e5c59e5ec FIX: image orientation wasn't properly working 2017-06-22 16:53:49 +02:00
Guo Xiang Tan
80e348d226 PERF: Speed up slow tests in our test suite.
Before

```
Finished in 7 minutes 23 seconds (files took 4.15 seconds to load)
7145 examples, 0 failures, 10 pending
```

After

```
Finished in 6 minutes 12 seconds (files took 4.41 seconds to load)
7145 examples, 0 failures, 10 pending
```
2017-06-22 11:23:31 +09:00
Régis Hanol
54e8fb0d89 FEATURE: new 'allow_staff_to_upload_any_file_in_pm' site setting 2017-06-12 22:41:29 +02:00