Commit Graph

41 Commits

Author SHA1 Message Date
Martin Brennan
7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
Mark VanLandingham
09d9baa6d7
FIX: Update S3 stubs for more aws-sdk API changes (#8534) 2019-12-11 11:26:52 -08:00
dependabot-preview[bot]
b90a592146 DEV: Bump aws-sdk-sns from 1.13.0 to 1.21.0 (#8490)
Bumps [aws-sdk-sns](https://github.com/aws/aws-sdk-ruby) from 1.13.0 to 1.21.0.
- [Release notes](https://github.com/aws/aws-sdk-ruby/releases)
- [Changelog](https://github.com/aws/aws-sdk-ruby/blob/master/gems/aws-sdk-sns/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-ruby/compare/1.13.0...1.21.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-12-11 06:13:17 -08:00
Guo Xiang Tan
20e783337d DEV: Remove use of fake external store in optimized images spec.
Testing against fake stuff is bad because we're not testing against reality when
reality changes.
2019-05-29 20:13:23 +08:00
Sam Saffron
4ea21fa2d0 DEV: use #frozen_string_literal: true on all spec
This change both speeds up specs (less strings to allocate) and helps catch
cases where methods in Discourse are mutating inputs.

Overall we will be migrating everything to use #frozen_string_literal: true
it will take a while, but this is the first and safest move in this direction
2019-04-30 10:27:42 +10:00
Guo Xiang Tan
839a54b97b FIX: Destroy OptimizedImage record even if Upload record is invalid. 2019-03-22 16:47:06 +08:00
Penar Musaraj
c50db76f5d FIX: do not treat TIFF, BMP, WEBP as images
Treating TIFF and BMP as images cause us to add them to IMG tags, this is very inconsistent across browsers.

You can still upload these files they will simply not be displayed in IMG tags.
2019-02-11 16:28:43 +11:00
Sam
570877da3c FEATURE: store thumbnail algorithm version in optimized image table
Previously we had no idea what algorithm generated thumbnails, this starts tracking the version.

We also bumped up the version to force all optimized images to be generated. This is important cause we recently introduced pngquant which results in much smaller images.
2019-01-03 17:07:30 +11:00
Neil Lalonde
3236f2668f add note to specs about ImageMagick 7 requirement 2019-01-02 12:03:57 -05:00
Sam
766e67ce57 FEATURE: introduce lossy color optimization on resized pngs
This feature ensures optimized images run via pngquant, this results extreme amounts of savings for resized images. Effectively the only impact is that the color palette on small resized images is reduced to 256.

To ensure safety we only apply this optimisation to images smaller than 500k.

This commit also makes a bunch of image specs less fragile.
2019-01-02 17:20:02 +11:00
Jeff Atwood
96aca20fd7 add image optim version note to relevant tests 2019-01-01 01:25:00 -08:00
Robin Ward
662cfc416b FEATURE: Show a blurry preview when lazy loading images
This generates a 10x10 PNG thumbnail for each lightboxed image.
If Image Lazy Loading is enabled (IntersectionObserver API) then
we'll load the low res version when offscreen. As the image scrolls
in we'll swap it for the high res version.

We use a WeakMap to track the old image attributes. It's much less
memory than storing them as `data-*` attributes and swapping them
back and forth all the time.
2018-12-19 01:57:30 +08:00
Sam
0a442e319c FIX: correct svg handling for images
We regressed and optimized images no longer worked with svg

The following adds the correct logic to simply copy file for svgs
and bypasses resizing for svg avatars
2018-11-07 15:29:26 +11:00
Joffrey JAFFEUX
78954672f9 FIX: uses hex to compare images
It prevents some terminals from crashing in case of errors and dumping the whole file content into the terminal.
2018-11-05 09:47:15 -05:00
Sam
adab7a3a48 improve test, also ensure no zero size is generated 2018-10-23 08:50:07 +11:00
Sam
bea8d337b2 DEV: ensure resizing test does not raise bad error
Current resizing test was showing binary diff in terminal and failing
in latest image magick 7, this fixes both issues
2018-10-23 08:45:06 +11:00
Sam
9ab1fb7dfc FEATURE: correctly store width and height on uploads
Previously we used width and height for thumbnails, new code ensures

1. We auto correct width and height
2. We added extra columns for thumbnail_width and height, this is determined
 by actual upload and no longer passed in as a side effect
3. Optimized Image now stores filesize which can be used for analysis, decisions

Also

- fixes Android image manifest as a side effect
- fixes issue where a thumbnail generated that is smaller than the upload is no longer used
2018-08-28 12:59:22 +10:00
Sam
8b5e42ea16 FIX: always test and coerce to image on upload
In the past the filename of the origin was used as the source
for the extension of the file when optimizing on upload.

We now use the actual calculated extension based on upload data.
2018-08-20 12:18:59 +10:00
Guo Xiang Tan
f8b367cc9c Fix the build. 2018-07-26 10:17:51 +08:00
Guo Xiang Tan
30242972d3 Add spec for 4bf3bf6786. 2018-07-26 09:16:14 +08:00
Régis Hanol
4bf3bf6786 SECURITY: force IM decoder based on file extension 2018-07-25 22:00:04 +02:00
Guo Xiang Tan
d839a42bf9 Skip imagemagick tests on Travis. 2018-07-17 17:11:05 +08:00
Guo Xiang Tan
1d74ccaaf8 Add compatibility for ImageMagick7. 2018-07-17 15:50:58 +08:00
Sam
15b5fddd49 SECURITY: protect upload params, only allow very strict filenames 2016-12-19 10:16:18 +11:00
Andy Waite
3e50313fdc Prepare for separation of RSpec helper files
Since rspec-rails 3, the default installation creates two helper files:
* `spec_helper.rb`
* `rails_helper.rb`

`spec_helper.rb` is intended as a way of running specs that do not
require Rails, whereas `rails_helper.rb` loads Rails (as Discourse's
current `spec_helper.rb` does).

For more information:

https://www.relishapp.com/rspec/rspec-rails/docs/upgrade#default-helper-files

In this commit, I've simply replaced all instances of `spec_helper` with
`rails_helper`, and renamed the original `spec_helper.rb`.

This brings the Discourse project closer to the standard usage of RSpec
in a Rails app.

At present, every spec relies on loading Rails, but there are likely
many that don't need to. In a future pull request, I hope to introduce a
separate, minimal `spec_helper.rb` which can be used in tests which
don't rely on Rails.
2015-12-01 20:39:42 +00:00
Sam Saffron
97f03e699d FIX: mixed local s3 store missing avatars 2015-06-01 11:17:42 +10:00
Régis Hanol
5a143c0c6e storage engines refactor 2015-05-29 18:39:47 +02:00
Sam
eeda367e70 FIX: should be able to serve optimized image from local if its ... local... 2015-05-26 12:32:52 +10:00
Régis Hanol
e36272cb68 fix the build 2015-02-09 17:13:22 +01:00
Luciano Sousa
b52f12948a models with rspec3 syntax 2015-01-05 13:04:23 -03:00
Régis Hanol
de76b512c1 fix most deprecations in the specs (still some left) 2014-09-25 17:44:48 +02:00
Sam
ffa11bd1d6 remove cruft 2014-09-25 09:50:43 +10:00
Sam
504cfcff96 Fix specs for avatars
Implement avatar picker
Correct avatar related jobs
2014-05-27 10:08:03 +10:00
Régis Hanol
2505d18aa9 FEATURE: support email attachments 2014-04-14 22:55:57 +02:00
Régis Hanol
37fd7ab574 pull hotlinked images 2013-11-05 19:07:29 +01:00
Régis Hanol
c867b67a0b custom avatar support 2013-08-13 22:08:29 +02:00
Régis Hanol
ed9417fa3b enable thumbnailing on S3
- added url to optimized image model
- refactored s3_store & local_store
2013-07-31 23:26:34 +02:00
Régis Hanol
ac7253a938 refactor CookedPostProcessor & specs 2013-07-08 01:39:08 +02:00
Régis Hanol
af45b5a11e proper column naming
silly schemaless database habits are hard to kill
2013-06-17 04:02:17 +02:00
Régis Hanol
510bac4b27 refactored a bit & tested thumbnails creation 2013-06-17 02:49:34 +02:00
Régis Hanol
5de03814fb created optimized_image model 2013-06-17 02:49:34 +02:00