Commit Graph

61 Commits

Author SHA1 Message Date
Vinoth Kannan
b7830680b6 DEV: use cdn url to download the external uploads to local. 2019-06-06 19:17:19 +05:30
Penar Musaraj
f00275ded3 FEATURE: Support private attachments when using S3 storage (#7677)
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
2019-06-06 13:27:24 +10:00
Guo Xiang Tan
8d1b0224ac Fix the build a3938f98f8. 2019-05-29 18:53:31 +08:00
Guo Xiang Tan
f0620e7118 FEATURE: Support [description|attachment](upload://<short-sha>) in MD take 2.
Previous attempt was missing `post_uploads` records.
2019-05-29 09:26:32 +08:00
Penar Musaraj
7c9fb95c15 Temporarily revert "FEATURE: Support [description|attachment](upload://<short-sha>) in MD. (#7603)"
This reverts commit b1d3c678ca.

We need to make sure post_upload records are correctly stored.
2019-05-28 16:37:01 -04:00
Guo Xiang Tan
b1d3c678ca FEATURE: Support [description|attachment](upload://<short-sha>) in MD. (#7603) 2019-05-28 11:18:21 -04:00
David Taylor
ef660d5a3e FIX: Return consistent character encodings when downloading S3 uploads
Net::HTTP always returns ASCII-8BIT encoding. File.read auto-detects the encoding. This leads to an encoding inconsistency between a fresh download, and a cached download. This commit ensures all downloaded files are treated equally, by always returning the cached version from the filesystem, even during initial download.

One symptom of this problem is during theme exports: https://meta.discourse.org/t/116907

Related ruby ticket: https://bugs.ruby-lang.org/issues/2567
2019-05-17 11:27:00 +01:00
Daniel Waterworth
e219588142 DEV: Prefabrication (test optimization) (#7414)
* Introduced fab!, a helper that creates database state for a group

It's almost identical to let_it_be, except:

 1. It creates a new object for each test by default,
 2. You can disable it using PREFABRICATION=0
2019-05-07 13:12:20 +10:00
Sam Saffron
4ea21fa2d0 DEV: use #frozen_string_literal: true on all spec
This change both speeds up specs (less strings to allocate) and helps catch
cases where methods in Discourse are mutating inputs.

Overall we will be migrating everything to use #frozen_string_literal: true
it will take a while, but this is the first and safest move in this direction
2019-04-30 10:27:42 +10:00
Sam Saffron
45285f1477 DEV: remove update_attributes which is deprecated in Rails 6
See: https://github.com/rails/rails/pull/31998

update_attributes is a relic of the past, it should no longer be used.
2019-04-29 17:32:25 +10:00
Guo Xiang Tan
b0c8fdd7da FIX: Properly support defaults for upload site settings. 2019-03-13 16:36:57 +08:00
Vinoth Kannan
cc496de10e FIX: Remove double quotes from etag value in API response
https://github.com/aws/aws-sdk-ruby/issues/1134
2019-02-08 14:31:19 +05:30
Robin Ward
bee68bba2e FIX: Heisentest
We use the `id` of the upload to calculate a `depth` partition in the
filename. This test would fail if your database had a higher seed
because the depth it was looking for was hard coded to 1.

The solution was to not save the records (which is faster anyway) and
specify the `id` of the upload to make the hash deterministic.
2019-01-16 15:01:50 -05:00
Vinoth Kannan
f94c0283b2
FIX: Use correct version when generating file path for optimized image (#6871) 2019-01-11 18:35:38 +05:30
Vinoth Kannan
75dbb98cca FEATURE: Add S3 etag value to uploads table (#6795) 2019-01-04 14:16:22 +08:00
Guo Xiang Tan
c666ef556d Fix the build.
Ref 570877da3c
2019-01-03 15:34:39 +08:00
Guo Xiang Tan
ce6a0a5e9e FIX: Moving upload to tombstone should update modification time.
A upload created a long time ago will be nuked from the tombstone
immediately if it gets deleted.
2018-09-18 10:48:29 +08:00
Raul Tambre
2271918be2 FEATURE: Use S3 dualstack endpoints
Allows S3 without a CDN to serve images from dualstack domains that also support ipv6
2018-08-27 11:22:46 +10:00
Sam
5d96809abd FIX: improve support for subfolder S3 CDN 2018-08-22 12:31:13 +10:00
Sam
f5142861e5 Revert "Revert "FIX: upload URLs from S3 on subfolder installs""
This reverts commit 26c96e97e5.

We have no choice but to run this code
2018-08-22 11:31:33 +10:00
Sam
26c96e97e5 Revert "FIX: upload URLs from S3 on subfolder installs"
This reverts commit 357df2ff4f.
2018-08-22 10:51:40 +10:00
Neil Lalonde
357df2ff4f FIX: upload URLs from S3 on subfolder installs 2018-08-21 14:58:55 -04:00
Guo Xiang Tan
1ea23b1eae FIX: Wrong order for S3Helper#copy_file. 2018-08-08 15:58:54 +08:00
Guo Xiang Tan
aafff740d2 Add FileStore::S3Store#copy_file. 2018-08-08 11:30:34 +08:00
Andrew Schleifer
dba22bbde2 rollback changes
This reverts:
* 1baba84c438e "fix s3 subfolders harder"
* ea5e57938edf "fix test for absolute_base_url change"
2018-07-06 17:16:40 -05:00
Andrew Schleifer
52e9f49ec1 fix s3 subfolders harder
specifically, include the folder in absolute_base_url
2018-07-06 16:28:40 -05:00
Sam
89ad2b5900 DEV: Rails 5.2 upgrade and global gem upgrade
This updates tests to use latest rails 5 practice
and updates ALL dependencies that could be updated

Performance testing shows that performance has not regressed
if anything it is marginally faster now.
2018-06-07 14:21:33 +10:00
Sam
70bb2aa426 FEATURE: allow specifying s3 config via globals
This refactors handling of s3 so it can be specified via GlobalSetting

This means that in a multisite environment you can configure s3 uploads
without actual sites knowing credentials in s3

It is a critical setting for situations where assets are mirrored to s3.
2017-10-06 16:20:01 +11:00
Sam Saffron
3bdade8970 correct fragile spec 2017-08-31 15:55:56 -04:00
Guo Xiang Tan
8cc8010564 Maintain backwards compatibility before Jobs::MigrateUploadExtensions runs. 2017-08-03 11:56:55 +09:00
Guo Xiang Tan
5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Guo Xiang Tan
13f3de4bf6 Nuke all SiteSetting.stubs from our codebase. 2017-07-07 15:09:14 +09:00
Guo Xiang Tan
3141c179f7 REFACTOR: Get bucket name from S3Helper. 2016-08-19 14:08:37 +08:00
Guo Xiang Tan
7ff1f6cb9d Allow custom bucket name for FileStore::S3Store. 2016-08-16 15:25:42 +08:00
Guo Xiang Tan
0433163866 FEATURE: Support subfolders in SiteSetting.s3_backup_bucket. 2016-08-15 16:14:51 +08:00
Guo Xiang Tan
aa5de3c40a FEATURE: Support subfolders in S3 bucket name.
This commit also fixes a bug where s3 uploads are not
moved to a tombstone folder when removed.
2016-08-15 13:07:41 +08:00
Guo Xiang Tan
3378ee223f FIX: Incorrect path being passed to S3Store#remove_file. 2016-08-15 11:35:30 +08:00
Hu Ming
f8a12d4940 Add support for AWS cn (#4327) 2016-07-14 16:56:09 +02:00
Guo Xiang Tan
bd07658a37
PERF: Split queries when cleaning uploads.
This reduces the number of scans that the db has to do in the query
to fetch orphan uploads. Futheremore, we were not batching our
records which bloats memory.
2016-07-04 16:34:32 +08:00
Guo Xiang Tan
3a140a982f
Fix build. 2016-05-23 11:22:25 +08:00
Neil Lalonde
cf1fefe236 add more specs for Image Lightbox on Subfolder Install 2016-04-20 15:42:40 -04:00
Andy Waite
3e50313fdc Prepare for separation of RSpec helper files
Since rspec-rails 3, the default installation creates two helper files:
* `spec_helper.rb`
* `rails_helper.rb`

`spec_helper.rb` is intended as a way of running specs that do not
require Rails, whereas `rails_helper.rb` loads Rails (as Discourse's
current `spec_helper.rb` does).

For more information:

https://www.relishapp.com/rspec/rspec-rails/docs/upgrade#default-helper-files

In this commit, I've simply replaced all instances of `spec_helper` with
`rails_helper`, and renamed the original `spec_helper.rb`.

This brings the Discourse project closer to the standard usage of RSpec
in a Rails app.

At present, every spec relies on loading Rails, but there are likely
many that don't need to. In a future pull request, I hope to introduce a
separate, minimal `spec_helper.rb` which can be used in tests which
don't rely on Rails.
2015-12-01 20:39:42 +00:00
Régis Hanol
cbc470baf1 fix build 2015-06-01 11:22:58 +02:00
Régis Hanol
5a143c0c6e storage engines refactor 2015-05-29 18:39:47 +02:00
Régis Hanol
4f1a72f729 fix build 2015-05-28 01:38:24 +02:00
Régis Hanol
8e7bfd0f29 FIX: automatically growing uploads tree 2015-05-28 01:03:24 +02:00
Régis Hanol
83d2b59fc3 FIX: s3 endpoint when using 'us-east-1' region 2015-05-27 17:50:49 +02:00
Sam
e17f614771 FIX: fallback to local store when uploads are not on S3 2015-05-26 13:08:31 +10:00
Régis Hanol
bb0c2813ac FEATURE: generate (avatar) thumbnails in a background task
FIX: keep the "uploading..." indicator until the server replies via the MessageBus
FIX: text was disapearing when uploading an avatar

PERF: always use a region for S3 (defaults to 'us-east-1')
FEATURE: ApplyCDN middleware when using S3
FIX: use the same pattern to store files on S3 and locally
PERF: keep a local cache of uploads when generating thumbnails
FEATURE: migrate_to_s3 rake task
2015-05-25 17:59:00 +02:00
Régis Hanol
9ded21e4c6 FIX: consistent and future-proof upload storage pattern 2015-05-19 12:31:12 +02:00