Penar Musaraj
9f6c4ad71a
FIX: inconsistency in S3 inventory config ( #11112 )
...
Ensures it matches S3 inventory config generation in our hosting.
2020-11-05 08:39:40 -05:00
Jarek Radosz
e00abbe1b7
DEV: Clean up S3 specs, stubs, and helpers
...
Extracted commonly used spec helpers into spec/support/uploads_helpers.rb, removed unused stubs and let definitions. Makes it easier to write new S3-related specs without copy and pasting setup steps from other specs.
2020-09-28 12:02:25 +01:00
Martin Brennan
80268357e7
DEV: Change upload verified column to be integer ( #10643 )
...
Per review https://review.discourse.org/t/dev-add-verified-to-uploads-and-fill-in-s3-inventory-10406/14180
Change the verified column for Upload to a verified_status integer column, to avoid having NULL as a weird implicit status.
2020-09-17 13:35:29 +10:00
David Taylor
bd0a7553c4
DEV: Detect when s3 inventory failure is caused by etag difference ( #10427 )
2020-08-13 09:30:28 +10:00
Martin Brennan
b950b3fb3f
DEV: Add verified to uploads and fill in S3 inventory ( #10406 )
...
When we run the S3 inventory, mark uploads that exist as verified true, those that don't as verified false, and uploads not included in the check / not yet checked as verified nil.
2020-08-11 14:43:51 +10:00
David Taylor
16c65a94f7
PERF: Preload S3 inventory data for multisite clusters
2020-07-29 10:31:55 +01:00
David Taylor
ec4024fe6d
FIX: Keep by_users check in S3 inventory
...
Partial revert of 8515d8fa
2020-07-21 17:19:56 +01:00
David Taylor
7f2b5a446a
PERF: Remove post_upload recovery in daily EnsureS3UploadsExistence job ( #10173 )
...
This is a very expensive process, and it should only be required in exceptional circumstances. It is possible to run a similar recovery using `rake uploads:recover` (5284d41a8e/lib/upload_recovery.rb (L135-L184)
2020-07-06 16:26:40 +01:00
Jarek Radosz
73b04976e5
FIX: Use updated_at in the S3 inventory job ( #8823 )
...
When we change upload's sha1 (e.g. when resizing images) it won't match the data in the most recent S3 inventory index. With this change the uploads that have been updated since the inventory has been generated are ignored.
2020-01-31 11:02:44 +01:00
Vinoth Kannan
9919ee1900
FIX: remove the tmp inventory files after the s3 uploads check.
2019-08-13 11:52:57 +05:30
Vinoth Kannan
ad04ce9f43
FIX: remove post upload record creation inside 'find_missing_uploads' method.
2019-07-19 01:44:08 +05:30
Vinoth Kannan
35d6fff69e
PERF: use url instead of file key in temporary inventory table.
2019-06-13 22:03:58 +05:30
Vinoth Kannan
2a7065c505
FIX: skip uploads without etag in s3 inventory check.
2019-05-20 00:09:52 +05:30
Vinoth Kannan
e8fafbc123
List and restore missing post uploads from S3 inventory.
2019-05-04 01:16:20 +05:30
Sam Saffron
4ea21fa2d0
DEV: use #frozen_string_literal: true on all spec
...
This change both speeds up specs (less strings to allocate) and helps catch
cases where methods in Discourse are mutating inputs.
Overall we will be migrating everything to use #frozen_string_literal: true
it will take a while, but this is the first and safest move in this direction
2019-04-30 10:27:42 +10:00
Daniel Waterworth
ad44243a57
Removed unused let blocks ( #7446 )
...
The bodies of these blocks were never evaluated.
2019-04-29 15:08:56 +08:00
Vinoth Kannan
a8f410a9c5
FEATURE: Create new helper method 'Discourse.stats' ( #7388 )
2019-04-17 12:45:04 +05:30
Vinoth Kannan
df6ef856e6
DEV: save missing s3 uploads count in redis
2019-04-04 19:05:57 +05:30
Guo Xiang Tan
243fb8d9ad
Fix the build.
2019-03-13 17:39:07 +08:00
Vinoth Kannan
563b953224
DEV: Add 'backfill_etags_' to the method name since it also backfilling the etags
2019-02-19 21:54:35 +05:30
Vinoth Kannan
0472bd4adc
FIX: Remove 'backfill_etags' keyword argument from 'uploads:missing' rake task
...
And etags backfilling code is optimized
2019-02-15 00:34:35 +05:30
Vinoth Kannan
b5fbd7385f
FIX: run the rake task only for uploads created before a day from inventory date
2019-02-14 17:53:08 +05:30
Vinoth Kannan
f7d03b2406
Stop the automatic creation of inventory 'gz' file in fixtures
2019-02-14 14:25:45 +05:30
Vinoth Kannan
7b5931013a
Update rake task to backfill etags from s3 inventory
2019-02-14 05:18:06 +05:30
Vinoth Kannan
1045bbc35b
FIX: S3 inventory data can be splitted into multiple csv files
2019-02-14 03:41:52 +05:30
Vinoth Kannan
ff12c4b2d4
FIX: Bucket name is missing in S3 inventory data path
2019-02-06 19:16:08 +05:30
Vinoth Kannan
b4f713ca52
FEATURE: Use amazon s3 inventory to manage upload stats ( #6867 )
2019-02-01 10:10:48 +05:30
