Commit Graph

203 Commits

Author SHA1 Message Date
Robin Ward
9ba8bfb1aa FIX: Multisite DB was leaving old data in test mode
This commit introduces a new helper to enable transactional fixtures
when testing multisite. This would show up as tests that passed the
first time then failed the second time due to stale data being leftover.
2019-01-09 15:20:37 -05:00
Kyle Zhao
dec8e5879a FEATURE: set CSP base-uri and object-src to none (#6863) 2019-01-09 15:04:50 -05:00
Vinoth Kannan
75dbb98cca FEATURE: Add S3 etag value to uploads table (#6795) 2019-01-04 14:16:22 +08:00
Sam
766e67ce57 FEATURE: introduce lossy color optimization on resized pngs
This feature ensures optimized images run via pngquant, this results extreme amounts of savings for resized images. Effectively the only impact is that the color palette on small resized images is reduced to 256.

To ensure safety we only apply this optimisation to images smaller than 500k.

This commit also makes a bunch of image specs less fragile.
2019-01-02 17:20:02 +11:00
Régis Hanol
5381096bfd PERF: new 'migrate_to_s3' rake task 2018-12-26 17:34:49 +01:00
Gerhard Schlager
2bdbca3801 DEV: Remove unnecessary to_not raise_error from specs
Follow-up to 01cdbd3a13
2018-12-17 16:10:10 +01:00
Gerhard Schlager
01cdbd3a13 FEATURE: Prohibit S3 bucket reusage
This validation makes sure that the s3_upload_bucket and the
s3_backup_bucket have different values. The backup bucket is
allowed to be a subfolder of the upload bucket. The other way
around is forbidden because the backup system searches by
prefix and would return all files stored within the backup
bucket and its subfolders.
2018-12-17 11:35:28 +01:00
Gerhard Schlager
1a8ca68ea3 FEATURE: Improve backup stats on admin dashboard
* Dashboard doesn't timeout anymore when Amazon S3 is used for backups
* Storage stats are now a proper report with the same caching rules
* Changing the backup_location, s3_backup_bucket or creating and deleting backups removes the report from the cache
* It shows the number of backups and the backup location
* It shows the used space for the correct backup location instead of always showing used space on local storage
* It shows the date of the last backup as relative date
2018-12-17 11:35:11 +01:00
Kyle Zhao
b0c2e9bb05
minor changes to default script-src (#6770)
- add report-sample to force require a sample of the violating code
- do not whitelist GA/GTM's entire domain
2018-12-14 08:17:31 -05:00
Gerhard Schlager
43cfdb1cb9 FIX: Wizard tries harder to find existing Welcome Topic
The wizard searches for:

* a topic that with the "is_welcome_topic" custom field
* a topic with the correct slug for the current default locale
* a topic with the correct slug for the English locale
* the oldest globally pinned topic

It gives up if it didn't find any of the above.
2018-12-06 10:27:22 +01:00
Gerhard Schlager
99117d664c FEATURE: Multisite support for S3 backup store (#6700) 2018-12-05 10:10:39 +08:00
David Taylor
9248ad1905 DEV: Enable Style/SingleLineMethods and Style/Semicolon in Rubocop (#6717) 2018-12-04 11:48:13 +08:00
Kyle Zhao
488fba3c5f
FEATURE: allow plugins and themes to extend the default CSP (#6704)
* FEATURE: allow plugins and themes to extend the default CSP

For plugins:

```
extend_content_security_policy(
  script_src: ['https://domain.com/script.js', 'https://your-cdn.com/'],
  style_src: ['https://domain.com/style.css']
)
```

For themes and components:

```
extend_content_security_policy:
  type: list
  default: "script_src:https://domain.com/|style_src:https://domain.com"
```

* clear CSP base url before each test

we have a test that stubs `Rails.env.development?` to true

* Only allow extending directives that core includes, for now
2018-11-30 09:51:45 -05:00
Penar Musaraj
03deda2147
Upgrade to FontAwesome 5 (take two) (#6673)
* Add missing icons to set

* Revert FA5 revert

 This reverts commit 42572ff

* use new SVG syntax in locales

* Noscript page changes (remove login button, center "powered by" footer text)

* Cast wider net for SVG icons in settings

- include any _icon setting for SVG registry (offers better support for plugin settings)

- let themes store multiple pipe-delimited icons in a setting

- also replaces broken onebox image icon with SVG reference in cooked post processor

* interpolate icons in locales

* Fix composer whisper icon alignment

* Add support for stacked icons

* SECURITY: enforce hostname to match discourse hostname

This ensures that the hostname rails uses for various helpers always matches
the Discourse hostname

* load SVG sprite with pre-initializers

* FIX: enable caching on SVG sprites

* PERF: use JSONP for SVG sprites so they are served from CDN

This avoids needing to deal with CORS for loading of the SVG

Note, added the svg- prefix to the filename so we can quickly tell in
dev tools what the file is

* Add missing SVG sprite JSONP script to CSP

* Upgrade to FA 5.5.0

* Add support for all FA4.7 icons

- adds complete frontend and backend for renamed FA4.7 icons

- improves performance of SvgSprite.bundle and SvgSprite.all_icons

* Fix group avatar flair preview

- adds an endpoint at /svg-sprites/search/:keyword

- adds frontend ajax call that pulls icon in avatar flair preview even when it is not in subset

* Remove FA 4.7 font files
2018-11-26 16:49:57 -05:00
Sam
86255faa08 FEATURE: do not switch to JPEG unless you meet 75k byte savings
This also adjusts the algorithm to expect

- 30% saving for JPEG conversion

AND

- Minimum of 75K bytes saved

The reasoning for increase of saving requirements is cause PNG may have been
uploaded unoptimized, 30% saving on PNG is very possible
2018-11-21 11:01:08 +11:00
Kyle Zhao
962fbd1ec7 include '/plugins/' directory for script-src and blob for worker-src
- plugins may include additional static JS assets
- ACE.js editor register a service worker with a blob for syntax
checking
2018-11-16 16:31:01 -05:00
Kyle Zhao
055d59373a
CSP: drop 'self' in script-src (#6611) 2018-11-15 12:14:16 -05:00
Guo Xiang Tan
44391ee8ab
FEATURE: Upload Site Settings. (#6573) 2018-11-14 15:03:02 +08:00
Guo Xiang Tan
57f92ac808 Revert "Swtich to regexp for DbHelper.remap."
Regexp is so much slower.

This reverts commit c3f89e3cd7.
2018-11-08 14:20:09 +08:00
Guo Xiang Tan
c3f89e3cd7 Swtich to regexp for DbHelper.remap. 2018-11-08 14:08:38 +08:00
Guo Xiang Tan
9737938a4a Add option to skip tabels when using DbHelper.remap. 2018-11-08 12:29:37 +08:00
Guo Xiang Tan
3365753bd0 PERF: Reduce number of database queries for DbHelper.remap
* Cuts number of queries from 273 to 89
* Add some specs
* For a table with 500 posts, benchmarks locally shows a runtime
  reduction from 0.046929135 to 0.032694705.
2018-11-08 10:54:39 +08:00
Kyle Zhao
e9a971a2b6
FEATURE: [Experimental] Content Security Policy (#6514)
do not register new MIME type, parse raw body instead
2018-10-22 13:22:23 -04:00
Bianca Nenciu
99b43f281b FIX: Fix browser detection for Microsoft Edge. (#6516)
cool!
2018-10-22 23:15:41 +11:00
Kyle Zhao
dca830cb73 Revert "FEATURE: [Experimental] Content Security Policy (#6504)"
This reverts commit fb8231077a.
2018-10-19 11:53:29 -04:00
Kyle Zhao
fb8231077a
FEATURE: [Experimental] Content Security Policy (#6504) 2018-10-19 10:39:22 -04:00
Guo Xiang Tan
84d4c81a26 FEATURE: Support backup uploads/downloads directly to/from S3.
This reverts commit 3c59106bac.
2018-10-15 09:43:31 +08:00
Guo Xiang Tan
3c59106bac Revert "FEATURE: Support backup uploads/downloads directly to/from S3."
This reverts commit c29a4dddc1.

We're doing a beta bump soon so un-revert this after that is done.
2018-10-11 11:08:23 +08:00
Gerhard Schlager
c29a4dddc1 FEATURE: Support backup uploads/downloads directly to/from S3. 2018-10-11 10:38:43 +08:00
Bianca Nenciu
1d26a473e7 FEATURE: Show "Recently used devices" in user preferences (#6335)
* FEATURE: Added MaxMindDb to resolve IP information.

* FEATURE: Added browser detection based on user agent.

* FEATURE: Added recently used devices in user preferences.

* DEV: Added acceptance test for recently used devices.

* UX: Do not show 'Show more' button if there aren't more tokens.

* DEV: Fix unit tests.

* DEV: Make changes after code review.

* Add more detailed unit tests.

* Improve logging messages.

* Minor coding style fixes.

* DEV: Use DropdownSelectBoxComponent and run Prettier.

* DEV: Fix unit tests.
2018-10-09 22:21:41 +08:00
Guo Xiang Tan
de85bb0a39 FIX: Don't update user_profile URLs unless upload is persisted. 2018-10-01 14:21:39 +08:00
Guo Xiang Tan
e262a08350 Add UploadRecovery#recover_user_profile_backgrounds. 2018-10-01 10:51:54 +08:00
Guo Xiang Tan
1a64b3a487 FIX: Don't try to recover an invalid sha1. 2018-09-20 14:21:57 +08:00
Sam
df45e82377 SECURITY: only allow picking of avatars created by self (#6417)
* SECURITY: only allow picking of avatars created by self

Also adds origin tracking to all uploads including de-duplicated uploads
2018-09-19 22:33:10 -07:00
Guo Xiang Tan
d403883d16 DEV: Improve specs for 293cf600f0. 2018-09-19 16:03:52 +08:00
Guo Xiang Tan
bc7f58191e FIX: UploadRecovery should look at links too. 2018-09-19 11:52:57 +08:00
Guo Xiang Tan
4a92c5b2d6 UploadRecovery should recover attachments too. 2018-09-19 10:44:36 +08:00
Guo Xiang Tan
0a06b3d977 Accept custom AR relation for UploadRecovery. 2018-09-13 16:33:14 +08:00
Guo Xiang Tan
5eb65ad612 FIX: Do not try to recover invalid Upload#short_url in UploadRecovery. 2018-09-13 13:59:17 +08:00
Guo Xiang Tan
1afe7162e1 Fix the build. 2018-09-13 13:41:38 +08:00
Guo Xiang Tan
d99dd840e4 Add basic test case for UploadRecovery. 2018-09-13 13:26:23 +08:00
Guo Xiang Tan
71caf7521d Drop tif, tiff, webp and bmp from supported images.
https://meta.discourse.org/t/cr2-raw-files-are-being-treated-as-tiff-files/96775/3?u=tgxworld
2018-09-12 09:29:54 +08:00
Régis Hanol
d1607a387a FIX: only allow printable characters in uploads filename 2018-08-21 18:11:01 +02:00
Sam
f5fe58384f correct regression around file renaming 2018-08-20 16:08:05 +10:00
Sam
d7b1919ead correct specs 2018-08-20 12:46:14 +10:00
Sam
8b5e42ea16 FIX: always test and coerce to image on upload
In the past the filename of the origin was used as the source
for the extension of the file when optimizing on upload.

We now use the actual calculated extension based on upload data.
2018-08-20 12:18:59 +10:00
Guo Xiang Tan
4b7c5ba162 Update UploadCreator specs to include cropping code path. 2018-08-17 15:22:12 +08:00
Guo Xiang Tan
a9e502936f FIX: Converting PNG to JPEG does not set the correct extension. 2018-08-17 13:09:48 +08:00
Gerhard Schlager
a115aae45f Use rchardet instead of charlock_holmes gem 2018-08-01 10:41:20 +02:00
Gerhard Schlager
ff942ed2f3 FIX: Try detecting encoding of RSS feed 2018-08-01 10:41:20 +02:00
David Taylor
abddb48260 Rubocop fix 2018-07-24 10:49:20 +01:00
David Taylor
20a21b1240 Move into MiniSQLMultisiteConnection, and add test for rollback 2018-07-24 09:41:55 +01:00
Michael Brown
63a1e9b60a backup restorer: tidy pg_dump schema portability logic, add test 2018-03-20 10:32:39 +08:00