Commit Graph

279 Commits

Author SHA1 Message Date
Penar Musaraj
f00275ded3 FEATURE: Support private attachments when using S3 storage (#7677)
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
2019-06-06 13:27:24 +10:00
Bianca Nenciu
7716617c0d DEV: Add spec for BrowserDetection and Chromebook. 2019-05-30 16:31:28 +03:00
Dan Ungureanu
6bd082feab
FIX: Update mapping between locales and Postgres dictionaries. (#7606) 2019-05-27 16:52:09 +03:00
Gerhard Schlager
b788948985 FEATURE: English locale with international date formats
Makes en_US the new default locale
2019-05-20 13:47:20 +02:00
Régis Hanol
b5c8f5f900 SPEC: ensure never remap readonly columns
Follow-up to bfcbfd78
2019-05-09 18:01:35 +02:00
Daniel Waterworth
e219588142 DEV: Prefabrication (test optimization) (#7414)
* Introduced fab!, a helper that creates database state for a group

It's almost identical to let_it_be, except:

 1. It creates a new object for each test by default,
 2. You can disable it using PREFABRICATION=0
2019-05-07 13:12:20 +10:00
Sam Saffron
9be70a22cd DEV: introduce new API to look up dynamic site setting
This removes all uses of both `send` and `public_send` from consumers of
SiteSetting and instead introduces a `get` helper for dynamic lookup

This leads to much cleaner and safer code long term as we are always explicit
to test that a site setting is really there before sending an arbitrary
string to the class

It also removes a couple of risky stubs from the auth provider test
2019-05-07 11:00:30 +10:00
Guo Xiang Tan
24347ace10 FIX: Properly associate user_profiles background urls via upload id.
`Upload#url` is more likely and can change from time to time. When it
does changes, we don't want to have to look through multiple tables to
ensure that the URLs are all up to date. Instead, we simply associate
uploads properly to `UserProfile` so that it does not have to replicate
the URLs in the table.
2019-05-02 14:58:24 +08:00
Sam Saffron
4ea21fa2d0 DEV: use #frozen_string_literal: true on all spec
This change both speeds up specs (less strings to allocate) and helps catch
cases where methods in Discourse are mutating inputs.

Overall we will be migrating everything to use #frozen_string_literal: true
it will take a while, but this is the first and safest move in this direction
2019-04-30 10:27:42 +10:00
Vinoth Kannan
d0fe42e2ef FIX: should look through posts for image markdown
Downloaded onebox images only included in the cooked HTML content. So we have to check 'post.cooked' instead of 'raw'. bfdd0fe64c
2019-04-10 13:52:35 +05:30
Guo Xiang Tan
6a95d3fded FIX: UploadRecovery should look through posts for img src and bbcode. 2019-04-02 11:41:00 +08:00
Sam Saffron
9ebabc1de8 FEATURE: unconditionally update Topic updated_at when posts change in topic
Previously we would bypass touching `Topic.updated_at` for whispers and post
recovery / deletions.

This meant that certain types of caching can not be done where we rely on
this information for cache accuracy.

For example if we know we have zero unread topics as of yesterday and whisper
is made I need to bump this date so the cache remains accurate

This is only half of a larger change but provides the groundwork.

Confirmed none of our serializers leak out Topic.updated_at so this is safe
spot for this info

At the moment edits still do not change this but it is not relevant for the
unread cache.

This commit also cleans up some specs to use the new `eq_time` matcher for
millisecond fidelity comparison of times

Previously `freeze_time` would fudge this which is not that clean.
2019-03-28 17:28:01 +11:00
Gerhard Schlager
dc90133d29 FIX: Forcing permissions of seeded categories shouldn't fail
Less restrictive permissions of subcategories could make the seeding of categories fail.
2019-03-26 10:39:07 +01:00
Guo Xiang Tan
19c3c25db1 FIX: Handle BBCode in migrate_to_s3 task as well. 2019-03-22 16:47:06 +08:00
Gerhard Schlager
3fd04df781
FEATURE: Locale support for seeded categories and topics (#7110) 2019-03-18 21:09:13 +01:00
Robin Ward
fa5a158683 REFACTOR: Move queue_jobs out of SiteSetting
It is not a setting, and only relevant in specs. The new API is:

```
Jobs.run_later!        # jobs will be thrown on the queue
Jobs.run_immediately!  # jobs will run right away, avoid the queue
```
2019-03-14 10:47:38 -04:00
Robin Ward
d1d9a4f128 Add new run_jobs_synchronously! helper for tests
Previously if you wanted to have jobs execute in test mode, you'd have
to do `SiteSetting.queue_jobs = false`, because the opposite of queue
is to execute.

I found this very confusing, so I created a test helper called
`run_jobs_synchronously!` which is much more clear about what it does.
2019-03-11 16:58:35 -04:00
David Taylor
705c898c21
FEATURE: Calculate CSP based on active themes (#6976) 2019-02-11 12:32:04 +00:00
Gerhard Schlager
84a10f8212 Fix failing spec
The Report class created its own BackupStore without a stubbed S3 client
2019-02-08 21:38:10 +01:00
Robin Ward
b75a3e391a Use skip not pending 2019-02-08 13:48:29 -05:00
Robin Ward
fb18c57372 For now, make spec pending since it is broken. 2019-02-08 12:46:19 -05:00
David Taylor
95b5c5898e FIX: Prevent duplicate params for raw template theme handlebars helpers 2019-02-08 12:54:00 +00:00
Gerhard Schlager
45b056b615 FIX: Do not show backups stored in subfolder of bucket 2019-01-24 22:28:03 +01:00
David Taylor
0b6be2cebc FIX: Correct deprecated theme settings in handlebars templates
Was overriding the incorrect `themeSetting.blah` syntax, instead of `themeSettings.blah`
2019-01-18 15:39:50 +00:00
David Taylor
880311dd4d
FEATURE: Support for localized themes (#6848)
- Themes can supply translation files in a format like `/locales/{locale}.yml`. These files should be valid YAML, with a single top level key equal to the locale being defined. For now these can only be defined using the `discourse_theme` CLI, importing a `.tar.gz`, or from a GIT repository.

- Fallback is handled on a global level (if the locale is not defined in the theme), as well as on individual keys (if some keys are missing from the selected interface language).

- Administrators can override individual keys on a per-theme basis in the /admin/customize/themes user interface.

- Theme developers should access defined translations using the new theme prefix variables:
  JavaScript: `I18n.t(themePrefix("my_translation_key"))`
  Handlebars: `{{theme-i18n "my_translation_key"}}` or `{{i18n (theme-prefix "my_translation_key")}}`

- To design for backwards compatibility, theme developers can check for the presence of the `themePrefix` variable in JavaScript

- As part of this, the old `{{themeSetting.setting_name}}` syntax is deprecated in favour of `{{theme-setting "setting_name"}}`
2019-01-17 11:46:11 +00:00
Penar Musaraj
e11c6ffa89 FEATURE: allow extending CSP base-uri and object-src
Plus, ensure :none is stripped, it cannot be combined with other sources
2019-01-09 15:34:14 -05:00
Robin Ward
9ba8bfb1aa FIX: Multisite DB was leaving old data in test mode
This commit introduces a new helper to enable transactional fixtures
when testing multisite. This would show up as tests that passed the
first time then failed the second time due to stale data being leftover.
2019-01-09 15:20:37 -05:00
Kyle Zhao
dec8e5879a FEATURE: set CSP base-uri and object-src to none (#6863) 2019-01-09 15:04:50 -05:00
Vinoth Kannan
75dbb98cca FEATURE: Add S3 etag value to uploads table (#6795) 2019-01-04 14:16:22 +08:00
Sam
766e67ce57 FEATURE: introduce lossy color optimization on resized pngs
This feature ensures optimized images run via pngquant, this results extreme amounts of savings for resized images. Effectively the only impact is that the color palette on small resized images is reduced to 256.

To ensure safety we only apply this optimisation to images smaller than 500k.

This commit also makes a bunch of image specs less fragile.
2019-01-02 17:20:02 +11:00
Régis Hanol
5381096bfd PERF: new 'migrate_to_s3' rake task 2018-12-26 17:34:49 +01:00
Gerhard Schlager
2bdbca3801 DEV: Remove unnecessary to_not raise_error from specs
Follow-up to 01cdbd3a13
2018-12-17 16:10:10 +01:00
Gerhard Schlager
01cdbd3a13 FEATURE: Prohibit S3 bucket reusage
This validation makes sure that the s3_upload_bucket and the
s3_backup_bucket have different values. The backup bucket is
allowed to be a subfolder of the upload bucket. The other way
around is forbidden because the backup system searches by
prefix and would return all files stored within the backup
bucket and its subfolders.
2018-12-17 11:35:28 +01:00
Gerhard Schlager
1a8ca68ea3 FEATURE: Improve backup stats on admin dashboard
* Dashboard doesn't timeout anymore when Amazon S3 is used for backups
* Storage stats are now a proper report with the same caching rules
* Changing the backup_location, s3_backup_bucket or creating and deleting backups removes the report from the cache
* It shows the number of backups and the backup location
* It shows the used space for the correct backup location instead of always showing used space on local storage
* It shows the date of the last backup as relative date
2018-12-17 11:35:11 +01:00
Kyle Zhao
b0c2e9bb05
minor changes to default script-src (#6770)
- add report-sample to force require a sample of the violating code
- do not whitelist GA/GTM's entire domain
2018-12-14 08:17:31 -05:00
Gerhard Schlager
43cfdb1cb9 FIX: Wizard tries harder to find existing Welcome Topic
The wizard searches for:

* a topic that with the "is_welcome_topic" custom field
* a topic with the correct slug for the current default locale
* a topic with the correct slug for the English locale
* the oldest globally pinned topic

It gives up if it didn't find any of the above.
2018-12-06 10:27:22 +01:00
Gerhard Schlager
99117d664c FEATURE: Multisite support for S3 backup store (#6700) 2018-12-05 10:10:39 +08:00
David Taylor
9248ad1905 DEV: Enable Style/SingleLineMethods and Style/Semicolon in Rubocop (#6717) 2018-12-04 11:48:13 +08:00
Kyle Zhao
488fba3c5f
FEATURE: allow plugins and themes to extend the default CSP (#6704)
* FEATURE: allow plugins and themes to extend the default CSP

For plugins:

```
extend_content_security_policy(
  script_src: ['https://domain.com/script.js', 'https://your-cdn.com/'],
  style_src: ['https://domain.com/style.css']
)
```

For themes and components:

```
extend_content_security_policy:
  type: list
  default: "script_src:https://domain.com/|style_src:https://domain.com"
```

* clear CSP base url before each test

we have a test that stubs `Rails.env.development?` to true

* Only allow extending directives that core includes, for now
2018-11-30 09:51:45 -05:00
Penar Musaraj
03deda2147
Upgrade to FontAwesome 5 (take two) (#6673)
* Add missing icons to set

* Revert FA5 revert

 This reverts commit 42572ff

* use new SVG syntax in locales

* Noscript page changes (remove login button, center "powered by" footer text)

* Cast wider net for SVG icons in settings

- include any _icon setting for SVG registry (offers better support for plugin settings)

- let themes store multiple pipe-delimited icons in a setting

- also replaces broken onebox image icon with SVG reference in cooked post processor

* interpolate icons in locales

* Fix composer whisper icon alignment

* Add support for stacked icons

* SECURITY: enforce hostname to match discourse hostname

This ensures that the hostname rails uses for various helpers always matches
the Discourse hostname

* load SVG sprite with pre-initializers

* FIX: enable caching on SVG sprites

* PERF: use JSONP for SVG sprites so they are served from CDN

This avoids needing to deal with CORS for loading of the SVG

Note, added the svg- prefix to the filename so we can quickly tell in
dev tools what the file is

* Add missing SVG sprite JSONP script to CSP

* Upgrade to FA 5.5.0

* Add support for all FA4.7 icons

- adds complete frontend and backend for renamed FA4.7 icons

- improves performance of SvgSprite.bundle and SvgSprite.all_icons

* Fix group avatar flair preview

- adds an endpoint at /svg-sprites/search/:keyword

- adds frontend ajax call that pulls icon in avatar flair preview even when it is not in subset

* Remove FA 4.7 font files
2018-11-26 16:49:57 -05:00
Sam
86255faa08 FEATURE: do not switch to JPEG unless you meet 75k byte savings
This also adjusts the algorithm to expect

- 30% saving for JPEG conversion

AND

- Minimum of 75K bytes saved

The reasoning for increase of saving requirements is cause PNG may have been
uploaded unoptimized, 30% saving on PNG is very possible
2018-11-21 11:01:08 +11:00
Kyle Zhao
962fbd1ec7 include '/plugins/' directory for script-src and blob for worker-src
- plugins may include additional static JS assets
- ACE.js editor register a service worker with a blob for syntax
checking
2018-11-16 16:31:01 -05:00
Kyle Zhao
055d59373a
CSP: drop 'self' in script-src (#6611) 2018-11-15 12:14:16 -05:00
Guo Xiang Tan
44391ee8ab
FEATURE: Upload Site Settings. (#6573) 2018-11-14 15:03:02 +08:00
Guo Xiang Tan
57f92ac808 Revert "Swtich to regexp for DbHelper.remap."
Regexp is so much slower.

This reverts commit c3f89e3cd7.
2018-11-08 14:20:09 +08:00
Guo Xiang Tan
c3f89e3cd7 Swtich to regexp for DbHelper.remap. 2018-11-08 14:08:38 +08:00
Guo Xiang Tan
9737938a4a Add option to skip tabels when using DbHelper.remap. 2018-11-08 12:29:37 +08:00
Guo Xiang Tan
3365753bd0 PERF: Reduce number of database queries for DbHelper.remap
* Cuts number of queries from 273 to 89
* Add some specs
* For a table with 500 posts, benchmarks locally shows a runtime
  reduction from 0.046929135 to 0.032694705.
2018-11-08 10:54:39 +08:00
Kyle Zhao
e9a971a2b6
FEATURE: [Experimental] Content Security Policy (#6514)
do not register new MIME type, parse raw body instead
2018-10-22 13:22:23 -04:00
Bianca Nenciu
99b43f281b FIX: Fix browser detection for Microsoft Edge. (#6516)
cool!
2018-10-22 23:15:41 +11:00
Kyle Zhao
dca830cb73 Revert "FEATURE: [Experimental] Content Security Policy (#6504)"
This reverts commit fb8231077a.
2018-10-19 11:53:29 -04:00
Kyle Zhao
fb8231077a
FEATURE: [Experimental] Content Security Policy (#6504) 2018-10-19 10:39:22 -04:00
Guo Xiang Tan
84d4c81a26 FEATURE: Support backup uploads/downloads directly to/from S3.
This reverts commit 3c59106bac.
2018-10-15 09:43:31 +08:00
Guo Xiang Tan
3c59106bac Revert "FEATURE: Support backup uploads/downloads directly to/from S3."
This reverts commit c29a4dddc1.

We're doing a beta bump soon so un-revert this after that is done.
2018-10-11 11:08:23 +08:00
Gerhard Schlager
c29a4dddc1 FEATURE: Support backup uploads/downloads directly to/from S3. 2018-10-11 10:38:43 +08:00
Bianca Nenciu
1d26a473e7 FEATURE: Show "Recently used devices" in user preferences (#6335)
* FEATURE: Added MaxMindDb to resolve IP information.

* FEATURE: Added browser detection based on user agent.

* FEATURE: Added recently used devices in user preferences.

* DEV: Added acceptance test for recently used devices.

* UX: Do not show 'Show more' button if there aren't more tokens.

* DEV: Fix unit tests.

* DEV: Make changes after code review.

* Add more detailed unit tests.

* Improve logging messages.

* Minor coding style fixes.

* DEV: Use DropdownSelectBoxComponent and run Prettier.

* DEV: Fix unit tests.
2018-10-09 22:21:41 +08:00
Guo Xiang Tan
de85bb0a39 FIX: Don't update user_profile URLs unless upload is persisted. 2018-10-01 14:21:39 +08:00
Guo Xiang Tan
e262a08350 Add UploadRecovery#recover_user_profile_backgrounds. 2018-10-01 10:51:54 +08:00
Guo Xiang Tan
1a64b3a487 FIX: Don't try to recover an invalid sha1. 2018-09-20 14:21:57 +08:00
Sam
df45e82377 SECURITY: only allow picking of avatars created by self (#6417)
* SECURITY: only allow picking of avatars created by self

Also adds origin tracking to all uploads including de-duplicated uploads
2018-09-19 22:33:10 -07:00
Guo Xiang Tan
d403883d16 DEV: Improve specs for 293cf600f0. 2018-09-19 16:03:52 +08:00
Guo Xiang Tan
bc7f58191e FIX: UploadRecovery should look at links too. 2018-09-19 11:52:57 +08:00
Guo Xiang Tan
4a92c5b2d6 UploadRecovery should recover attachments too. 2018-09-19 10:44:36 +08:00
Guo Xiang Tan
0a06b3d977 Accept custom AR relation for UploadRecovery. 2018-09-13 16:33:14 +08:00
Guo Xiang Tan
5eb65ad612 FIX: Do not try to recover invalid Upload#short_url in UploadRecovery. 2018-09-13 13:59:17 +08:00
Guo Xiang Tan
1afe7162e1 Fix the build. 2018-09-13 13:41:38 +08:00
Guo Xiang Tan
d99dd840e4 Add basic test case for UploadRecovery. 2018-09-13 13:26:23 +08:00
Guo Xiang Tan
71caf7521d Drop tif, tiff, webp and bmp from supported images.
https://meta.discourse.org/t/cr2-raw-files-are-being-treated-as-tiff-files/96775/3?u=tgxworld
2018-09-12 09:29:54 +08:00
Régis Hanol
d1607a387a FIX: only allow printable characters in uploads filename 2018-08-21 18:11:01 +02:00
Sam
f5fe58384f correct regression around file renaming 2018-08-20 16:08:05 +10:00
Sam
d7b1919ead correct specs 2018-08-20 12:46:14 +10:00
Sam
8b5e42ea16 FIX: always test and coerce to image on upload
In the past the filename of the origin was used as the source
for the extension of the file when optimizing on upload.

We now use the actual calculated extension based on upload data.
2018-08-20 12:18:59 +10:00
Guo Xiang Tan
4b7c5ba162 Update UploadCreator specs to include cropping code path. 2018-08-17 15:22:12 +08:00
Guo Xiang Tan
a9e502936f FIX: Converting PNG to JPEG does not set the correct extension. 2018-08-17 13:09:48 +08:00
Gerhard Schlager
a115aae45f Use rchardet instead of charlock_holmes gem 2018-08-01 10:41:20 +02:00
Gerhard Schlager
ff942ed2f3 FIX: Try detecting encoding of RSS feed 2018-08-01 10:41:20 +02:00
David Taylor
abddb48260 Rubocop fix 2018-07-24 10:49:20 +01:00
David Taylor
20a21b1240 Move into MiniSQLMultisiteConnection, and add test for rollback 2018-07-24 09:41:55 +01:00
Michael Brown
63a1e9b60a backup restorer: tidy pg_dump schema portability logic, add test 2018-03-20 10:32:39 +08:00