Commit Graph

773 Commits

Author SHA1 Message Date
David Taylor
110fdf0189
DEV: Remove dependence on dartsass-sprockets (#23665)
Discourse has a custom stylesheet pipeline which compiles things 'just in time'. The only place we were still running sass files through sprockets was for the `/tests` route in development mode. This use can be removed by compiling the relevant stylesheets through ember-cli instead (which we were already doing for testem runs)

This work was prompted by the incompatibility of dartsass-sprockets with the latest sass-embedded release (https://github.com/tablecheck/dartsass-sprockets/issues/13)
2023-09-26 16:25:07 +01:00
Martin Brennan
cf42466dea
DEV: Add S3 upload system specs using minio (#22975)
This commit adds some system specs to test uploads with
direct to S3 single and multipart uploads via uppy. This
is done with minio as a local S3 replacement. We are doing
this to catch regressions when uppy dependencies need to
be upgraded or we change uppy upload code, since before
this there was no way to know outside manual testing whether
these changes would cause regressions.

Minio's server lifecycle and the installed binaries are managed
by the https://github.com/discourse/minio_runner gem, though the
binaries are already installed on the discourse_test image we run
GitHub CI from.

These tests will only run in CI unless you specifically use the
CI=1 or RUN_S3_SYSTEM_SPECS=1 env vars.

For a history of experimentation here see https://github.com/discourse/discourse/pull/22381

Related PRs:

* https://github.com/discourse/minio_runner/pull/1
* https://github.com/discourse/minio_runner/pull/2
* https://github.com/discourse/minio_runner/pull/3
2023-08-23 11:18:33 +10:00
Mark VanLandingham
fbf7b106cc
DEV: Bump selenium-webdriver version to fix system spec running (#23117)
We can no long user Webdriver - SeleniumHQ/selenium#11066. Bumping selenium-webdriver did the trick, as well as manually setting the user_agent for mobile system specs. Unsure what changed to make this necessary, but it is necessary to get the app to boot in mobile view.
2023-08-16 15:07:03 -05:00
Alan Guo Xiang Tan
0187ad0d37
DEV: Upgrade Rails to 7.0.7 (#23069) 2023-08-11 09:00:42 +08:00
Gerhard Schlager
0b29dc5d38 DEV: Add experimental generic bulk import script 2023-08-09 20:56:14 +02:00
Joffrey JAFFEUX
05aa55e172
DEV: moves logic from job to a service (#22691)
`Jobs::AutoJoinChannelBatch` was holding a lot of logic which should be in a service. Moreover, this refactoring is the opportunity to address a bug which could cause a duplicate key error.

From now when trying to insert a new membership it won't fail if a membership is already present.

Example error:

```
Job exception: ERROR:  duplicate key value violates unique constraint "user_chat_channel_unique_memberships"
DETAIL:  Key (user_id, chat_channel_id)=(1, 2) already exists.

Backtrace
rack-mini-profiler-3.1.0/lib/patches/db/pg.rb:110:in `exec'
rack-mini-profiler-3.1.0/lib/patches/db/pg.rb:110:in `async_exec'
(eval):29:in `async_exec'
mini_sql-1.4.0/lib/mini_sql/postgres/connection.rb:209:in `run'
mini_sql-1.4.0/lib/mini_sql/active_record_postgres/connection.rb:38:in `block in run'
mini_sql-1.4.0/lib/mini_sql/active_record_postgres/connection.rb:34:in `block in with_lock'
activesupport-7.0.5.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
activesupport-7.0.5.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
activesupport-7.0.5.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
activesupport-7.0.5.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
mini_sql-1.4.0/lib/mini_sql/active_record_postgres/connection.rb:34:in `with_lock'
mini_sql-1.4.0/lib/mini_sql/active_record_postgres/connection.rb:38:in `run'
mini_sql-1.4.0/lib/mini_sql/postgres/connection.rb:64:in `query_single'
/var/www/discourse/plugins/chat/app/jobs/regular/chat/auto_join_channel_batch.rb:38:in `execute'
```

Note this commit is also using main branch of `shoulda-matchers` as the gem has not been released yet.

Co-authored-by: Loïc Guitaut <5648+Flink@users.noreply.github.com>
2023-07-27 10:25:41 +02:00
Alan Guo Xiang Tan
384068f4c7
DEV: Remove webrick dependency which we no longer need (#22738)
The `discourse-prometheus` plugin has since specificed the depedency on
webrick in the plugin so we no longer need to carry this in core.

See c4b675f0fe
2023-07-21 13:12:32 +08:00
Alan Guo Xiang Tan
351005ef1b
DEV: Upgrade MessageBus to latest version. (#22355)
Previously it was pinned to 4.3.2 because 4.3.3 was broken. This has now
been resolved so we no longer need to pin MessageBus.
2023-06-30 08:14:16 +08:00
Gerhard Schlager
50f710bcae
DEV: Bump oj from 3.13.14 to 3.15.0 (#22352)
https://github.com/ohler55/oj/issues/789 has been fixed and SSE4.2 is disabled by default.
2023-06-30 07:39:55 +08:00
Greg Molnar
4d3999de10
DEV: Upgrade Rails to 7.0.5.1 (#22305)
See discuss.rubyonrails.org/t/cve-2023-28362-possible-xss-via-user-supplied-values-to/83132

Impact of this vulnerability has been assess to be very low for Discourse since XSS attacks are mitigated by Discourse's default CSP.
2023-06-28 07:18:49 +08:00
Sam
9e241e82e9
DEV: use HTML5 version of loofah (#21522)
https://meta.discourse.org/t/markdown-preview-and-result-differ/263878

The result of this markdown had different results in the composer preview and the post. This is solved by updating Loofah to the latest version and using html5 fragments like our user had reported. While the change was only needed in cooked_post_processor.rb for this fix, other areas also had to be updated due to various side effects.
2023-06-20 09:49:22 +08:00
Martin Brennan
a510de1f6e
DEV: Roll back MessageBus version (#22199)
We are having issues with a lot of MessageBus updates not coming
through, it seems like the poll is not reconnecting after hanging
up. Pinning to the version before this commit to check:

a2a46fde87
2023-06-20 10:24:47 +10:00
David Taylor
d3e5251704
PERF: Use OpenSSL::KDF for Pbkdf2 implementation (#20982)
This was introduced to the standard library in Ruby 2.4. In my testing, it produces the same result, and is around 8x faster than our pure-ruby implementation
2023-04-05 17:00:05 +01:00
Jarek Radosz
ef0016cdba
DEV: Drop our mail gem fork (#20805)
Didn't happen in https://github.com/discourse/discourse/pull/16622 but now that the official release is fixed - let's do this :P
2023-03-24 20:10:28 +01:00
Blake Erickson
d0c6b33cc2
SECURITY: Bump Rails to v7.0.4.3 (#20675) 2023-03-14 10:19:31 -06:00
Alan Guo Xiang Tan
d5ebcc3309
DEV: Remove ruby-lsp from Gemfile (#20595) 2023-03-09 05:34:54 +08:00
Alan Guo Xiang Tan
990b710ade
DEV: Add ruby_lsp gem to development (#20517)
In order to use the ruby-lsp vscode extension, the ruby_lsp gem needs to
be added to the project's Gemfile. That may soon change with
https://github.com/Shopify/vscode-ruby-lsp/pull/419 but this will do for
now.
2023-03-03 06:45:52 +08:00
Penar Musaraj
3cadeaf90f
FIX: Add support for Europe/Kyiv timezone (#20394) 2023-02-21 11:52:04 -05:00
Osama Sayegh
75ae70c27c
DEV: rtlcss_wrapper renamed to rtlcss (#20331)
The `rtlcss_wrapper` gem has been renamed to `rtlcss` per bd89847a39.
2023-02-16 18:15:56 +03:00
Martin Brennan
60ad836313
DEV: Chat service object initial implementation (#19814)
This is a combined work of Martin Brennan, Loïc Guitaut, and Joffrey Jaffeux.

---

This commit implements a base service object when working in chat. The documentation is available at https://discourse.github.io/discourse/chat/backend/Chat/Service.html

Generating documentation has been made as part of this commit with a bigger goal in mind of generally making it easier to dive into the chat project.

Working with services generally involves 3 parts:

- The service object itself, which is a series of steps where few of them are specialized (model, transaction, policy)

```ruby
class UpdateAge
  include Chat::Service::Base

  model :user, :fetch_user
  policy :can_see_user
  contract
  step :update_age

  class Contract
    attribute :age, :integer
  end

  def fetch_user(user_id:, **)
    User.find_by(id: user_id)
  end

  def can_see_user(guardian:, **)
    guardian.can_see_user(user)
  end

  def update_age(age:, **)
    user.update!(age: age)
  end
end
```

- The `with_service` controller helper, handling success and failure of the service within a service and making easy to return proper response to it from the controller

```ruby
def update
  with_service(UpdateAge) do
    on_success { render_serialized(result.user, BasicUserSerializer, root: "user") }
  end
end
```

- Rspec matchers and steps inspector, improving the dev experience while creating specs for a service

```ruby
RSpec.describe(UpdateAge) do
  subject(:result) do
    described_class.call(guardian: guardian, user_id: user.id, age: age)
  end

  fab!(:user) { Fabricate(:user) }
  fab!(:current_user) { Fabricate(:admin) }

  let(:guardian) { Guardian.new(current_user) }
  let(:age) { 1 }

   it { expect(user.reload.age).to eq(age) }
end
```

Note in case of unexpected failure in your spec, the output will give all the relevant information:

```
  1) UpdateAge when no channel_id is given is expected to fail to find a model named 'user'
     Failure/Error: it { is_expected.to fail_to_find_a_model(:user) }

       Expected model 'foo' (key: 'result.model.user') was not found in the result object.

       [1/4] [model] 'user' 
       [2/4] [policy] 'can_see_user'
       [3/4] [contract] 'default'
       [4/4] [step] 'update_age'

       /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/update_age.rb:32:in `fetch_user': missing keyword: :user_id (ArgumentError)
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:202:in `instance_exec'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:202:in `call'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:219:in `call'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:417:in `block in run!'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:417:in `each'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:417:in `run!'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:411:in `run'
       	from <internal:kernel>:90:in `tap'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:302:in `call'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/spec/services/update_age_spec.rb:15:in `block (3 levels) in <main>'
```
2023-02-13 13:09:57 +01:00
Rafael dos Santos Silva
6e522e4aad
DEV: Move to Sass compilation to dart-sass (#19910)
This PR is a major change to Sass compilation in Discourse.

The new version of sass-ruby moves to dart-sass putting we back on the supported version of Sass. It does so while keeping compatibility with the existing method signatures, so minimal change is needed in Discourse for this change.

This moves us

From:
  - sassc 2.0.1 (Feb 2019)
  - libsass 3.5.2 (May 2018)

To:
  - dart-sass 1.58

This update applies the following breaking changes:

> 
> These breaking changes are coming soon or have recently been released:
> 
>  [Functions are stricter about which units they allow](https://sass-lang.com/documentation/breaking-changes/function-units) beginning in Dart Sass 1.32.0.
> 
>  [Selectors with invalid combinators are invalid](https://sass-lang.com/documentation/breaking-changes/bogus-combinators) beginning in Dart Sass 1.54.0.
> 
>  [/ is changing from a division operation to a list separator](https://sass-lang.com/documentation/breaking-changes/slash-div) beginning in Dart Sass 1.33.0.
> 
>  [Parsing the special syntax of @-moz-document will be invalid](https://sass-lang.com/documentation/breaking-changes/moz-document) beginning in Dart Sass 1.7.2.
> 
>  [Compound selectors could not be extended](https://sass-lang.com/documentation/breaking-changes/extend-compound) in Dart Sass 1.0.0 and Ruby Sass 4.0.0.


SCSS files have been migrated automatically using `sass-migrator division app/assets/stylesheets/**/*.scss`
2023-02-07 12:24:57 -03:00
Osama Sayegh
f94951147e
FIX: Replace R2 gem with rtlcss for generating RTL CSS (#19636)
We've had a couple of problems with the R2 gem where it generated a broken RTL CSS bundle that caused a badly broken layout when Discourse is used in an RTL language, see a3ce93b and 5926386. For this reason, we're replacing R2 with `rtlcss` that can handle modern CSS features better than R2 does.

`rltcss` is written in JS and available as an npm package. Calling the `rltcss` from rubyland is done via the `rtlcss_wrapper` gem which contains a distributable copy of the `rtlcss` package and loads/calls it with Mini Racer. See https://github.com/discourse/rtlcss_wrapper for more details.

Internal topic: t/76263.
2023-02-01 14:21:15 +03:00
David Taylor
87316d7a10
SECURITY: Bump Rails to v7.0.4.1 (#19956) 2023-01-23 15:38:49 -05:00
David Taylor
ef437a1e41
DEV: Bump Rails to 7.0.4 (#19881) 2023-01-17 09:54:50 +00:00
David Taylor
ce6335693a
DEV: Bump sprockets to include ERB kwargs fix (#19850)
This should resolve these warnings under Ruby 3.1

```
warning: Passing safe_level with the 2nd argument of ERB.new is deprecated
```

Unfortunately Sprockets 3.x has not seen a rubygems release since 2018, so we need to fetch these improvements via git.
2023-01-13 10:52:05 +00:00
Rafael dos Santos Silva
192d8c25e6
DEV: Move back to web-push gem (#19849)
Our fork was needed for OpenSSL 3 and Ruby 2.X compatibility.

The OpenSSL 3 part was merged into the gem for version 3.

Discourse dropped support for Ruby 2.X.

That means we don't need our fork anymore.
2023-01-12 12:12:19 -03:00
David Taylor
0cf6421716
DEV: Apply syntax_tree formatting to Gemfile 2023-01-09 11:13:33 +00:00
David Taylor
93e2dad656
DEV: Introduce syntax_tree code formatter (#19775)
This commit introduces the necessary gems and config, but adds all our ruby code directories to the `--ignore-files` list.

Future commits will apply syntax_tree to parts of the codebase, removing the ignore patterns as we go
2023-01-07 11:11:08 +00:00
Rafael dos Santos Silva
7b53973bd8
DEV: Use WebPush fork for OpenSSL 3 compat (#19627)
* DEV: Use WebPush fork for OpenSSL 3 compat

* add some context on gemfile changes
2022-12-27 15:28:13 -03:00
Rafael dos Santos Silva
e430f22936
FIX: Ensure we have a patched version of CGI gem (#19588)
* FIX: Ensure we have a patched version of CGI gem

Per https://github.com/ruby/cgi/pull/29 the current shipped version of
the CGI gem doesn't allow for leading dots in domain names, which breaks
setting cookies like `.example.com`.

* Update Gemfile

Co-authored-by: Jarek Radosz <jradosz@gmail.com>

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2022-12-22 19:52:20 -03:00
Rafael dos Santos Silva
aa1e8790e5
FIX: Prometheus plugin expects webrick in production (#19562)
* FIX: Prometheus plugin expects webrick in production

* newline
2022-12-21 18:20:07 -03:00
Rafael dos Santos Silva
7c5744a4cb
DEV: Ruby 3.1 compat (#19543)
* DEV: Ruby 3.1 compat

* DEV: Add webrick to test/dev

* DEV: Review ruby2_keywords license

* move ruby2_keywords to ignored

* move openssl to ignored
2022-12-21 01:20:43 -03:00
Leonardo Mosquera
bfecbde837
Fixes for vBulletin bulk importer (#17618)
* Allow taking table prefix from env var

* FIX: remove unused column references

The columns `filedata` and `extension` are not present in a v4.2.4
database, and they aren't used in the method anyways.

* FIX: report progress for tables without imported_id

* FIX: effectively check for AR validation errors

NOTE: other migration scripts also have this problem; see /t/58202

* FIX: properly count Posts when importing attachments

* FIX: improve logging

* Remove leftover comment

* FIX: show progress when exporting Permalink file

* PERF: stream Permalink file

The current way results in tons of memory usage; write once per line instead

* Document fixes needed

* WIP - deduplicate category names

* Ignore non alphanumeric chars for grouping

* FIX: properly deduplicate user emails by merging accounts

* FIX: don't merge empty UserEmails

* Improve logging

* Merge users AFTER fixing primary key sequences

* Parallelize user merging

* Save duplicated users structure for debugging purposes

* Add progress logging for the (multiple hour) user merging step
2022-11-28 16:30:19 -03:00
David Taylor
84bec1cbae
DEV: Cleanup legacy asset compilation gems and code (#19177)
We now use Ember CLI (core/plugins) and DiscourseJSProcessor (themes) for all Ember and template compilation. This commit removes the remnants of the legacy Sprockets-based Ember compilation system.

Sprockets, and its DiscourseJSProcess-based Babel transformations, is still in use for a few assets. Ideally that will be removed/replaced in the near future.
2022-11-24 12:13:59 +00:00
Jarek Radosz
b5005e3574
DEV: Always use non-builtin net-* and digest gems (#18560)
This will possibly fix the issue we're having with Dependabot. It seems it now uses a different ruby version (i.e. 3.1+)
2022-10-12 15:22:22 +02:00
Krzysztof Kotlarek
e6b5b6eae3
DEV: update to patched discourse-seed-fu (#18493)
Original seed-fu was cloned and patched with David's fix 576b69a368
2022-10-07 09:16:04 +11:00
David Taylor
e8b1021cb6
DEV: Add faraday and faraday-retry as explicit dependencies (#18473)
`Faraday` is very commonly used in official and third-party plugins, and we will likely be increasing our use of it in core. This commit adds it as a direct dependency and adds the official faraday-retry gem which is very commonly used (e.g. by Octokit).
2022-10-05 13:19:04 +01:00
Martin Brennan
57caf08e13
DEV: Minimal first pass of rails system test setup (#16311)
This commit introduces rails system tests run with chromedriver, selenium,
and headless chrome to our testing toolbox.

We use the `webdrivers` gem and `selenium-webdriver` which is what
the latest Rails uses so the tests run locally and in CI out of the box.

You can use `SELENIUM_VERBOSE_DRIVER_LOGS=1` to show extra
verbose logs of what selenium is doing to communicate with the system
tests.

By default JS logs are verbose so errors from JS are shown when
running system tests, you can disable this with
`SELENIUM_DISABLE_VERBOSE_JS_LOGS=1`

You can use `SELENIUM_HEADLESS=0` to run the system
tests inside a chrome browser instead of headless, which can be useful to debug things
and see what the spec sees. See note above about `bin/ember-cli` to avoid
surprises.

I have modified `bin/turbo_rspec` to exclude `spec/system` by default,
support for parallel system specs is a little shaky right now and we don't
want them slowing down the turbo by default either.

### PageObjects and System Tests

To make querying and inspecting parts of the page easier
and more reusable inbetween system tests, we are using the
concept of [PageObjects](https://www.selenium.dev/documentation/test_practices/encouraged/page_object_models/) in
our system tests. A "Page" here is generally corresponds to
an overarching ember route, e.g. "Topic" for `/t/324345/some-topic`,
and this contains logic for querying components within the topic
such as "Posts".

I have also split "Modals" into their own entity. Further down the
line we may want to explore creating independent "Component"
contexts.

Capybara DSL should be included in each PageObject class,
reference for this can be found at https://rubydoc.info/github/teamcapybara/capybara/master#the-dsl

For system tests, since they are so slow, we want to focus on
the "happy path" and not do every different possible context
and branch check using them. They are meant to be overarching
tests that check a number of things are correct using the full stack
from JS and ember to rails to ruby and then the database.

### CI Setup

Whenever a system spec fails, a screenshot
is taken and a build artifact is produced _after the entire CI run is complete_,
which can be downloaded from the Actions UI in the repo.

Most importantly, a step to build the Ember app using Ember CLI
is needed, otherwise the JS assets cannot be found by capybara:

```
- name: Build Ember CLI
  run: bin/ember-cli --build
```

A new `--build` argument has been added to `bin/ember-cli` for this
case, which is not needed locally if you already have the discourse
rails server running via `bin/ember-cli -u` since the whole server is built and
set up by default.

Co-authored-by: David Taylor <david@taylorhq.com>
2022-09-28 11:48:16 +10:00
David Taylor
53ee8746f6
DEV: Bump rubocop config and switch back to rubygems (#18379) 2022-09-27 12:07:09 +01:00
Loïc Guitaut
cc515e2c1c DEV: Remove before(:all) from RSpec suite 2022-07-18 14:21:56 +02:00
Sam
8dad778fcc
FIX: support systems without SSE 4.2 (#17505)
Older CPUs do not support the instruction set used internally by the oj gem
starting with version 3.13.15.

This pins the version to a working one while oj patch the project to support
CPUs without the instructions (eg: AMD Opteron 4180).

see:

https://meta.discourse.org/t/discourse-update-keeps-failing/231862
https://github.com/ohler55/oj/issues/789
a455c389c0
2022-07-15 10:06:04 +10:00
David Taylor
ee07f6da7d
SECURITY: Bump Rails to 7.0.3.1 (#17469)
https://discuss.rubyonrails.org/t/81017
2022-07-13 11:17:46 +01:00
Loïc Guitaut
fff3ff11c0 FIX: Make disabling TLS in mail possible again
Following the Rails 7 upgrade, the `DISCOURSE_SMTP_ENABLE_START_TLS`
setting doesn’t work anymore. This is because Rails upgraded the
`net-smtp` gem to the 0.3.1 version which enables `starttls` by default.
The `mail` gem doesn’t support this new behavior yet and doesn’t know
how to disable TLS. This should be fixed in an upcoming release.

Meanwhile applying this patch allows us to get back the previous
behavior which is expected by many.
2022-06-08 14:04:05 +02:00
David Taylor
8bc8dbc134
Revert "DEV: Drop our mail gem fork (#16622)" (#16853)
This reverts commit 0d30c19b7e.

Pending resolution of https://github.com/mikel/mail/issues/1489
2022-05-17 16:11:39 +01:00
Loïc Guitaut
5fb6dd5664 DEV: Upgrade to Rails 7.0.3 2022-05-10 12:01:00 +02:00
Sam
2df3c65ba9
FIX: add support for pipelined and multi redis commands (#16682)
Latest redis interoduces a block form of multi / pipelined, this was incorrectly
passed through and not namespaced.

Fix also updates logster, we held off on upgrading it due to missing functions
2022-05-10 08:19:02 +10:00
Jarek Radosz
0d30c19b7e
DEV: Drop our mail gem fork (#16622)
Bianca's patch was finally released (in rc release for now, but still!)
2022-05-04 16:53:53 +02:00
Jarek Radosz
79c854d9be
DEV: Remove RTLit gem (#16620)
Its only use was removed 7 years ago in #3377.
2022-05-04 14:11:12 +08:00
Jarek Radosz
d8ce4228da
DEV: Lock logster to 2.11.0 (#16605)
2.11.1+ is not compatible with DiscourseRedis implementation. This fixes error reporting.
2022-05-03 13:41:09 +02:00
Loïc Guitaut
008b700a3f DEV: Upgrade to Rails 7
This patch upgrades Rails to version 7.0.2.4.
2022-04-28 11:51:03 +02:00
David Taylor
c88ca23e8f
Revert "DEV: Update to Sprockets 4.0 (#16467)" (#16524)
This reverts commit 01107e418e.

We have seen some random occurrences of corrupted assets, and think it may be related to the sprockets 4 update. Reverting for investigation
2022-04-20 22:17:29 +01:00
David Taylor
01107e418e
DEV: Update to Sprockets 4.0 (#16467)
The main difference is that Sprockets 4.0 no longer tries to compile everything by default. This is good for us, because we can remove all our custom 'exclusion' logic which was working around the old sprockets 3.0 behavior.

The other big change is that lambdas can no longer be added to the `config.assets.precompile` array. Instead, we can do the necessary globs ourselves, and add the desired files manually.

A small patch is required to make ember-rails compatible. Since we plan to remove this dependency in the near future, I do not intend to upstream this change.

I have compared the `bin/rake assets:precompile` output before and after this change, and verified that all files are present.
2022-04-13 15:03:50 +01:00
David Taylor
78f7e8fe2f
Revert "DEV: Update to Sprockets 4.0 (#16459)" (#16466)
This reverts commit ec7efbde1a.

This is causing problems in non-ember-cli environments. Reverting for now.
2022-04-13 11:04:13 +01:00
David Taylor
ec7efbde1a
DEV: Update to Sprockets 4.0 (#16459)
The main difference is that Sprockets 4.0 no longer tries to compile everything by default. This is good for us, because we can remove all our custom 'exclusion' logic which was working around the old sprockets 3.0 behavior.

The other big change is that lambdas can no longer be added to the `config.assets.precompile` array. Instead, we can do the necessary globs ourselves, and add the desired files manually.

A small patch is required to make ember-rails compatible. Since we plan to remove this dependency in the near future, I do not intend to upstream this change.

I have compared the `bin/rake assets:precompile` output before and after this change, and verified that all files are present.
2022-04-13 10:21:59 +01:00
Alan Guo Xiang Tan
01c5f9337a
DEV: Upgrade Rails to 6.1.4.7. (#16143)
There was a security release in 6.1.4.6 to fix: https://github.com/rails/rails/security/advisories/GHSA-wh98-p28r-vrc9

It doesn't affect us as Discourse doesn't use thread local variables
but we should still upgrade as a matter of caution.
2022-03-09 10:10:51 +08:00
Jarek Radosz
aaf432df86
DEV: Remove mock_redis (#15985)
Was used just in one spec file. And we prefer to run specs against a real redis server.
2022-02-18 01:14:38 +01:00
Noah Gibbs
3dbb3d09dc
DEV: Use non-builtin net-pop, net-imap and net-smtp for Ruby 3.1+ (#15692) 2022-02-03 16:25:29 +11:00
Jarek Radosz
fb25333b7f
DEV: Update oj gem (#15713)
https://github.com/ohler55/oj/issues/699 was fixed back in September 2021.
2022-02-01 09:45:52 +11:00
Vinoth Kannan
44b3b614bd
Revert "DEV: Avoid duplication of gems in gemfile." (#14784)
This reverts commit 2818ebd19f.
2021-11-01 17:58:24 +05:30
Osama Sayegh
228264d17c
Revert "DEV: add routes_lazy_route to boost boot-up time (#14545)" (#14581)
This reverts commit f5cf647e57.

The gem breaks usage of Rails URL helpers when used outside views and
controllers, for example in
88ecb83382/app/models/upload.rb (L239-L242)
the `upload_short_path` method call fails with an undefined method
exception when this gem is enabled.
2021-10-12 17:30:38 +03:00
Sam
f5cf647e57
DEV: add routes_lazy_route to boost boot-up time (#14545)
The lazy route initialization cuts down boot time of rails.

On my local system it cuts out 200ms of boot time taking me from 3.2 to 3 seconds.

This is not a radically enormous amount of time, but paper cuts add up, and a faster boot in dev will make everyone happy.

TBD if we want to also include this in production.

Gem is heavily maintained by @amatsuda, last commit 3 days ago.
2021-10-11 13:22:13 +11:00
Yasuo Honda
a29a6522d0
FIX: rss gem is bundled gem since Ruby 3 (#14350)
* FIX: rss gem is bundled gem since Ruby 3

Refer https://github.com/ruby/ruby/pull/2832
2021-09-20 11:51:47 +08:00
Alan Guo Xiang Tan
6a5b44be53
DEV: Pin oj gem to 3.13.2. (#14219)
Follow-up to 0183d51070
2021-09-02 11:01:54 +08:00
Alan Guo Xiang Tan
16cfbed89d
DEV: Upgrade Rails to 6.1.4.1. (#14098)
This pulls in a fix for CVE-2021-22942 but we're not tagging it as a
security fix because Discourse is not affected by it in production.
2021-08-20 10:20:08 +08:00
Alan Guo Xiang Tan
2818ebd19f DEV: Avoid duplication of gems in gemfile.
Follow-up to 08c6a7679d
2021-07-28 21:23:15 +08:00
Vinoth Kannan
08c6a7679d
DEV: allow dev:populate rake task in production environment. (#13777)
To use this rake task in production `ALLOW_DEV_POPULATE` environment variable should be set to "1".
2021-07-20 12:25:59 +05:30
David Taylor
8c370c3fe3 DEV: Add annotate rake tasks, and enforce via GitHub actions
`bin/rake annotate` is an alias of `bin/annotate --models`
`bin/rake annotate:clean` generates annotations by using a temporary, freshly migrated database. This should help us to produce more consistent annotations, even if development databases have been polluted by plugin migrations.

A GitHub actions task is also added which generates annotations on a clean database, and raises an error if they differ from the committed annotations.
2021-07-06 10:11:06 +01:00
Vinoth Kannan
6abc45e57b
DEV: move discourse_dev gem to the core. (#13360)
And get avatar images from `discourse_dev_assets` gem.
2021-06-14 20:34:44 +05:30
Arpit Jalan
283b08d45f
DEV: Absorb onebox gem into core (#12979)
* Move onebox gem in core library

* Update template file path

* Remove warning for onebox gem caching

* Remove onebox version file

* Remove onebox gem

* Add sanitize gem

* Require onebox library in lazy-yt plugin

* Remove onebox web specific code

This code was used in standalone onebox Sinatra application

* Merge Discourse specific AllowlistedGenericOnebox engine in core

* Fix onebox engine filenames to match class name casing

* Move onebox specs from gem into core

* DEV: Rename `response` helper to `onebox_response`

Fixes a naming collision.

* Require rails_helper

* Don't use `before/after(:all)`

* Whitespace

* Remove fakeweb

* Remove poor unit tests

* DEV: Re-add fakeweb, plugins are using it

* Move onebox helpers

* Stub Instagram API

* FIX: Follow additional redirect status codes (#476)

Don’t throw errors if we encounter 303, 307 or 308 HTTP status codes in responses

* Remove an empty file

* DEV: Update the license file

Using the copy from https://choosealicense.com/licenses/gpl-2.0/#

Hopefully this will enable GitHub to show the license UI?

* DEV: Update embedded copyrights

* DEV: Add Onebox copyright notice

* DEV: Add MIT license, convert COPYRIGHT.txt to md

* DEV: Remove an incorrect copyright claim

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Co-authored-by: jbrw <jamie@goatforce5.org>
2021-05-26 15:11:35 +05:30
Blake Erickson
fccf4d4375
DEV: Switch off of our image_optim fork (#13124)
The main image_optim gem now includes the timeout feature
that we had in our fork. So it is now safe to switch off of our fork and
back to the image_optim gem.

This is the link to the commit in the image_optim repo that adds the
timeout option:

ec3767dde0

One difference with the new timeout implementation is that image_optim
now handles the timeout exceptions instead of bubbling them up:

1ed0328587/lib/image_optim.rb (L128-L129)

```
 rescue Errors::TimeoutExceeded
   handler.result
```

So a timeout will just return `nil`, which is the same response if it
couldn't optimize an image. I don't think we were really watching for
or doing anything about these timeout warnings in our logs so I think
this is an okay change to have and we will have less warnings in our
logs now too.
2021-05-24 19:27:20 -06:00
David Taylor
c456e31d26
SECURITY: Bump Rails to 6.1.3.2 (#12963)
Includes fixes for

- CVE-2021-22902
- CVE-2021-22903
- CVE-2021-22904
- CVE-2021-22885

https://github.com/rails/rails/blob/v6.1.3.2/actionpack/CHANGELOG.md
2021-05-06 12:41:45 +01:00
Osama Sayegh
45ccadeeeb
DEV: Upgrade Rails to 6.1.3.1 (#12688)
Rails 6.1.3.1 deprecates a few API and has some internal changes that break our tests suite, so this commit fixes all the deprecations and errors and now Discourse should be fully compatible with Rails 6.1.3.1. We also have a new release of the rails_failover gem that's compatible with Rails 6.1.3.1.
2021-04-21 12:36:32 +03:00
Roman Rizzi
5e4c0e2caa
FEATURE: Treat site settings as plain text and add a new HTML type. (#12618)
To add an extra layer of security, we sanitize settings before shipping them to the client. We don't sanitize those that have the "html" type.

The CookedPostProcessor already uses Loofah for sanitization, so I chose to also use it for this. I added it to our gemfile since we installed it as a transitive dependency.
2021-04-07 12:51:19 -03:00
Penar Musaraj
0cd4da51c7
FIX: Include json_schemer in prod (#12351) 2021-03-10 21:32:23 -05:00
Sam
9e79f8321a
DEV: Update Rails to latest patch level (#12281)
This contains 3 security fixes that do not impact us, see:

https://discuss.rubyonrails.org/c/security-announcements/9
2021-03-05 12:07:55 +11:00
Vinoth Kannan
8d96713aa0
DEV: add discourse_dev gem in development dependencies. (#12285)
For more details visit https://github.com/discourse/discourse_dev
2021-03-04 23:04:51 +05:30
Bianca Nenciu
3246c3cc92
DEV: Update mail and use fork (#10639)
Version 2.8 brings some changes to how address fields are handled and
this commits updates that and should also include a fix which handles
encoded attachment filenames.

The fork contains a bugfix to correctly decode mail attachments.
2021-02-18 20:15:02 +02:00
Blake Erickson
7434116933
DEV: Add schema checking to api doc testing (#11721)
* DEV: Add schema checking to api doc testing

This commit improves upon rswag which lacks schema checking. rswag
really only checks that the https status matches, but this change adds
in the json-schema_builder gem which also has schema validation.

Now we can define schemas for each of our requests/responses in the
`spec/requests/api/schemas` directory which will make our documentation
specs a lot cleaner.

If we update a serializer by either adding or removing an attribute the
tests will now fail (this is a good thing!). Also if you change the type
of an attribute say from an array to a string the tests will now fail.
This will help significantly with keeping the docs in sync with actual
code changes! Now if you change how an endpoint will respond you will
have to update the docs too in order for the tests to pass. :D

This PR is inspired by:

 https://www.tealhq.com/post/how-teal-keeps-their-api-tests-and-documentation-in-sync

* Swap out json schema validator gem

Swapped out the outdated json-schema_builder gem with the json_schemer
gem.

* Add validation fields to schema

In order to have "strict" validation we need to add
`additionalProperties: false` to the schema, and we need to specify
which attributes are required.

Updated the debugging test output to print out the error details if
there are any.
2021-01-21 16:28:08 -07:00
Sam
c3fedae73a
DEV: update rack-mini-profiler (#11597)
Included support for flamegraphs using speedscope!
2020-12-29 13:54:41 +11:00
Vinoth Kannan
347423007a
DEV: remove instagram login site settings and auth classes. (#11073)
Instagram removed the support for login and should use Facebook login instead.
2020-10-30 09:09:56 +05:30
Guo Xiang Tan
122cf8d3fb
Update Rails to 6.0.3.3. 2020-09-10 15:41:06 +08:00
Blake Erickson
b16915d166 DEV: Switch over to a fork of ember-rails
We are switching over to a fork because we are currently on a pinned
version of ember-rails 0.18.5 which is pretty old. Upgrading to the
latest version causes many things to break which isn't really worth the
time to debug while we plan to completely switch over to ember-cli
somewhat soonish. Our fork contains a single cherry-pick commit

https://github.com/emberjs/ember-rails/pull/534

which will fix an issue when running the `rails g migration` command and
it spits out a bunch of deprecation warnings.
2020-09-04 14:47:56 -06:00
Blake Erickson
d79f50a3c6 DEV: Remove rb-inotify gem from Gemfile
We are no longer directly referencing the rb-inotify gem directly in
code. This was just a spec level dependency anyways.

Using `git log -S "Inotify"` resulted in these two commits as usages of
`Inotify`:

- b56b11d96a
- 9cf03b352c

both from 2013, but we no longer are using inotify in

https://github.com/discourse/discourse/blob/master/lib/tasks/autospec.rake

which appears to be the only file that was using it.

Based on this info we can safely remove rb-inotify from the Gemfile.

Just as a side note we still do have a couple of gems that do have
rb-inotify as a dependency: listen, and lru_redux.
2020-09-02 11:21:40 +08:00
Blake Erickson
43ffd4d28f
DEV: Switch our fast_xor gem for xorcist (#10565)
* DEV: Switch our fast_xor gem for xorcist

We use the `xor` function as part of password hashing and we want to use
a faster version than the native ruby xor'ing feature so we use a gem
for this.

fast_xor has been abandoned, and xorcist fixed our initial holdup for
switching in https://github.com/fny/xorcist/issues/4

xorcist also has jruby support so we can remove our jruby fallback
logic.

* Move using statement inside of class
2020-08-31 13:20:44 -06:00
Bianca Nenciu
f2e14a3946
FEATURE: Add site setting and wizard step to set base font (#10250)
Co-authored-by: Neil Lalonde <neillalonde@gmail.com>
2020-08-31 13:14:09 +03:00
romanrizzi
0f63a61fa4
Revert "DEV: upgrades sassc to 2.4.0 (#10447)"
We suspect it could be the cause of some issues with themes.

This reverts commit 2fac77cc48.
2020-08-21 13:23:01 -03:00
Joffrey JAFFEUX
2fac77cc48
DEV: upgrades sassc to 2.4.0 (#10447)
We were stuck on 2.0.1 but multiple reports on the original issue shows that the issue has been fixed since 2.3 and 2.4:

https://github.com/sass/sassc-ruby/issues/146#issuecomment-654522157
https://github.com/sass/sassc-ruby/issues/146#issuecomment-654522157

There's one report show it was not fixed in 2.3.0, but that's the only one, it's probably worth trying: https://github.com/sass/sassc-ruby/issues/146#issuecomment-639073736
2020-08-21 13:46:46 +02:00
Blake Erickson
ea2e58e622
DEV: Bump rotp gem to latest version (#10472)
The rotp gem is currently pinned to version 5.1.0 and this will bump it
up to version 6.0.1.

Follow up to: 85d4370f79

because this issue we were waiting on is now closed:

https://github.com/mdp/rotp/issues/98

Because version 6 is now encoding the params I needed to update the
tests as well.
2020-08-19 09:16:33 -06:00
Blake Erickson
c767fb1ed3
DEV: Upgrade highline gem to latest (#10471)
Currently we have pinned highline to version 1.7.0. This is the gem that
we use to have an interactive command line for tasks like `rake
admin:create`.

Upgrading to the latest version 2.0.3 will remove ruby 2.7 deprecation
warnings.

I'm not sure why *this* gem was pinned. I manually executed a couple of
our rake tasks that use this and everything seems fine.
2020-08-19 09:15:23 -06:00
Sam Saffron
620c223d50
DEV: change platform mri to platform ruby on some gems
This assists with truffleruby testing which needs to be able to install
these gems
2020-07-29 17:10:38 +10:00
Sam Saffron
85d4370f79
DEV: pin the rotp gem
Not ready for an upgrade due to: https://github.com/mdp/rotp/issues/98

The policy here is that for cases like this we pin the version and add
a comment explaining why it is pinned.

We can revisit in a few months depending on upstream.
2020-07-14 17:05:12 +10:00
Sam Saffron
bac25e6dd7
DEV: upgrade rack to version 2.2.3
This is very minor, see: https://github.com/advisories/GHSA-j6w9-fv6q-3q52

An attacker can elevate own cookie usage to bypass server cookie restrictions

Technically this is a security commit, but the surface area is extremely
low, we do not expect any real world impact.
2020-07-08 16:42:31 +10:00
Sam Saffron
8af5194e39
DEV: upgrade rails to version 6.0.3.2
This includes a fix for CVE-2020-8185 we are not vulnerable as we do not use
the impacted middleware. However it still makes sense to stay upgraded, other
small fixes exist in this release.
2020-07-08 16:34:29 +10:00
Guo Xiang Tan
58e52c0e4f
DEV: Use rails_failover gem for ActiveRecord and Redis failover handling 2020-06-15 15:47:07 +08:00
Guo Xiang Tan
2e2dcb7d93
DEV: Disable better_errors by default.
It adds a somewhat unnecessary middleware before `ActionDispatch::DebugExceptions` and totally bypasses it. Apps that register exception interceptors with `ActionDispatch::DebugExceptions` would therefore stop working if better_errors is used.
2020-06-12 09:22:40 +08:00
Martin Brennan
e84ffb4861
DEV: Add pry-byebug and optionally require pry behind a ENV var for better spec debugging (#9984)
We removed pry-nav a while back because it is not up to date with pry but it is super useful. Luckily pry-byebug is here to save us all from Satan's power.

To get this to work you need to add the following to your $HOME/.pryrc file.

```
if defined?(PryByebug)
  Pry.commands.alias_command 'c', 'continue'
  Pry.commands.alias_command 's', 'step'
  Pry.commands.alias_command 'n', 'next'
  Pry.commands.alias_command 'f', 'finish'
end

Pry::Commands.command /^$/, "repeat last command" do
  pry_instance.run_command Pry.history.to_a.last
end
```

The require-ing of pry, pry-rails, and pry-byebug in specs is controlled by the IMPROVED_SPEC_DEBUGGING flag (disabled by default).
2020-06-05 12:30:34 +10:00
Robin Ward
cae839ca96
DEV: Update gems to latest versions (#9909) 2020-05-28 13:13:10 -04:00
Guo Xiang Tan
8c86a109bb
DEV: Add ENV flag to test out ActiveRecord::Failover. 2020-05-28 16:24:22 +08:00
Michael Brown
d9a02d1336
Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eee.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
Jeff Atwood
20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6f, reversing
changes made to 2660c2e21d.
2020-05-22 20:25:56 -07:00