Commit Graph

8855 Commits

Author SHA1 Message Date
Vinoth Kannan
2c7cc40be3
DEV: only use the records that are auto populated by the task. (#14360)
Previously, it was using existing user and topic records to generate random posts.
2021-09-17 09:47:32 +05:30
Penar Musaraj
ab9c63e4ee
UX: Optionally show a "Summarize" button in topic timeline (#13533) 2021-09-16 15:15:00 -04:00
Vinoth Kannan
a6de4a5ce9
DEV: use upload id to save in theme setting instead of URL. (#14341)
When we use URL instead it creates the problem while changing the CDN hostname.
2021-09-16 07:58:53 +05:30
Martin Brennan
41e19adb0d
DEV: Ignore reminder_type for bookmarks (#14349)
We don't actually use the reminder_type for bookmarks anywhere;
we are just storing it. It has no bearing on the UI. It used
to be relevant with the at_desktop bookmark reminders (see
fa572d3a7a)

This commit marks the column as readonly, ignores it, and removes
the index, and it will be dropped in a later PR. Some plugins
are relying on reminder_type partially so some stubs have been
left in place to avoid errors.
2021-09-16 09:56:54 +10:00
Yasuo Honda
eed3773b97 FIX: Address ArgumentError to support Ruby 3 keyword arguments 2021-09-15 12:56:54 -04:00
Alan Guo Xiang Tan
27bad28c53
Partially revert "PERF: Improve query performance all inbox private messages. (#14304)" (#14344)
This partially reverts commit ddb458343d.

Seeing performance degrade on larger sites so back to drawing board on
this one. Instead of the DISTINCT LEFT JOIN, we switch back to
IN(subquery).
2021-09-15 11:32:10 +08:00
Alan Guo Xiang Tan
ddb458343d
PERF: Improve query performance all inbox private messages. (#14304)
First reported in https://meta.discourse.org/t/-/202482/19

There are two optimizations being applied here:

1. Fetch a user's group ids in a seperate query instead of including it
   as a sub-query. When I tried a subquery, the query plan becomes very
inefficient.

1. Join against the `topic_allowed_users` and `topic_allowed_groups`
   table instead of doing an IN against a subquery where we UNION the
`topic_id`s from the two tables. From my profiling, this enables PG to
do a backwards index scan on the `index_topics_on_timestamps_private`
index.

This commit fixes a bug where listing all messages was incorrectly
excluding topics if a topic has been archived by a group even if the
user did not belong to the group.

This commit also fixes another bug where dismissing private messages
selectively was subjected to the default limit of 30.
2021-09-15 10:29:42 +08:00
Martin Brennan
22208836c5
DEV: Ignore bookmarks.topic_id column and remove references to it in code (#14289)
We don't need no stinkin' denormalization! This commit ignores
the topic_id column on bookmarks, to be deleted at a later date.
We don't really need this column and it's better to rely on the
post.topic_id as the canonical topic_id for bookmarks, then we
don't need to remember to update both columns if the bookmarked
post moves to another topic.
2021-09-15 10:16:54 +10:00
Martin Brennan
0d809197aa
FIX: Make sure S3 object headers are preserved on copy (#14302)
When copying an existing upload stub temporary object
on S3 to its final destination we were not copying across
its additional headers such as content-disposition and
cache-control, which led to issues like attachments not
downloading with their original filename when clicking
the download links in posts.

This is because the metadata_directive = REPLACE option
was not being passed to object.copy_from(), so only the
source object's headers were being used. Added an option
for apply_metadata_to_destination to apply this option
conditionally, because we may not always want to replace
this metadata, but we definitely do when copying a temporary
upload.
2021-09-10 12:59:51 +10:00
Alan Guo Xiang Tan
bc23dcd30b
FIX: Don't publish PM archive events to acting user. (#14291)
When a user archives a personal message, they are redirected back to the
inbox and will refresh the list of the topics for the given filter.
Publishing an event to the user results in an incorrect incoming message
because the list of topics has already been refreshed.

This does mean that if a user has two tabs opened, the non-active tab
will not receive the incoming message but at this point we do not think
the technical trade-offs are worth it to support this feature. We
basically have to somehow exclude a client from an incoming message
which is not easy to do.

Follow-up to fc1fd1b416
2021-09-10 09:20:50 +08:00
Daniel Waterworth
05c356f7c6
DEV: Add rake task to check that the DB can be accessed (#14300) 2021-09-09 14:42:10 -05:00
Daniel Waterworth
6262396d8a
FIX: Don't attempt to migrate multisite test db while holding the mutex (#14298)
Since you'll have to wait for the mutex to timeout before it can
continue.
2021-09-09 13:31:52 -05:00
Jarek Radosz
02a6b991fe
FIX: Correct the play icon position (#14295) 2021-09-09 15:10:32 +02:00
Arpit Jalan
11a07b37e1
FIX: ignore canonical link for medium.com oneboxes (#14278)
https://meta.discourse.org/t/bug-in-onebox-link-being-rendered-as-a-gist-when-it-isnt/202463
2021-09-08 20:19:57 +05:30
Joffrey JAFFEUX
95b15acb1e
DEV: uses forked Mousetrap to avoid leaking listeners (#14198) 2021-09-08 14:48:13 +02:00
Krzysztof Kotlarek
e3793e6d7c
FIX: better filter for groups search (#14262)
Follow up of https://github.com/discourse/discourse/pull/14216

Allow plugins to register custom filter with block
2021-09-08 09:38:45 +10:00
Alan Guo Xiang Tan
fc1fd1b416
FEATURE: Display new/unread count in browse more messages for PMs. (#14188)
In order to include the new/unread count in the browse more message
under suggested topics, a couple of technical changes have to be made.

1. `PrivateMessageTopicTrackingState` is now auto-injected which is
   similar to how it is done for `TopicTrackingState`. This is done so
we don't have to attempt to pass the `PrivateMessageTopicTrackingState`
object multiple levels down into the suggested-topics component. While
the object is auto-injected, we only fetch the initial state and start
tracking when the relevant private messages routes has been hit and only
when a private message's suggested topics is loaded. This is
done as we do not want to add the extra overhead of fetching the inital
state to all page loads but instead wait till the private messages
routes are hit.

2. Previously, we would stop tracking once the `user-private-messages`
   route has been deactivated. However, that is not ideal since
navigating out of the route and back means we send an API call to the
server each time. Since `PrivateMessageTopicTrackingState` is kept in
sync cheaply via messageBus, we can just continue to track the state
even if the user has navigated away from the relevant stages.
2021-09-07 12:30:40 +08:00
Martin Brennan
9f36d8ad43
FIX: Capture CC addresses for forwarded emails (#14254)
When forwarding emails into the group inbox, we now use the
original sender email as the from_address since
2ac9fd9dff. However, we have not
been saving the original CC addresses of the forwarded email,
which are needed to include those recipients in on the conversation
when replying via the group inbox.

This commit captures the CC addresses on the incoming email, and
makes sure the emails are created as staged users and added to the
list of topic allowed users so they are included on CC's sent by
the GroupSmtpEmail and other jobs.
2021-09-07 08:46:28 +10:00
Jean
34ff7bfeeb
FEATURE: Hide suspended users from site-wide search to regular users (#14245) 2021-09-06 09:59:35 -04:00
Martin Brennan
7b392cee50
FIX: Correct the forwarded by user small post for group inbox (#14252)
When 2ac9fd9dff was done, this
affected the small post that is created when forwarding an email
into the group inbox. Instead of using the name and the email of
the user who forwarded the email, it used the original from email
and name to create the small post. So instead of something like
"Discourse Team forwarded the above email" we ended up with
"John Smith forwarded the above email" which is incorrect.

This fixes the issue by creating a staged user for the forwarding
email address (if such a user does not yet exist) and uses that
for the "forwarded" small post instead.
2021-09-06 15:02:13 +10:00
Martin Brennan
dd4b8c2afa
FIX: Use random file name for temporary uploads (#14250)
Other locale characters in file names (e.g. é, ä) as well
as special characters can cause issues on S3, notably the S3
copy object operation does not support these special characters.
Instead of storing the original file name in the key, which is
unnecessary, we now generate a random file name with the original
extension for the temporary file and use that for all external
upload stub operations.
2021-09-06 10:21:20 +10:00
Krzysztof Kotlarek
f859fd6bde
FEATURE: allow plugins to extend Groups (#14216)
* add_permitted_group_param API for plugins
* add groups-interaction-custom-options outlet
* custom search can use custom group scope
2021-09-06 10:18:51 +10:00
Daniel Waterworth
d7873dd823
FIX: Don't attempt to migrate concurrently with other migrations (#14231) 2021-09-03 10:22:25 -05:00
Arpit Jalan
763f48abc7
FIX: increase chunk size to fetch title tag correctly (#14144) 2021-09-03 13:15:58 +05:30
Penar Musaraj
90a23c6fc8
FEATURE: Enable auto dark mode on new instances (#14208) 2021-09-02 14:55:38 -04:00
Neil Lalonde
f0d2b0f2f0
Version bump to v2.8.0.beta6 (#14228)
v2.8.0.beta5 can cause segfaults due to the oj gem v3.13.3.
2021-09-02 14:25:33 -04:00
Joshua Rosenfeld
823354d177
DEV: Update Plugin Release Notes heading format (#14218) 2021-09-02 11:08:31 +08:00
Neil Lalonde
c6f1818b85
Version bump to v2.8.0.beta5 (#14210) 2021-09-01 13:29:36 -04:00
David Taylor
ed688bec8c
FIX: Ensure id sequences are not reset during db:migrate (#14184)
The seed-fu gem resets the sequence on all the tables it touches. In some situations, this can cause primary keys to be re-used. This commit introduces a freedom patch which ensures seed-fu only touches the sequence when it is less than the id of one of the seeded records.
2021-08-30 12:31:22 +01:00
David Taylor
b47f288b13
FIX: Update PresenceChannel#present to work for redis 6.0 (#14180)
The `LT` parameter was only added in redis 6.2. We can make do with `NX` instead
2021-08-27 19:44:28 +01:00
David Taylor
01e2836071
FIX: Allow PresenceChannel to work on Redis 6.0 (#14178)
The `ZRANGE ... BYSCORE` syntax was only added in 6.2. We can use the older `ZRANGEBYSCORE` instead
2021-08-27 18:55:54 +01:00
David Taylor
31db83527b DEV: Introduce PresenceChannel API for core and plugin use
PresenceChannel aims to be a generic system for allow the server, and end-users, to track the number and identity of users performing a specific task on the site. For example, it might be used to track who is currently 'replying' to a specific topic, editing a specific wiki post, etc.

A few key pieces of information about the system:
- PresenceChannels are identified by a name of the format `/prefix/blah`, where `prefix` has been configured by some core/plugin implementation, and `blah` can be any string the implementation wants to use.
- Presence is a boolean thing - each user is either present, or not present. If a user has multiple clients 'present' in a channel, they will be deduplicated so that the user is only counted once
- Developers can configure the existence and configuration of channels 'just in time' using a callback. The result of this is cached for 2 minutes.
- Configuration of a channel can specify permissions in a similar way to MessageBus (public boolean, a list of allowed_user_ids, and a list of allowed_group_ids). A channel can also be placed in 'count_only' mode, where the identity of present users is not revealed to end-users.
- The backend implementation uses redis lua scripts, and is designed to scale well. In the future, hard limits may be introduced on the maximum number of users that can be present in a channel.
- Clients can enter/leave at will. If a client has not marked itself 'present' in the last 60 seconds, they will automatically 'leave' the channel. The JS implementation takes care of this regular check-in.
- On the client-side, PresenceChannel instances can be fetched from the `presence` ember service. Each PresenceChannel can be used entered/left/subscribed/unsubscribed, and the service will automatically deduplicate information before interacting with the server.
- When a client joins a PresenceChannel, the JS implementation will automatically make a GET request for the current channel state. To avoid this, the channel state can be serialized into one of your existing endpoints, and then passed to the `subscribe` method on the channel.
- The PresenceChannel JS object is an ember object. The `users` and `count` property can be used directly in ember templates, and in computed properties.
- It is important to make sure that you `unsubscribe()` and `leave()` any PresenceChannel objects after use

An example implementation may look something like this. On the server:

```ruby
register_presence_channel_prefix("site") do |channel|
  next nil unless channel == "/site/online"
  PresenceChannel::Config.new(public: true)
end
```

And on the client, a component could be implemented like this:

```javascript
import Component from "@ember/component";
import { inject as service } from "@ember/service";

export default Component.extend({
  presence: service(),
  init() {
    this._super(...arguments);
    this.set("presenceChannel", this.presence.getChannel("/site/online"));
  },
  didInsertElement() {
    this.presenceChannel.enter();
    this.presenceChannel.subscribe();
  },
  willDestroyElement() {
    this.presenceChannel.leave();
    this.presenceChannel.unsubscribe();
  },
});
```

With this template:

```handlebars
Online: {{presenceChannel.count}}
<ul>
  {{#each presenceChannel.users as |user|}} 
    <li>{{avatar user imageSize="tiny"}} {{user.username}}</li>
  {{/each}}
</ul>
```
2021-08-27 16:26:06 +01:00
Martin Brennan
99ec8eb6df
FIX: Capture S3 metadata when calling create_multipart (#14161)
The generate_presigned_put endpoint for direct external uploads
(such as the one for the uppy-image-uploader) records allowed
S3 metadata values on the uploaded object. We use this to store
the sha1-checksum generated by the UppyChecksum plugin, for later
comparison in ExternalUploadManager.

However, we were not doing this for the create_multipart endpoint,
so the checksum was never captured and compared correctly.

Also includes a fix to make sure UppyChecksum is the last preprocessor to run.
It is important that the UppyChecksum preprocessor is the last one to
be added; the preprocessors are run in order and since other preprocessors
may modify the file (e.g. the UppyMediaOptimization one), we need to
checksum once we are sure the file data has "settled".
2021-08-27 09:50:23 +10:00
David Taylor
0aa8798046
DEV: Remove yarn install during assets:precompile (#14166)
Instead, we'll do it alongside `bundle install` in `web.template.yml`: https://github.com/discourse/discourse_docker/pull/565
2021-08-26 19:14:50 +01:00
Bianca Nenciu
f0005025c4
FIX: Correctly display GitHub code oneboxes (#14149)
Code fragments without the code lines were not displayed correctly.

Follow-up-to 721a946d77.
2021-08-26 12:55:46 +03:00
Martin Brennan
e43a8af3bd
FIX: Do not send emails to mailing_list_mode subscribers for PMs (#14159)
This bug was introduced by f66007ec83.

In PostJobsEnqueuer we previously did not fire the after_post_create
event and after_topic_create event for private message topics. This was
changed in the above commit in order to publish message bus messages
for topic tracking state updates. Unfortunately this caused the
NotifyMailingListSubscribers job to be enqueued for all posts including
private messages, and admins and the users involved in the PMs got
emailed the contents of the PMs if they had mailing list mode enabled.

Luckily the impact of this was mitigated by a Guardian#can_see? check
for each mailing list mode user in the NotifyMailingListSubscribers job.
We never want to notify mailing list mode subscribers for private messages
so an early return has been added there, plus the logic in PostJobsEnqueuer
has been fixed, and tests have been added to that class where there were
none before.
2021-08-26 15:16:35 +10:00
Martin Brennan
841e054907
FIX: Do not prefix temp/ S3 keys with s3_bucket_folder_path in S3Helper (#14145)
This is unnecessary, as when the temporary key is created
in S3Store we already include the s3_bucket_folder_path, and
the key will always start with temp/ to assist with lifecycle
rules for multipart uploads.

This was affecting Discourse.store.object_from_path,
Discourse.store.signed_url_for_path, and possibly others.

See also: e0102a5
2021-08-26 08:50:49 +10:00
Penar Musaraj
85b8fea262
UX: Add Styling step to wizard (#14132)
Refactors three wizard steps (colors, fonts, homepage style) into one new step called Styling.
2021-08-25 17:10:12 -04:00
Bianca Nenciu
721a946d77
FIX: Remove spacing from GitHub oneboxes (#14147)
Each code line was surrounded by a lot of white space. The new styles
are similar to what the web interface is using.
2021-08-25 16:34:01 +03:00
Bianca Nenciu
197532dc31
FIX: Add plugin event to topic list user lookup (#14116)
This can be used to change the list of topic posters. For example,
discourse-solved can use this to move the user who posted the solution
after the original poster.
2021-08-25 13:16:08 +03:00
Arpit Jalan
419d71abcb
FEATURE: allow admin to delete all posts by a user irrespectively (#14128)
This commit allows admin to delete all posts by a user irrespective of
site settings `delete_user_max_post_age` and `delete_all_posts_max`.
2021-08-25 10:14:22 +05:30
Alan Guo Xiang Tan
f66007ec83
FEATURE: Display unread and new counts for messages. (#14059)
There are certain design decisions that were made in this commit.

Private messages implements its own version of topic tracking state because there are significant differences between regular and private_message topics. Regular topics have to track categories and tags while private messages do not. It is much easier to design the new topic tracking state if we maintain two different classes, instead of trying to mash this two worlds together.

One MessageBus channel per user and one MessageBus channel per group. This allows each user and each group to have their own channel backlog instead of having one global channel which requires the client to filter away unrelated messages.
2021-08-25 11:17:56 +08:00
Penar Musaraj
1167b16913
FIX: Order outputted theme stylesheets (#14133) 2021-08-25 09:37:07 +08:00
Martin Brennan
e0102a533a
FIX: Restructure temp/ folders for direct S3 uploads (#14137)
Previously we had temp/ in the middle of the S3 key path like so

* /uploads/default/temp/randomstring/test.png (normal site)
* /sitename/uploads/default/temp/randomstring/test.png (s3 folder path site)
* /standard10/uploads/sitename/temp/randomstring/test.png (multisite site)

However this necessitates making a lifecycle rule to clean up incomplete
S3 multipart uploads for every site, something which we cannot do. It makes
much more sense to have a structure with /temp at the start of the key,
which is what this commit does:

* /temp/uploads/default/randomstring/test.png (normal site)
* /temp/sitename/uploads/default/randomstring/test.png (s3 folder path site)
* /temp/standard10/uploads/sitename/randomstring/test.png (multisite site)
2021-08-25 09:22:36 +10:00
Martin Brennan
d295a16dab
FEATURE: Uppy direct S3 multipart uploads in composer (#14051)
This pull request introduces the endpoints required, and the JavaScript functionality in the `ComposerUppyUpload` mixin, for direct S3 multipart uploads. There are four new endpoints in the uploads controller:

* `create-multipart.json` - Creates the multipart upload in S3 along with an `ExternalUploadStub` record, storing information about the file in the same way as `generate-presigned-put.json` does for regular direct S3 uploads
* `batch-presign-multipart-parts.json` - Takes a list of part numbers and the unique identifier for an `ExternalUploadStub` record, and generates the presigned URLs for those parts if the multipart upload still exists and if the user has permission to access that upload
* `complete-multipart.json` - Completes the multipart upload in S3. Needs the full list of part numbers and their associated ETags which are returned when the part is uploaded to the presigned URL above. Only works if the user has permission to access the associated `ExternalUploadStub` record and the multipart upload still exists.

  After we confirm the upload is complete in S3, we go through the regular `UploadCreator` flow, the same as `complete-external-upload.json`, and promote the temporary upload S3 into a full `Upload` record, moving it to its final destination.
* `abort-multipart.json` - Aborts the multipart upload on S3 and destroys the `ExternalUploadStub` record if the user has permission to access that upload.

Also added are a few new columns to `ExternalUploadStub`:

* multipart - Whether or not this is a multipart upload
* external_upload_identifier - The "upload ID" for an S3 multipart upload
* filesize - The size of the file when the `create-multipart.json` or `generate-presigned-put.json` is called. This is used for validation.

When the user completes a direct S3 upload, either regular or multipart, we take the `filesize` that was captured when the `ExternalUploadStub` was first created and compare it with the final `Content-Length` size of the file where it is stored in S3. Then, if the two do not match, we throw an error, delete the file on S3, and ban the user from uploading files for N (default 5) minutes. This would only happen if the user uploads a different file than what they first specified, or in the case of multipart uploads uploaded larger chunks than needed. This is done to prevent abuse of S3 storage by bad actors.

Also included in this PR is an update to vendor/uppy.js. This has been built locally from the latest uppy source at d613b849a6. This must be done so that I can get my multipart upload changes into Discourse. When the Uppy team cuts a proper release, we can bump the package.json versions instead.
2021-08-25 08:46:54 +10:00
Bianca Nenciu
ff367e22fb
FEATURE: Make allow_uploaded_avatars accept TL (#14091)
This gives admins more control over who can upload custom profile
pictures.
2021-08-24 10:46:28 +03:00
Martin Brennan
2ac9fd9dff
FIX: Use original from address when forwarding to group inbox (#14114)
When emails were forwarded to a group inbox by the email address
of the group, for example when an email ends up in spam and must
be manually forwarded to the group+site@discoursemail.com address,
the OP of the topic ended up being the group's email address instead
of the sender who originally sent the email to the group inbox.

This commit detects that an email has been forwarded using existing
tools, and if the from address matches one of the group incoming
email addresses, then we look at the forwarded email's from address
and use that instead for the incoming email from address as well as
the staged/regular user used for the Topic.user.

This will make it much cleaner to forward emails into a group inbox,
and will prevent issues with PostAlerter where the OP is double-notified
for these emails.
2021-08-24 08:57:28 +10:00
Kris
bde6f7e9b0
UX: Update "get a room" composer message (#14104) 2021-08-23 18:34:23 -04:00
Penar Musaraj
3d6cbb5ab3
DEV: Watch for changes in wizard.scss (#14119)
This ensures the stylesheet is recompiled. Devs still need to refresh
the page manually, since we don't include the message bus in the wizard
app.
2021-08-23 13:39:53 -04:00
Joffrey JAFFEUX
efbc2481d8
DEV: removes plugin generator (#14101)
I don't have the time to maintain this and https://github.com/discourse/discourse-plugin-skeleton is doing 80% of the job very easily.
2021-08-20 11:29:06 +02:00
Osama Sayegh
ec352a1969
FEATURE: Order pinned topics by their pinned_at column (#14090)
Currently, pinned topics are ordered by the `bumped_at` column. This behavior is not desired because it gives admins no control over the order of pinned topics. This PR makes pinned topics ordered by the `pinned_at` column. A topic that is pinned last appears first in topic lists. If an admin wants an already pinned topic to appear first in the list of pinned topics, they'll have to unpin that topic and pin it again.

Meta topic: https://meta.discourse.org/t/how-do-i-set-the-order-of-pinned-topics/16935/23?u=osama.
2021-08-19 14:43:58 +03:00
Dan Ungureanu
c0ec1e931e
FIX: Make themes:update work with multisites (#14082)
Running this Rake task in a multisite cluster updated only the default
site.
2021-08-19 09:41:58 +03:00
Arpit Jalan
c481f2ce16
UX: do not show selected composer education messages on whisper post (#14078)
This commit disables the "sequential_replies" and "duplicate_link"
education message on composer when creating a whipser post.
2021-08-19 09:32:32 +05:30
Bianca Nenciu
5c9abab0cd
FIX: Remove markers from all translated languages (#14081)
Emails can include the marker in a different language, depending on
site and user settings. The email receiver always looked for the marker
in default language.
2021-08-18 19:42:04 +03:00
Arpit Jalan
7db3888f17
FIX: log proper error message when SSO nonce verification fails (#14077) 2021-08-18 18:44:12 +05:30
Bianca Nenciu
4380ba34d5
FIX: Make site tasks work with duplicated uploads (#13972)
Uploads can be reused between site settings. This change allows the same
upload to be exported only once and then the same file is reused. The
same applies to import.
2021-08-18 15:57:31 +03:00
Alan Guo Xiang Tan
d13716286c
FIX: Unread group PMs should use GroupUser#first_unread_pm_at. (#14075)
This bug was causing unread PMs for groups to appear inaccurate.
2021-08-18 11:23:28 +08:00
Rafael dos Santos Silva
b136375582
FEATURE: Rate limit exceptions via ENV (#14033)
Allow admins to configure exceptions to our Rails rate limiter.

Configuration happens in the environment variables, and work with both
IPs and CIDR blocks.

Example:

```
env:
  DISCOURSE_MAX_REQS_PER_IP_EXCEPTIONS: >-
    14.15.16.32/27
    216.148.1.2
```
2021-08-13 12:00:23 -03:00
Rafael dos Santos Silva
b2e4c91818
DEV: Allow removing local assets after s3 sync (#14013) 2021-08-12 14:20:05 -03:00
Chema Balsas
d27d7c8cca FIX: Unescapes hash section with present to account for url-encoded chars
Sections with unreserverd characters will appear url-encoded and need to
be unescaped before using it.

Wikipedia generates 2 different spans in this case in the same page, one
with an id resulting of replacing the % symbols with . and the other with
the decoded version of the string. For example, for /wiki/foo#A%C3%A1A it
will generate:

<span id="A.C3.A1A"></span>
<span id="AáA">AáA</span>

Unescaping the `m_url_hash_name` should work in all cases to target the
proper section span.
2021-08-12 10:43:50 -04:00
jbrw
20a6bad87e
FIX: Category group moderators can read flagged post meta_topics (#14014)
When a post is flagged with the reason of 'Something Else' a brief message can be added by the user which subsequently creates a `meta_topic` private message. The group `moderators` is automatically added to this topic.

If category group moderation is enabled, and the post belongs to a category with a reviewable group, that group should also be added to the meta_topic.

Note: This extends the `notify_moderators` logic, and will add the reviewable group to the meta_topic, regardless of the settings of that group.
2021-08-11 18:11:22 -04:00
Rafael dos Santos Silva
8e45fdfbb1
DEV: Log duration of ember-cli asset build (#13980) 2021-08-10 23:43:08 -03:00
Krzysztof Kotlarek
195f7346ba
FIX: check if BasicBadge is enabled for TL1 welcome message (#13983)
In 2018 check was added that TL1 welcome message is sent unless user already has BasicBadge granted.

I think we should also check if BasicBadge is even enabled. Otherwise, each time group is assigned to a user and trust level is recalculated, they will receive a welcome message.
2021-08-11 08:39:25 +10:00
Andrei Prigorshnev
622859dbe6
FEATURE: add Unseen view (#13977)
This view is the same as Latest except it hides the topics you have fully read. Based on this plugin of @davidtaylorhq https://meta.discourse.org/t/simple-unread-list-plugin-discourse-simple-unread/70013.
2021-08-10 18:30:34 +04:00
Dmitry Klimensky
d54b339809 Escape values of HTML attributes 2021-08-10 10:25:15 -04:00
Joffrey JAFFEUX
644441852e
FIX: prevents exception on malformatted messages (#13997)
The following example message would generate an exception:

```
Return-Path: <discourse@bar.com>
From: Foo Bar <discourse@bar.com>
To: reply+4f97315cc828096c9cb34c6f1a0d6fe8@bar.com
Date: Fri, 15 Jan 2016 00:12:43 +0100
Message-ID: <21@foo.bar.mail>
Mime-Version: 1.0
Content-Type: text/html; charset=UTF-8

</div>

```

Exception:

```
NoMethodError:
       undefined method `split' for nil:NilClass
```
2021-08-10 15:49:32 +02:00
Arpit Jalan
3006de39d1
REVERT "FIX: do not show private group flair on user avatars" (#13991)
This reverts commit fe3e18f981 and 0d8fd9ace6
2021-08-10 17:25:11 +05:30
Laurent Arnoud
03896a9c43
FIX: Errno::EXDEV when across filesystem boundaries (#13976)
ref https://bugs.ruby-lang.org/issues/12615

```
        W: rake aborted!
        W: Errno::EXDEV: Invalid cross-device link @ rb_file_s_rename - (/app/tmp/adeeee62504de67238341871bda1aaf0, /app/app/assets/javascripts/plugins/discourse-cakeday.js.e
rb)
        W: lib/discourse.rb:65:in `rename'
        W: lib/discourse.rb:65:in `atomic_write_file'
        W: /app/lib/plugin/instance.rb:683:in `activate!'
        W: lib/discourse.rb:246:in `block in activate_plugins!'
        W: lib/discourse.rb:243:in `each'
        W: lib/discourse.rb:243:in `activate_plugins!'
```
2021-08-09 12:20:26 +02:00
Robin Ward
3fdb64184b FIX: Don't swallow an error if we can't run yarn ember build 2021-08-06 15:13:18 -04:00
Andrei Prigorshnev
09ad3ed41d
FEATURE: revert disallowing putting URLs in titles for TL0 users (#13970)
This reverts a part of changes introduced by https://github.com/discourse/discourse/pull/13947

In that PR I:
1. Disallowed topic feature links for TL-0 users
2. Additionally, disallowed just putting any URL in topic titles for TL-0 users

Actually, we don't need the second part. It introduced unnecessary complexity for no good reason. In fact, it tries to do the job that anti-spam plugins (like Akismet plugin) should be doing.

This PR reverts this second change.
2021-08-06 20:07:42 +04:00
Arpit Jalan
bb2c48b065
FIX: update iframe url for simplecast onebox (#13957)
https://meta.discourse.org/t/onebox-regression-simplecast-com/187911
2021-08-05 18:29:04 +05:30
Robin Ward
18c5e9338f DEV: Allow us to use Ember CLI assets in production
This adds an optional ENV variable, `EMBER_CLI_PROD_ASSETS`. If truthy,
compiling production assets will be done via Ember CLI and will replace
the assets Rails would otherwise use.
2021-08-05 08:32:33 -04:00
Arpit Jalan
a341dba5d9
FIX: update oEmbed URL for simplecast onebox (#13956) 2021-08-05 17:42:38 +05:30
Andrei Prigorshnev
0c0a11b66a
FEATURE: Disallow putting urls in the title for TL-0 users (#13947)
This disallows putting URLs in topic titles for TL0 users, which means that:

If a TL-0 user puts a link into the title, a topic featured link won't be generated (as if it was disabled in the site settings)
Server methods for creating and updating topics will be refusing featured links when they are called by TL-0 users
TL-0 users won't be able to put any link into the topic title. For example, the title "Hey, take a look at https://my-site.com" will be rejected.

Also, it improves a bit server behavior when creating or updating feature links on topics in the categories with disabled featured links. Before the server just silently ignored a featured link field that was passed to him, now it will be returning 422 response.
2021-08-05 13:38:39 +04:00
Alan Guo Xiang Tan
0bf27242ec FIX: Group inbox new filter not accounting for dismissed topics.
Follow-up to 2c046cc670
2021-08-05 16:53:12 +08:00
Alan Guo Xiang Tan
2c046cc670 FEATURE: Dismiss new and unread for PM inboxes. 2021-08-05 12:56:15 +08:00
Krzysztof Kotlarek
354c939656
FIX: remove Nokogumbo references (#13951)
Specs broken after f4720205c0
2021-08-05 11:46:25 +10:00
Martin Brennan
d8a0d2262c
DEV: Update pretender and fake-xml-http-request (#13937)
We are still on a version of pretender since 2017
https://github.com/pretenderjs/pretender/releases/tag/v1.6.1

Since then many changes have been made, including adding support
for xhr.upload. Upgrading will let us write proper acceptance
tests for uppy, which uses XmlHTTPRequest internally including
xhr.upload.

Updates pretender to 3.4.7 and fake-xml-http-request to 2.1.2.

Note: There have been no breaking changes in the releases that would
affect us, mainly dropping support for old node versions.
2021-08-05 08:23:01 +10:00
Bianca Nenciu
d9843d757a
FIX: Update draft count when sequence is increased (#13940)
* FIX: Update draft count when sequence is increased

Sometimes users ended up having a draft count higher than the actual
number of drafts.

* FIX: Do not update draft count twice

The call to DraftSequence.next! above already does it.
2021-08-04 13:30:37 +03:00
Bianca Nenciu
e2af2a2219
FIX: Make rake site:export_structure export uploads (#13938)
It failed because of an undefined variable.
2021-08-04 11:14:30 +03:00
Bianca Nenciu
54e2b95539
FIX: Remove 'reply above line' marker (#13927)
This marker is automatically added by Discourse's group SMTP mailer.
When user responded to emails sent by it, Discourse did not trim this
marker.
2021-08-03 20:08:19 +03:00
Bianca Nenciu
e2c415457c
FEATURE: Attach backup log as upload (#13849)
Discourse automatically sends a private message after backup or
restore finished. The private message used to contain the log inline
even when it was very long. A very long log can create issues because
the length of the post will be over the maximum allowed length of a
post. When that happens, Discourse will try to create an upload with
the logs. If that fails, it will trim the log and inline it.
2021-08-03 20:06:50 +03:00
Bianca Nenciu
52520638ca
FIX: Inline secure images with duplicated names (#13926)
Inlining secure images with the same name was not possible because they
were indexed by filename. If an email contained two files with the same
name, only the first image was used for both of them. The other file
was still attached to the email.
2021-08-03 18:58:34 +03:00
Martin Brennan
b88d8c8894
FIX: Use reply-to address for incoming emails if present (#13896)
When the Reply-To header is present for incoming emails we
want to use it instead of the from address. This is usually the
case when forwarding an email via a mailing list into Discourse.

For now we are only using the Reply-To header if the email has
been forwarded via Google Groups, which is why we are checking the
X-Original-From header too. In future we may want to use the Reply-To
header in more cases.
2021-08-03 08:01:17 +10:00
Jean
e7b8e75583
FEATURE: Add post edits count to user activity (#13495) 2021-08-02 10:15:53 -04:00
Bianca Nenciu
fbf7627c8e
FIX: Make search work with sub-sub-categories (#13901)
Searching in a category looked only one level down, ignoring the site
setting max_category_nesting. The user interface did not support the
third level of categories and did not display them in the "Categorized"
input of the advanced search options.
2021-08-02 14:04:13 +03:00
Alan Guo Xiang Tan
016efeadf6
FEATURE: New and Unread messages for user personal messages. (#13603)
* FEATURE: New and Unread messages for user personal messages.

Co-authored-by: awesomerobot <kris.aubuchon@discourse.org>
2021-08-02 12:41:41 +08:00
Arpit Jalan
fe3e18f981
FIX: do not show private group flair on user avatars (#13872)
Meta ref: https://meta.discourse.org/t/visible-flair-for-invisible-groups-is-that-on-purpose/167674
2021-08-02 06:21:00 +05:30
jbrw
2f28ba318c
FEATURE: Onebox can match engines based on the content_type (#13876)
* FEATURE: Onebox can match engines based on the content_type

`FinalDestination` now returns the `content_type` of a resolved URL.

`Oneboxer` passes this value to `Onebox` itself. Onebox engines can now specify a `matches_content_type` regex of content_types that the engine can handle, regardless of the URL.

`ImageOnebox` will match URLs with a content type of `image/png`, `jpg`, `gif`, `bmp`, `tif`, etc.

This will allow images that exist at a URL without a file type extension to be correctly rendered, assuming a valid `content_type` is returned.
2021-07-30 13:36:30 -04:00
Gerhard Schlager
a37b5b177b
FIX: Remapping of uploads could fail during restore of backup (#13897)
Sidekiq should be paused until the end of the restore, otherwise it might interfere (deadlock) with the remap of upload URLs in posts.
2021-07-30 11:25:28 +02:00
Alan Guo Xiang Tan
849827841f Revert "DEV: Move private message topic tracking state publish into sidekiq."
This reverts commit c51c80466c.

There is a conditional in post jobs enqueuer that only enqueues the job
for non-PM posts.
2021-07-30 11:36:01 +08:00
Bianca Nenciu
300db3d3fa
FIX: Update draft count after creating a post (#13884)
When a post is created, the draft sequence is increased and then older
drafts are automatically executing a raw SQL query. This skipped the
Draft model callbacks and did not update user's draft count.

I fixed another problem related to a raw SQL query from Draft.cleanup!
method.
2021-07-29 17:06:11 +03:00
Alan Guo Xiang Tan
c51c80466c DEV: Move private message topic tracking state publish into sidekiq.
Same intend as b1f32f2f57.
2021-07-29 14:57:19 +08:00
Andrei Prigorshnev
5a2ad7e386
DEV: remove calls to guardian from GroupActionLogger (#13835)
We shouldn't be checking if a user is allowed to do an action in the logger. We should be checking it just before we perform the action. In fact, guardians in the logger can make things even worse in case of a security bug. Let's say we forgot to check user's permissions before performing some action, but we still have a call to the guardian in the logger. In this case, a user would perform the action anyway, and this action wouldn't even be logged!

I've checked all cases and I confirm that we're safe to delete this calls from the logger.

I've added two calls to guardians in admin/user_controller. We didn't have security bugs there, because regular users can't access admin/... routes at all. But it's good to have calls to guardian in these methods anyway, neighboring methods have them.
2021-07-28 15:04:04 +04:00
Martin Brennan
b500949ef6
FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787)
This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads`  settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader.

A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used. 

### Starting a direct S3 upload

When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded.

Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage.

### Completing a direct S3 upload

Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`.

1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this.

2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues.

    We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large.

3. Finally we return the serialized upload record back to the client

There are several errors that could happen that are handled by `UploadsController` as well.

Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
2021-07-28 08:42:25 +10:00
wbhouston
efe38efb0a
FIX: Don't grant sharing badges to users who don't exist (#13851)
In badge queries for 'First Share' and 'Nice/Good/Great Share' badges,
check that the user exists.

For 'Nice+ Share' badges, also grant badges if the number of shares is
equal to the threshhold count to better match the descriptions.
2021-07-27 16:32:59 +10:00
Alan Guo Xiang Tan
02b84dbff2 DEV: Fix flaky instagram onebox spec by not mutating constant. 2021-07-27 13:54:14 +08:00
Martin Brennan
e1d2b67178
DEV: Fix stylesheet manager flaky spec (#13846)
The cache_fullpath for the Stylesheet::Manager was the same for
every test runner in a parallel test environment, so when other
specs or other places e.g. the stylesheets_controller_spec ran
rm -rf Stylesheet::Manager.cache_fullpath this caused errors
for other specs running that went through the
Stylesheet::Manager::Builder#compile path, causing the error

```
Errno::ENOENT:
       No such file or directory @ rb_sysopen
```


Also fixed the stylesheet_controller which was interpolating Rails.root + CACHE_PATH
itself instead of just using Stylesheet::Manager.cache_fullpath
2021-07-26 14:22:54 +10:00
Joffrey JAFFEUX
ed9e63b00e DEV: update doc as current would raise 2021-07-26 08:54:34 +08:00
Robin Ward
7b45a5ce55 FIX: Better and more secure validation of periods for TopicQuery
Co-authored-by: Martin Brennan <mjrbrennan@gmail.com>
2021-07-23 14:24:44 -04:00
Alan Guo Xiang Tan
2923abdec7 SECURITY: Don't leak user of previous whisper post when deleting a topic.
A topic's last poster can be incorrectly set to a user of a whisper post
if the whisper post is before the last post and the last post is
deleted.
2021-07-23 15:42:23 +08:00
Alan Guo Xiang Tan
303e9e42b6 SECURITY: Do not reveal post whisperer in personal messages.
Prior to this fix, post whisperer in personal messages are revealed in
the topic's participants list even though non-staff users are unable to
see the whisper.
2021-07-23 15:42:23 +08:00
Neil Lalonde
cb858af8c7
Version bump to v2.8.0.beta4 (#13824) 2021-07-22 12:23:50 -04:00
Bianca Nenciu
18c32a809b
FIX: Validate email_accent_bg_color color (#13778)
Using an invalid value was allowed. This commit tries to automatically
fix the color by adding missing # symbol or will show an error to the
user if it is not possible and it is not a CSS color either.
2021-07-22 17:42:47 +03:00
Martin Brennan
f41908ad5b
SECURITY: Validate period param for top topic routes (#13818)
Fixes a possible SQL injection vector
2021-07-22 16:31:53 +10:00
jbrw
09d23a37a5
DEV: Add default Accept-Language to FinalDestination requests (#13817)
Not specifying an `Accept-Language` should be equivalent to specifying an `Accept-Language` of `*`, however some webservers seem to prefer it if we are explicit about being able to handle a response of content in any language.
2021-07-22 15:49:59 +10:00
Penar Musaraj
8a470e508e
UX: Improve quick search suggestions (#13813) 2021-07-21 14:00:27 -04:00
Bianca Nenciu
4da0a33524
FIX: Allow only groups with flairs to be selected (#13744)
It used the same permission check as for primary groups which is
wrong because not all groups that can be primary have a flair.
2021-07-21 14:41:04 +03:00
Jarek Radosz
4223541bc5
DEV: Fix release_note task warnings (#13794)
There was a bunch of warnings repeated over and over during spec runs:

```
/var/www/discourse/lib/tasks/release_note.rake:3: warning: already initialized constant DATE_REGEX
/var/www/discourse/lib/tasks/release_note.rake:3: warning: previous definition of DATE_REGEX was here
/var/www/discourse/lib/tasks/release_note.rake:5: warning: already initialized constant CHANGE_TYPES
/var/www/discourse/lib/tasks/release_note.rake:5: warning: previous definition of CHANGE_TYPES was here
```
2021-07-20 20:25:48 +02:00
Kris
5f6b9e36ed
UX: New text and style for dominating topic message (#13789) 2021-07-20 13:58:38 -04:00
Andrei Prigorshnev
1a8c949900
UX: suspend forever time period messages (#13776)
When the Forever option is selected for suspending a user, the user is suspended for 1000 years. Without customizing the site’s text, this time period is displayed to the user in the suspension email that is sent to the user, and if the user attempts to log back into the site. Telling someone that they have been suspended for 1000 years seems likely to come across as a bad attempt at humour.

This PR special case messages when a user suspended or silenced forever.
2021-07-20 14:42:08 +04:00
Vinoth Kannan
08c6a7679d
DEV: allow dev:populate rake task in production environment. (#13777)
To use this rake task in production `ALLOW_DEV_POPULATE` environment variable should be set to "1".
2021-07-20 12:25:59 +05:30
Michael Brown
784e929499 upload_creator: force ImageMagick to use internal SVG code to determine size
This change largely targets dev users, but it could potentially change
behaviour in production.

Jamie Wilson & I debugged a problem where "should not be larger than the
maximum thumbnail size" would fail due to timeouts.

On our systems, on ImageMagick 7.1.0-2, with inkscape installed, IM would
attempt to rasterise the svg then check the resulting filesize, causing the
test to timeout.

As of now, we haven't found a way to cause this to behave better, but have a
workaround in that forcing IM to use the internal renderer (`MSVG:`) seems to
make it perform the same on development workstations as it does in our docker
container.
2021-07-19 22:03:57 -04:00
Michael Brown
5f7e60d9dc discourse/discourse change from 'master' to 'main' 2021-07-19 11:46:15 -04:00
Alan Guo Xiang Tan
a1047f5ef4
FEATURE: Add new plugin API to allow plugins to extend Site#categories (#13773) 2021-07-19 13:54:19 +08:00
Neil Lalonde
b0f06b8ed0
FIX: don't allow category and tag tracking settings on staged users (#13688)
Configuring staged users to watch categories and tags is a way to sign
them up to get many emails. These emails may be unwanted and get marked
as spam, hurting the site's email deliverability.
Users can opt-in to email notifications by logging on to their
account and configuring their own preferences.

If staff need to be able to configure these preferences on behalf of
staged users, the "allow changing staged user tracking" site setting
can be enabled. Default is to not allow it.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2021-07-16 14:50:40 -04:00
Neil Lalonde
5f8fa976d4
Version bump to v2.8.0.beta3 (#13703) 2021-07-15 14:54:02 -04:00
David Taylor
8b89787426
SECURITY: Sanitize YouTube Onebox data (#13748)
CVE-2021-32764
2021-07-15 19:31:50 +01:00
Penar Musaraj
a23153fdca
FIX: Add order to outputted stylesheet link tags (#13735)
See PR for details. (Disabled by default in this commit.)
2021-07-15 12:51:46 -04:00
Vinoth Kannan
74b3730143
DEV: return populated data at the end of the method. (#13739)
And some minor refactoring.
2021-07-15 17:45:32 +05:30
David Taylor
4ce58fbf0b
DEV: Improve rake release_note:generate date handling (#13726)
* DEV: Improve rake `release_note:generate` date handling

A commit-ish value like HEAD@{2021-01-01} is based on the **local state** of HEAD on that date. It does not use dates attached to commits.

Instead, the rake task now detects date-like strings and supplies them to `git log` via the `--after` and `--before` flags

* Skip printing plugin when there are no changes found

A list of skipped plugins is printed on a single line at the end of the output
2021-07-15 16:15:57 +10:00
Vinoth Kannan
76fe3a16a9
DEV: trigger new discourse events after_create_dev_record & after_populate_dev_records. (#13733)
After every new random record created using the `dev:populate` rake task a new Discourse event will be triggered. So the plugins can modify the records if needed.
2021-07-15 11:11:41 +05:30
Penar Musaraj
f7ab852e12
FIX: Issues with custom icons in themes (#13732)
Fixes two issues:
- ignores invalid XML in custom icon sprite SVG file (and outputs an error if sprite was uploaded via admin UI)
- clears SVG sprite cache when deleting an `icons-sprite` upload in a theme
2021-07-14 15:18:29 -04:00
Jarek Radosz
c750bfb4af
FIX: A memoization bug in UserLookup and refactor (#13692)
`@group_lookup` memoization bug was introduced in #13587.
2021-07-14 13:51:33 +02:00
Gerhard Schlager
2a95f892af
DEV: Update i18n:check rake task to detect invalid Markdown links (#13728)
In addition to that it fixes a problem where the check failed on empty locale files and allows calling the rake task with multiple locales.
2021-07-14 13:26:12 +02:00
Kim Lindberger
7e52eada20
FIX: Use Terser for minification even if uglify-js is not available (#13683) 2021-07-14 14:20:57 +08:00
jbrw
a64aea38b7
FIX: Don’t use user_generated images as avatar images in Oneboxed Twitter content (#13712)
By default, Twitter will return the URL for the avatar image of the tweet poster as the `og:image` value.

However, if the `user_generated` attribute is true, we should not use this as the avatar URL as this will be an URL of an image in the tweet itself (e.g., an image belonging to a tweeted news story).
2021-07-13 14:54:28 -04:00
Guhyoun Nam
0dc96ce817
FEATURE: Setting to allow moderators to change post ownership (#13708) 2021-07-13 09:40:11 -05:00
Dan Ungureanu
49090c3524
FIX: Suggest current username for staged users (#13706)
If user had a staged account and logged in using a third party service
a different username was suggested. This change will try to use the
username given by the authentication provider first, then the current
staged username and last suggest a new one.
2021-07-13 09:15:06 +10:00
David Taylor
76f279dfb4
DEV: Update release notes output format, and add plugin task (#13709) 2021-07-12 21:23:50 +01:00
Bianca Nenciu
f74640ac58
FIX: Show correct dates in topic timelines (#13670)
The date shown in topic timeline was one day later if the post at that
position was made near midnight. This happened because the days number
was rounded down.
2021-07-12 19:35:24 +03:00
Kane York
81d1a61d17
DEV: Report message-id, skip reason in rake email:test (#13674)
meta/t/185109

Additionally, print an extra warning if the `disable_emails` site setting is set.
2021-07-09 11:14:18 -07:00
Bianca Nenciu
87c1e98571
FEATURE: Let users select flair (#13587)
User flair was given by user's primary group. This PR separates the
two, adds a new field to the user model for flair group ID and users
can select their flair from user preferences now.
2021-07-08 10:46:21 +03:00
Blake Erickson
448a564845
DEV: Remove super old code comment TODO (#13663)
This TODO comment has existed for 8 years. Sort must be working just
fine or we would have prioritized fixing it.

Removing this comment as a tiny step toward keeping our codebase nice
and tidy.
2021-07-07 14:34:05 -06:00
Arpit Jalan
dec7e19da3
FIX: fix error message for fix_missing_s3 rake task (#13661) 2021-07-07 19:59:03 +05:30
Penar Musaraj
35110f6681
FIX: Set CSP base-uri to self (#13654) 2021-07-07 09:43:48 -04:00
Arpit Jalan
236d6d91b2
FIX: fix_missing_s3 task fails on failed upload (take 2) (#13660)
ref: 935aadbfdd
2021-07-07 18:53:43 +05:30
Arpit Jalan
935aadbfdd
FIX: do not stop fix_missing_s3 task if saving an upload failed (#13658)
This commit logs an error and moves to next upload when saving a single
upload record fails when running `uploads:fix_missing_s3` task.
2021-07-07 16:57:24 +05:30
Martin Brennan
35f6441938
DEV: Add uppy.js to build and project (#13645)
This PR adds uppy to the project with a custom JS build and the shims needed to import it into our JS code. We need a custom build of Uppy because we do not use webpack for our JS modules/build. The only way to get what you want from Uppy is to use the webpack modules or to include the entire Uppy project including all plugins in a single JS file. This way we can just use the plugins we actually want. Future PRs will actually use Uppy!
2021-07-07 10:39:33 +10:00
dependabot[bot]
dda41cf253
Build(deps): Bump rubocop from 1.18.2 to 1.18.3 (#13653)
* Build(deps): Bump rubocop from 1.18.2 to 1.18.3

Bumps [rubocop](https://github.com/rubocop/rubocop) from 1.18.2 to 1.18.3.
- [Release notes](https://github.com/rubocop/rubocop/releases)
- [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rubocop/rubocop/compare/v1.18.2...v1.18.3)

---
updated-dependencies:
- dependency-name: rubocop
  dependency-type: indirect
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix offences

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2021-07-07 01:51:43 +02:00
Penar Musaraj
95b5794331
DEV: Compile core and plugin stylesheets independently of themes (#13638)
Take 2 of https://github.com/discourse/discourse/pull/13466. 

Fixes a few issues with the original PR: 

- color definition stylesheet target now includes the theme id, to avoid themes set to use the default color scheme loading the same stylesheet 
- changes the internal cache key for color definition stylesheet to reset the pre-existing cache
2021-07-06 13:11:10 -04:00
David Taylor
8c370c3fe3 DEV: Add annotate rake tasks, and enforce via GitHub actions
`bin/rake annotate` is an alias of `bin/annotate --models`
`bin/rake annotate:clean` generates annotations by using a temporary, freshly migrated database. This should help us to produce more consistent annotations, even if development databases have been polluted by plugin migrations.

A GitHub actions task is also added which generates annotations on a clean database, and raises an error if they differ from the committed annotations.
2021-07-06 10:11:06 +01:00
David Taylor
2955d64703
DEV: Allow annotations to work in symlinked plugins, add binstub (#13635) 2021-07-05 15:43:10 +01:00
Alan Guo Xiang Tan
37b8ce79c9
FEATURE: Add last visit indication to topic view page. (#13471)
This PR also removes grey old unread bubble from the topic badges by
dropping `TopicUser#highest_seen_post_number`.
2021-07-05 14:17:31 +08:00
Penar Musaraj
02999f5eb2
Skip compressing theme test files (#13619) 2021-07-02 12:16:16 -04:00
Roman Rizzi
6be956339c
DEV: Tests can reset a single registry (#13618)
Plugins can use this method to reset their specific registries without touching the ones defined in core.
2021-07-02 12:40:50 -03:00
Dan Ungureanu
6ea4bbd2ec
DEV: Prefer .pluck_first over .pluck.first (#13607) 2021-07-02 10:03:54 +08:00
Arpit Jalan
05bdbd9f97
SECURITY: Onebox canonical links bypassing FinalDestination checks (#13605) 2021-07-01 20:09:29 +05:30
Alan Guo Xiang Tan
3312e09d02
DEV: Incorrect name in precompile output. (#13602)
Follow-up to c54d58e28f
2021-07-01 10:52:21 +08:00
Alan Guo Xiang Tan
c54d58e28f
FIX: Child themes being precompiled multiple times take 2. (#13599)
This fix was reverted in 128fdf9d9c but
fix is still relevant.
2021-07-01 10:09:39 +08:00
Martin Brennan
59582102d8
FIX: Rename ninja edit terminology to grace period (#13598)
We renamed the site setting for this long ago, but there
were a few places left in the code base where "ninja edit"
needed to be turned into "grace period". Doing this here
to avoid combatative language.
2021-07-01 11:27:11 +10:00
Penar Musaraj
128fdf9d9c
DEV: Revert stylesheet refactors (#13584)
* Revert "FIX: Clear appropriate cache when updating font settings (#13582)"

This reverts commit de6cc7a924.

* Revert "DEV: Improve output of `Stylesheet::Mananger.precompile_theme_css`."

This reverts commit 95038856c9.

* Revert "FIX: Child themes being precompiled multiple times."

This reverts commit 6986b36985.

* Revert "Update spec/components/stylesheet/manager_spec.rb"

This reverts commit ddaa7cc7ea.

* Revert "Refactor scss live refreshing"

This reverts commit a838293aaf.

* Revert "Precompile core stylesheets independently of themes"

This reverts commit 99d259d39b.

* Revert "DEV: Add simple digest for core stylesheets"

This reverts commit d82c58e6cc.
2021-06-30 21:33:15 +08:00
Bianca Nenciu
715ecabd85
FIX: TL4 users cannot delete others posts (#13554) 2021-06-30 15:51:35 +03:00
Alan Guo Xiang Tan
95038856c9 DEV: Improve output of Stylesheet::Mananger.precompile_theme_css. 2021-06-30 16:58:12 +08:00
Arpit Jalan
b63c9febe8
FIX: ignore canonical link to localhost (#13577) 2021-06-30 13:55:17 +05:30
Alan Guo Xiang Tan
6986b36985 FIX: Child themes being precompiled multiple times. 2021-06-30 16:09:45 +08:00
Penar Musaraj
a838293aaf Refactor scss live refreshing 2021-06-30 13:42:40 +08:00
Penar Musaraj
99d259d39b Precompile core stylesheets independently of themes 2021-06-30 13:42:40 +08:00
Penar Musaraj
d82c58e6cc DEV: Add simple digest for core stylesheets
And move fonts + category_backgrounds to color definitions stylesheet.
This will let us use the same core + plugin stylesheets in multisite.
2021-06-30 13:42:40 +08:00
mintsaxon
7d0d13c32e FEATURE: add slow_mode_prevents_editing setting..
..as per https://meta.discourse.org/t/slow-mode-for-a-category/179574/16
2021-06-28 13:08:10 -04:00
Jarek Radosz
04baca593b
UX: Tweak the timestamp line in Twitter onebox (#13551)
Fixed alignment and made the color less intrusive to make the actual content pop out more.
2021-06-28 15:04:33 +02:00
Andrei Prigorshnev
6be4699954
FIX: topic level bookmark button (#13530)
We changed (https://github.com/discourse/discourse/pull/13407) behaviour of the topic level bookmark button recently. That PR made the button be opening the edit bookmark modal when there is only one bookmark on the topic instead of just removing that bookmark as it was before.

This PR fixes the next problems that weren't taken into account in the previous PR:

1. Everything should work fine even on very big topics when a bookmarked post is unloaded from the post stream. I've added code that loads the post we need and makes everything work as expected
2. When at least one bookmark on the topic has a reminder, we should always be showing the icon with a clock on the topic level bookmark button
3. We should show correct tooltips for the topic level bookmark button
2021-06-28 12:24:23 +04:00
Alan Guo Xiang Tan
7719453fb7 DEV: Don't eager load tags when tagging is not enabled. 2021-06-28 16:00:10 +08:00
Alan Guo Xiang Tan
ff1c53dd6f FIX: Missing category edit icon.
Follow-up to 0e4b8c5318
2021-06-28 10:54:23 +08:00
Martin Brennan
d3e27cabf6
FIX: Improve participant display in group SMTP emails (#13539)
This PR makes several changes to the group SMTP email contents to make it look more like a support inbox message.

* Remove the context posts, they only add clutter to the email and replies
* Display email addresses of staged users instead of odd generated usernames
* Add a "please reply above this line" message to sent emails
2021-06-28 10:42:06 +10:00
Martin Brennan
87684f7c5e
FEATURE: Use group SMTP job and mailer instead of UserNotifications change (#13489)
This PR backtracks a fair bit on this one https://github.com/discourse/discourse/pull/13220/files.

Instead of sending the group SMTP email for each user via `UserNotifications`, we are changing to send only one email with the existing `Jobs::GroupSmtpEmail` job and `GroupSmtpMailer`. We are changing this job and mailer along with `PostAlerter` to make the first topic allowed user the `to_address` for the email and any other `topic_allowed_users` to be the CC address on the email. This is to cut down on emails sent via SMTP, which is subject to daily limits from providers such as Gmail. We log these details in the `EmailLog` table now.

In addition to this, we have changed `PostAlerter` to no longer rely on incoming email email addresses for sending the `GroupSmtpEmail` job. This was unreliable as a user's email could have changed in the meantime. Also it was a little overcomplicated to use the incoming email records -- it is far simpler to reason about to just use topic allowed users.

This also adds a fix to include cc_addresses in the EmailLog.addressed_to_user scope.
2021-06-28 08:55:13 +10:00
Jarek Radosz
203d56719d
UX: Improve blockquote styling in emails (#13527) 2021-06-25 20:13:46 +02:00
Roman Rizzi
fa4e5e8dad
FEATURE: Render emojis on GitHub labels when oneboxing an issue. (#13531) 2021-06-25 14:48:36 -03:00
Jarek Radosz
61472d6aaa
DEV: Rename hilight to highlight (#13526) 2021-06-25 18:05:50 +02:00
Martin Brennan
c3394ed9bb
DEV: Update aws-sdk-s3 gem for S3 multipart uploads (#13523)
We are a few versions behind on this gem. We need to update it
for S3 multipart uploads. In the current version we are using, we
cannot do this:

```ruby
Discourse.store.s3_helper.object(key).presigned_url(:upload_part, part_number: 1, upload_id: multipart_upload_id)
```

The S3 client raises an error, saying the operation is undefined. Once
I updated the gem this operation works as expected and returns a
presigned URL for the upload_part operation.

Also remove use of Aws::S3::FileUploader::FIFTEEN_MEGABYTES.
This was part of a private API and should not have been used.
2021-06-25 14:22:31 +10:00
Arpit Jalan
0adeddde61
FIX: follow redirects for inline/mini onebox (#13512) 2021-06-24 19:53:39 +05:30
Alan Guo Xiang Tan
0e4b8c5318 PERF: Cache categories in Site model take 3.
Previous attempt resulted in custom fields going missing in the
serialized output.

This reverts commit 83a6ad32ff.
2021-06-24 13:30:51 +08:00
Jarek Radosz
046a875222
DEV: Improve script/downsize_uploads.rb (#13508)
* Only shrink images that are used in Posts and no other models
* Don't save the upload if the size is the same
2021-06-24 00:09:40 +02:00
David Taylor
d2c5165052 FIX: Check all migrations for dropped columns/tables during restore
Previously only post-deploy migrations were being checked for DROPPED_(COLUMNS|TABLES) constants
2021-06-23 17:43:38 +01:00
Rafael dos Santos Silva
fa4a462517
FEATURE: Optimize images before upload (#13432)
Integrates [mozJPEG](https://github.com/mozilla/mozjpeg) and [Resize](https://github.com/PistonDevelopers/resize) using WebAssembly to optimize user uploads in the composer on the client-side.

NPM libraries are sourced from our [Squoosh fork](https://github.com/discourse/squoosh/tree/discourse), which was needed because we have an older asset pipeline.
2021-06-23 12:31:12 -03:00
Penar Musaraj
cfc60f41f0
DEV: Rename emoji icon (#13499) 2021-06-23 11:12:48 -04:00
Osama Sayegh
fa62b5e83b
DEV: Add a way to exclude ENV vars from getting unset in themes:isolated_test (#13494) 2021-06-23 14:50:54 +03:00
Osama Sayegh
d3a3d1b94c
DEV: Introduce TemporaryRedis and unset DISCOURSE_* env vars in the themes:isolated_test rake task (#13401)
The `themes:isolated_test` rake task will now unset all `DISCOURSE_*` env variables if `UNSET_DISCOURSE_ENV_VARS` env var is set and will also spin up a temporary redis server so the unicorn web server that's spun up for the tests doesn't leak into the "main" redis server.
2021-06-23 07:38:43 +03:00
Jarek Radosz
425ead8f35 DEV: Remove a perf fix that's no longer needed
It has been added to Rails in cc2d614e63 (v6.0.0.beta2)
2021-06-23 11:21:12 +08:00
Penar Musaraj
fc0da499f8
DEV: Refactor custom svg icon caching (#13483)
Previously, we were storing custom svg sprite paths in the cache. This is a problem because sprites in themes get stored as uploads, and the returned paths were files in the temporary download cache which could sometimes be cleaned up, resulting in a broken cache.

I previously tried to fix this by skipping the missing files and clearing the cache, but that didn't work out well with CDNs. This PR stores the contents of the files in the custom_svg_sprites cache to avoid the problem of missing temp files.

Also, plugin custom icons are only included if the plugin is enabled.
2021-06-22 14:07:23 -04:00
Mark VanLandingham
7fc3d7bdde
DEV: Plugin API to add directory columns (#13440) 2021-06-22 13:00:04 -05:00
Roman Rizzi
e0e1e24c14
FIX: Ignore posts needing approval when calculating reviewable counts. (#13464)
In #12841, we started setting the ReviewableQueuedPost's target and topic after approving it instead of storing them in the payload. As a result, the reviewable_counts query started to include queued posts.

When a category is set to require approval, every post has an associated reviewable. Pointing that each post has an associated queued post is not necessary in this case, so I added a WHERE clause to skip them.
2021-06-22 12:12:39 -03:00
David Taylor
e76c583b91
DEV: Promote old post-deploy migrations to pre-deploy migrations (#13477)
Having a large number of post-deploy migrations running out-of-numerical-sequence with pre-deploy migrations can be problematic. For example, if we have the sequence

- db/migrate/2017... - add column
- db/post_migrate/2018... - drop the column
- db/migrate/2021... - add the same column again

It will work fine in numerical order. But if you run the pre-deploy migrations **followed by** the post-deploy migrations, you will not get the same result.

Our post-deploy system is designed to allow for seamless upgrades of Discourse. However, it is reasonable for us to only support this totally seamless experience for a limited period of time. This commit moves all post_deploy migrations which are more than 1 year old (i.e. more than 2 major Discourse versions ago) into the regular pre-deploy migrations directory. This limits the impact of any edge cases caused by out-of-numerical-sequence migrations.
2021-06-22 16:02:24 +01:00
Gerhard Schlager
820068ddaf
FIX: fix_missing_s3 rake task could fail due to missing upload (#13479) 2021-06-22 17:00:55 +02:00
Penar Musaraj
f343cfd92e
DEV: Remove IntersectionObserver polyfill (#13445) 2021-06-22 09:30:44 -04:00
Jarek Radosz
cd8a82a82f
FIX: ShrinkUploadedImage was broken since cb12a721 (#13474) 2021-06-22 14:34:38 +02:00
Alan Guo Xiang Tan
83012b6473 FIX: Previewing theme components broken when component has no SCSS. 2021-06-22 09:48:48 +08:00
Martin Brennan
5222247746
FEATURE: Add more columns to outbound EmailLog (#13449)
This adds the following columns to EmailLog:

* cc_addresses
* cc_user_ids
* topic_id
* raw

This is to bring the EmailLog table closer in parity to
IncomingEmail so it can be better utilized for Group SMTP
and IMAP mailing.

The raw column contains the full content of the outbound email,
but _only_ if the new hidden site setting
enable_raw_outbound_email_logging is enabled. Most sites do not
need it, and it's mostly required for IMAP and SMTP sending.

In the next pull request, there will be a migration to backfill
topic_id on the EmailLog table, at which point we can remove the
topic fallback method on EmailLog.
2021-06-22 08:32:01 +10:00
Kane York
83a6ad32ff Revert "PERF: Cache categories in Site model take 2."
This reverts commit 06fa1efd3d.

Breakage in solved plugin
2021-06-21 12:25:04 -07:00
Bianca Nenciu
e70e8d8d8f
FIX: TL4 users can see their deleted posts (#13364) 2021-06-21 19:10:02 +03:00
Joffrey JAFFEUX
e50b7e9111
SECURITY: ensures timeouts are correctly used on connect (#13455) 2021-06-21 17:34:01 +02:00
Penar Musaraj
515fd8a4c3
DEV: Change method used to reference custom font assets (#13446)
The `asset-url` helper has been deprecated.
2021-06-21 09:33:12 -04:00
Dan Ungureanu
c69b58e22e
FIX: Downsize image last when saving uploads (#13458)
The other processing operations, such as fixing orientation or cropping,
can in rare cases increase the size of the uploaded image. Running the
downsize step after all these operations should create the best image
possible.
2021-06-21 14:13:24 +03:00
Alan Guo Xiang Tan
e64f9c87fb DEV: Fix broken stylesheet watcher.
Follow-up to 8e3691d537
2021-06-21 16:43:07 +08:00
Alan Guo Xiang Tan
7901f73597 FIX: Multiple links being generated for non-theme targets.
Follow-up to 8e3691d537
2021-06-21 16:26:49 +08:00
Alan Guo Xiang Tan
be655762d8 FIX: Color scheme stylesheet not loaded when no default theme is set.
Follow-up to 8e3691d537
2021-06-21 12:50:45 +08:00
Martin Brennan
7b31d8a11b
DEV: Move chrome binary check into a shared lib (#13451)
We had checks for the chrome binary in 3 different places
for tests and only one of them checked for google-chrome-stable,
which is problematic for Arch linux users (there are dozens of us!)

This PR moves all the code to one place and references it instead
of copying and pasting.
2021-06-21 13:28:48 +10:00
Alan Guo Xiang Tan
44aa46ca05 Code review comments. 2021-06-21 11:06:58 +08:00
Alan Guo Xiang Tan
8e3691d537 PERF: Eager load Theme associations in Stylesheet Manager.
Before this change, calling `StyleSheet::Manager.stylesheet_details`
for the first time resulted in multiple queries to the database. This is
because the code was modelled in a way where each `Theme` was loaded
from the database one at a time.

This PR restructures the code such that it allows us to load all the
theme records in a single query. It also allows us to eager load the
required associations upfront. In order to achieve this, I removed the
support of loading multiple themes per request. It was initially added
to support user selectable theme components but the feature was never
completed and abandoned because it wasn't a feature that we thought was
worth building.
2021-06-21 11:06:58 +08:00
Alan Guo Xiang Tan
06fa1efd3d PERF: Cache categories in Site model take 2.
Follow-up to aa4f0aee67.

Fixed the security problem in the previous attempt.
2021-06-21 09:47:05 +08:00
Martin Brennan
22b96c9ce1
FIX: Prevent resurrecting old topics via email reply for group inboxes with SMTP enabled (#13382)
We already reject email replies to public topics via `SiteSetting.disallow_reply_by_email_after_days` and raising the `OldDestinationError`. This PR introduces similar behaviour for group inboxes, but without the rejection, and **only when SMTP is enabled for the group**.

If a reply is sent via email and the post is older than `SiteSetting.disallow_reply_by_email_after_days` days ago, then we create a new topic instead of making a reply in the old one and link back to the original topic. This is done to prevent long running group inbox discussions.
2021-06-21 11:45:00 +10:00
Bianca Nenciu
1e992d9193
FIX: Do not check for duplicate links in Onebox (#13345)
If a user posted a URL that appeared inside a Onebox, then the user
got a duplicate link notice. This was fixed by skipping those links in
Ruby.

If a user posted a URL that was Oneboxes and contained other links that
appeared in previous posts, then the user got a duplicate link notice.
This was fixed by skipping those links in JavaScript.
2021-06-18 18:55:24 +03:00
Joffrey JAFFEUX
0fd55acf84 Update lib/site_settings/type_supervisor.rb
Co-authored-by: David Taylor <david@taylorhq.com>
2021-06-18 16:36:23 +08:00
jjaffeux
dc63613c96 DEV: improve error message for invalid setting’s value
Before this fix we would display this exception:

```
Discourse::InvalidParameters:
  value
```

After this fix we will display:

```
Discourse::InvalidParameters:
  Invalid `x` value for `s3_region`
```
2021-06-18 16:36:23 +08:00
Martin Brennan
ff6114d83f
FIX: Do not add mailing list headers to group SMTP emails (#13431)
When we are emailing people from a group inbox, we are having
a PM conversation with them, as a support account would. In this
case mailing list headers do not make sense. It is not like a forum
topic where you may have tens or hundreds of participants -- it is a
conversation between the group and a small handful of people
directly contacting the group, often just one person.

The only header left in tact was List-Unsubsribe which is important
for letting people opt out to notifications.
2021-06-18 14:36:17 +10:00
jbrw
fbfd1fd80b
FIX: Allow SVG uploads if dimensions are a fraction of a unit (#13409)
* FIX: Allow SVG uploads if dimensions are a fraction of a unit

`UploadCreator` counts the number of pixels in an file to determine if it is valid. `pixels` is calculated by multiplying the width and height of the image, as determined by FastImage.

SVG files can have their width/height expressed in a variety of different units of measurement. For example, ‘px’, ‘in’, ‘cm’, ‘mm’, ‘pt’, ‘pc’, etc are all valid within SVG files. If an image has a width of `0.5in`, FastImage may interpret this as being a width of `0`, meaning it will report the `size` as being `0`.

However, we don’t need to concern ourselves with the number of ‘pixels’ in a SVG files, as that is irrelevant for this file format, so we can skip over the check for `pixels == 0` when processing this file type.

* DEV: Speed up getting SVG dimensions

The `-ping` flag prevents the entire image from being rasterized before a result is returned. See:

https://imagemagick.org/script/command-line-options.php#ping
2021-06-17 15:56:11 -04:00
Mark VanLandingham
95b51669ad
DEV: Revert 3 commits for plugin API to add directory columns (#13423) 2021-06-17 12:37:37 -05:00
Osama Sayegh
c47f55253f
DEV: Add optional theme test step to the smoke:test rake task (#13418)
The purpose of this is to allow us to catch regressions for a feature we've built recently that allows theme tests to run in production. We recently had a regression that we didn't notice for days, so to prevent that from happening again we'll use this in our internal CI pipelines.
2021-06-17 20:09:29 +03:00
Mark VanLandingham
0c42a29dc4
DEV: Plugin API to allow creation of directory columns with item query (#13402)
The first thing we needed here was an enum rather than a boolean to determine how a directory_column was created. Now we have `automatic`, `user_field` and `plugin` directory columns.

This plugin API is assuming that the plugin has added a migration to a column to the `directory_items` table.

This was created to be initially used by discourse-solved. PR with API usage - https://github.com/discourse/discourse-solved/pull/137/
2021-06-17 09:06:18 -05:00
Alan Guo Xiang Tan
aa4f0aee67 Revert "PERF: Cache categories in Site model."
This reverts commit 7dc0f88acd.
2021-06-17 15:20:35 +08:00
Alan Guo Xiang Tan
7dc0f88acd PERF: Cache categories in Site model.
Profiling showed that we were roughly 10% of a request time creating all
the ActiveRecord objects for categories in the `Site` model on a site with 61 categories.
Instead of querying for the categories each time based on which categories the user can see,
we can just preload all of the categories upfront and filter out the
categories that the user can not see.
2021-06-17 13:17:42 +08:00
Martin Brennan
6fe78cd542
FIX: Make sure reset-new for tracked is not limited by per_page count (#13395)
When dismissing new topics for the Tracked filter, the dismiss was
limited to 30 topics which is the default per page count for TopicQuery.
This happened even if you specified which topic IDs you were
selectively dismissing. This PR fixes that bug, and also moves
the per_page_count into a DEFAULT_PER_PAGE_COUNT for the TopicQuery
so it can be stubbed in tests.

Also moves the unused stub_const method into the spec helpers
for cases like this; it is much better to handle this in one place
with an ensure. In a follow up PR I will clean up other specs that
do the same thing and make them use stub_const.
2021-06-17 08:20:09 +10:00
Brandon Gastelo
20dbcbf022
FIX: Sort filelists to ensure consistant asset precompilation hash (#13393)
Dir.glob does not guarantee file order and can change when ran on different machines.
This means that running asset precompilation on the exact same codebase will output
different content hashes.
2021-06-16 11:04:21 -04:00
David Taylor
b0416cb1c1
FEATURE: Upload to s3 in parallel to speed up backup restores (#13391)
Uploading lots of small files can be made significantly faster by parallelizing the `s3.put_object` calls. In testing, an UPLOAD_CONCURRENCY of 10 made a large restore 10x faster. An UPLOAD_CONCURRENCY of 20 made the same restore 18x faster.

This commit is careful to parallelize as little as possible, to reduce the chance of concurrency issues. In the worker threads, no database transactions are performed. All modification of shared objects is controlled with a mutex.

Unfortunately we do not have any existing tests for the `ToS3Migration` class. This change has been tested with a large site backup (120k uploads totalling 45GB)
2021-06-16 10:34:39 +01:00
Martin Brennan
c659e3e95b
FIX: Make sure topic_user.bookmarked is synced in more places (#13383)
When we call Bookmark.cleanup! we want to make sure that
topic_user.bookmarked is updated for topics linked to the
bookmarks that were deleted. Also when PostDestroyer calls
destroy and recover. We have a job for this already --
SyncTopicUserBookmarked -- so we just utilize that.
2021-06-16 08:30:40 +10:00
Osama Sayegh
503017474c
DEV: Skip CSS watcher when running QUnit tests and expose more Chrome logs (#13390)
There are 2 changes in this PR:

1) Add a new environment variable called `DISCOURSE_SKIP_CSS_WATCHER` to disable our stylesheet watcher, and make the `qunit:test` rake task set this variable on the Unicorn/Rails server it spins up to disable our stylesheet watcher when running the tests because it doesn't really need it.

2) Print more Chrome logs (such as network/security errors) to the console.
2021-06-15 18:27:15 +03:00
Roman Rizzi
fa57316a4e
FIX: Validate upload is still valid after calling the "before_upload_creation" event (#13091)
Since we use the event to perform additional validations on the file, we should check if it added any errors to the upload before saving it. This change makes the UploadCreator more consistent since we no longer have to rely on exceptions.
2021-06-15 10:10:03 -03:00
Martin Brennan
7fca7fb7ff
DEV: Add SMTP group ID to EmailLog (#13381)
Adds a new `smtp_group_id` column to `EmailLog` which is filled in if the mail `from_address` matches a group's `email_username`. This is for easier debugging, so we know which emails have been sent via group SMTP.
2021-06-15 11:29:46 +10:00
Alan Guo Xiang Tan
5f8f139ac9 PERF: Remove extra PG query.
In `Theme.list_baked_fields`, common is always included as a target.
2021-06-15 08:51:00 +08:00
Kane York
c780ae9d25
FEATURE: Add a messages view for all official warnings of a user (#12659)
Moderators are allowed to see the warnings list, with an access warning.

https://meta.discourse.org/t/why-arent-warnings-easily-accessible-like-suspensions-are/164043
2021-06-14 14:01:17 -07:00
Jarek Radosz
091beaf4a2
DEV: Work around a Docker issue (#13368)
Fixes our backend spec suite in GitHub Actions CI. For more information about the Docker issue see: https://github.com/docker/for-linux/issues/1015

(It's possible that error could also happen in dev/production, though thankfully that hasn't happened yet afaik)
2021-06-14 17:38:36 +02:00
Vinoth Kannan
6abc45e57b
DEV: move discourse_dev gem to the core. (#13360)
And get avatar images from `discourse_dev_assets` gem.
2021-06-14 20:34:44 +05:30
Martin Brennan
e9dc88a7b6
FIX: Link up reply to post correctly when emailing group (#13339)
When replying to a user_private_message email originating from
a group PM that does _not_ have a reply key (e.g. when replying
directly to the group's SMTP address), we were mistakenly linking
the new post created from the reply to the OP and the user who
created the topic, based on the first IncomingEmail message ID in
the topic, rather than using the correct reply to user and post number
that the user actually replied to.

We now use the In-Reply-To header to look up the corresponding EmailLog
record when the user who replied was sent a user_private_message email,
and use the post from that as the reply_to_user/post.

This also removes superfluous filtering of incoming_email records. After
already filtering by message_id and then addressed_to_user (which only
returns incoming emails where the to, from, or cc address includes any
of the user's emails), we were filtering again but in the ruby code for
the exact same conditions. After removing this all existing tests still
pass.
2021-06-10 15:28:50 +10:00
Penar Musaraj
6f76479054
FEATURE: Add upgrade-insecure-requests to CSP when force_https is enabled (#13348)
If force_https is enabled all resource (including markdown preview and so on) will be accessed using HTTPS

If for any reason you attempt to link to non HTTPS reachable content content may appear broken
2021-06-10 10:53:10 +10:00
Alan Guo Xiang Tan
2a4a20ad67
PERF: Avoid running a pointless PG query when theme has no variables. (#13342)
When `Theme#all_theme_variables` returns an empty array, we were running
a pointless query in `StyleSheet::Manager#uploads_digest`.

`SELECT "sha1" FROM "theme_fields" INNER JOIN "uploads" ON
"uploads"."id" = "theme_fields"."upload_id" WHERE 1=0`
2021-06-10 10:50:17 +10:00
Penar Musaraj
a96f6fbdf5
FIX: Do not block SVG sprite bundle if a file is missing (#13338) 2021-06-09 08:29:00 -04:00
Neil Lalonde
c2c647b990
FIX: errors loading secure uploads when secure uploads is disabled (#13047) 2021-06-08 13:25:51 -04:00
David Taylor
9811a1c5d9
DEV: Allow transformed values to be used in all widget hbs statements (#13331)
Previously, the `transformed.blah` shortcut could only be used in top-level hbs statements like {{transformed.blah}}. When attempting to use it in a sub-expression like `{{concat "hello" transformed.world}}`, it would raise a "transformed is not defined" error.

This commit updates the shortcut logic to make `transformed.blah` and `attrs.blah` work consistently in all hbs expressions.

Co-authored-by: Jordan Vidrine <jordan@jordanvidrine.com>
2021-06-08 16:46:07 +01:00
Neil Lalonde
41038d6cdb
Version bump to v2.8.0.beta2 (#13329) 2021-06-08 11:27:50 -04:00
Penar Musaraj
8336e732d3
DEV: Add manifest-src to CSP (#13319)
Defaults to `manifest-src: 'self'` and allows plugins/themes to extend it.
2021-06-08 09:32:31 -04:00
Alan Guo Xiang Tan
2717615f11
PERF: Preload client settings json while booting app. (#13283)
Generating the client settings json involves santizing all string based
site settings. This is expensive as per our profile in production (~120ms) and one request after each deploy has
to pay this penalty.
2021-06-08 13:15:55 +10:00
Alan Guo Xiang Tan
19a92fbadc
PERF: Switch ActiveRecord PG connection active check to use empty query. (#13323)
See https://github.com/rails/rails/pull/42368

The impact is not quantifiable at the time of this writing but
prelimary investigation shows that `SELECT 1` accounts for 0.09 of CPU
time on a database. Note that Discourse runs thousands of databases so
the small impact may be amplified by the large number of databases that
we run.
2021-06-08 13:15:07 +10:00
Penar Musaraj
f90c4bd6a1
DEV: Allow plugins to extend frame-ancestors (#13316) 2021-06-07 14:59:15 -04:00
jbrw
09bc95d46b
FIX: Quoting Oneboxed content should exclude formatting (#13296)
* FIX: Quoting Oneboxed content should exclude formatting

When a post is quoted that includes Oneboxed content, we should not include the formatting generated by the Onebox. Rather, we should attempt to collapse the link referenced by the Onebox to a single line text link.

* DEV: fix tests
2021-06-07 13:03:53 -04:00
Dan Ungureanu
36e0e6a322
FIX: Allow invites if must_approve_users is true (#13257) 2021-06-07 18:57:08 +03:00
Arpit Jalan
2e4f07678e
FIX: IMDb links were being oneboxed as posters (#13310)
IMDb movie links were being rendered as posters. This was because
IMDb was sending `og:type` as `image` randomly in some cases. To
fix this we'll now default all IMDb links as article type. This will
ensure that the IMDb onebox link includes all the information instead
of showing just a poster without any context.
2021-06-07 18:45:59 +05:30
Alan Guo Xiang Tan
064bca430c PERF: Preload Post#image_upload in TopicView.
This fixes an N+1 queires problem when generating `Post#image_url` which
requires the upload to be loaded.

Follow-up to 141f16eb6b
2021-06-07 14:43:49 +08:00
Alan Guo Xiang Tan
cadf5eafe6 DEV: Move Discourse app specific concern out of unicorn conf. 2021-06-04 09:13:34 +08:00
Alan Guo Xiang Tan
a8667b5454 PERF: Defer setting of distributed cache in more spots.
See follow up commit for rational.

Follow-up to 8cfe203383
2021-06-04 09:13:18 +08:00
Kris
cd9941e0ca
UX: more consistent setting/edit buttons (#13276) 2021-06-03 18:33:36 -04:00
David Taylor
4134173bbf
FEATURE: Add global admin api key rate limiter (#12527) 2021-06-03 10:52:43 +01:00
Alan Guo Xiang Tan
58b30fb510 PERF: Preload settings, groups and badge icons in SvgSprite.
Identified as a hot path in production. Preload it early instead of
executing the queries in a live request.
2021-06-03 16:45:56 +08:00
Martin Brennan
eb2c399445
FEATURE: Use group SMTP settings for sending user notification emails (initial) (#13220)
This PR changes the `UserNotification` class to send outbound `user_private_message` using the group's SMTP settings, but only if:

* The first allowed_group on the topic has SMTP configured and enabled
* SiteSetting.enable_smtp is true
* The group does not have IMAP enabled, if this is enabled the `GroupSMTPMailer` handles things

The email is sent using the group's `email_username` as both the `from` and `reply-to` address, so when the user replies from their email it will go through the group's SMTP inbox, which needs to have email forwarding set up to send the message on to a location (such as a hosted site email address like meta@discoursemail.com) where it can be POSTed into discourse's handle_mail route.

Also includes a fix to `EmailReceiver#group_incoming_emails_regex` to include the `group.email_username` so the group does not get a staged user created and invited to the topic (which was a problem for IMAP), as well as updating `Group.find_by_email` to find using the `email_username` as well for inbound emails with that as the TO address.

#### Note

This is safe to merge without impacting anyone seriously. If people had SMTP enabled for a group they would have IMAP enabled too currently, and that is a very small amount of users because IMAP is an alpha product, and also because the UserNotification change has a guard to make sure it is not used if IMAP is enabled for the group. The existing IMAP tests work, and I tested this functionality by manually POSTing replies to the SMTP address into my local discourse.

There will probably be more work needed on this, but it needs to be tested further in a real hosted environment to continue.
2021-06-03 14:47:32 +10:00
Bianca Nenciu
d184fe59ca
FEATURE: Censor Oneboxes (#12902)
Previously onebox content was not passed by the censor regex, meaning you could sneak in censored words via onebox.
2021-06-03 11:39:12 +10:00
Alan Guo Xiang Tan
58cb120aa2
DEV: Minor code clean up in assets.rake. (#13245) 2021-06-03 11:37:06 +10:00
Alan Guo Xiang Tan
8cfe203383 PERF: Defer setting of distributed cache in performance critical paths.
Setting a key/value pair in DistributedCache involves waiting on the
write to Redis to finish. In most cases, we don't need to wait on the
setting of the cache to finish. We just need to take our return value
and move on.
2021-06-03 09:30:52 +08:00
Jarek Radosz
3bb765ac92
DEV: Remove the remaining Travis code (#13255)
The second attempt at #10041 now that all our plugins use GitHub Actions CI instead.
2021-06-02 20:29:47 +02:00
Penar Musaraj
fd9ef14ec0
FIX: Show required tags to staff by default and override limit (#13242)
This improves the display of available tags in categories that are
configured to require at least (x) tags from a tag group.

There are two changes included:
- regular users will now see all the available tags in the required tag
group (previously they could see a max. of 5 tags)
- staff users will now see the tags from the required tag group when
the tag group contains more tags than the default limit (also set to 5)

Both changes only apply to the default query (i.e. no search terms).
2021-06-02 12:43:34 -04:00
Penar Musaraj
9d6780f03d
DEV: Remove emoji_one files (#13236)
And add a symlink so requests to those assets fallback to the default
emoji set (Twitter).
2021-06-02 09:18:25 -04:00
Bianca Nenciu
d9484db718
FIX: Split link watched words from replace (#13196)
It was not clear that replace watched words can be used to replace text
with URLs. This introduces a new watched word type that makes it easier
to understand.
2021-06-02 15:36:49 +10:00
Alan Guo Xiang Tan
2e4182b4b3 PERF: Warm up caches in SiteSerializer while preloading app.
Otherwise, time will be spent warming up the caches in the initial
requests.
2021-06-02 13:33:31 +08:00
Alan Guo Xiang Tan
6a79864f14 PERF: Cache categories query in Stylesheet::Manager.color_scheme_digest.
The query is being executed each time we try and generate the link path
for a stylesheet within the duration of a reqeust. Categories are not
updated that often so repeating this query multiple times a request is
wasteful.

At the time of this commit, there is a `publish_discourse_stylesheet`
ActiveRecord callback on the `Category` model which clears the cache of
`Stylesheet::Manager` each time a category is saved.
2021-06-02 13:33:31 +08:00
Alan Guo Xiang Tan
4390b50b12 DEV: Fix incorrect optimization in SvgSprite take 2.
Follow-up to 0700e9382d
2021-06-02 11:17:11 +08:00
Alan Guo Xiang Tan
ca9e0ee40b DEV: Fix incorrect optimization in SvgSprite.
`SvgSprite.custom_sprite_paths` does not just fetch custom SVG sprite
from themes. It also picks up any custom svg icons from a pre-defined
plugin folder.

Follow-up to 0700e9382d
2021-06-02 10:42:23 +08:00
Alan Guo Xiang Tan
0700e9382d PERF: Memoize core svgs in memory to avoid expensive XML parsing.
The XML parsing of SVGs is done whenever the cache expires or on the
first load after a reboot. In one of our production instance, parsing
ranges from 30ms to 70ms which is not ideal. Instead, we've decided to
make a small memory trade off here by memoizing the core SVGs once on
boot to avoid parsing of the SVG files during the duration of a request.
The memozied hash will take up 57440 bytes or 0.05744 megabytes in size.
2021-06-02 09:16:37 +08:00
Martin Brennan
e15c86e8c5
DEV: Topic tracking state improvements (#13218)
I merged this PR in yesterday, finally thinking this was done https://github.com/discourse/discourse/pull/12958 but then a wild performance regression occurred. These are the problem methods:

1aa20bd681/app/serializers/topic_tracking_state_serializer.rb (L13-L21)

Turns out date comparison is super expensive on the backend _as well as_ the frontend.

The fix was to just move the `treat_as_new_topic_start_date` into the SQL query rather than using the slower `UserOption#treat_as_new_topic_start_date` method in ruby. After this change, 1% of the total time is spent with the `created_in_new_period` comparison instead of ~20%.

----

History:

Original PR which had to be reverted **https://github.com/discourse/discourse/pull/12555**. See the description there for what this PR is achieving, plus below.

The issue with the original PR is addressed in 92ef54f402

If you went to the `x unread` link for a tag Chrome would freeze up and possibly crash, or eventually unfreeze after nearly 10 mins. Other routes for unread/new were similarly slow. From profiling the issue was the `sync` function of `topic-tracking-state.js`, which calls down to `isNew` which in turn calls `moment`, a change I had made in the PR above. The time it takes locally with ~1400 topics in the tracking state is 2.3 seconds.

To solve this issue, I have moved these calculations for "created in new period" and "unread not too old" into the tracking state serializer.

When I was looking at the profiler I also noticed this issue which was just compounding the problem. Every time we modify topic tracking state we recalculate the sidebar tracking/everything/tag counts. However this calls `forEachTracked` and `countTags` which can be quite expensive as they go through the whole tracking state (and were also calling the removed moment functions).

I added some logs and this was being called 30 times when navigating to a new /unread route because  `sync` is being called from `build-topic-route` (one for each topic loaded due to pagination). So I just added a debounce here and it makes things even faster.

Finally, I changed topic tracking state to use a Map so our counts of the state keys is faster (Maps have .size whereas objects you have to do Object.keys(obj) which is O(n).)

<!-- NOTE: All pull requests should have tests (rspec in Ruby, qunit in JavaScript). If your code does not include test coverage, please include an explanation of why it was omitted. -->
2021-06-02 09:06:29 +10:00
jbrw
461a2c334b
FIX: return an empty result if response from Amazon is missing expected attributes (#13173)
* FIX: return an empty result if response from Amazon is missing attributes

Check we have the basic attributes requires to construct a Onebox for Amazon.

This is an attempt to handle scenarios where we receive a valid 200-status response from an Amazon request that does not include the data we’re expecting.

* Update lib/onebox/engine/amazon_onebox.rb

Co-authored-by: Régis Hanol <regis@hanol.fr>

Co-authored-by: Régis Hanol <regis@hanol.fr>
2021-06-01 16:23:18 -04:00
Gerhard Schlager
41ee5b7c86
FIX: Don't store translated trust level names in anonymous cache (#13224)
Refactors `TrustLevel` and moves translations from server to client

Additional changes:
  * "staff" and "admin" wasn't translatable in site settings
  * it replaces a concatenated string with a translation
  * uses translation for trust levels in users_by_trust_level report
  * adds a DB migration to rename keys of translation overrides affected by this commit
2021-06-01 22:11:48 +02:00
Penar Musaraj
4d4c3fe1e4
FIX: Delete internal links when moderator deletes a post (#13233) 2021-06-01 14:02:53 -04:00
Joffrey JAFFEUX
1cd0424ccd
FEATURE: lets users favorite 2 badges to show on user-card (#13151) 2021-06-01 10:33:40 +02:00
Gerhard Schlager
c1c090419b DEV: Fix warning about already initialized constants in FileStore::BaseStore 2021-06-01 09:30:37 +08:00
Dan Ungureanu
ccbe3bea79
UX: Improve style of GitHub PR body in emails (#13198)
On the web, we display only an excerpt in a monospace font and the rest
of the body is hidden under ellipsis. The email displayed both of them
and it did not use the same style. This commit leaves only the excerpt
in emails and makes it use a monospace font to display it.
2021-05-31 14:03:19 +03:00
Osama Sayegh
b81b24dea2
Revert "DEV: Topic tracking state improvements (#12958)" (#13209)
This reverts commit 002c676344.

Perf regression, we will redo it.
2021-05-31 17:47:42 +10:00
Jarek Radosz
bbcd70e8b5 DEV: Remove dead code
There's no such thing as `@in_spoiler` anymore.
2021-05-31 10:22:50 +08:00
Martin Brennan
002c676344
DEV: Topic tracking state improvements (#12958)
Original PR which had to be reverted **https://github.com/discourse/discourse/pull/12555**. See the description there for what this PR is achieving, plus below.

The issue with the original PR is addressed in 92ef54f402

If you went to the `x unread` link for a tag Chrome would freeze up and possibly crash, or eventually unfreeze after nearly 10 mins. Other routes for unread/new were similarly slow. From profiling the issue was the `sync` function of `topic-tracking-state.js`, which calls down to `isNew` which in turn calls `moment`, a change I had made in the PR above. The time it takes locally with ~1400 topics in the tracking state is 2.3 seconds.

To solve this issue, I have moved these calculations for "created in new period" and "unread not too old" into the tracking state serializer.

When I was looking at the profiler I also noticed this issue which was just compounding the problem. Every time we modify topic tracking state we recalculate the sidebar tracking/everything/tag counts. However this calls `forEachTracked` and `countTags` which can be quite expensive as they go through the whole tracking state (and were also calling the removed moment functions).

I added some logs and this was being called 30 times when navigating to a new /unread route because  `sync` is being called from `build-topic-route` (one for each topic loaded due to pagination). So I just added a debounce here and it makes things even faster.

Finally, I changed topic tracking state to use a Map so our counts of the state keys is faster (Maps have .size whereas objects you have to do Object.keys(obj) which is O(n).)
2021-05-31 09:22:28 +10:00
Penar Musaraj
06e1af2b1d
FIX: Giphy oneboxing when the response is an image (#13199) 2021-05-28 15:10:32 -04:00
Penar Musaraj
47e09700fe
FIX: Support pausing GIFs for giphy/tenor oneboxes (#13194) 2021-05-28 08:40:30 -04:00
Martin Brennan
501de809da
FIX: Do not mark badge image uploads as secure (#13193)
* FIX: Do not mark badge image uploads as secure

We do not need badge_image upload types to be marked as secure.
Post migration is the same as
https://github.com/discourse/discourse/pull/12081.

See
https://meta.discourse.org/t/secure-media-uploads/140017/122?u=martin
2021-05-28 12:35:52 +10:00
Martin Brennan
964da21817
FEATURE: Improve group email settings UI (#13083)
This overhauls the user interface for the group email settings management, aiming to make it a lot easier to test the settings entered and confirm they are correct before proceeding. We do this by forcing the user to test the settings before they can be saved to the database. It also includes some quality of life improvements around setting up IMAP and SMTP for our first supported provider, GMail. This PR does not remove the old group email config, that will come in a subsequent PR. This is related to https://meta.discourse.org/t/imap-support-for-group-inboxes/160588 so read that if you would like more backstory.

### UI

Both site settings of `enable_imap` and `enable_smtp` must be true to test this. You must enable SMTP first to enable IMAP.

You can prefill the SMTP settings with GMail configuration. To proceed with saving these settings you must test them, which is handled by the EmailSettingsValidator.

If there is an issue with the configuration or credentials a meaningful error message should be shown.

IMAP settings must also be validated when IMAP is enabled, before saving.

When saving IMAP, we fetch the mailboxes for that account and populate them. This mailbox must be selected and saved for IMAP to work (the feature acts as though it is disabled until the mailbox is selected and saved):

### Database & Backend

This adds several columns to the Groups table. The purpose of this change is to make it much more explicit that SMTP/IMAP is enabled for a group, rather than relying on settings not being null. Also included is an UPDATE query to backfill these columns. These columns are automatically filled when updating the group.

For GMail, we now filter the mailboxes returned. This is so users cannot use a mailbox like Sent or Trash for syncing, which would generally be disastrous.

There is a new group endpoint for testing email settings. This may be useful in the future for other places in our UI, at which point it can be extracted to a more generic endpoint or module to be included.
2021-05-28 09:28:18 +10:00
Robin Ward
2082abbb75
DEV: Add support for class properties in babel (#13189)
This allows us to start using JS classes instead of Ember's classes.
2021-05-27 16:13:14 -04:00
Penar Musaraj
726500bc59
FEATURE: Enable pausing images from Giphy and Tenor (#13185) 2021-05-27 15:00:38 -04:00
Bianca Nenciu
571ee4537a
FEATURE: Silence watched word (#13160)
This is a new type of watched word to replace auto_silence_first_post_
regex site setting.
2021-05-27 19:19:58 +03:00
Gerhard Schlager
157f10db4c
FEATURE: Use path from existing URL of uploads and optimized images (#13177)
Discourse shouldn't dynamically calculate the path of uploads and optimized images after a file has been stored on disk or S3. Otherwise it might calculate the wrong path if the SHA1 or extension stored in the database doesn't match the actual file path.
2021-05-27 17:42:25 +02:00
Penar Musaraj
855e854cb8
DEV: Enable optional chaining in all contexts (#13180)
* Revert "FIX: We can't use `?.` yet (#13168)"
2021-05-27 09:56:35 -04:00
Dan Ungureanu
723d7de18c
Various GitHub Onebox improvements (#13163)
* FIX: Improve GitHub folder regexp in Onebox

It used to match any GitHub URL that was not matched by the other GitHub
Oneboxes and it did not do a good job at handling those. With this
change, the generic Onebox will handle the remaining URLs.

* FEATURE: Add Onebox for GitHub Actions

* FEATURE: Add Onebox for PR check runs

* FIX: Remove image from GitHub folder Oneboxes

It is a generic, auto-generated image which does not provide any value.

* DEV: Add tests

* FIX: Strip HTML comments from PR body
2021-05-27 12:38:42 +03:00
Penar Musaraj
1270c7ad15
UX: Twitter onebox layout adjustments (#13181) 2021-05-27 15:35:32 +10:00
Arpit Jalan
283b08d45f
DEV: Absorb onebox gem into core (#12979)
* Move onebox gem in core library

* Update template file path

* Remove warning for onebox gem caching

* Remove onebox version file

* Remove onebox gem

* Add sanitize gem

* Require onebox library in lazy-yt plugin

* Remove onebox web specific code

This code was used in standalone onebox Sinatra application

* Merge Discourse specific AllowlistedGenericOnebox engine in core

* Fix onebox engine filenames to match class name casing

* Move onebox specs from gem into core

* DEV: Rename `response` helper to `onebox_response`

Fixes a naming collision.

* Require rails_helper

* Don't use `before/after(:all)`

* Whitespace

* Remove fakeweb

* Remove poor unit tests

* DEV: Re-add fakeweb, plugins are using it

* Move onebox helpers

* Stub Instagram API

* FIX: Follow additional redirect status codes (#476)

Don’t throw errors if we encounter 303, 307 or 308 HTTP status codes in responses

* Remove an empty file

* DEV: Update the license file

Using the copy from https://choosealicense.com/licenses/gpl-2.0/#

Hopefully this will enable GitHub to show the license UI?

* DEV: Update embedded copyrights

* DEV: Add Onebox copyright notice

* DEV: Add MIT license, convert COPYRIGHT.txt to md

* DEV: Remove an incorrect copyright claim

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Co-authored-by: jbrw <jamie@goatforce5.org>
2021-05-26 15:11:35 +05:30
Roman Rizzi
a5fddd5cdb
FIX: Hide a post's pending flag count from TL4 users. (#13129)
They can't see the review queue, so there's no reason to show a button with the post's pending flag count in the post controls.
2021-05-26 12:58:00 +10:00
Martin Brennan
7a79bd7da3
FEATURE: Allow selective dismissal of new and unread topics (#12976)
This PR improves the UI of bulk select so that its context is applied to the Dismiss Unread and Dismiss New buttons. Regular users (not just staff) are now able to use topic bulk selection on the /new and /unread routes to perform these dismiss actions more selectively.

For Dismiss Unread, there is a new count in the text of the button and in the modal when one or more topic is selected with the bulk select checkboxes.

For Dismiss New, there is a count in the button text, and we have added functionality to the server side to accept an array of topic ids to dismiss new for, instead of always having to dismiss all new, the same as the bulk dismiss unread functionality. To clean things up, the `DismissTopics` service has been rolled into the `TopicsBulkAction` service.

We now also show the top Dismiss/Dismiss New button based on whether the bottom one is in the viewport, not just based on the topic count.
2021-05-26 09:38:46 +10:00
Rafael dos Santos Silva
9118bb2076
FEATURE: Normalize the service worker route (#12343)
Re-lands the change initially proposed on #8359 but without a new nginx
location block, so it has less change surface.

Co-authored-by: Jeff Wong <awole20@gmail.com>

Co-authored-by: Jeff Wong <awole20@gmail.com>
2021-05-25 19:39:31 -03:00
Penar Musaraj
6e9ee3db03
FIX: Sort user bookmarks by reminder date (#13145) 2021-05-25 15:23:46 -04:00
Jeff Atwood
50926f6143
FIX: Simplify post and topic deletion language (#13128)
Based on feedback from Matt Haughey, we don't need to use so many words when describing a deleted topic or post.

Co-authored-by: Martin Brennan <martin@discourse.org>
2021-05-25 12:04:10 +10:00
Blake Erickson
fccf4d4375
DEV: Switch off of our image_optim fork (#13124)
The main image_optim gem now includes the timeout feature
that we had in our fork. So it is now safe to switch off of our fork and
back to the image_optim gem.

This is the link to the commit in the image_optim repo that adds the
timeout option:

ec3767dde0

One difference with the new timeout implementation is that image_optim
now handles the timeout exceptions instead of bubbling them up:

1ed0328587/lib/image_optim.rb (L128-L129)

```
 rescue Errors::TimeoutExceeded
   handler.result
```

So a timeout will just return `nil`, which is the same response if it
couldn't optimize an image. I don't think we were really watching for
or doing anything about these timeout warnings in our logs so I think
this is an okay change to have and we will have less warnings in our
logs now too.
2021-05-24 19:27:20 -06:00
Daniel Waterworth
ca809d2c40
FEATURE: Add on_preload for TopicView (#13122)
This allows plugins to load data on fetched posts before each post is
individually serialized.
2021-05-24 11:46:57 -05:00
David Taylor
93ae1b2b1c
DEV: Default to development RAILS_ENV when running theme tests (#13106)
When testing theme components in development, it doesn't make sense to use the `test` environment. The `test` environment almost certainly has 0 themes installed.

This change still works fine when using the `themes:install_and_test` rake task, because that rake task explicitly specifies environment/database-config.
2021-05-24 13:35:10 +01:00
Andrei Prigorshnev
c62efc0f0f
FIX: Ignore max excerpt length for div excerpts too (#13058)
We support two types of custom excerpts. It can be <div class="excerpt"> or <span class="excerpt">: b21f74060e/lib/excerpt_parser.rb (L120)

We also ignore max excerpt length for custom excerpts. But we forgot to process div when ignoring max length.
2021-05-24 13:05:24 +04:00
Gerhard Schlager
332ae97555
FEATURE: Correctly convert topic title to uppercase and lowercase for Turkish default locale (#13115) 2021-05-24 18:13:30 +10:00
Bianca Nenciu
7c29ff3d85
FEATURE: Add tasks to export and import site structure (#12584) 2021-05-24 12:38:42 +10:00
Martin Brennan
292017dd25
FIX: Do not call :post_edited webhook twice when editing OP (#13112)
When editing the first post for the topic we do two AJAX requests
to two separate controllers in this order:

PUT /t/topic-name
PUT /posts/2489523

This causes two post revisor calls, which end up triggering the
:post_edited DiscourseEvent twice. This is then picked up and sent
as a WebHook event twice. However we do not need to send a :post_edited
webhook event if the first post is being edited and topic_changed is
true from the :post_edited DiscourseEvent, because a second event will
shortly come through for just the post.

See https://meta.discourse.org/t/post-webhook-fires-two-times-on-post-edited-for-first-post-in-a-topic/162408

Continued on from https://github.com/discourse/discourse/pull/10590
2021-05-24 09:10:22 +10:00
Bianca Nenciu
38af28d58b
FIX: Allow add email to group if user can invite (#13097)
It used to allow adding email addresses to a group even if invites were
disabled for the site. This does not allow user to input email address
if they cannot invite.

The second thing this commit improves is the message that is displayed
to the user when they hit the invite rate limit.
2021-05-21 11:34:17 +03:00
Josh Soref
13d40ead97
DEV: Correct spelling mistakes in comments 2021-05-21 13:37:17 +10:00
Martin Brennan
afb2b4c6b9
FIX: Update upload secure status when revising posts (#13110)
When uploads are created from the composer (editing or creating a post),
for sites with secure uploads enabled we assume security by default and
that new upload is set to secure. When the post is created, we then
check whether the post uploads _actually_ need to be secure and adjust
accordingly.

We were not doing this when revising a post, so when a new upload was
created when editing a post in a public topic, the secure status stayed
true erroneously causing issues with image previews, among other things.
2021-05-21 13:32:32 +10:00
Josh Soref
59097b207f
DEV: Correct typos and spelling mistakes (#12812)
Over the years we accrued many spelling mistakes in the code base. 

This PR attempts to fix spelling mistakes and typos in all areas of the code that are extremely safe to change 

- comments
- test descriptions
- other low risk areas
2021-05-21 11:43:47 +10:00
David Taylor
bd8c2b4615
DEV: Correct theme:qunit help text (#13104)
- Task name is themes:qunit, not themes:unit
- Some shells try to expand the square brackets. The whole thing should be enclosed in quotes to avoid this
2021-05-20 22:30:12 +01:00
Penar Musaraj
29f3621f45
FIX: Disable lightboxing of animated images (#13099) 2021-05-20 15:19:44 -04:00
Dan Ungureanu
e2f9da9795
DEV: Remove reference to very old plugin (#13098) 2021-05-20 16:18:22 +03:00
Alan Guo Xiang Tan
958b56dc6c
PERF: Precompile child theme stylesheets. (#13040)
Previously, we only precompiled the CSS for parent themes but not for
the child themes. As a result, the CSS for child themes were being
compiled during the first request which made the respond time high for
that request.
2021-05-20 16:01:54 +10:00
Arpit Jalan
626b8465ba
FIX: do not validate uploads when running uploads:fix_missing_s3 task (#13096) 2021-05-20 15:29:09 +10:00
Arpit Jalan
130160537c
FEATURE: add support for "skip_validations" option in UploadCreator (#13094)
FIX: do not validate uploads when running `uploads:fix_missing_s3` task
2021-05-19 20:54:52 +05:30
Andrei Prigorshnev
3e0f72f57f
FEATURE: Increase daily edit limits proportionally to trust level (#13090) 2021-05-19 13:57:21 +04:00
Arpit Jalan
f96f534f3e
FIX: do not include contact url & email in client site settings payload (#13004) 2021-05-19 16:15:24 +10:00
Neil Lalonde
05c1d3c9cd
Version bump to v2.8.0.beta1 (#13092) 2021-05-18 16:38:17 -04:00
Bianca Nenciu
c1dfd76658
FIX: Make replace watched words work with wildcard (#13084)
Watched words are always regular expressions, despite watched_words_
_regular_expressions being enabled or not. Internally, wildcard
characters are replaced with a regular expression that matches any non
whitespace character.
2021-05-18 12:09:47 +03:00
jbrw
a24b6daa87
FIX: An unresolved blank uri should attempt an alternate Oneboxing strategy, if available (#13070) 2021-05-14 15:23:20 -04:00
Bianca Nenciu
3a1b05f219
FIX: Make autotag watched words case insensitive (#13043)
* FIX: Hide tag watched words if tagging is disabled

These 'autotag' words were shown even if tagging was disabled.

* FIX: Make autotag watched words case insensitive

This commit also fixes the bug when no tag was applied if no other tag
was already present.
2021-05-14 16:52:10 +03:00
jbrw
19182b1386
DEV: Oneboxer wildcard subdomains (#13015)
* DEV: Allow wildcards in Oneboxer optional domain Site Settings

Allows a wildcard to be used as a subdomain on Oneboxer-related SiteSettings, e.g.:

- `force_get_hosts`
- `cache_onebox_response_body_domains`
- `force_custom_user_agent_hosts`

* DEV: fix typos

* FIX: Try doing a GET after receiving a 500 error from a HEAD

By default we try to do a `HEAD` requests. If this results in a 500 error response, we should try to do a `GET`

* DEV: `force_get_hosts` should be a hidden setting

* DEV: Oneboxer Strategies

Have an alternative oneboxing ‘strategy’ (i.e., set of options) to use when an attempt to generate a Onebox fails. Keep track of any non-default strategies that were used on a particular host, and use that strategy for that host in the future.

Initially, the alternate strategy (`force_get_and_ua`) forces the FinalDestination step of Oneboxing to do a `GET` rather than `HEAD`, and forces a custom user agent.

* DEV: change stubbed return code

The stubbed status code needs to be a value not recognized by FinalDestination
2021-05-13 15:48:35 -04:00
Alan Guo Xiang Tan
f05f5fde0f
PERF: Preload JsLocaleHelper.load_translations for default locale. (#13041)
In production, each Unicorn child process will currently hold the
default locale in memory on first load. Instead, we should preload it in
the Unicorn master process so that the memory is immediately shared when
forking.

Also, the translations are only memoized on first load now and is
adding considerable overhead to the first few requests after a fresh
boot.
2021-05-13 16:16:01 +10:00
Martin Brennan
3d2cace94f
DEV: Add service to validate email settings (#13021)
We have a few places in the code where we need to validate various email related settings, and will have another soon with the improved group email settings UI. This PR introduces a class which can validate POP3, IMAP, and SMTP credentials and also provide a friendly error message for issues if they must be presented to an end user.

This PR does not change any existing code to use the new service. I have added a TODO to change POP3 validation and the email test rake task to use the new validator post-release.
2021-05-13 15:11:23 +10:00
Dan Ungureanu
60be1556fc
FIX: Various invite system fixes (#13003)
* FIX: Ensure the same email cannot be invited twice

When creating a new invite with a duplicated email, the old invite will
be updated and returned. When updating an invite with a duplicated email
address, an error will be returned.

* FIX: not Ember helper does not exist

* FIX: Sync can_invite_to_forum? and can_invite_to?

The two methods should perform the same basic set of checks, such as
check must_approve_users site setting.

Ideally, one of the methods would call the other one or be merged and
that will happen in the future.

* FIX: Show invite to group if user is group owner
2021-05-12 13:06:39 +03:00
Krzysztof Kotlarek
4a2912233c
FIX: flaky specs after topic view custom filters (#13019)
* FIX: flaky specs after topic view custom filters

When ensuring TopicView class variables return to the original state it should use empty Hash instead of empty Array. That

https://github.com/discourse/discourse/blob/master/lib/topic_view.rb#L60

* FIX: convert to string for topic view custom filter
2021-05-11 11:24:14 +10:00
Blake Erickson
37812b8e35
Revert "DEV: Drop old IE11 intersection-observer references" (#13017)
This reverts commit 7360a0f70f.

iOS still wants this sometimes. Probably best to revert for now and we can
always remove this again later.

See: https://meta.discourse.org/t/189799/11?u=blake
2021-05-10 17:47:09 -06:00
Neil Lalonde
02f0acc41b
Version bump to v2.7.0.beta9 (#13006) 2021-05-10 11:29:47 -04:00
Krzysztof Kotlarek
a4bd1806d9
FEATURE: ability to register custom filters for posts (#12938)
Allow plugins to extend TopicView to filter posts
2021-05-10 08:57:58 +10:00
Alan Guo Xiang Tan
4bad1e7e93 PERF: Preload LetterAvatar.image_magick_version in master process.
It is wasteful to run the clean up and command to generate the version
digest on every unicorn child process.
2021-05-07 14:00:38 +08:00
Martin Brennan
5f7bef6d20
FEATURE: Add email_encoded parameter to accept inbound base64 encoded emails (#12947)
We have found when receiving and posting inbound emails to the handle_mail route, it is better to POST the payload as a base64 encoded string to avoid strange encoding issues. This introduces a new param of `email_encoded` and maintains the legacy param of email, showing a deprecation warning. Eventually the old param of `email` will be dropped and the new one `email_encoded` will be the only way to handle_mail.
2021-05-06 12:59:52 +10:00
Blake Erickson
1b02dce594
DEV: Drop old IE11 intersection-observer references (#12942)
It's been awhile since we have supported IE11 so it should be safe to remove
IntersectionObserver now.

From a TODO task in this repo:
> drop when we eventually drop IE11

Announcement of when we removed IE11 support:

https://meta.discourse.org/t/137984/40?u=blake
2021-05-04 17:54:54 -06:00
Roman Rizzi
5794787300
FIX: Make the docker:test task install dependencies before calling yarn ember test. (#12943) 2021-05-04 16:30:35 -03:00
Joffrey JAFFEUX
374ff4c54c
FIX: topic might be night at this stage (#12929) 2021-05-03 10:37:43 +02:00
Dan Ungureanu
3ff9dbb15c
FIX: Show warning only if new value is different (#12893)
A warning is logged when a shadowed site setting is being changed. This
should not be the case when the new value is the same as the old one.
2021-05-03 09:25:02 +02:00
Joffrey JAFFEUX
d16a39dc53
FIX: prevents exception when text input is nil (#12922)
nil was converted to "" and the matching regex would return [] and then be converted to nil with max usage.

Example exception:

```
NoMethodError (undefined method `<=' for nil:NilClass)

lib/text_sentinel.rb:71:in `seems_unpretentious?'
lib/validators/quality_title_validator.rb:13:in `validate_each'
lib/topic_creator.rb:25:in `valid?'
```
2021-05-03 09:21:35 +02:00
Joffrey JAFFEUX
64dda7112d
FIX: correctly use timeouts in FileHelper and FinalDestination (#12921)
Previous refactors have lost usage of read_timeout in `FileHelper.download` and `FinalDestination` was incorrectly using `Net::HTTP.start` by setting `open_timeout` in the block instead of directly during the invocation.

Couldn't figure how to write a good test for this without slowing the spec.
2021-05-03 09:21:11 +02:00
Jeff Wong
656b0ae39e
DEV: Add an option to skip a theme update from the themes:install task. (#12905)
A theme can now specify `skip_update: true` in the yml config for
update allowing for the theme to be installed only if it does not already
exist.
2021-04-30 09:31:41 -07:00
Gerhard Schlager
4f36f604c8
FIX: Looking up translation overrides by symbol failed when count is used (#12896) 2021-04-30 13:54:45 +02:00
David Taylor
cac7725e28
DEV: Improve bin/unicorn boot time in development environment (#12900)
This commit makes a few changes to improve boot time in development environments. It will have no effect on production boot times.

- Skip the SchemaCache warmup. In development mode, the SchemaCache is refreshed every time there is a code change, so warmup is of limited use.

- Skip warming up PrettyText. This adds ~2s to each web worker's boot time. The vast majority of requests do not use PrettyText, so it is more efficient to defer its warmup until it's needed

- Skip the intentional 1 second pause during Unicorn worker forking. The comment (which also exists in Unicorn's documentation) suggests this works around a Unix signal handling bug, but I haven't been able to locate any more information. Skipping it in dev will significantly speed up boot. If we start to see issues, we can revert this change.

On my machine, this improves `/bin/unicorn` boot time from >10s to ~4s
2021-04-30 11:32:13 +01:00
David Taylor
1bc1a439ee
PERF: Ensure locales are always handled as symbols internally (#12897)
Sometimes, parts of the application pass in the locale as a string, not a symbol. This was causing the translate_accelerator to cache two versions of the locale separately: one cache for the symbol version, and one cache for the string version. For example, in a running production process:

```
irb(main):001:0> I18n.instance_variable_get(:@loaded_locales)
=> [:en, "en"]
```

This commit ensures the `locale` key is always converted to a symbol, and adds a spec to ensure the same locale cannot appear twice in `@loaded_locales`
2021-04-30 10:56:27 +01:00
David Taylor
ad8c7714c8
DEV: Use filesystem-based SchemaCache in development (#12901)
In development we regularly restart/reload Rails, which wipes out the schema cache. This then has to be regenerated using DDL queries on the database.

Instead, we can make use of the `rake db:schema:cache:dump` command. This will dump the schema cache to a YAML file, and then load it when needed. This is significantly faster than rebuilding the cache from DDL queries every time.
2021-04-30 10:54:49 +01:00
Roman Rizzi
7423943ffe
DEV: The docker:test task can run JS tests using the Ember CLI. (#12879)
Set the EMBER_CLI env var to 1 to include these tests.
2021-04-29 14:51:34 -03:00
Neil Lalonde
a7261a1f68
Version bump to v2.7.0.beta8 (#12894) 2021-04-29 13:37:14 -04:00
Bianca Nenciu
e6329d3007
FIX: Gracefully handle inline images in emails (#12855) 2021-04-29 10:17:33 +03:00
Osama Sayegh
4f88f2eb15
FEATURE: Allow theme tests to be run in production (take 2) (#12845)
This commit allows site admins to run theme tests in production via a new `/theme-qunit` route. When you visit `/theme-qunit`, you'll see a list of the themes/components installed on your site that have tests, and from there you can select a theme or component that you run its tests.

We also have a new rake task `themes:install_and_test` that can be used to install a list of themes/components on a temporary database and run the tests of the themes/components that are installed. This rake task can be useful when upgrading/deploying a Discourse instance to make sure that the installed themes/components are compatible with the new Discourse version being deployed, and if the tests fail you can abort the build/deploy process so you don't end up with a broken site.
2021-04-28 23:12:08 +03:00
David Taylor
7568c7a101
PERF: Make stylesheet max_file_mtime calculation more efficient (#12876)
Plugins always store their stylesheets under `/assets/stylesheets`, so we can make the glob pattern much more specific. In my local development environment, this increases the speed of `Stylesheet::Manager.max_file_mtime` from ~65ms to ~3ms (20x faster). This significantly improves stylesheet regeneration time, and the responsiveness of the theme admin UI.

Note that this will have negligible effect in production, because in production the value of `max_file_mtime` is aggressively cached.
2021-04-28 17:27:18 +01:00
Régis Hanol
cd93d1b5f7
FEATURE: new 'trim_incoming_emails' site setting (#12874)
This setting allows admin to de/activate automatic trimming of incoming email.
There are instances where it does wonders in trimming all the garbage content and other
instances where it's so bad that it trims the most important part of the email.

FIX: don't remove hidden content using the style attribute when converting HTML to Markdown.
The regexp used was doing more harm than good. It was way too broad.

FIX: properly elide signatures from emails sent with Front App.
This is fairly safe as Front App nicely identifies signatures in the HTML part.
2021-04-28 17:08:48 +02:00
Penar Musaraj
548c044809
FIX: Improvements to animated image pausing (#12839) 2021-04-28 10:48:00 -04:00
Martin Brennan
6d53005e8b
Revert "DEV: Improving topic tracking state code (#12555)" (#12864)
This reverts commit 45df579db0.

This was causing huge browser freezes and crashes.
2021-04-28 11:29:54 +10:00
Martin Brennan
45df579db0
DEV: Improving topic tracking state code (#12555)
The aim of this PR is to improve the topic tracking state JavaScript code and test coverage so further modifications can be made in plugins and in core. This is focused on making topic tracking state changes easier to respond to with callbacks, and changing it so all state modifications go through a single method instead of modifying `this.state` all over the place. I have also tried to improve documentation, make the code clearer and easier to follow, and make it clear what are public and private methods.

The changes I have made here should not break backwards compatibility, though there is no way to tell for sure if other plugin/theme authors are using tracking state methods that are essentially private methods. Any name changes made in the tracking-state.js code have been reflected in core.

----

We now have a `_trackedTopicLimit` in the tracking state. Previously, if a topic was neither new nor unread it was removed from the tracking state; now it is only removed if we are tracking more than `_trackedTopicLimit` topics (which is set to 4000). This is so plugins/themes adding topics with `TopicTrackingState.register_refine_method` can add topics to track that aren't necessarily new or unread, e.g. for totals counts.

Anywhere where we were doing `tracker.states["t" + data.topic_id] = newObject` has now been changed to flow through central `modifyState` and `modifyStateProp` methods. This is so state objects are not modified until they need to be (e.g. sometimes properties are set based on certain conditions) and also so we can run callback functions when the state is modified.

I added `onStateChange` and `onMessageIncrement` methods to register callbacks that are called when the state is changed and when the message count is incremented, respectively. This was done so we no longer need to do things like `@observes("trackingState.states")` in other Ember classes.

I split up giant functions like `sync` and `establishChannels` into smaller functions for readability and testability, and renamed many small functions to _functionName to designate them as private functions which not be called by consumers of `topicTrackingState`. Public functions are now all documented (well...at least ones that are not immediately obvious).

----

On the backend side, I have changed the MessageBus publish events for TopicTrackingState to send back tags and tag IDs for more channels, and done some extra code cleanup and refactoring. Plugins may override `TopicTrackingState.report` so I have made its footprint as small as possible and externalised the main parts of it into other methods.
2021-04-28 09:54:45 +10:00
Robin Ward
7a3f2eb3fd
FIX: Ember CLI Live Reload was not supported by the CSP (#12859) 2021-04-27 13:55:23 -04:00
David Taylor
02f1f85306
FIX: Remove whitespace from theme git versions (#12857)
Followup to 0a6fab564f
2021-04-27 17:45:54 +01:00
Régis Hanol
bea1fc1ab8
FEATURE: TL4 & category moderators can merge posts (#12843)
This gives TL4 users and category moderators the ability to merge posts in topics they have moderation privileges.
2021-04-27 18:24:27 +02:00
David Taylor
0a6fab564f
PERF: Add timeout to theme import git commands (#12856)
This ensures that large git repositories cannot cause performance issues on the server. In normal use, this 20s timeout should never be hit.
2021-04-27 17:04:15 +01:00
David Taylor
1fd8f6df5f
PERF: Improve theme stylesheet compilation performance (#12850)
When building the `scss_load_paths`, we were creating a full export of the theme (including uploads), and not cleaning it up. With many uploads, this can be extremely slow (because it downloads every upload from S3), and the lack of cleanup could cause a disk to fill up over time.

This commit updates the ZipExporter to provide a `with_export_dir` API, which takes care of cleanup. It also adds a kwarg which allows exporting only extra_scss fields. This should make things much faster for themes with many uploads.
2021-04-27 14:33:43 +01:00
Krzysztof Kotlarek
e29605b79f
FEATURE: the ability to search users by custom fields (#12762)
When the admin creates a new custom field they can specify if that field should be searchable or not.

That setting is taken into consideration for quick search results.
2021-04-27 15:52:45 +10:00
Bianca Nenciu
8aeeadd8b0
FIX: Replace use of regular expression (#12838)
It used a regular expression to check if message IDs were in RFC format.
2021-04-27 08:48:51 +03:00
Osama Sayegh
a169dc6832
Revert "FEATURE: Allow theme tests to be run in production (#12815)" (#12840)
This reverts commit 7217dcb67a.

https://meta.discourse.org/t/failed-to-bootstrap-due-to-out-of-memory-killer/188141/18?u=osama

Precompiling test_helper.js is so expensive that it can make bootstrap
fail on servers with limited resources (2GB RAM). We will find another
way that doesn't require much resources.
2021-04-26 23:05:58 +03:00
Bianca Nenciu
765ba1ab2d
FEATURE: Ignore anonymous page views on private sites (#12800)
For sites with login_required set to true, counting anonymous pageviews is
confusing. Requests to /login and other pages would make it look like
anonymous users have access to site's content.
2021-04-26 14:19:47 +03:00
Osama Sayegh
7217dcb67a
FEATURE: Allow theme tests to be run in production (#12815)
This commit allows site admins to run theme tests in production via a new `/theme-qunit` route. When you visit `/theme-qunit`, you'll see a list of the themes/components installed on your site that have tests, and from there you can select a theme or component that you run its tests.

We also have a new rake task `themes:install_and_test` that can be used to install a list of themes/components on a temporary database and run the tests of the themes/components that are installed. This rake task can be useful when upgrading/deploying a Discourse instance to make sure that the installed themes/components are compatible with the new Discourse version being deployed, and if the tests fail you can abort the build/deploy process so you don't end up with a broken site.
2021-04-26 12:56:45 +03:00
Robin Ward
e3b1d1a718
DEV: Improve Ember CLI's bootstrap logic (#12792)
* DEV: Give a nicer error when `--proxy` argument is missing

* DEV: Improve Ember CLI's bootstrap logic

Instead of having Ember CLI know which URLs to proxy or not, have it try
the URL with a special header `HTTP_X_DISCOURSE_EMBER_CLI`. If present,
and Discourse thinks we should bootstrap the application, it will
instead stop rendering and return a HTTP HEAD with a response header
telling Ember CLI to bootstrap.

In other words, any time Rails would otherwise serve up the HTML for the
Ember app, it stops and says "no, you do it."

* DEV: Support asset filters by path using a new options object

Without this, Ember CLI's bootstrap would not get the assets it wants
because the path it was requesting was different than the browser path.
This adds an optional request header to fix it.

So far this is only used by the styleguide.
2021-04-23 10:24:42 -04:00
Bianca Nenciu
8c4a11c006
DEV: Move autotag to topic creator (#12790)
This move was necessary to automatically tag the topic with the right
tags from creation time. The process post job may be delayed for a
short time.
2021-04-23 16:55:34 +03:00
Penar Musaraj
c11d75da87
FEATURE: Allow pausing animated images in posts (#12795)
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2021-04-22 11:28:35 -04:00
Joffrey JAFFEUX
0eeedf307a
DEV: replaces huge generated emoji list by a simpler regex (#11053)
Note that this commit is also fixing various mistakes in emojis.

Some of them have been fixed manually in db.json/data.js/groups.json and will need to be fixed in emoji-db gem.
2021-04-22 08:43:06 +02:00
Penar Musaraj
886cf72b0e
FIX: Fall back to hardcoded version when period check disabled (#12784)
When `SiteSetting.version_checks?` is disabled, the Discourse version is
not stored in Redis, so we need to fall back to the hardcoded value.
2021-04-21 12:40:27 -04:00
Roman Rizzi
6b613e3076
FEATURE: Review every post using the review queue. (#12734)
* FEATURE: Review every post using the review queue.

If the `review_every_post` setting is enabled, posts created and edited by regular uses are sent to the review queue so staff can review them. We'll skip PMs and posts created or edited by TL4 or staff users.

Staff can choose  to:

- Approve the post (nothing happens)
- Approve and restore the post (if deleted)
- Approve and unhide the post (if hidden)
- Reject and delete it
- Reject and keep deleted (if deleted)
- Reject and suspend the user
- Reject and silence the user

* Update config/locales/server.en.yml

Co-authored-by: Robin Ward <robin.ward@gmail.com>

Co-authored-by: Robin Ward <robin.ward@gmail.com>
2021-04-21 08:41:36 -03:00
Osama Sayegh
45ccadeeeb
DEV: Upgrade Rails to 6.1.3.1 (#12688)
Rails 6.1.3.1 deprecates a few API and has some internal changes that break our tests suite, so this commit fixes all the deprecations and errors and now Discourse should be fully compatible with Rails 6.1.3.1. We also have a new release of the rails_failover gem that's compatible with Rails 6.1.3.1.
2021-04-21 12:36:32 +03:00
Penar Musaraj
c47e6a2004
FIX: Use CDN urls for theme settings of type upload (#12773) 2021-04-20 18:42:02 -04:00
Martin Brennan
e3b1f5721b
FIX: Enforce tag group count validation before sending to review queue (#12728)
There is a category setting that enforces 1 or more tags must be added to a topic from a specific tag group before creating it. This validation was not being run before the topic was being sent to a review queue for categories that have that setting enabled.

There was an existing validation in `TopicCreator` but it was not correct; it was only validating when the tags did _not_ exist and also only happened on `create`. I now run the validation in `TopicCreator.valid?`

I also improved the error message shown to the user when they have not added the tags required (showing the tag names from the tag group), and changed the composer tag selector to not show "optional" if there are N tags required from a certain group.
2021-04-19 09:43:50 +10:00
Joffrey JAFFEUX
3157d5ee1b
DEV: ensures stylesheet watcher isn't crashing with gems plugins (#12733)
* DEV: ensures stylesheet watcher isn't crashing with gems plugins

This bug has been exhibited since discourse_dev is now including an auth plugin which was loaded as a relative path instead of an absolute path, eg:

`Users/bob/.gem/ruby/2.6.6/gems/discourse_dev-0.1.0/auth/plugin.rb`

Instead of

`/Users/bob/.gem/ruby/2.6.6/gems/discourse_dev-0.1.0/auth/plugin.rb`
2021-04-16 15:25:20 +02:00
David Taylor
3436fef5e3
FIX: Update database_restorer to avoid shell use (#12731)
Follow-up to 0ec5fd5262
2021-04-16 13:39:45 +01:00
Régis Hanol
c43a7bb23b
FIX: ensure plugin's gems are in the gem path (#12727)
Let's say you want to use a gem in a plugin that has a dependencie.

You would write something like this:

```ruby
gem 'dependency-gem', '1.2.3'
gem 'amazing-gem', '4.5.6'
```

However, since when we install a plugin gem we install it in the `gems`
directory created inside the plugins directory, when it comes the time
to install the `amazing-gem`, it won't be able to find the `dependency-gem`.

This fixes that issue by adding the `gems` plugins directory to the global gem path.

Also fixed a frozen string error when specifying a source.
2021-04-16 10:21:39 +02:00
Vinoth Kannan
e08432ddde
DEV: Bump discourse_dev to 0.1.0 (#12722)
And add the "discourse-development-auth" plugin and client locale files.
2021-04-16 06:42:34 +05:30
David Taylor
0ec5fd5262
DEV: Raise exception when execute_command will spawn a shell (#12716) 2021-04-15 16:29:37 +01:00
Dan Ungureanu
85d4b60a45
FIX: Improve error messages for invites (#12714)
The error messages used to include an unnecessary 'Validation failed:
Email' prefix which was removed.
2021-04-15 14:46:32 +03:00
Mark VanLandingham
1d85574d52
FIX: Don't show the get a room composer message in private categories (#12702) 2021-04-14 12:34:13 -05:00
Bianca Nenciu
21d1ee1065
FIX: Use Nokogiri and Loofah consistently (#12693)
CookedPostProcessor used Loofah to parse the cooked content of a post
and Nokogiri to parse cooked Oneboxes. Even though Loofah is built on
top of Nokogiri, replacing an element from the cooked post (a Nokogiri
node) with a parsed onebox (a Loofah node) produced a strange result
which included XML namespaces. Removing the mix and using Loofah
to parse Oneboxes fixed the problem.
2021-04-14 18:09:55 +03:00
Neil Lalonde
07ca35670a
Version bump to v2.7.0.beta7 (#12696) 2021-04-14 10:54:08 -04:00
David Taylor
9430185493
SECURITY: Improve theme git import (#12694) 2021-04-14 15:32:47 +01:00
Bianca Nenciu
e4e2c7c66f
FIX: Improve anchor links (#12683)
* FIX: Use theme color for anchor icon

* FIX: Do not count anchor links

* FIX: Do not count hashtags links either

* DEV: Add tests for link_count

* FIX: Disable anchors in quotes and preview

* FIX: Try building some anchor slugs for unicode

* DEV: Fix tests
2021-04-14 10:27:07 +03:00
Martin Brennan
eeaecd4fd2
FEATURE: Category setting to allow unlimited first post edits by the owner of the topic (#12690)
This PR adds a new category setting which is a column in the `categories` table, `allow_unlimited_owner_edits_on_first_post`.

What this does is:

* Inside the `can_edit_post?` method of `PostGuardian`, if the current user editing a post is the owner of the post, it is the first post, and the topic's category has `allow_unlimited_owner_edits_on_first_post`, then we bypass the check for `LimitedEdit#edit_time_limit_expired?` on that post.
* Also, similar to wiki topics, in `PostActionNotifier#after_create_post_revision` we send a notification to all users watching a topic when the OP is edited in a topic with the category setting `allow_unlimited_owner_edits_on_first_post` enabled.

This is useful for forums where there is a Marketplace or similar category, where topics are created and then updated indefinitely by the OP rather than the OP making new topics or additional replies. In a way this acts similar to a wiki that only one person can edit.
2021-04-14 15:54:09 +10:00
David Taylor
65647000a1
DEV: Allow Discourse::Utils.execute_command timeout with env (#12672)
Followup to 5deda5ef3e

The first argument to `Open3.capture3` can be an environment variable hash. In this case, we need to insert the `timeout` command after the env hash.
2021-04-12 13:53:41 +01:00
Osama Sayegh
cd24eff5d9
FEATURE: Introduce theme/component QUnit tests (take 2) (#12661)
This commit allows themes and theme components to have QUnit tests. To add tests to your theme/component, create a top-level directory in your theme and name it `test`, and Discourse will save all the files in that directory (and its sub-directories) as "tests files" in the database. While tests files/directories are not required to be organized in a specific way, we recommend that you follow Discourse core's tests [structure](https://github.com/discourse/discourse/tree/master/app/assets/javascripts/discourse/tests).

Writing theme tests should be identical to writing plugins or core tests; all the `import` statements and APIs that you see in core (or plugins) to define/setup tests should just work in themes.

You do need a working Discourse install to run theme tests, and you have 2 ways to run theme tests:

* In the browser at the `/qunit` route. `/qunit` will run tests of all active themes/components as well as core and plugins. The `/qunit` now accepts a `theme_name` or `theme_url` params that you can use to run tests of a specific theme/component like so: `/qunit?theme_name=<your_theme_name>`.

* In the command line using the `themes:qunit` rake task. This take is meant to run tests of a single theme/component so you need to provide it with a theme name or URL like so: `bundle exec rake themes:qunit[name=<theme_name>]` or `bundle exec rake themes:qunit[url=<theme_url>]`.

There are some refactors to how Discourse processes JavaScript that comes with themes/components, and these refactors may break your JS customizations; see https://meta.discourse.org/t/upcoming-core-changes-that-may-break-some-themes-components-april-12/186252?u=osama for details on how you can check if your themes/components are affected and what you need to do to fix them.

This commit also improves theme error handling in Discourse. We will now be able to catch errors that occur when theme initializers are run and prevent them from breaking the site and other themes/components.
2021-04-12 15:02:58 +03:00
David Taylor
35e1e009fa
FIX: Allow restoring non-subfolder backup to subfolder site (#12537)
`GlobalSetting.relative_url_root` comes from the destination site. We
can't be sure whether it was the same on the original site. It's safer
to use a wildcard here, so we can backup/restore sites with different
relative_url_root values.
2021-04-12 14:00:52 +10:00
Penar Musaraj
abb0a4bae2
DEV: Add SCSS helper to replace asset-uri and image-uri (#12664) 2021-04-12 13:57:39 +10:00
Sam
5deda5ef3e
FIX: automatically timeout long running image magick commands (#12670)
Previously certain images may lead to convert / identify to run for unreasonable
amounts of time

This adds a maximum amount of time these commands can run prior to forcing
them to stop
2021-04-12 13:55:54 +10:00
Sam
5b342ae505
FIX: remove superfluous spaces from CJK blurbs (#12629)
Previously we used the raw data indexed to generate blurbs even for cases
when Chinese/Korean/Japanese text was used.

This caused superfluous spaces to show up in excerpts.
2021-04-12 12:46:42 +10:00
Penar Musaraj
43d09dbabe
DEV: Skip looking for svgo, gifsicle binaries (#12655) 2021-04-09 08:52:06 -04:00
Penar Musaraj
e77b9dfd45
DEV: Use Terser for JS minification/compression if available (#12656) 2021-04-09 08:51:49 -04:00
Penar Musaraj
eb7ff576e5
FEATURE: Use SVG icons for some oneboxes (#12654) 2021-04-08 09:58:12 -04:00
Neil Lalonde
3d4e2595d3
Version bump to v2.7.0.beta6 (#12636) 2021-04-07 15:10:07 -04:00
jbrw
50252d803e
DEV: stub youtube embed requests (#12637)
* DEV: stub youtube embed requests

* DEV: Ignore redirects on youtube.com when oneboxing
2021-04-07 13:32:27 -04:00
Roman Rizzi
5e4c0e2caa
FEATURE: Treat site settings as plain text and add a new HTML type. (#12618)
To add an extra layer of security, we sanitize settings before shipping them to the client. We don't sanitize those that have the "html" type.

The CookedPostProcessor already uses Loofah for sanitization, so I chose to also use it for this. I added it to our gemfile since we installed it as a transitive dependency.
2021-04-07 12:51:19 -03:00
Osama Sayegh
2b9ab3a0d9
Revert "FEATURE: Introduce theme/component QUnit tests (#12517)" (#12632)
This reverts commit a53d8d3e61 and 105634435f.

Reverted because the change broke some components. Will be added back in a few days.
2021-04-07 17:45:49 +03:00
Ahmed Gagan
2308a58113
DEV: Added support for custom site setting 'emoji_list' (#12414)
Example usage:

```
best_emojis:
    type: emoji_list
    default: laughing|open_mouth|cry|angry|hugs
    client: true
```
2021-04-07 15:32:05 +02:00
Osama Sayegh
105634435f
FIX: Prevent double slashes in Ember templates paths (#12630)
Follow-up to https://github.com/discourse/discourse/pull/12517
2021-04-07 14:08:29 +03:00
Osama Sayegh
a53d8d3e61
FEATURE: Introduce theme/component QUnit tests (#12517)
This commit allows themes and theme components to have QUnit tests. To add tests to your theme/component, create a top-level directory in your theme and name it `test`, and Discourse will save all the files in that directory (and its sub-directories) as "tests files" in the database. While tests files/directories are not required to be organized in a specific way, we recommend that you follow Discourse core's tests [structure](https://github.com/discourse/discourse/tree/master/app/assets/javascripts/discourse/tests).

Writing theme tests should be identical to writing plugins or core tests; all the `import` statements and APIs that you see in core (or plugins) to define/setup tests should just work in themes.

You do need a working Discourse install to run theme tests, and you have 2 ways to run theme tests:

* In the browser at the `/qunit` route. `/qunit` will run tests of all active themes/components as well as core and plugins. The `/qunit` now accepts a `theme_name` or `theme_url` params that you can use to run tests of a specific theme/component like so: `/qunit?theme_name=<your_theme_name>`.

* In the command line using the `themes:qunit` rake task. This take is meant to run tests of a single theme/component so you need to provide it with a theme name or URL like so: `bundle exec rake themes:qunit[name=<theme_name>]` or `bundle exec rake themes:qunit[url=<theme_url>]`.

There are some refactors to internal code that's responsible for processing themes/components in Discourse, most notably:

* `<script type="text/discourse-plugin">` tags are automatically converted to modules.

* The `theme-settings` service is removed in favor of a simple `lib` file responsible for managing theme settings. This was done to allow us to register/lookup theme settings very early in our Ember app lifecycle and because there was no reason for it to be an Ember service.

These refactors should 100% backward compatible and invisible to theme developers.
2021-04-07 10:39:57 +03:00
Neil Lalonde
e8a9917db1
FEATURE: Allow setting avatar flair for automatic groups (#12586) 2021-04-06 11:13:06 -04:00
Bianca Nenciu
e026af11ac
FIX: Simplify send PM to email settings (#12583)
This feature used to be controlled by two site settings 
enable_personal_email_messages and min_trust_to_send_email_messages.
I removed enable_personal_email_messages and unhide 
min_trust_to_send_email_messages to simplify the process of 
enabling / disabling this feature.
2021-04-06 15:39:27 +03:00
Jeff Wong
3a22c654e2
FIX: rake themes:update should fail if a theme update fails (#12608) 2021-04-05 15:53:34 -07:00
Arpit Jalan
6234d7455b
FEATURE: add maximum limit for secondary emails (#12599) 2021-04-05 20:31:42 +05:30
Arpit Jalan
c478ffc662
FIX: post merging was failing silently (#12566)
https://meta.discourse.org/t/merging-very-long-posts-removes-them/183597
2021-04-01 06:46:18 +05:30
jbrw
68d0916eb5
FEATURE: Oneboxer cache response body (#12562)
* FEATURE: Cache successful HTTP GET requests during Oneboxing

Some oneboxes may fail if when making excessive and/or odd requests against the target domains. This change provides a simple mechanism to cache the results of succesful GET requests as part of the oneboxing process, with the goal of reducing repeated requests and ultimately improving the rate of successful oneboxing.

To enable:

Set `SiteSetting.cache_onebox_response_body` to `true`

Add the domains you’re interesting in caching to `SiteSetting. cache_onebox_response_body_domains` e.g. `example.com|example.org|example.net`

Optionally set `SiteSetting.cache_onebox_user_agent` to a user agent string of your choice to use when making requests against domains in the above list.

* FIX: Swap order of duration and value in redis call

The correct order for `setex` arguments is `key`, `duration`, and `value`.

Duration and value had been flipped, however the code would not have thrown an error because we were caching the value of `1.day.to_i` for a period of 1 seconds… The intention appears to be to set a value of 1 (purely as a flag) for a period of 1 day.
2021-03-31 13:19:34 -04:00
Dan Ungureanu
dce48d8aa7
FIX: Redirect to provided origin after auth (#12558)
It used to redirect to the destination_url cookie which sometimes is set
incorrectly.
2021-03-31 10:23:12 +01:00
Bianca Nenciu
52a5c7e438
FIX: Improve error messages if user cannot send PM emails (#12547) 2021-03-30 12:18:57 +03:00
Penar Musaraj
5096920500
FEATURE: Implement nonces for Google Tag Manager integration (#12531) 2021-03-26 11:19:31 -04:00
Martin Brennan
8de07181a8
FEATURE: Add new :topic_tags_changed DiscourseEvent (#12530)
This is called in DiscourseTagging.tag_topic_by_names only after
all the validations etc. have been passed, and after topic.tags = X
has been called (because this is when the associations are created/
destroyed). The event has the topic, then a second param with the
old and new tag names in arrays for easy inspection.
2021-03-26 13:53:47 +10:00
David Taylor
f637bf1b58
DEV: Deprecate message parameter in auth provider plugin API (#12523)
This has been unused since d2bceff133
2021-03-25 13:23:48 +00:00
Gerhard Schlager
acc73f2989
FIX: uploads:fix_missing_s3 rake task used wrong SHA1 (#12495) 2021-03-25 11:35:29 +01:00
Krzysztof Kotlarek
c03c85e661
FIX: delete orphan post revisions (#12502)
I was adding specs to ensure that post actions and uploads are removed for permanently deleted posts.

I noticed that post revisions were not permanently destroyed. I added a migration to fix old data.
2021-03-25 12:34:53 +11:00
Martin Brennan
ea6f9af08b
FIX: Add topic_diff to PostRevisor (#12518)
The instance of the PostRevisor is passed to the post_edited
event. It is useful to know what has happened to the topic in
this event (we already pass a boolean for topic_changed? but that
is not so helpful by itself).
2021-03-25 10:24:50 +10:00
Bianca Nenciu
9dca7fe9a3
DEV: Replace Rails ends_with? with Ruby end_with? (#12507) 2021-03-24 19:51:21 +02:00
Mark VanLandingham
371afc45e0
DEV: API for plugins to add post update params and handlers (#12505) 2021-03-24 10:22:16 -05:00
Bianca Nenciu
e7fb45cc29
FEATURE: Treat emoji or similar characters as one (#12482)
Long messages consisting only of emojis, dots or commas used to be
valid because character-wise they were over the limit.
2021-03-24 16:47:35 +02:00
Sam
dc6b547ed8
FIX: rake db:validate_indexes was broken (#12463)
A file was moved, but zeitwerk can not find it due to custom inflection.

Renamed so it can be properly found.
2021-03-24 08:53:01 +11:00
Sam
5a1318e3c3
DEV: upgrade mini_sql (#12465)
* DEV: upgrade mini_sql

Even though we are not planning on using this quite yet, mini_sql now supports
prepared statements.

Would like this upgrade merged so we can do some benchmarking.

Note, this will not work with pg_bouncer, but sites that are not using it
may benefit from the feature.

* implement multisite friendly prepared statements
2021-03-24 08:48:04 +11:00
Jarek Radosz
6ff888bd2c
DEV: Retry-after header values should be strings (#12475)
Fixes `Rack::Lint::LintError: a header value must be a String, but the value of 'Retry-After' is a Integer`. (see: 14a236b4f0/lib/rack/lint.rb (L676))

I found it when I got flooded by those warning a while back in a test-related accident 😉 (ember CLI tests were hitting a local rails server at a fast rate)
2021-03-23 20:32:36 +01:00
Penar Musaraj
066c59d0e3
FIX: Regression in colors used by non-default theme (#12492)
eb7f0ec caused this regression, where a non-default theme set to use
the base color scheme was resolving to the default theme's color scheme.
2021-03-23 12:46:25 -04:00
Rafael dos Santos Silva
fb4486d5f1
FEATURE: Add CSP frame-ancestors support (#12404) 2021-03-22 16:00:25 -03:00
Dan Ungureanu
4e46732346
FEATURE: Implement browser update in crawler view (#12448)
browser-update script does not work correctly in some very old browsers
because the contents of <noscript> is not accessible in JavaScript.
For these browsers, the server can display the crawler page and add the
browser update notice.

Simply loading the browser-update script in the crawler view is not a
solution because that means all crawlers will also see it.
2021-03-22 19:41:42 +02:00
Bianca Nenciu
129aeb8c2f
FIX: Improvements to email styles (#12451) 2021-03-22 14:09:38 +02:00
Martin Brennan
6eb0d0c38d
SECURITY: Fix is_private_ip for RateLimiter to cover all cases (#12464)
The regular expression to detect private IP addresses did not always detect them successfully.
Changed to use ruby's in-built IPAddr.new(ip_address).private? method instead
which does the same thing but covers all cases.
2021-03-22 13:56:32 +10:00
Martin Brennan
49f4c548ef
FEATURE: Bookmark pinning (#12431)
Users can now pin bookmarks from their bookmark list. This will anchor the bookmark to the top of the list, and show a pin icon next to it. This also applies in the nav bookmarks panel. If there are multiple pinned bookmarks they sort by last updated order.
2021-03-22 09:50:22 +10:00
David Taylor
56a573ab4b
DEV: Remove String#match? and Regexp.match? polyfills (#12454)
This was added in Ruby 2.4. Discourse will no longer run on ruby versions older than that, so we can drop this polyfill
2021-03-19 19:55:44 +00:00
Robin Ward
af5adc440e
DEV: We hadn't enabled object-rest-spread in all contexts (#12452)
See: https://babeljs.io/docs/en/babel-plugin-proposal-object-rest-spread

This fixes Discourse in browsers that don't support this syntax yet.
2021-03-19 13:57:04 -04:00
Martin Brennan
fb0b4e26a9
DEV: Add output_sql_to_stderr! to MethodProfiler (#12445)
This PR adds MethodProfiler.output_sql_to_stderr! for easier debugging of SQL queries and their timings from the console.

This is almost the same as ensure_discourse_instrumentation! but should not
be used in production (save for debugging in the console), and is only instrumenting
PostgresSQL queries.

This is almost the same as ensure_discourse_instrumentation! but should not
be used in production. This logs all SQL queries run and their durations
between start and stop.

It also works for super long running queries. If you interrupt the long-running
query the latest query data will still be logged after stopping the profiler.

Usage:

```
MethodProfiler.output_sql_to_stderr!(filter_transactions: true)
MethodProfiler.start

# some code that runs queries

timings = MethodProfiler.stop
```
2021-03-19 17:48:30 +10:00
Martin Brennan
355d51afde
FEATURE: Allow using invites when DiscourseConnect SSO is enabled (#12419)
This PR allows invitations to be used when the DiscourseConnect SSO is enabled for a site (`enable_discourse_connect`) and local logins are disabled. Previously invites could not be accepted with SSO enabled simply because we did not have the code paths to handle that logic.

The invitation methods that are supported include:

* Inviting people to groups via email address
* Inviting people to topics via email address
* Using invitation links generated by the Invite Users UI in the /my/invited/pending route

The flow works like this:

1. User visits an invite URL
2. The normal invitation validations (redemptions/expiry) happen at that point
3. We store the invite key in a secure session
4. The user clicks "Accept Invitation and Continue" (see below)
5. The user is redirected to /session/sso then to the SSO provider URL then back to /session/sso_login
6. We retrieve the invite based on the invite key in secure session. We revalidate the invitation. We show an error to the user if it is not valid. An additional check here for invites with an email specified is to check the SSO email matches the invite email
7. If the invite is OK we create the user via the normal SSO methods
8. We redeem the invite and activate the user. We clear the invite key in secure session.
9. If the invite had a topic we redirect the user there, otherwise we redirect to /

Note that we decided for SSO-based invites the `must_approve_users` site setting is ignored, because the invite is a form of pre-approval, and because regular non-staff users cannot send out email invites or generally invite to the forum in this case.

Also deletes some group invite checks as per https://github.com/discourse/discourse/pull/12353
2021-03-19 10:20:10 +10:00
Penar Musaraj
aee7ef0dc9
DEV: Fix build due to highlight.js branch issue (#12441)
Highlight.js changed their default branch from master to main. This switches to the @highlightjs/cdn-assets package, thus sidestepping the problem. It's a slightly cleaner integration though (no need to build locally anymore).
2021-03-18 18:21:23 -04:00
Dan Ungureanu
033d6b6437
FEATURE: Obfuscate emails on invite show page (#12433)
The email should not be ever displayed in clear text, except the case
when the user authenticates using another service.
2021-03-18 19:09:23 +02:00
Penar Musaraj
eb7f0ec766
FIX: In components, fall back to default theme color variables (#12423)
Component SCSS compilation should use the current theme's SCSS color
variables as a fallback before using the default core colors.

This is mostly a backwards-compatibility fix, new themes and components
should use CSS custom properties, which offer better support for on-the-fly
color scheme changes (dark mode support, etc.).
2021-03-17 13:34:15 -04:00
Osama Sayegh
d56b2e85aa
FIX: Escape Font Awesome icons (#12421)
This is not a security issue because regular users are not allowed to insert FA icons anywhere in the app. Admins can insert icons via custom badges, but they do have the ability to create themes with JS.
2021-03-17 16:11:40 +03:00
Arpit Jalan
64d2f260a9
FIX: do not convert format for site setting uploads (#12410) 2021-03-16 20:14:41 +05:30
Krzysztof Kotlarek
e10a74694a
FIX: deprecation warning - initialization autoloaded the constant (#12400)
Get rid of deprecation related to Zeitwerk autoloader.

Original PR was reverted because of multisite bug #12381 - thank you @davidtaylorhq for fixing it.

I added the last commit to fix that multisite problem.
2021-03-16 09:47:57 +11:00
Rafael dos Santos Silva
f8d9f09370
FEATURE: Fallback to system users when creating new TopicEmbed (#12386)
Previosly, if the topic embed request had a missing username parameter,
and SiteSetting.embed_by_username was empty we would fail to create the
new topic and not show any errors.

Now we will fallback using the priority:

1. Username parameter
2. SiteSetting.embed_by_username
3. SiteSetting.site_contact_username
4. system user
2021-03-15 11:58:53 -03:00
Penar Musaraj
de05c410c5
DEV: Use fewer threads when watching for SCSS changes (#12393) 2021-03-12 16:18:00 -05:00
Daniel Waterworth
e7ac906f21
DEV: Move TemporaryDB class to its own file (#12383)
rake tasks get repeated reloaded during the tests and it causes
redefinition warnings.
2021-03-12 11:27:13 -06:00
Penar Musaraj
52d833472c
DEV: Refactor plugin SCSS compilation (#12359) 2021-03-12 11:17:42 -05:00
David Taylor
1533cbb38b
Revert "FIX: deprecation warning - initialization autoloaded the constant (#12347)" (#12381)
This reverts commit 9a5a38a529.

This change was preventing `rake multisite:migrate` from working
2021-03-12 13:48:46 +00:00
Rafael dos Santos Silva
9737ce8c5f
FEATURE: New DiscourseEvent trigger for likes (#12374) 2021-03-11 21:07:29 -03:00
Krzysztof Kotlarek
9a5a38a529
FIX: deprecation warning - initialization autoloaded the constant (#12347)
Get rid of deprecation related to Zeitwerk autoloader.
2021-03-12 08:42:53 +11:00
Roman Rizzi
56262349a9
FIX: When destroying a topic, destroy its associated published page. (#12358)
Destroying a topic should remove the associated published page so another topic could re-use the slug.
2021-03-11 11:34:54 -03:00
Roman Rizzi
8fcad73b36
FEATURE: Admins can flag posts so they can review them later. (#12311)
Staff can send a post to the review queue by clicking the "Flag Post" button next to "Take Action...". Clicking it flags the post using the "Notify moderators" score type and hides it. A custom message will be sent to the user.
2021-03-11 08:21:24 -03:00
Penar Musaraj
10780d2448
DEV: support json_schema in theme settings (#12294) 2021-03-10 20:15:04 -05:00
jbrw
aed97c7bab
FIX: Add amazon sites to force_get_hosts (#12341)
It has been observed that doing a HEAD against an Amazon store URL may result in a 405 error being returned.

Skipping the HEAD request may result in an improved oneboxing experience when requesting these URLs.
2021-03-10 14:42:17 -05:00
Neil Lalonde
5b8941c568
Version bump to v2.7.0.beta5 (#12287) 2021-03-10 13:45:19 -05:00
Penar Musaraj
4071af1d09
DEV: Refactor font and category background importers (#12312) 2021-03-10 11:05:56 -05:00
Bianca Nenciu
9bd436c20b
FIX: Do not add same email multiple times (#12322)
The user and an admin could create multiple email change requests for
the same user. If any of the requests was validated and it became
primary, the other request could not be deleted anymore.
2021-03-10 14:49:26 +02:00
Sam
10001e4e8d
DEV: add db:validate_indexes task (#12273)
Added a new task to test if indexes are coherent with a blank database

This allows us to detect for cases where somehow indexes are out of sync

FIX_INDEXES=1 or `rake db:validate_indexes[fix]` to correct the issues it finds. 

Detects: 

- Badly named indexes that need to be renamed
- Missing indexes
- Extra indexes

Can correct all 3 with the fix option
2021-03-10 10:59:20 +11:00
jbrw
ac7773a30d
FEATURE: allow category group moderators to pin/unpin topics (#12325)
* FEATURE: allow category group moderators to pin/unpin topics

Category group moderators should be able to pin/unpin any topics within a category where they have appropraite category group moderator permissions.
2021-03-09 16:05:11 -05:00
Alan Guo Xiang Tan
ce04db8610 FEATURE: Allow invites redemption with Omniauth providers. 2021-03-09 09:27:18 +08:00
Alan Guo Xiang Tan
ebe4896e48 FEATURE: Change very high/low search priority to rank at absolute ends.
Prior to this change, we had weights for very_high, high, low and
very_low. This means there were 4 weights to tweak and what weights to
use for `very_high/high` and `very_low/low` pair was hard to explain.
This change makes it such that `very_high` search priority will always
ensure that the posts are ranked at the top while `very_low` search
priority will ensure that the posts are ranked at the very bottom.
2021-03-09 09:20:37 +08:00
Penar Musaraj
0a656767c7
DEV: use Redis data for version check in DiscourseUpdates (#12293)
A version bump caused the specs to fail, this switches to using what is
stored in Redis, which is also what is stubbed in the specs.
2021-03-08 09:05:08 -05:00
Vinoth Kannan
8d96713aa0
DEV: add discourse_dev gem in development dependencies. (#12285)
For more details visit https://github.com/discourse/discourse_dev
2021-03-04 23:04:51 +05:30
Rafael dos Santos Silva
e06076268a
FIX: Missing quote in emoji cdn setting caused by 83f332b (#12280) 2021-03-03 17:39:00 -03:00
Gerhard Schlager
58c218a4bf
FIX: Remap old S3 endpoints during backup restore (#12276)
It also starts outputting exceptions on the console.
2021-03-03 21:10:09 +01:00
Dan Ungureanu
c047640ad4
FEATURE: Various improvements to invite system (#12023)
The user interface has been reorganized to show email and link invites
in the same screen. Staff has more control over creating and updating
invites. Bulk invite has also been improved with better explanations.

On the server side, many code paths for email and link invites have
been merged to avoid duplicated logic. The API returns better responses
with more appropriate HTTP status codes.
2021-03-03 11:45:29 +02:00
Krzysztof Kotlarek
38ab45cb93
FIX: check min tag count requirement when change category (#12252)
When a category is created, we can set `minimum_required_tags` property.

When the topic is created, we are checking that field and ensuring that the minimum amount of tags were added - unless topic created by a staff member.

Problem is that validation is skipped when we change the category from for example category with 0 tags required to the category with 1 tag required.

Changing category is kind of the unicorn as it is a complicated operation:

https://github.com/discourse/discourse/blob/master/lib/post_revisor.rb#L84

https://github.com/discourse/discourse/blob/master/app/models/topic.rb#L911

https://github.com/discourse/discourse/blob/master/app/models/topic.rb#L823

Before we start to try to change the category, we should ensure that the tags requirement is fulfilled.

https://meta.discourse.org/t/the-category-setting-for-tags-is-not-respected/181214
2021-03-03 10:59:23 +11:00
Rafael dos Santos Silva
83f332b5a5
FEATURE: Add a site setting to allow emojis to come from an external URL (#12180) 2021-03-02 16:04:16 -03:00
Gerhard Schlager
a96a5db0fb
DEV: Add option to send system message to groups (#12256) 2021-03-02 18:51:50 +01:00
David Taylor
8fd46c04ea
Drop flash video onebox (#12261)
Flash was discontinued by Adobe at the end of 2020. There is no need to continue OneBox support for it
2021-03-02 17:11:14 +00:00
Mark VanLandingham
4adce0d844
DEV: APIs for plugin to add custom reviewable confirm modal (#12246) 2021-03-02 10:28:27 -06:00
Gerhard Schlager
154bfcf750
FEATURE: Include details about S3 backup storage errors (#12257) 2021-03-02 15:29:37 +01:00
Martin Brennan
6b4d066834
FIX: Skip sending PM email for user silence (#12240)
We were sending 2 emails for user silencing if a message was provided in the UI. Also always send email for user silence and user suspend with reason regardless of whether message provided.
2021-03-02 09:18:09 +10:00
Robin Ward
79502b5b10
FIX: Use addRawTemplates even when compiling .hbr files (#12228)
Without this patch, we'd be using the `__RAW_TEMPLATES` global which is
incompatible with Ember CLI.
2021-03-01 11:46:50 -05:00
jbrw
a9b6f4d829
FIX - use ImageMagick to determine size of svg images (#12230)
SVG files can have dimensions expressed in inches, centimeters, etc., which may lead to the dimensions being misinterpreted (e.g. “8in” ends up as 8 pixels).

If the file type is `svg`, ask ImageMagick to work out what size the SVG file should be rendered on screen.

NOTE: The `pencil.svg` file was obtained from https://freesvg.org/1534028868, which has placed the file in to the public domain.
2021-03-01 11:44:00 -05:00
Penar Musaraj
6809cccd88
DEV: Add experimental json_scheme site setting type (#12226) 2021-03-01 09:15:17 -05:00
Penar Musaraj
cf0192018e
FIX: Do not output empty style tags for components (#12229) 2021-03-01 09:14:25 -05:00
Penar Musaraj
aa1442fdc3
DEV: Use separate files for theme component stylesheets (take 2) (#12225)
This switches to outputting a separate file for each theme component CSS
asset. We have separate CSS plugin files, separate JS files
(for plugins/themes/components), it makes sense to do the same for
component CSS assets.

Benefits:
- easier debugging
- fixes a regression with theme component sourcemaps
- changes to theme components are updated individually

With HTTP/2, there is also no performance downside to having additional
files in the initial request.
2021-02-26 12:30:23 -05:00
Penar Musaraj
e74bdfdf8e
Revert "DEV: Use separate files for theme component stylesheets (#12214)" (#12224)
This reverts commit f57a49c2f9.

This had some unexpected side effects, needs some more work.
2021-02-26 08:20:39 -05:00
Bianca Nenciu
4b0496b2fc
FIX: Reintroduce auto_silence_first_post_regex (#12223)
Follow-up to 533800a87b.
2021-02-26 15:07:04 +02:00
Penar Musaraj
45c5fd2172
DEV: Remove JoyPixels emoji option (#12197)
- removes the option from site settings
- deletes the site setting on existing sites that have it
- marks posts using emojis as requiring a rebake

Note that the actual image files are not removed here, the plan is to
remove them in a few weeks/months (when presumably the rebaking of old
posts has been completed).
2021-02-26 07:44:52 -05:00
Penar Musaraj
f57a49c2f9
DEV: Use separate files for theme component stylesheets (#12214)
This switches to outputting a separate file for each theme component CSS
asset. We have separate CSS plugin files, separate JS files
(for plugins/themes/components), it makes sense to do the same for
component CSS assets.

Benefits:
- easier debugging
- fixes a regression with theme component sourcemaps
- changes to theme components are updated individually

With HTTP/2, there is also no performance downside to having additional
files in the initial request.
2021-02-26 07:44:15 -05:00
Alan Guo Xiang Tan
3fc72543de DEV: Avoid duplicating method attributes in Auth::Result.
This ensures that within the class, we have a consistent order of
attributes that is passed to `UsernameSuggester`.
2021-02-26 10:20:31 +08:00
jbrw
fff8a24f2b
FIX: Don’t display error if only error is a missing image (#12216)
`Onebox.preview` can return 0-to-n errors, where the errors are missing OpenGraph attributes (e.g. title, description, image, etc.). If any of these attributes are missing, we construct an error message and attach it to the Oneboxer preview HTML. The error message is something like:

 “Sorry, we were unable to generate a preview for this web page, because the following oEmbed / OpenGraph tags could not be found: description, image”

However, if the only missing tag is `image` we don’t need to display the error, as we have enough other data (title, description, etc.) to construct a useful/complete Onebox.
2021-02-25 14:30:40 -05:00
Bianca Nenciu
533800a87b
Add watched words of type "replace" (#12020)
This commit includes other various improvements to watched words.

auto_silence_first_post_regex site setting was removed because it overlapped
with 'require approval' watched words.
2021-02-25 14:00:58 +02:00
Bianca Nenciu
a9a93b15ec
FIX: Strip whitespace from URL before validating (#12174) 2021-02-25 13:31:38 +02:00
Martin Brennan
13c2a4886f
FEATURE: Add disable_onebox_media_download_controls hidden site setting (#12208)
Uses discourse/onebox@ff9ec90

Adds a hidden site setting called disable_onebox_media_download_controls which will add controlslist="nodownload" to video and audio oneboxes, and also to the local video and audio oneboxes within Discourse.
2021-02-25 12:39:15 +10:00
Bianca Nenciu
0c8d658ba8
SECURITY: Prefer Loofah for processing cooked HTML 2021-02-24 17:17:49 +02:00
Penar Musaraj
c0fd4ab9d6
DEV: Remove deprecated plugins variables importer (#12168) 2021-02-23 16:20:59 -05:00
Rafael dos Santos Silva
01584c5b22
FIX: emails.rake-use domain when sending mail test (#12188)
`rake emails:test` task was always sending `localhost` as the domain name rather than using `smtp[:domain]` (aka `DISCOURSE_SMTP_DOMAIN`. `discourse-setup` has recently been modified to always set `DISCOURSE_SMTP_DOMAIN`, so it's important that this test rake task actually use the value.

I tested this change on a standard production install, and it's working as expected. Hopefully I managed not to bungle the copy/paste of the single line here in the github edit window.

Co-authored-by: Jay Pfaffman <jay@literatecomputing.com>
2021-02-23 14:16:51 -03:00
Penar Musaraj
0620f6298e
UX: Remove emoji step in wizard (#12169) 2021-02-22 12:14:18 -05:00
David Taylor
a040f72f96
FIX: Make email_valid handling consistent (#11556)
Previously we were checking truthiness in some places, and `== true` in
others. That can lead to some inconsistent UX where the interface says
the email is valid, but account creation fails.

This commit ensures values are boolean when set, and raises an error for
other value types.

If this safety check is triggered, it means the specific auth provider
needs to be updated to pass booleans.
2021-02-22 12:05:36 +00:00
David Taylor
04c75d417b
UX: Skip github commit avatars for topic/post thumbnails (#12157)
GitHub oneboxes use `.onebox-avatar-inline`, not `.onebox-avatar`
2021-02-22 10:40:40 +00:00
Penar Musaraj
5604ce70d4
DEV: More refactoring of SCSS importers (#12143) 2021-02-19 11:22:24 -05:00
Penar Musaraj
7ee660a017
FIX: Sort admin dashboard new updates by latest (#12146) 2021-02-19 11:03:36 -05:00
Neil Lalonde
180249112f Version bump to v2.7.0.beta4 2021-02-18 14:21:37 -05:00
Bianca Nenciu
3246c3cc92
DEV: Update mail and use fork (#10639)
Version 2.8 brings some changes to how address fields are handled and
this commits updates that and should also include a fix which handles
encoded attachment filenames.

The fork contains a bugfix to correctly decode mail attachments.
2021-02-18 20:15:02 +02:00
jbrw
447ea3f888
FIX: NewPostManager should respect category_group_moderator settings (#12116)
NewPostManager’s `post_needs_approval_in_its_category` method should allow category group moderators to create topics/reply to topics that where they have appropraite permissions.

(ie, if a user has permission to moderate a post, any posts made by them shouldn’t be sent to moderation)
2021-02-18 13:00:06 -05:00
Joshua Rosenfeld
b79c06f3b7
Include UI commits when generating release notes. (#12127)
`UX` is the officially supported prefix per https://meta.discourse.org/t/19392, but sometimes `UI` is used instead. We should still include those commits.
2021-02-18 10:53:23 -05:00
Bianca Nenciu
08acf51be0
FEATURE: Use diffhtml to update composer preview (#11237)
Displaying videos, animated GIFs or any kind of rich content in preview
used to refresh on every keystroke, which could cause performance
problems.
2021-02-18 16:07:26 +02:00
David Taylor
13d2a1f82c
SECURITY: Attach DiscourseConnect (SSO) nonce to current session (#12124) 2021-02-18 10:35:10 +00:00
Roman Rizzi
07cf0f9460
FIX: Allow plugins to correctly extend API key scopes. (#12113)
Adding a scope from a plugin was broken. This commit fixes it and adds a test.

It also documents the instance method and renames the serialized "id" attribute to "scope_id" to avoid a conflict when the scope also has a parameter with the same name.
2021-02-17 14:42:44 -03:00
Arpit Jalan
a174c8b8d4
FIX: hide sso payload behind a button click and log views (#12110) 2021-02-17 21:27:51 +05:30
David Taylor
9d7dc89d79
DEV: Replace deprecated sso site settings with discourse_connect (#12099) 2021-02-17 09:31:20 +00:00
Jeff Wong
f17e745fe3
DEV: log rake plugin:update_all plugin_path on error (#12100)
* DEV: log rake plugin:update_all plugin_path on error

Allows a dev to know which plugin was failing to update
2021-02-16 11:50:27 -08:00
Martin Brennan
9f0f801ae3
FIX: Do not mark group_flair images as secure on upload (#12081)
See https://meta.discourse.org/t/secure-media-uploads-breaks-group-flair-image/173671/4

Group flair image uploads definitely do not need to be secure.
2021-02-16 12:34:03 +10:00
Arpit Jalan
85c4e8fd32
FEATURE: support mark tag (#12088)
This commit adds support for `mark` tag for highlighting text content.
2021-02-15 21:47:30 +05:30
Gerhard Schlager
4b05fc2d2d
FIX: Restoring backup via UI was broken (#12089) 2021-02-15 16:46:44 +01:00
jbrw
70050a8ba3
FIX: should_alter_quality should respect png_to_jpg_quality (#12055)
`convert_to_jpeg!` is only called if `convert_png_to_jpeg?` and/or `should_alter_quality?` is true.

`convert_png_to_jpeg?` can be disabled by setting `SiteSetting.png_to_jpg_quality` to 100.

However, `should_alter_quality?` could be true if `SiteSetting.recompress_original_jpg_quality` was lower than the quality of the uploaded file, regardless of file type.

This commits changes `should_alter_quality?` so that uploaded png files will use the `SiteSetting.png_to_jpg_quality` value, rather than ``SiteSetting.recompress_original_jpg_quality` value.
2021-02-12 13:37:35 -05:00
Penar Musaraj
74ca5ec743
FIX: Regression in admin new features (#12068) 2021-02-12 10:49:10 -05:00
Penar Musaraj
900d4187ef
DEV: Prevents rate limits for new feature checks on multisite (#12053) 2021-02-12 08:52:59 -05:00
Daniel Waterworth
df8436cd7f
FIX: Don't add a slug to constructed quote urls (#12052)
A topic with the slug 'topic' might exist and may end up being linked to
by mistake when malformed (i.e. cross-site) quotes are posted.
2021-02-11 12:21:13 -06:00
David Taylor
b770c30391
FEATURE: Allow onebox images to be used as topic thumbnails (#12050)
Still excludes GitHub avatars. Those were the original reason for adding
this broad exclusion. Context at https://meta.discourse.org/t/165713/4

If we find more oneboxes which are unsuitable for thumbnails, we can add
them to this selector.
2021-02-11 17:50:42 +00:00
David Taylor
66151d8056
FIX: Handle empty email address from authentication provider (#12046)
If no email is provided, email_valid should be set false, so that
Discourse can prompt the user for an email and verify it.

This fixes signups via twitter for accounts with no email address.
2021-02-11 16:26:43 +00:00
David Taylor
830797a9c3
FEATURE: Allow post/topic thumbnails to be prioritized via markdown (#12044)
Previously we would always take the first image in a post to use as the
thumbnail. On media-heavy sites, users may want to manually select a
specific image as the topic thumbnail. This commit allows this to be
done via a `|thumbnail` attribute in markdown.

For example, in this case, bbb would be chosen as the thumbnail:

```
![alttext|100x100](upload://aaa)
![alttext|100x100|thumbnail](upload://bbb)
```
2021-02-11 15:44:41 +00:00
Jarek Radosz
abe688beb3 DEV: Use discourse/discourse_test in CI workflows
Using our testing Docker image (`discourse/discourse_test:release`) allows us to drop "Update imagemagick" step which shaves ~10 minutes from all runs.
2021-02-11 15:24:15 +01:00
Penar Musaraj
a1955b0542
Dev: --dev option does nothing in yarn (#12032) 2021-02-10 19:19:47 -05:00
Krzysztof Kotlarek
a1aa37758c
FIX: do not notify when the hidden tag is added or removed (#12025)
The bug was mentioned on meta https://meta.discourse.org/t/users-are-seeing-handling-of-unhandled-tag-again/155367

It was related to users who are watching a specific topic. In that case, when the hidden tag was added or removed to the topic they were notified by `NotifyTagChangeJob`.

That job should take hidden tags into consideration. If all changed tags are in a hidden group, it should exclude user not belong to that group.

At the same time, if visible to anyone tag is added or removed users watching topic should be notified.
2021-02-11 10:03:45 +11:00
Penar Musaraj
544a4e4b48
UX: Changes to new features section in admin dashboard (#12029) 2021-02-10 13:12:04 -05:00
Jarek Radosz
43948f6a10
DEV: Support for main branches in plugin:update (#12027)
Based on https://github.com/discourse/docker_manager/pull/94
2021-02-10 11:34:17 -05:00
T
1a159de36f
tasks/emails.rake: catch exception and puts message (#12024)
* tasks/emails.rake: catch exception and puts message

* add username and auth
2021-02-10 10:42:21 -05:00
Gerhard Schlager
0b05302cfe FIX: Restoring could fail due to missing path 2021-02-09 17:28:03 +01:00
Gerhard Schlager
4f5ea4fbde FIX: Restoring backup could fail due to missing uploads
Clearing theme and emoji cache might require uploaded files.
2021-02-09 17:28:03 +01:00
Gerhard Schlager
4d719725c8
FEATURE: Allow overriding the backup location when restoring via CLI (#12015)
You can use `discourse restore --location=local FILENAME` if you want to restore a backup that is stored locally even though the `backup_location` has the value `s3`.
2021-02-09 16:02:44 +01:00
David Taylor
821bb1e8cb
FEATURE: Rename 'Discourse SSO' to DiscourseConnect (#11978)
The 'Discourse SSO' protocol is being rebranded to DiscourseConnect. This should help to reduce confusion when 'SSO' is used in the generic sense.

This commit aims to:
- Rename `sso_` site settings. DiscourseConnect specific ones are prefixed `discourse_connect_`. Generic settings are prefixed `auth_`
- Add (server-side-only) backwards compatibility for the old setting names, with deprecation notices
- Copy `site_settings` database records to the new names
- Rename relevant translation keys
- Update relevant translations

This commit does **not** aim to:
- Rename any Ruby classes or methods. This might be done in a future commit
- Change any URLs. This would break existing integrations
- Make any changes to the protocol. This would break existing integrations
- Change any functionality. Further normalization across DiscourseConnect and other auth methods will be done separately

The risks are:
- There is no backwards compatibility for site settings on the client-side. Accessing auth-related site settings in Javascript is fairly rare, and an error on the client side would not be security-critical.
- If a plugin is monkey-patching parts of the auth process, changes to locale keys could cause broken error messages. This should also be unlikely. The old site setting names remain functional, so security-related overrides will remain working.

A follow-up commit will be made with a post-deploy migration to delete the old `site_settings` rows.
2021-02-08 10:04:33 +00:00
Joffrey JAFFEUX
8957e4d9d0
A11Y: makes user notifications list more accessible (#11992)
Previous markup used to be

```
<div>
  <div>
    <li>
```

Instead we will now have:

```
<ul>
  <li>
    <div>
```

Note this commit also adds two things:
- ability to override tagName of a widget when attaching it
- ability to pass opts and otherOpts to {{attach}}, it could be useful in templates but is mostly useful to test `tagName` for now
2021-02-08 08:45:14 +01:00
Kris
ded0f5b4e5
UX: Remove Helvetica from our font stack (#11876)
Arial is more consistently available across operating systems and aligns better.
2021-02-05 17:01:21 -05:00
Alan Guo Xiang Tan
4b3f65bb26 FIX: Select earliest post when aggregating posts in a topic for search.
This is a revert of
d8c796bc44
and
5bf0a0893b.

Linking to the post within a topic that has the highest rank was
confusing users and hard to explain because ranking is determined via
the PG ranking function. See the following meta topics for the
complaints after we switch to the new ordering:

1. https://meta.discourse.org/t/title-search-not-working-as-expected/157737
2. https://meta.discourse.org/t/search-results-should-prioritize-first-post-in-topic-when-title-matches-search-term/175154
2021-02-05 09:52:53 +08:00
Martin Brennan
4af77f1e38
FEATURE: Allow durations < 1 hour and < 1 day for topic timers where duration is specified (auto delete replies, close based on last post) (#11961)
This PR allows entering a float value for topic timers e.g. 0.5 for 30 minutes when entering hours, 0.5 for 12 hours when entering days. This is achieved by adding a new column to store the duration of a topic timer in minutes instead of the ambiguous both hours and days that it could be before.

This PR has ommitted the post migration to delete the duration column in topic timers; it will be done in a subsequent PR to ensure that no data is lost if the UPDATE query to set duration_mintues fails.

I have to keep the old keyword of duration in set_or_create_topic_timer for backwards compat, will remove at a later date after plugins are updated.
2021-02-05 10:12:56 +10:00
David Taylor
414224ce07
DEV: Allow autospec to run full test suite in parallel (#11976)
The definition of 'multiple_files' did not consider that 'spec' refers to the entire `spec/` directory, and therefore includes multiple files
2021-02-04 14:44:40 +00:00
Penar Musaraj
12ffba771c
FIX: Improve SCSS handling in components (#11963)
- ignores errors when including component SCSS in parent theme
- adds support for SCSS `@import`s in components' `color_definitions.scss` files
2021-02-04 08:51:18 -05:00
Penar Musaraj
3da2e85855
FIX: Save updated remote theme, only update enabled themes/components (#11960) 2021-02-04 14:44:07 +11:00
Krzysztof Kotlarek
f39e7fe81d
FEATURE: New way to dismiss new topics (#11927)
This is a try to simplify logic around dismiss new topics to have one solution to work in all places - dismiss all-new, dismiss new in a specific category or even in a specific tag.
2021-02-04 11:27:34 +11:00
Robin Ward
61f5d501cb
DEV: Migrate to Ember CLI (#11932)
This encompasses a lot of work done over the last year, much of which
has already been merged into master. This is the final set of changes
required to get Ember CLI running locally for development.

From here on it will be bug fixes / enhancements.

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Co-authored-by: romanrizzi <rizziromanalejandro@gmail.com>

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Co-authored-by: romanrizzi <rizziromanalejandro@gmail.com>
2021-02-03 14:22:20 -05:00
Jarek Radosz
8ad5284cf7
FIX: Don't create email invites when SSO is on or local logins are off (#11951)
A more general, lower-level change in addition to #11950.

Most code paths already check if SSO is enabled or if local logins are disabled before trying to create an email invite.
This is a safety net to ensure no invalid invites sneak by. 

Also includes:
FIX: Don't allow to bulk invite when SSO is on (or when local logins are disabled)
This mirrors can_invite_to_forum? and other email invite code paths.
2021-02-03 19:01:23 +01:00
Arpit Jalan
7f86a310ef
FEATURE: check blocked_onebox_domains setting for inline oneboxes (#11944) 2021-02-03 21:45:22 +05:30
Penar Musaraj
2309032e68
FIX: Include extra SCSS in child theme (#11952) 2021-02-03 11:02:53 -05:00
Penar Musaraj
e8b82724fd
DEV: Refactor theme SCSS compilation (#11919) 2021-02-02 13:09:41 -05:00
Gerhard Schlager
f88def5f5b
PERF: Avoid lookbehinds when replacing links in imported emails (#11931)
Follow-up to 3c678df942
2021-02-02 17:34:00 +01:00
Vinoth Kannan
9d2eaec88f
DEV: enable CORS to all CDN get requests from workbox. (#11896)
To prevent opaque cache files, now all the CDN files will be requested in 'cors' mode if the cdn_cors_enabled global setting is enabled. Before enabling the setting, should enable the cors in the CDN server by adding the response header `access-control-allow-origin: *` or `access-control-allow-origin: https://discourse.example.com.`

And other external file requests other than CDN will not be cached if the response type is opaque.
2021-02-02 11:38:29 +05:30
Roman Rizzi
e040de0c2c
DEV: New option to always destroy posts. (#11898) 2021-02-01 16:57:31 -05:00
Dan Ungureanu
dd175537f3
FIX: Existing shared drafts should be accessible (#11915)
Disabling shared drafts used to leave topics in an inconsistent state
where they were not displayed as shared drafts and thus there was no
way of publishing them. Moreover, they were accessible just to users
who have permissions to create shared drafts.

This commit adds another permission check that is used for most
operations and the old can_create_shared_draft? remains used just when
creating a new shared draft.
2021-02-01 16:16:34 +02:00
Vinoth Kannan
a5923ad603
DEV: apply allow origin response header for CDN requests. (#11893)
Currently, it creates a CORS error while accessing those static files.
2021-01-29 07:44:49 +05:30
Martin Brennan
4af4d36175
FIX: IMAP allow unknown senders to reply to group topics via email (#11877)
Adds a new column/setting to groups, allow_unknown_sender_topic_replies, which is default false. When enabled, this scenario is allowed via IMAP:

* OP sends an email to the support email address which is synced to a group inbox via IMAP, creating a group topic
* Group user replies to the group topic
* An email notification is sent to the OP of the topic via GroupSMTPMailer
* The OP has several email accounts and the reply is sent to all of them, or they forward their reply to another email account
* The OP replies from a different email address than the OP (gloria@gmail.com instead of gloria@hey.com for example)
* The a new staged user is created, the new reply is accepted and added to the topic, and the staged user is added to the topic allowed users

Without allow_unknown_sender_topic_replies enabled the new reply creates an entirely new topic (because the email address it is sent from is not previously part of the topic email chain).
2021-01-29 09:59:10 +10:00
Martin Brennan
f49e3e5731
DEV: Add security_last_changed_at and security_last_changed_reason to uploads (#11860)
This PR adds security_last_changed_at and security_last_changed_reason to uploads. This has been done to make it easier to track down why an upload's secure column has changed and when. This necessitated a refactor of the UploadSecurity class to provide reasons why the upload security would have changed.

As well as this, a source is now provided from the location which called for the upload's security status to be updated as they are several (e.g. post creator, topic security updater, rake tasks, manual change).
2021-01-29 09:03:44 +10:00
Joshua Rosenfeld
b70e0d88f7
Include accessibility commits when generating release notes (#11884) 2021-01-28 12:58:04 -05:00
Neil Lalonde
c4dc610498 Version bump to v2.7.0.beta3 2021-01-28 12:53:10 -05:00
Robert
d8846e4fcd
FEATURE: add contact emails metadata entry to optional plugin meta (#11879) 2021-01-28 11:17:00 -05:00
Penar Musaraj
568bad75c1
FIX: Support version in new feature payload (#11841)
Followup to 4f01ca87e3
2021-01-25 16:30:43 -05:00
Joffrey JAFFEUX
8edd2b38cb
FIX: ensures timeline_lookup includes last tuple (#11829)
A simplified version of the logic used in the function before my fix is as follow:

```ruby
result = []
things = [0,1,2,3]
max_values = 2
every = (things.size.to_f / max_values).ceil

things.each_with_index do |t, index|
  next unless (t % every) === 0
  result << t
end

p result # [0, 2]
# 3 doesn’t get included
```

The problem is that if you get unlucky two times you won't get last tuple(s) and might get a very erroneous date.

Double unlucky:
- last tuple index % computed every !== 0 and you don't get the last tuple
- the last tuple is related to a post with a very different date than the previous tuples (on year difference in our case)
2021-01-25 11:30:59 +01:00
Penar Musaraj
4f01ca87e3
FEATURE: Add new features section in admin dashboard (#11731) 2021-01-22 10:09:02 -05:00
Neil Lalonde
c232b5a291 Version bump to v2.7.0.beta2 2021-01-21 14:52:01 -05:00
Robin Ward
53ab3dda5d
FIX: Embedded comments should only return regular posts (#11773)
There shoudln't be a situation where you'd want to see moderator actions
or small posts.
2021-01-21 12:47:03 -05:00
Martin Brennan
9ee8a01c3a
FIX: Change default for IncomingEmail#created_via to 0 (unknown) and make NOT NULL (#11782)
Follow up to https://review.discourse.org/t/dev-add-created-via-column-to-incomingemail-pr-11751/18366/6
2021-01-21 12:59:50 +10:00
Martin Brennan
f34fa999a2
DEV: IMAP debugging improvements (#11784)
Improvements to make console access to IncomingEmail more pleasant, and stopping certain IMAP logs from landing in the DB because they just create too much noise,
2021-01-21 11:37:47 +10:00
Gerhard Schlager
3b2f6e129a
FEATURE: Add English (UK) as locale (#11768)
* "English" gets renamed into "English (US)"
* "English (UK)" replaces "English"

@discourse-translator-bot keep_translations_and_approvals
2021-01-20 21:32:22 +01:00
Penar Musaraj
a9dfc5a547
DEV: Use absolute path for popper.js sourcemap (#11762) 2021-01-20 08:59:34 -05:00
Bianca Nenciu
8536521a72
DEV: Add DiscourseEvent before post changes are published (#11759)
This can be used to add to the message notifying the client about the
update.
2021-01-20 10:51:31 +02:00
Martin Brennan
fb184fed06
DEV: Add created_via column to IncomingEmail (#11751)
This should make it easier to track down how the incoming email was created, which is one of four locations:

The POP3 poller (which picks up reply via email replies)
The admin email controller #handle_mail (which is where hosted mail is sent)
The IMAP sync tool
The group SMTP mailer, which sends emails when replying to IMAP topics, pre-emptively creating IncomingEmail records to avoid double syncing
2021-01-20 13:22:41 +10:00
Dan Ungureanu
1f2f84a6df
FIX: Add Retry-Header to rate limited responses (#11736)
It returned a 429 error code with a 'Retry-After' header if a
RateLimiter::LimitExceeded was raised and unhandled, but the header was
missing if the request was limited in the 'RequestTracker' middleware.
2021-01-19 11:35:46 +02:00
Roman Rizzi
ea8b5c18db
UX: Text customization for different languages. (#11729)
Admins can now edit translations in different languages without having to change their locale. We display a warning when there's a fallback language set.
2021-01-18 14:53:45 -03:00
Justin DiRose
be1b75187e
DEV: Rename Knowledge Explorer to Docs (#11716)
Removes the old discourse-knowledge-explorer repository and adds the renamed version, discourse-docs.
2021-01-18 10:58:05 -06:00
Jarek Radosz
650e3d18c4
DEV: Add a note to S3 migration task (#11738)
A followup to #11703
2021-01-18 17:12:47 +01:00
Michael K Johnson
2a23e54632
FIX: remove migrate_from_s3 task that silently corrupts data (#11703)
Transient errors in migration are ignored, silently corrupting
data, and the migration is incomplete and misses many sources of
uploads, which will lead to an incorrect expectation of independence
from the remote object storage after announcing that the migration
was successful, regardles of whether transient errors permanently
corrupted the data.

Remove this migration until such time as it is re-written to
follow the same pattern has the migration to s3, moving the
core logic out of the task.
2021-01-17 22:33:29 +01:00
Martin Brennan
bd25627198
FIX: IMAP post alerter race condition and code improvements (#11711)
This PR fixes a race condition with the IMAP notification code. In the `Email::Receiver` we call the `NewPostManager` to create the post and enqueue jobs and sends alerts via `PostAlerter`. However, if the post alerter reaches the `notify_pm_users` and the `group_notifying_via_smtp` method _before_ the incoming email is updated with the post and topic, we unnecessarily send a notification to the person who just posted. The result of this is that the IMAP syncer re-imports the email sent to the user about their own post, which looks like this in the group inbox:

To fix this, we skip the jobs enqueued by `NewPostManager` and only enqueue them with `PostJobsEnqueuer` manually _after_ the incoming email record has been updated with the post and topic.

Other improvements:

* Moved code to calculate email addresses from `IncomingEmail` records into the topic, with a group passed in, for easier testing and debugging. It is not the responsibility of the post alerter to figure this stuff out.
* Add shortcut methods on `IncomingEmail` to split or provide an empty array for to and cc addresses to avoid repetition.
2021-01-15 10:54:46 +10:00
Krzysztof Kotlarek
06b7c44593
FEATURE: reason to reject user signup (#11700)
Feature for `Must Approve Users` setup. When a user is rejected, a staff member can optionally set a reason for audit purposes. In addition, feedback email can be sent to the user.

Meta: https://meta.discourse.org/t/account-rejection-email/103112/8
2021-01-15 09:43:26 +11:00
Martin Brennan
87961534ea
FEATURE: IMAP detect spammed email and delete associated Discourse topic (#11654)
This PR adds functionality for the IMAP sync code to detect if a UID that is missing from the mail group mailbox is in the Spam/Junk folder for the mail account, and if so delete the associated Discourse topic. This is identical to what we do for emails that are moved for Trash.

If an email is missing but not in Spam or Trash, then we mark the incoming email record with imap_missing: true. This may be used in future to further filter or identify these emails, and perhaps go hunting for them in the email account in bulk.

Note: This adds some code duplication because the trash and spam email detection and handling is very similar. I intend to do more refactors/improvements to the IMAP sync code in time because there is a lot of room for improvement.
2021-01-14 09:54:18 +10:00
Bianca Nenciu
74b95c88ac
FIX: Use ImageMagick to detect animated images (#11702)
This is a fallback when FastImage cannot be used (animated WEBP images).
2021-01-13 19:01:30 +02:00
Martin Brennan
2404fa7a23
DEV: Split toggle topic close job (#11679)
Splits the `ToggleTopicClosed` job into two distinct `OpenTopic` and `CloseTopic` jobs to make the code clearer. The old job cannot be deleted yet because of outstanding sidekiq schedules, so a todo has been added to do so later this year.

Also replaced mentions of `topic_status_update` with `topic_timer` in some files, because the `topic_status_update` model is obsolete and replaced by topic timer.

Added some shortcut methods for checking if a topic is open/whether a user can change an open topic.
2021-01-13 08:49:29 +10:00
Andrew Prigorshnev
0f1c9a2180
FIX: Use the same time moment for related Redis calls in rate limiter (#11692)
Co-authored-by: Robin Ward <robin.ward@gmail.com>
2021-01-12 15:09:15 -05:00
Andrew Prigorshnev
e25dd41aee
FIX: sliding window end time in rate limiter (#11691)
If the sliding window size is N seconds, then a moment at the Nth second
should be considered as the moment outside of the sliding window.

Otherwise, if the sliding window is already full, at the Nth second,
a new call wouldn't be allowed, but a time to wait before the next call
would be equal to zero, which is confusing.

In other words, the end of the time range shouldn't be included in the
sliding window.

Let's say we start at the second 0, and the sliding window size is 10
seconds. In the current version of rate limiter, this sliding window will
be considered as a time range [0, 10] (including the end of the range),
which actually is 11 seconds in length.

After this fix, the time range will be considered as [0, 10)
(excluding the end of the range), which is exactly 10 seconds in length.
2021-01-12 13:26:43 -05:00
Bianca Nenciu
ec0212e56b
FIX: Make category slugs lowercase (#11277)
Admins could specify category slug with upper case characters and same slug,
but with different cases could be used simultaneously.
2021-01-12 17:28:33 +02:00
Bianca Nenciu
499a594754
FIX: Do not downsize or crop GIF images (#10989)
It was a problem because during this operation only the first frame
is kept. This commit removes the alternative solution to check if a GIF
image is animated.
2021-01-12 17:07:07 +02:00
jbrw
b1f32f2f57
DEV: TopicTrackingState calls should happen in the background (#11682)
* DEV: TopicTrackingState calls should happen in the background

It was observed that calling TopicTrackingState on popular topics could result in a large number of calls to redis, resulting in slow response times when posting replies.

These calls should be moved to a background job.

* DEV: PostUpdateTopicTrackingState should execute on default queue
2021-01-11 15:58:27 -05:00
Krzysztof Kotlarek
efaa63bd1d
FIX: always truncate uncomplete emojis in excerpts (#11678)
Additional fix after https://github.com/discourse/discourse/pull/11667

Always truncate "broken" emojis from excerpts.
2021-01-11 13:43:11 +11:00