Commit Graph

468 Commits

Author SHA1 Message Date
Mark VanLandingham
3dd2f2f701
DEV: Remove RSS feed polling in favor of plugin (#8233) 2019-11-12 09:49:02 -06:00
David Taylor
52c5cf33f8
FEATURE: Overhaul of admin API key system (#8284)
- Allow revoking keys without deleting them
- Auto-revoke keys after a period of no use (default 6 months)
- Allow multiple keys per user
- Allow attaching a description to each key, for easier auditing
- Log changes to keys in the staff action log
- Move all key management to one place, and improve the UI
2019-11-05 14:10:23 +00:00
Krzysztof Kotlarek
c32bd8ae48 FEATURE: Remove attachments and truncate raw field for incoming emails (#8253)
Adds the settings: 

raw_email_max_length, raw_rejected_email_max_length, delete_rejected_email_after_days. 

These settings control retention of the "raw" emails logs.

raw_email_max_length ensures that if we get incoming email that is huge we will truncate it removing uploads from the raw log.

raw_rejected_email_max_length introduces an even more aggressive truncation for rejected incoming mail. 

delete_rejected_email_after_days controls how many days we will keep rejected emails for (default 90)
2019-10-30 16:54:35 +11:00
Sam Saffron
3d85cc1e69 PERF: run expensive clean up uploads less frequently
Previously every hour we would run a full scan of the entire DB searching
for expired uploads that need to be moved to the tombstone folder.

This commit amends it so we only run the job 2 times per clean_orpha_uploads_grace_period_hours

There is a upper bound of 7 days so even if the grace period is set really
high it will still run at least once a week.

By default we have a 48 grace period so this amends it to run this cleanup
daily instead of hourly. This eliminates 23 times we run this ultra expensive
query.
2019-10-28 11:14:52 +11:00
Dan Ungureanu
1358312584
FIX: Zeitwerk-related fixes for jobs. (#8219) 2019-10-21 20:25:35 +03:00
Joffrey JAFFEUX
b6ef473a31
DEV: prevents redefinition of various constants (#8199)
CLEANUP_GRACE_PERIOD
MAX_AWARDED
POLL_MAILBOX_TIMEOUT_ERROR_KEY
2019-10-17 09:18:07 +02:00
Dan Ungureanu
086b46051c
FIX: Zeitwerk-related fixes for jobs. (#8187) 2019-10-14 13:03:22 +03:00
Krzysztof Kotlarek
427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Robin Ward
d251f12c9c Tweak calculation for reviewable sensitivities/priorities
Previously, calculating thresholds for reviewables was done based on the
50th and 85th percentile across all reviewables. However, many forum
owners provided feedback that these thresholds were too easy to hit, in
particular when it came to auto hiding content.

The calculation has been adjusted to base the priorities on reviewables
that have a minimum of 2 scores (flags). This should push the amount of
flags required to hide something higher then before.
2019-09-19 14:07:56 -04:00
Robin Ward
d5b52abf2f FIX: Require a min amount of reviewables before calculating thresholds
On forums with very few flags you don't want to calculate averages
because they won't be very useful. Stick with the defaults until we hit
15 reviewables at least.
2019-09-19 13:42:50 -04:00
Roman Rizzi
568232052e
DEV: Remove FlagQuery class and old code (#8064) 2019-09-12 13:21:33 -03:00
Gerhard Schlager
56c6e8e961 FIX: POP3 doesn't work with TLS 1.3 2019-09-11 18:43:19 +02:00
David Taylor
e2449f9f23 Revert "Revert "Revert "FIX: Heartbeat check per sidekiq process (#7873)"""
This reverts commit c3497559be.
2019-08-30 11:26:16 +01:00
David Taylor
5d4fcc62bb Revert "FIX: heartbeat should be per host"
This reverts commit 17bed44405.
2019-08-30 11:25:59 +01:00
Sam Saffron
17bed44405 FIX: heartbeat should be per host
We need to schedule heartbeat on the correct host, otherwise some queues
may end up starving.
2019-08-30 16:10:08 +10:00
Sam Saffron
c3497559be Revert "Revert "FIX: Heartbeat check per sidekiq process (#7873)""
This reverts commit e805d44965.
We now have mechanisms in place to ensure heartbeat will always
be scheduled even if the scheduler is overloaded per: 098f938b
2019-08-30 10:12:10 +10:00
Sam Saffron
4fce6484fe PERF: reduce window of consistency on user actions
Databases can have a lot of user actions, self joining and running an
aggregate on millions of rows can be very costly

This optimisation will reduce the regular window of consistency down to 13
hours, this ensures the job runs much faster
2019-08-29 13:27:04 +10:00
OsamaSayegh
e805d44965 Revert "FIX: Heartbeat check per sidekiq process (#7873)"
This reverts commit 340855da55.
2019-08-27 11:56:23 +00:00
Osama Sayegh
340855da55
FIX: Heartbeat check per sidekiq process (#7873)
* FIX: Heartbeat check per sidekiq process

* Rename method

* Remove heartbeat queues of previous bootups

* Regis feedback

* Refactor before_start

* Update lib/demon/sidekiq.rb

Co-Authored-By: Régis Hanol <regis@hanol.fr>

* Update lib/demon/sidekiq.rb

Co-Authored-By: Régis Hanol <regis@hanol.fr>

* Expire redis keys after 3600 seconds

* Don't use redis to store the list of queues
2019-08-26 09:33:49 +03:00
Gerhard Schlager
00b75b4f4e FIX: Don't try to delete staged, unused admins and mods 2019-08-21 15:29:51 +02:00
Gerhard Schlager
cc7b24b88b FIX: Regularly reset unknown extension of uploads
Discourse will try to detect the extension the next time the upload is needed. Maybe there was just a transient error the last time.
2019-08-21 10:23:20 +02:00
Bianca Nenciu
f63b8bb79d FIX: Periodically ensure consistency of categories. (#7663) 2019-06-06 11:30:52 +02:00
Neil Lalonde
ecc9c76692
FEATURE: dynamically update the topic heat settings monthly (#7670)
The site settings beginning with "topic views heat" and "topic post like
heat" are set to defaults when installing Discourse, but there has not
been a process or guidance for updating these values based on
community activity.

This feature will update them once a month. The low, medium, and
high settings will be based on the minimums of the 45th, 25th, and
10th percentile topics respectively, so that 45% of topics will have
some "heat".

Disable automatic changes with the automatic_topic_heat_values setting.
2019-06-04 10:34:07 -04:00
Sam Saffron
22abad4151 PERF: stop reindexing and skipping deleted posts 2019-06-04 17:53:35 +10:00
Sam Saffron
74c4f926fc FIX: drop deleted posts from search index
This does two things

1. Our "index grace period" has been wound down to 1 day, there is no point
keeping a bloated index for a week, usually when people delete stuff they
mean for it to be removed

2. We were never dropping deleted posts from the index, only posts from
deleted topics

These changes speed up search a tiny bit and reduce background work.
2019-06-04 17:19:59 +10:00
Sam Saffron
77300c1d8d DEV: reindex old data in a more consistent way
Previously we were grabbing arbitrary rows in many cases which makes
diagnosing issues in the indexer more complex
2019-06-04 11:47:24 +10:00
Robin Ward
b85fa62882 FIX: Don't send replies to flags that are ignored by expiry
The `AutoQueueHandler` will ignore really old flags. In that case, don't
notify the user that the moderator is looking into it. They probably
never saw it because it didn't meet the reviewable minimum priority.
2019-05-27 17:24:36 -04:00
Robin Ward
e74cd54fc6 REFACTOR: Replace score bonuses with low/med/high priorities
We removed score from the UX so it makes more sense to have sites set
priorities instead of score bonuses.
2019-05-23 11:54:45 -04:00
Robin Ward
225593d445 FIX: Small tweak to percentiles based on feedback 2019-05-16 13:45:36 -04:00
Gerhard Schlager
bbab60fa9b FIX: Don’t try to delete inactive moderators
follow-up to 6d77156a
2019-05-13 21:44:03 +02:00
Guo Xiang Tan
8165ceb320 Make rubocop happy. 2019-05-13 09:55:44 +08:00
Sam Saffron
30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Robin Ward
5af7c90bab FEATURE: Hide Reviewable scores, change score filter to Priority
We found score hard to understand. It is still there behind the scenes
for sorting purposes, but it is no longer shown.

You can now filter by minimum priority (low, med, high) instead of
score.
2019-05-07 14:05:23 -04:00
Guo Xiang Tan
152238b4cf DEV: Prefer public_send over send. 2019-05-07 09:33:21 +08:00
Sam Saffron
9be70a22cd DEV: introduce new API to look up dynamic site setting
This removes all uses of both `send` and `public_send` from consumers of
SiteSetting and instead introduces a `get` helper for dynamic lookup

This leads to much cleaner and safer code long term as we are always explicit
to test that a site setting is really there before sending an arbitrary
string to the class

It also removes a couple of risky stubs from the auth provider test
2019-05-07 11:00:30 +10:00
Sam Saffron
f8eddd40ad PERF: remove avg_time calculations and regular jobs from posts and topics
After careful analysis of large data-sets it became apparent that avg_time
had no impact whatsoever on "best of" topic scoring. Calculating avg_time
was a very costly operation especially on large databases.

We have some longer term plans of introducing other weighting that is read
time based into our scoring for "best of" and "top" topics, but in the
interim to stop a large amount of work that is not achieving any value we
are removing the jobs.

Column removal will follow once we decide on a new replacement metric.
2019-05-06 15:59:01 +10:00
Guo Xiang Tan
24347ace10 FIX: Properly associate user_profiles background urls via upload id.
`Upload#url` is more likely and can change from time to time. When it
does changes, we don't want to have to look through multiple tables to
ensure that the URLs are all up to date. Instead, we simply associate
uploads properly to `UserProfile` so that it does not have to replicate
the URLs in the table.
2019-05-02 14:58:24 +08:00
Sam Saffron
8fc2b0124f PERF: destroy users in batches of 50 users
This speeds up the process as we no longer need to commit a transaction
per user deleted
2019-04-26 22:51:33 +10:00
Gerhard Schlager
6d77156a94 FIX: Don't try to delete inactive admins 2019-04-25 11:59:50 +02:00
Guo Xiang Tan
e351b2ee26 Revert "FEATURE: stop running migrate to new scheme once done"
This reverts commit 2e346e6ea2.

Super seeded by 573ddcbbd0.
2019-04-24 14:03:31 +08:00
Sam Saffron
2e346e6ea2 FEATURE: stop running migrate to new scheme once done
This setting will automatically disable itself when there are no longer any
problem uploads
2019-04-24 14:32:39 +10:00
Guo Xiang Tan
149411ec90 PERF: Speed up Upload.migrate_to_new_scheme by limiting remap scope.
Doing a `LIKE` on `Post#raw` and `Post#cooked` takes forever on large
sites.
2019-04-24 11:56:48 +08:00
Robin Ward
6f56fba016 UX: Update post actions to "Approve Post" and "Reject Post"
This should be more clear.
2019-04-23 12:19:11 -04:00
Robin Ward
14f9d40e48 FEATURE: Clarify Reviewable User Actions
"Approve" is now "Approve User" and "Delete" is a dropdown with a choice
that allows you to block.
2019-04-17 11:44:17 -04:00
Dan Ungureanu
e23c87c442
FIX: Prevent exception in rescue block.
> Job exception: undefined method `id' for nil:NilClass
> clean_up_inactive_users.rb:25:in `rescue in block in execute'
2019-04-16 01:27:39 +03:00
Robin Ward
f7ebfb1acc FIX: Only consider pending queued posts for cleaning up uploads 2019-04-12 14:39:32 -04:00
Régis Hanol
ab79060468 FIX: don't send activation reminder to staged users 2019-04-10 17:29:08 +02:00
Bianca Nenciu
3d545d66df FEATURE: Send user activation reminders. (#7280) 2019-04-10 16:53:52 +02:00
Tarek Khalil
442fb2facb FEATURE: Remove ignore feature SiteSetting and enable ignore by default (#7349) 2019-04-10 12:54:59 +02:00
Sam Saffron
ec1c3559da PERF: correct clean up inactive so it does not clog scheduler
also add a hard limit of 1000 users per job run so we do not clog the
scheduler

destroyer.destroy has a transaction and this can have some serious complications
with the open record set find_each has going
2019-04-09 22:25:05 +10:00