Figuring out what unread topics a user has is a very expensive
operation over time.
Users can easily accumulate 10s of thousands of tracking state rows
(1 for every topic they ever visit)
When figuring out what a user has that is unread we need to join
the tracking state records to the topic table. This can very quickly
lead to cases where you need to scan through the entire topic table.
This commit optimises it so we always keep track of the "first" date
a user has unread topics. Then we can easily filter out all earlier
topics from the join.
We use pg functions, instead of nested queries here to assist the
planner.
Added order and direction parameters for sorting admin user pages. This
commit only includes backend api changes.
https://meta.discourse.org/t/make-admin-users-list-sortable-suggestion/47649
Now you can pass in `order` and `asc` parameters to the
`/admin/users/list/<query>.json` endpoint.
Example:
`/admin/users/list/active.json?&order=post_count` which defaults to desc
and
`/admin/users/list/active.json?order=post_count&asc=true`
Revamped system for managing authentication tokens.
- Every user has 1 token per client (web browser)
- Tokens are rotated every 10 minutes
New system migrates the old tokens to "legacy" tokens,
so users still remain logged on.
Also introduces weekly job to expire old auth tokens.
Scrubbing an ASCII-8BIT string isn't ever going to remove anything, because
there's no code point that isn't valid 8-bit ASCII. Since we'd really
prefer it if everything were UTF-8 anyway, we'll just assume, for now, that
whatever comes out of SimpleRSS is probably UTF-8, and just nuke anything
that isn't a valid UTF-8 codepoint.
Of course, the *real* bug here is that SimpleRSS [unilaterally converts
everything to
ASCII-8BIT](https://github.com/cardmagic/simple-rss/issues/15). It's
presumably *far* too much to ask that it detects the encoding of the source
RSS feed and marks the parsed strings with the correct encoding...
This reduces the number of scans that the db has to do in the query
to fetch orphan uploads. Futheremore, we were not batching our
records which bloats memory.
- All unsubscribes go to the exact same page
- You may unsubscribe from watching a category on that page
- You no longer need to be logged in to unsubscribe from a topic
- Simplified footer on emails
* Rearrange frontend to account for mailing list mode
* Allow update of user preference for mailing list frequency
* Add mailing list frequency estimate
* Simplify frequency estimate; disable activity summary for mailing list mode
* Remove combined updates
* Add specs for enqueue mailing list mode job
* Write mailing list method for mailer
* Fix linting error
* Account for stale topics
* Add translations for default mailing list setting
* One query for mailing list topics
* Fix failing spec
* WIP
* Flesh out html template
* First pass at text-based mailing list summary
* Add user avatar
* Properly format posts for mailing list
* Move make_all_links_absolute into Email::Styles
* Apply first_seen_at to user
* Send mailing list email summary hourly based on first_seen_at
* Branch and test cleanup
* Use existing mailing list mode estimate
* Fix failing specs
- Show bounce score on user admin page
- Added reset bounce score button on user admin page
- Only whitelisted email types are sent to emails with high bounce score
- FIX: properly detect bounces even when there is no TO: header in the email
- Don't desactivate a user when reaching the bounce threshold
As it stands we load up user records quite frequently on the topic pages,
this in turn pulls all the columns for the users being selected, just to
discard them after they are loaded
New structure keeps all options in a discrete table, this is better organised
and allows us to easily add more column without worrying about bloating the
user table
There were actually 2 bugs:
1/ Calling '.try(:key)' on a hash doesn't work. So EmailLogs were never associated to a user.
2/ Turns out that we update the 'user.last_emailed_at' whenever we create an EmailLog (in the 'after_create' callback).
So we need to always create an EmailLog (whenever the email is sent or skipped).
There are people who have RSS feeds set up that do HTTPS -> HTTP
redirects which throw errors. Since RSS feeds are all configured
by admins I think it's OK if they allow an unsafe redirect as the
content is public anyway. This will reduce many server side errors.
We cap new and unread at 2/5th of SiteSetting.max_tracked_new_unread
This dynamic capping is applied under 2 conditions:
1. New capping is applied once every 15 minutes in the periodical job, this effectively ensures that usually even super active sites are capped at 200 new items
2. Unread capping is applied if a user hits max_tracked_new_unread,
meaning if new + unread == 500, we defer a job that runs within 15 minutes that will cap user at 200 unread
This logic ensures that at worst case a user gets "bad" numbers for 15 minutes and then the system goes ahead and fixes itself up
FEATURE: allow backup interval of up to 30 days
FIX: if a custom file exists in backup directory look at its date
FEATURE: site setting automatic_backups_enabled default true
SimpleRss is unreliable with parsing RSS feeds that contain German Umlauts.
For example this feed http://www.lauffeuer-lb.de/api/v2/articles.xml can't be
parsed by SimpleRss. Discourse's logs are full of
```
Job exception: Wrapped Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8
Job exception: incompatible character encodings: ASCII-8BIT and UTF-8
```
The embedding fails because the feed can't be parsed.
This change forces the encoding (using #scrub) which prevents the numerous
encoding errors.
- new hidden site setting 'migrate_to_new_scheme' (defaults to false)
- new rake tasks to toggle migration to new scheme
- FIX: migrate_to_new_scheme also works with CDN
- PERF: improve perf of the DbHelper.remap method
- REFACTOR: UrlHelper is now a class
