Commit Graph

42 Commits

Author SHA1 Message Date
Alan Guo Xiang Tan
3193afe7ca
FIX: Rescue and warn when error is encountered in DiscourseIpInfo.mmdb_download ()
Since switching to Maxmind permalinks to download the databases in
7079698cdf, we have received multiple
reports about rebuilds failing as `maxminddb:refresh` runs during
the rebuilds and failing to download the databases cases the rebuilds to
fail.

Downloading Maxmind databases should not sit in the critical rebuild
path but since we are close to the Discourse 3.3 release, we have opted
to just rescue all errors encountered when downloading the databases.

In the near future after the Discourse 3.3 release, we will be looking
at moving the downloading of maxmind databases out of the rebuild path.
2024-07-30 11:33:20 +08:00
Jay Pfaffman
6a5ee0693d
FIX: rescue mmdb download failure ()
* rescue mmdb download failure

If mmdb database fails to download a bootstrap fails. This is a trivial fix for that problem. A more elegant solution might check whether the dataabase was downloaded and provide a helpful error message.

* linting

---------

Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2024-06-28 05:10:14 +08:00
Alan Guo Xiang Tan
7079698cdf
FIX: Use MaxMind supplied permalinks to download MaxMind databases ()
This commit switches `DiscourseIpInfo.mmdb_download` to use the
permalinks supplied by MaxMind to download the MaxMind databases as
specified in
https://dev.maxmind.com/geoip/updating-databases#directly-downloading-databases
which states:

```
To directly download databases, follow these steps:

1. In the "Download Links" column, click "Get Permalink(s)" for the desired database.
2. Copy the permalink(s) provided in the modal window.
3. Provide your account ID and your license key using Basic Authentication to authenticate.
```

Previously we are downloading from `https://download.maxmind.com/app/geoip_download` but this is not
documented anyway on MaxMind's docs so this URL can in theory break
in the future without warning. Therefore, we are taking a proactive
approach to download the databases from MaxMind the recommended way
instead of relying on a hidden URL. This old way of downloading the
databases with only a license key will be deprecated in 3.3 and be
removed in 3.4.
2024-05-09 15:11:56 +08:00
Alan Guo Xiang Tan
6edadeab27
FIX: DiscourseIpInfo.mmdb_download incorrectly joining URLs ()
This commit changes `DiscourseIpInfo.mmdb_download` to use `File.join`
instead of `URI.join` when `GlobalSetting.maxmind_mirror_url` has been
configured. This is necessary because `URI.join` does not work the way I
expect it to work when I implemented it previously.

`URI.join("http://www.example.com/mirror", "test.tar.gz") results in
`http://www.example.com/test.tar.gz` instead of our expected
`http://www.exmaple.com/mirror/test.tar.gz`. For our simple use case
here, `File.join` is sufficient.
2024-04-08 10:27:40 +08:00
Alan Guo Xiang Tan
9182501366
DEV: Introduce maxmind_mirror_url GlobalSetting ()
Why this change?

This allows downloading the MaxMind databases from a mirror in cases
where downloading directly from MaxMind's API endpoint is problematic
due to API limits.
2024-04-02 14:53:53 +08:00
Jay Pfaffman
63f6c1e81a
FIX: Allow redirect when downloading maxmind database ()
Per https://dev.maxmind.com/geoip/release-notes/2024#presigned-urls-for-database-downloads 

MaxMind users who download databases should make sure that their HTTP client follows redirects and there are no proxy or firewall settings that would block requests to the host we are redirecting to.
2024-03-26 09:39:09 +11:00
Sam
c2fd090d7d
DEV: revert missing license for maxmind changes ()
Reverts
 
 - DEV: maxmind license checking failing tests  
 - UX: Show if MaxMind key is missing on IP lookup 

These changes are leading to surprising results, our logs are now filling up with warnings on dev environments 

We need the change to be redone
2023-11-24 11:31:11 +11:00
Sam
85d2b5fa48
DEV: maxmind license checking failing tests ()
This improves the implementation of 

1. Error message displayed to user is clearer
2. open_db will also be called, even if license key is blank, as it was previously
3. This in turn means no need to keep stubbing 'maxmind_license_key'
2023-11-24 09:38:46 +11:00
MichaIng
c58a41cb3e
UX: Show on IP lookup if MaxMind key is missing ()
as discussed in https://meta.discourse.org/t/maxminddb-not-found-error/148512/7.
 
shows a warning to the admin if no license for maxmind is found
2023-11-24 08:02:05 +11:00
David Taylor
6417173082
DEV: Apply syntax_tree formatting to lib/* 2023-01-09 12:10:19 +00:00
Alan Guo Xiang Tan
7f9c0e476f
DEV: Typo. () 2022-03-03 09:24:58 +08:00
Bianca Nenciu
b21bf840cb
FEATURE: Return geoname IDs from MaxMindDb ()
The geoname IDs can be used by plugins to locate the user more precisely.
2022-03-02 23:51:42 +02:00
Sam Saffron
d0630ea6ee FIX: MaxMind DB file not downloading correctly
Previously we had the ability to download a simple .gz file
new changes mean we have a a tar.gz file that needs some levels
of fiddling to get extracted correctly
2020-01-05 22:08:13 +11:00
Sam Saffron
a8ffb6949c FEATURE: support MaxMind DB downloads using a license key
MaxMind now requires an account with a license key to download files.

Discourse admins can register for such an account at:

https://www.maxmind.com/en/geolite2/signup

License key generation is available in the profile section.

Once registered you can set the license key using `DISCOURSE_MAXMIND_LICENSE_KEY`

This amends it so we unconditionally skip MaxMind DB downloads if no license key exists.
2020-01-03 16:32:48 +11:00
Sam Saffron
769501c63a FIX: exception which was meant to be ignored and logged was failing
Thanks to Darix!
2019-05-28 11:45:12 +10:00
Sam Saffron
7429700389 FIX: ensure we can download maxmind without redis or db config
This also corrects FileHelper.download so it supports "follow_redirect"
correctly (it used to always follow 1 redirect) and adds a `validate_url`
param that will bypass all uri validation if set to false (default is true)
2019-05-28 10:28:57 +10:00
Sam Saffron
6580025af9 FEATURE: add backup directory for mmdb files
This new `DISCOURSE_MAXMIND_BACKUP_PATH` can be used a secondary location
for maxmind db. That way a build machine, for example can cache it on the
host and reuse between builds.

Also per 5bfeef77 added proper error raising for download fails from
dedicated rake task

This also moves "refresh_maxmind_db_during_precompile_days" to a global
setting, it did not make sense in a site setting
2019-05-27 16:51:24 +10:00
Régis Hanol
d6aacadc4e make 🤖 happy 2019-05-25 00:32:29 +02:00
Régis Hanol
4e5305cb67
FIX: catch proper exception 2019-05-25 00:14:54 +02:00
Michael Brown
5bfeef7705 FIX: warn instead of excepting if MaxMind DBs can't be downloaded () 2019-05-25 00:08:00 +02:00
Régis Hanol
3bc99e5372
FIX: gunzip the downloaded file 2019-05-25 00:02:23 +02:00
Michael Brown
14c7aa8829 FIX: gz_file might be nil here 2019-05-24 16:11:10 -04:00
Régis Hanol
82f162962f DEV: use different MaxMindDB download endpoint 2019-05-24 20:11:24 +02:00
Régis Hanol
2f5938c3bf DEV: use FileHelper to download MaxMindDB 2019-05-24 18:11:39 +02:00
Régis Hanol
8c9ffee3af TEMP: log more information when failing to download db 2019-05-24 17:33:52 +02:00
Régis Hanol
a51e2271af DEV: remove temporary rescue when downloading MaxMindDB
follow-up 8c8c925d1b
2019-05-24 15:13:19 +02:00
Régis Hanol
f9bb8dde3c TEMP: output the error to STDERR 2019-05-23 16:55:04 +02:00
Régis Hanol
8c8c925d1b TEMP: catch exception when failing to download MaxMindDB 2019-05-23 16:30:21 +02:00
Régis Hanol
cb3ddeca40 DEV: remove unnecessary requires 2019-05-23 16:01:40 +02:00
Sam Saffron
0f76e3090c FIX: use tar -xzvf to extract maxmind vs Ruby wrappers
The tar/gz extractor in Ruby is somewhat buggy, instead rely on battle
tested `tar` and `gzip` Linux commands
2019-05-21 10:48:18 +10:00
Bianca Nenciu
a723699519 FIX: Ensure TarReader is closed. 2019-04-10 13:37:04 +03:00
Bianca Nenciu
4555d0c598 FEATURE: Refresh MaxmindDb during assets:precompile. () 2019-04-10 11:37:29 +02:00
Joffrey JAFFEUX
03014b0d05
FEATURE: adds security tab to dashboard ()
This commit also includes the new staff_logins report
2018-12-14 13:47:59 +01:00
Sam
d1e3c213a7 Do not look up blank ips
No point attempting any kind of lookup on blank ip addresses
this simply adds log noise

Also ... freeze strings
2018-11-19 15:50:21 +11:00
Bianca Nenciu
ab02b9a5d8 FIX: Use 'require' for dependencies. () 2018-11-02 09:16:45 +11:00
Sam
23423ba112 correct spec and error reporting
previous commit misused warn_exception which caused a spec to fail
2018-10-31 13:38:05 +11:00
Sam
31ad7a60ab FIX: improve error handling for missing maxmind dbs
Correct issue where we kept on looking up data on a Logster::Message object
2018-10-31 12:57:18 +11:00
Sam
32b1f34910 PERF: avoid DNS lookups when getting IP info
Also cleans up interface in DiscourseIpInfo
grew cache to 2000 entries
2018-10-31 12:38:57 +11:00
Bianca Nenciu
e1e392f15b DEV: Use DiscourseIpInfo for all IP queries. ()
* DEV: Use DiscourseIpInfo for all IP queries.

* UX: Use latitude and longitude for more precision.
2018-10-30 22:08:57 +00:00
Bianca Nenciu
effbef7d0b UX: Use user locale for locations. ()
* UX: Use user locale for locations.

* DEV: Added MaxMindDB test data and fixed test.
2018-10-25 10:54:01 +00:00
Bianca Nenciu
6a3767cde7 FEATURE: Warn users via email about suspicious logins. ()
* FEATURE: Warn users via email about suspicious logins.

* DEV: Move suspicious login check to a job.
2018-10-25 09:45:31 +00:00
Bianca Nenciu
1d26a473e7 FEATURE: Show "Recently used devices" in user preferences ()
* FEATURE: Added MaxMindDb to resolve IP information.

* FEATURE: Added browser detection based on user agent.

* FEATURE: Added recently used devices in user preferences.

* DEV: Added acceptance test for recently used devices.

* UX: Do not show 'Show more' button if there aren't more tokens.

* DEV: Fix unit tests.

* DEV: Make changes after code review.

* Add more detailed unit tests.

* Improve logging messages.

* Minor coding style fixes.

* DEV: Use DropdownSelectBoxComponent and run Prettier.

* DEV: Fix unit tests.
2018-10-09 22:21:41 +08:00