Sam Saffron
30990006a9
DEV: enable frozen string literal on all files
...
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.
Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Sam Saffron
3dc4ab905c
DEV: annotate models
...
(also looks like renaming of bigint(8) -> bigint in annotate
which seems fine)
2019-05-03 08:34:12 +10:00
Guo Xiang Tan
214dac05de
Update annotations.
2018-07-16 14:19:07 +08:00
Arpit Jalan
9ca6ebe8fe
FEATURE: enforce tagging on categories
2018-04-11 07:15:24 +05:30
Neil Lalonde
2bd44bbf13
WebCrawlerRequest.clear_cache needs to clear user agent list too
2018-03-27 15:11:48 -04:00
Neil Lalonde
4d12ff2e8a
when writing cache, remove elements from the user agents list. also return a message and content type when blocking a crawler.
2018-03-27 13:44:14 -04:00
Neil Lalonde
ced7e9a691
FEATURE: control which web crawlers can access using a whitelist or blacklist
2018-03-22 15:41:02 -04:00
