Commit Graph

16 Commits

Author SHA1 Message Date
David Taylor
436b3b392b
DEV: Apply syntax_tree formatting to script/* 2023-01-09 11:13:22 +00:00
Jarek Radosz
2fc70c5572
DEV: Correctly tag heredocs (#16061)
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Jarek Radosz
6f6406ea03
DEV: Fix random typos (#16066) 2022-02-28 10:20:58 +08:00
Peter Zhu
c5fd8c42db
DEV: Fix methods removed in Ruby 3.2 (#15459)
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
2022-01-05 18:45:08 +01:00
Jay Pfaffman
3c72cbc5de
FIX: Google groups import changed login URL (#9432)
I'm not clear why changing only the `wait_for_url` address was necessary and not also the `get` a few lines above, but this change seems to work for me on both literatecomputing.com Groups and a public group.
2020-04-15 16:45:14 -04:00
Gerhard Schlager
e825f47daa DEV: Better handling of incremental scrapes for Google Groups 2020-03-14 00:00:36 +01:00
Gerhard Schlager
d7ccb58559 FIX: Google Groups scraper failed to login 2020-03-02 17:24:48 +01:00
Gerhard Schlager
c218036107 FIX: Make Google Groups scraper work for G Suite users 2019-11-28 02:09:51 +01:00
Gerhard Schlager
ed1e5ef6cc FIX: By default, don't abort Google Groups crawling on error 2019-09-18 18:14:04 +02:00
Gerhard Schlager
ab96239f2a FIX: Google Groups crawler failed to login
Trying to automate the login into a Google account is quite hard. This makes the crawler use the content of a cookies.txt file instead. It also removes a couple of deprecation warnings and adds some color to the output.
2019-09-18 13:09:20 +02:00
Gerhard Schlager
f0fea5991f FIX: Latest Selenium gem broke Google Groups import script
Selenium uses Keep-Alive since version 3.141, so the net-http-persistent gem shouldn't be needed anymore.
2019-07-10 09:45:33 +02:00
Sam Saffron
30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Gerhard Schlager
453ba2da7b Make Google Groups scraper work with latest chromedriver 2019-03-25 16:11:22 +01:00
Gerhard Schlager
2349ba3bc4 Improve Google Groups scraper
* Better error detection during login phase
* Experimental support for 2FA and SMS codes
* Detect missing permissions to scrape email addresses
2019-03-24 23:15:13 +01:00
Gerhard Schlager
d6f89a85ef Make Rubocop happy 2018-10-31 01:30:14 +01:00
Gerhard Schlager
65db9326b4 FEATURE: Add download script for Google Groups 2018-10-31 01:12:05 +01:00