Commit Graph

42 Commits

Author SHA1 Message Date
Jarek Radosz
2fc70c5572
DEV: Correctly tag heredocs (#16061)
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Michael Brown
3bf3b9a4a5 DEV: pull email address validation out to a new EmailAddressValidator
We validate the *format* of email addresses in many places with a match against
a regex, often with very slightly different syntax.

Adding a separate EmailAddressValidator simplifies the code in a few spots and
feels cleaner.

Deprecated the old location in case someone is using it in a plugin.

No functionality change is in this commit.

Note: the regex used at the moment does not support using address literals, e.g.:
* localpart@[192.168.0.1]
* localpart@[2001:db8::1]
2022-02-17 21:49:22 -05:00
Peter Zhu
c5fd8c42db
DEV: Fix methods removed in Ruby 3.2 (#15459)
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
2022-01-05 18:45:08 +01:00
Michael Maroszek
5bec0e5763
fix vbulletin importer to import unreferenced attachments (#12187) 2021-04-19 21:05:16 +02:00
Michael Maroszek
144584aacb
fix vbulletin importer to hide soft-deleted posts (#12057)
equal to theads posts can be soft-deleted which results in a visibile = 2 state. at the moment those posts will be imported fully visible.
2021-02-12 14:29:05 +01:00
Sam Saffron
30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Guo Xiang Tan
24347ace10 FIX: Properly associate user_profiles background urls via upload id.
`Upload#url` is more likely and can change from time to time. When it
does changes, we don't want to have to look through multiple tables to
ensure that the URLs are all up to date. Instead, we simply associate
uploads properly to `UserProfile` so that it does not have to replicate
the URLs in the table.
2019-05-02 14:58:24 +08:00
Régis Hanol
afa22a0c6f REFACTOR: more 'fake_email' to base importer 2018-10-22 11:12:40 +02:00
Régis Hanol
8b20e2500a
Remove unnecessary line 2018-10-19 15:48:48 +02:00
Régis Hanol
637123ff6f Merge users based on their email in vBulletin importer 2018-10-19 15:16:45 +02:00
Régis Hanol
53aa0344bf FIX: properly import vBulletin's hashed password 2018-10-18 10:22:55 +02:00
Sam
5f64fd0a21 DEV: remove exec_sql and replace with mini_sql
Introduce new patterns for direct sql that are safe and fast.

MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API

- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder

See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
discoursehosting
fc973f9363 Improve the VBulletin importer (#5922) 2018-06-12 20:41:21 +02:00
Guo Xiang Tan
5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Régis Hanol
9641d2413d REFACTOR: upload workflow creation into UploadCreator
- Automatically convert large-ish PNG/BMP to JPEG
- Updated fast_image to latest version
2017-05-11 00:16:57 +02:00
Régis Hanol
0d250c3935 50% faster vBulletin 4 importer 2017-02-01 14:33:09 +01:00
Jay Pfaffman
3a5c0c5605 add env variables for vBulletin import script 2017-01-31 10:31:18 +05:30
Mohamad Abras
bc621a704e fix close topics vb4 importer 2016-12-15 14:20:05 +02:00
Arpit Jalan
59523aef9d more improvements to vBulletin import script 2016-12-05 17:44:18 +05:30
Mohamad Abras
f68194cf8e fix nil for vb4 importer 2016-12-05 04:16:59 +02:00
Mohamad Abras
52749c0121 imporoving vb4 importer 2016-12-04 00:02:47 +02:00
Arpit Jalan
92e716a1fd fix vbulletin import script 2016-09-14 08:15:48 +05:30
Arpit Jalan
201d344a2d improve vBulletin import script 2016-07-10 14:51:31 +05:30
Gerhard Schlager
6238a43f93 Fix base and vBulletin importer 2015-10-21 19:07:31 +02:00
Kane York
821124f3fd FEATURE: Skip batches if all records exist
Update all import scripts to take advantage of all_records_exist?
2015-09-21 17:13:02 -07:00
Erick Guan
7737cc2828 vBulletin: ability to import attachments from filesystem 2015-05-15 19:39:15 +08:00
Gerhard Schlager
a412e9bede Fix gem load order for all importers
https://meta.discourse.org/t/migrating-from-mybb/25563/8
2015-03-18 20:31:02 +01:00
Gerhard Schlager
b5426763e4 FIX: Importers should allow categories with existing name if the parent category is different
This changes the content of `@categories_lookup` from `Category` objects
to IDs since the category names aren't needed anymore. The lookup
method has been renamed too.
2015-03-12 21:15:02 +01:00
Daniel Phin
c35cdfcf6b Update vbulletin.rb 2015-03-10 00:31:12 +10:30
Régis Hanol
31340de446 don't break the vBulletin importer when PrettyText times out 2015-01-26 20:35:30 +01:00
Régis Hanol
eecc573fbc FIX: don't break import when raw can't be preprocessed (vBulletin importer) 2015-01-21 09:36:46 +01:00
Régis Hanol
6c4d852011 Improve vBulletin importer
- FEATURE: TopicCreator now supports 'pinned_at' parameter
- FIX: 🐛 FIX TopicQuerySQL to support pinned topic older than 2010
- FIX: 🐛 Properly remove all HTML Entities from Usernames/Titles/Category Names/Groups in vBulletin importer
- FIX: 🐛 Properly handle specific vBulletin BBCode (quotes/mentions)
- FIX: 🐛 Make sure we generate a username from the name of the user instead of a fake email
- FEATURE: Allow for custom timezone in vBulletin importer
- FEATURE: Support for profile pictures/background in vBulletin importer
- FIX: 🐛 merge the categories tree to only 2 levels in vBulletin importer
2015-01-19 15:00:55 +01:00
Régis Hanol
5045d8677c new vBulletin importer (uses mysql dump instead of multiple csv files) 2014-12-22 13:22:16 +01:00
Régis Hanol
d1608bdfe9 FIX: better error handling in vBulletin importer 2014-09-08 11:02:54 +02:00
Régis Hanol
4907053cc4 FIX: vBulletin pre-processing regexes order 2014-09-05 18:44:34 +02:00
Régis Hanol
85cbb001ae FIX: properly close topics in vBulletin importer
FEATURE: add backtrace when an exception happen (importers)
FIX: post-processing should also happen on first posts (vBulletin
importer)
PERF: faster topic bypass when already imported
2014-09-04 17:55:05 +02:00
Régis Hanol
e823f568a7 FIX: preprocessing regexes in vBulletin importer 2014-08-29 13:11:56 +02:00
Régis Hanol
9b86d1507d VBULLETIN: add some stats about the imported groups 2014-08-25 12:33:08 +02:00
Régis Hanol
7aaf718cf3 update bbcode dialect and fix vBulletin importer 2014-08-25 10:48:29 +02:00
Régis Hanol
841d547a48 FIX: vBulletin wasn't importing posters... 2014-08-20 23:04:43 +02:00
Régis Hanol
66aaa9329e FEATURE: vBulletin importer 2014-08-18 13:04:08 +02:00
Régis Hanol
fab2c8f816 Create vbulletin.rb 2014-07-09 22:44:06 +02:00