mirror of
https://github.com/discourse/discourse.git
synced 2025-03-13 11:18:04 +08:00
fbe0e4c78c
This commit makes sure we don't load all data into memory when doing CSV exports. The most important change here made to the recently introduced export of chat messages (3ea31f4). We were loading all data into memory in the first version, with this commit it's not the case anymore. Speaking of old exports. Some of them already use find_each, and it worked as expected, without loading all data into memory. And it will proceed working as expected after this commit. In general, I made sure this change didn't break other CSV exports, first manually, and then by writing system specs for them. Sadly, I haven't managed yet to make those specs stable, they work fine locally, but flaky in GitHub actions, so I've disabled them for now. I'll be making more changes to the CSV exports code soon, those system specs will be very helpful. I'll be running them locally, and I hope I'll manage to make them stable while doing that work.
59 lines
1.3 KiB
Ruby
59 lines
1.3 KiB
Ruby
# frozen_string_literal: true
|
|
|
|
module Chat
|
|
module MessagesExporter
|
|
LIMIT = 10_000
|
|
|
|
def chat_message_export
|
|
Chat::Message
|
|
.unscoped
|
|
.where(created_at: 6.months.ago..Time.current)
|
|
.includes(:chat_channel)
|
|
.includes(:user)
|
|
.includes(:last_editor)
|
|
.limit(LIMIT)
|
|
.find_each do |chat_message|
|
|
yield(
|
|
[
|
|
chat_message.id,
|
|
chat_message.chat_channel.id,
|
|
chat_message.chat_channel.name,
|
|
chat_message.user.id,
|
|
chat_message.user.username,
|
|
chat_message.message,
|
|
chat_message.cooked,
|
|
chat_message.created_at,
|
|
chat_message.updated_at,
|
|
chat_message.deleted_at,
|
|
chat_message.in_reply_to&.id,
|
|
chat_message.last_editor&.id,
|
|
chat_message.last_editor&.username,
|
|
]
|
|
)
|
|
end
|
|
end
|
|
|
|
def get_header(entity)
|
|
if entity === "chat_message"
|
|
%w[
|
|
id
|
|
chat_channel_id
|
|
chat_channel_name
|
|
user_id
|
|
username
|
|
message
|
|
cooked
|
|
created_at
|
|
updated_at
|
|
deleted_at
|
|
in_reply_to_id
|
|
last_editor_id
|
|
last_editor_username
|
|
]
|
|
else
|
|
super
|
|
end
|
|
end
|
|
end
|
|
end
|