If you're a staff member you might've noticed that the logs database wasn't carried over since we had our last move. The reason for this is, well, I did the entire move from the windows box to Debian in about two hours, which included 45 minutes uploading our MySQL database. So now that a few days have passed, and I had time to import the logs database, I figured I'd set that up to run overnight. All I had to do was open the SQL file and remove the top 20ish lines.

Those top lines drop the database table that's existing and create a new one. Which would've been fine if I was importing the file right away, but I'm not, and there are a few days worth of logs that would be lost if I recreated the table.

So that is problem number one. Problem number two? The file is 5.94 Gigabytes and Notepad++ just goes "Naaaaaah". Sublime shows a lovely progress bar, looks promising. Fast forward two and a half hours later. The progress bar is at 90%, I've closed every program running on my laptop - which has 16 Gigabytes of RAM - and I am now at 94% usage of my RAM.
My computer running out of memory

Basically, my computer isn't powerful enough to open the SQL file, so I am uploading the file to the dedicated server, in hopes of being able to use vim or nano to open the file, cut the top 20 lines, and import the log into the database.

The challenges of working with large MySQL files (and the results of not splitting your export of structure & data)!


Vim was able to open the file without issue! So it is currently importing into the database, and hopefully we will have all of our logs available again. To those people excited about how quickly logs load now, they will probably be rather slow again. We're going to look into porting them over to elastic search in the future.