Hello everyone, it’s time for some news, as it is long overdue, so we figured we’d provide a longer update in an official capacity, rather than providing bits and pieces all over the place in the various HepBoat chats.
We wanted to take the time to go ahead with updating everyone regarding HepBoat. While we are currently aware that there are ongoing issues with HepBoat that has been causing some instability and unresponsiveness, it is important to note that we are taking the time to do some upgrades and modifications to the bot and the infrastructure that runs it. In my head (me being JakeyPrime, in this case) it’s been rough as Discord is the main reason we’ve been plagued with issues, and we were given absolutely zero notice of the gateway changes.
For context, because of the scale of Discord, they have gateway servers that bots connect to (and users, but that’s not what we are focused on) in the past, bots would connect and stay connected to these servers. Simple enough makes sense. However again, due to the scale of Discord bots can stay online for a large amount of time and some gateway servers would be handling all of the load and some would have none, Discord doesn’t seem to have code to realize how big a bot is before it connects. So they implemented forced reconnects, meaning that hepboat can just get disconnected whenever Discord needs to rebalance a node. We get around 15000 events a second when HepBoat reconnects. So on top of the load already being high on the bot due to the popularity of some of the servers reading this, we have to then re-process some 20k events live, and due to the nature of the bot, they need to be processed quickly. (Think bans, kicks, etc, we might have missed when we were reconnecting). On top of reprocessing these events, we typically have to re-cache a bunch of users in the bot, which causes the catching up after a reconnect to be even slower. This can ultimately slow down the heartbeater in the bot and it will fail to reconnect, causing it either to loop and try the reconnect again, or give up and go offline completely.
While I disagree with some of the brunt anger of some users complaining about a service I provide for free, out of my own pocket (patreon still doesn’t cover the cost of running the old server, let alone the new server I am currently announcing in this exact post) I do feel we’ve outgrown the old machine running the current instance of the boat. However, those of you who have been patient and welcoming and have seemingly had endless patience, this is dedicated to you. No pun intended.
Currently, HepBoat runs on a virtual machine shared with a few other projects I run. Very low resource projects, but any CPU and RAM used up is resources that HepBoat cannot use, plus its ultimately, technically shared hardware with my other projects, which means there’s reservations and CPU interrupts from other things.
Well, in short, this will be changing. A new server has been purchased (again might I mention the lovely Patreons, toss a coin to your local Patrons, because they helped in funding this! Not fully, but the support from them means a lot more to me than the actual monetary value, so I happily paid the other half, love you guys)
We’ve tripled the ram and gotten a cpu that scores 4 times higher than the previous box, and it’s fully dedicated to being for HepBoat, and nothing else. Oh and did I mention we are moving from a plain on SATA hard drive to a redundant RAID of 2x NVME SSD’s?
This is beyond exciting news for us, and hopefully exciting news for HepBoats users as well! However, with all this great news, there is some bad news, but not really.
In order to get the migration done, there needs to be… Well… a migration. HepBoat currently rocks a 70GB database, which even with top port speeds in the industry can take a while, on top of setup, exporting the db itself, and importing the database once again, making sure it’s all stable, and re-adding a bunch of data. This process will take a WHILE.
Not only will I need copious amounts of cute animal pictures in the boats chat, but I will need your patience and understanding. I am a one man crew dealing with 70GB database and the migration of it and setting the whole bot up, fresh, on a new server.
Also, just to put this in here before Lacking sends more :eyes: emojis in chat, HepBoat Prime. Yes it’s a thing, yes it’s coming soon. No I will not give more information.
The migration is expected to take around 24 hours. This is not something anyone wants to hear, and also not the announcement I want to be giving. However, this is a maximum, worst case scenario. We lost an entire disk and had corrupted the entire HepBoat database previously, and our team had it fixed with no file loss in less than 14 hours, this migration should go a lot smoother. The only reason it’s estimated to be so long, is because the data transfer time, on top of the export and import time of the database is something I have NO idea how long it will take. Transfer should be pretty quick once it happens, but again, it’s completely a mystery. Another thing to note, is that once the migration happens, and it’s on the new server, if the migration goes better than expected I will take time to make some DB modifications and upgrades to make it perform much better, as well as vacuuming out some old data that we don’t need anymore, and then running an actual PG vacuum to free up the taken space and optimize the database even further.
The migration date is planned to be Thursday, May 7th, at around 11PM EST. From then the bot will be unavailable. If you would like, you can send a request / email to https://hep.gg/request with the following, and we can get you whitelisted and setup with your current config on the beta bot, however the dashboard will be unavailable, but we can copy your main config over.
Send the following with your request:
A file upload with your config in it.
Your desired prefix. Please make sure to make it different from the main boat.
I’ll reply to your ticket with an invite link for the boat, once you invite it, please let me know in the same ticket, so that I can go ahead and force setup the bot for you, once that’s done I’ll reply and let you know it’s good to go with your config! 🙂
Once the migration is done, don’t be upset if the bot leaves your server, the beta bot that is.
All in all, what I hope from this is that it will help HepBoat to be more stable, for all of you, no matter what. A lot of you have been so extremely supportive of me and the rest of the team. I want to make sure it’s the best experience it can be, and it hasn’t been, and I really want to change that and have been working super hard. Again, I would like to thank the patrons that made this all possible, and those of you who have been super patient with us, we wouldn’t be here without you. If you have ANY questions whatsoever, please reach out and we will do our best to answer your questions in #hepboat-users
Thanks everyone, I love ya’ll
- JakeyPrime & The Rest of the HepBoat Crew.