Admin of mstdn.io @[email protected]

Done, thanks Hetzner for instant VM scaling

Well, a backup has started so this will wait for tomorrow

@EdwardTorvalds that's weird, I'm not sure how to fix that and my only idea is broken https://github.com/tootsuite/mastodon/issues/12916

We're going down in a few minutes for a rescale

Right now I'm deleting all the objects that have already been transferred from the old bucket, in order to reduce the load on the backend databases, so that maybe we can transfer the remaining object to the new bucket at OVH.

Once the bucket is "complete" I will run some major cleanup tasks such as removing orphans, foreign emojis etc

More technical details:

So basically the issue at Scaleway is the metadata database, which goes out of sync for large datasets.

Some objects appears to be broken, and can't be accessed as of rn. Out of 1.8m we're missing about 250k.

I decided to move the instance to the new bucket even though the sync was not finished because it will take some time for the last objects to be transfer and the instance was down for too long. Also the backlog was taking too much memory

To handle the huge backlog of toots of the past weeks, I added power to the server. Now that everything's fine, I'll go back to the previous specs. This will take a few minutes tonight.

Other than that, everything should be working properly.

2/2

State of the instance: easy version

We have a new provider for the image storage. Due to some technical issues on the old provider side, some images are missing.
You might see missing images such as avatars, headers, emojis, attachments...
This should hopefully be fixed in the next days.

1/2

We're back.

@spla how did you delete these users?

@spla issues with object storage for weeks

Almost there...

Well, 500k jobs remaining

It should be OK in about 15 hours.

Under 2 millions enqueued jobs now.

upgraded the server a bit until the backlog is ok

Wow, it's 2x to 6x more efficient. The backlog size is finally going down!

I've split the queues so that the default queue doesn't use all the threads

@codewiz me too.