Show more

More technical details:

So basically the issue at Scaleway is the metadata database, which goes out of sync for large datasets.

Some objects appears to be broken, and can't be accessed as of rn. Out of 1.8m we're missing about 250k.

I decided to move the instance to the new bucket even though the sync was not finished because it will take some time for the last objects to be transfer and the instance was down for too long. Also the backlog was taking too much memory

To handle the huge backlog of toots of the past weeks, I added power to the server. Now that everything's fine, I'll go back to the previous specs. This will take a few minutes tonight.

Other than that, everything should be working properly.

2/2

Show thread

State of the instance: easy version

We have a new provider for the image storage. Due to some technical issues on the old provider side, some images are missing.
You might see missing images such as avatars, headers, emojis, attachments...
This should hopefully be fixed in the next days.

1/2

I started this instance because I wanted to make a cool place for people to meet and communicate. Thank you for your support, 🇧🇷 :blobheart:

instagram.com/p/B7RF2H-gPHV/

Wow, it's 2x to 6x more efficient. The backlog size is finally going down!

Show thread

I've split the queues so that the default queue doesn't use all the threads

Show thread

Your timelines should start getting filled faster but it will take some time before we get back to real time

sidekiq is only processing 50 jobs at once, increasing it seems to exceed the pool of postgres connections

Show thread

I disabled full text search so that I could turn off Elasticsearch, because I'm very short on RAM.

Show thread

Background jobs are processing right now, faster and with less errors because the bucket is usable. From what I see the number of jobs is still going up. Let's wait and see.

Show thread

The thing is, the background jobs queue is huge now, because there is a backlog of 3.5 millions jobs. So Redis is using 5-6GB of RAM which is messing up with the server.

Since this is getting pretty bad, I decided to switch to the OVH bucket with the missing files. I would have preferred not to but it's the best option right now.

Show thread
Show more
Mastodon

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!