Show more

Wow, it's 2x to 6x more efficient. The backlog size is finally going down!

Show thread

I've split the queues so that the default queue doesn't use all the threads

Show thread

Your timelines should start getting filled faster but it will take some time before we get back to real time

sidekiq is only processing 50 jobs at once, increasing it seems to exceed the pool of postgres connections

Show thread

I disabled full text search so that I could turn off Elasticsearch, because I'm very short on RAM.

Show thread

Background jobs are processing right now, faster and with less errors because the bucket is usable. From what I see the number of jobs is still going up. Let's wait and see.

Show thread

The thing is, the background jobs queue is huge now, because there is a backlog of 3.5 millions jobs. So Redis is using 5-6GB of RAM which is messing up with the server.

Since this is getting pretty bad, I decided to switch to the OVH bucket with the missing files. I would have preferred not to but it's the best option right now.

Show thread

Stuck at 85%. About 250k objects are missing but their metadata is supposedly broken. I'll see tomorrow with the provider

Show thread

75% as of now. The majority of what's left is broken objects so it will take longer

Show thread

Lol. I suddenly can't access any IPv4 sites at home. Which means this wonderful instance works while birdsite doesn't.

Migration status:

980k objects / 1.8m
409 GB / 699 GB

Show thread

So the Storage Lead Developer of Scaleway reached to me saying that they had real issues with buckets with more than 1m+ objects, the databases can't handle it. He said my bucket was really stuck so he couldn't do anything.

They know I'm migrating so he's keeping a eye on it and we keep in touch while it's ongoing.

They will offer this month's bill, which is cool.

Sometimes stuff goes wrong, :shrug_yui:

About 40% of the media files have been transferred, but the pace is quite slow and there are about 25% of requests that are failing. The weekend is over so hopefully Scaleway will do something.

Show more
Mastodon

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!