So the Storage Lead Developer of Scaleway reached to me saying that they had real issues with buckets with more than 1m+ objects, the databases can't handle it. He said my bucket was really stuck so he couldn't do anything.
They know I'm migrating so he's keeping a eye on it and we keep in touch while it's ongoing.
They will offer this month's bill, which is cool.
Sometimes stuff goes wrong,
980k objects / 1.8m
409 GB / 699 GB
75% as of now. The majority of what's left is broken objects so it will take longer
85% but it's kinda stuck. No ETA as of now
Stuck at 85%. About 250k objects are missing but their metadata is supposedly broken. I'll see tomorrow with the provider
The thing is, the background jobs queue is huge now, because there is a backlog of 3.5 millions jobs. So Redis is using 5-6GB of RAM which is messing up with the server.
Since this is getting pretty bad, I decided to switch to the OVH bucket with the missing files. I would have preferred not to but it's the best option right now.
Background jobs are processing right now, faster and with less errors because the bucket is usable. From what I see the number of jobs is still going up. Let's wait and see.
I disabled full text search so that I could turn off Elasticsearch, because I'm very short on RAM.
See for yourself:
sidekiq is only processing 50 jobs at once, increasing it seems to exceed the pool of postgres connections
(also yeah you can upload images, nice)
Well I've raised it to 100
I've split the queues so that the default queue doesn't use all the threads
Wow, it's 2x to 6x more efficient. The backlog size is finally going down!
upgraded the server a bit until the backlog is ok
@admin still going on?
@admin Just a quick check-in and data point: some intermittent difficulties with 500, 502, 504 errors on Local and Federated Timeline as of ~2030 UTC on 17 Jan. If migration is ongoing, it would probably account for that, but I wanted to make sure you knew. Thanks for the ETRs.
@admin Thank you for resurrecting mstdn.io!
I missed it *so* much this weekend!
@codewiz me too.
@admin Congratulations on the progress!
@admin I often refer to myself as an "IT guy", but I have no idea what you are doing right now
@admin My personal opinion is that it does not matter that much if an image posted two months ago or weeks ago are missing. It would matter if it was a blog or a wiki. On a social network where more content is posted by the hour than I could read in a day.. it doesn't really matter. And I know some instances will simply drop older media content after XX days as a matter of routine.
Not being able to attach images to new posts is far more annoying. I am quite happy with that being restored.
@admin related to the missing avatars issue from earlier possibly?
@admin excellent progress Stan! Grateful that you're following through.
@admin He is nice. Thank you for telling me before I use this service. 😺
@admin weird, I have 1.4 million files in a scaleway bucket at the moment and it's running fine 🤔
@kindly not for long
@admin that's good... always nice to get a real update. thanks for informing us :)
@admin more than stuff going wrong, sometimes or rather most times, even companies named Scaleway don't plan to scale enough.
@admin interesting strategy to include 'scale' in your company's name and then create products that don't scale
> doesn't scale
The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!