More technical details:

So basically the issue at Scaleway is the metadata database, which goes out of sync for large datasets.

Some objects appears to be broken, and can't be accessed as of rn. Out of 1.8m we're missing about 250k.

I decided to move the instance to the new bucket even though the sync was not finished because it will take some time for the last objects to be transfer and the instance was down for too long. Also the backlog was taking too much memory

Right now I'm deleting all the objects that have already been transferred from the old bucket, in order to reduce the load on the backend databases, so that maybe we can transfer the remaining object to the new bucket at OVH.

Once the bucket is "complete" I will run some major cleanup tasks such as removing orphans, foreign emojis etc

Show thread

@admin some irony there with Scaleaway not being able to scale up for large datasets

Sign in to participate in the conversation

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!