So the PostgreSQL server was under really high load whenever the web worker was running, which affected all of the instance.
The solution was on the PostgreSQL side: https://pgtune.leopard.in.ua
I don't know which parameter was important, but it's day and night
(Thanks a lot to Jackster on the Discord server)
@angristan there is a mstdn.io Discord ?
@Matthieu Mastodon discord
mstdn.io is one of the instance in the fediverse. We're an open-minded generalistic instance. Learn more here!