updates
- subscribers tasks [before: 1.81.14.0][after: 1.82.0.0] deployment
- assistant [before: 1.4.15.1][after: 1.5.5.0] deployment
- subscribers postponed [before: 1.72.3.1][after: 1.82.0.0] deployment
- subscribers updates [before: 1.56.11.1][after: 1.82.0.0] deployment
- subscribers cache [before: 1.78.0.1][after: 1.82.0.0] deployment
- directory API [before: 1.81.15.0][after: 1.82.0.0] deployment
- manager API [before: 1.107.0.0][after: 1.108.9.0] deployment
- directory messages [before: 1.80.2.1][after: 1.82.0.0] deployment
update plan
- deploy until directory messages
- static files to be updated (manager-api)
- migrations for directory database to be applied
- proxy servers to be expanded in order to avoid throttling issues
deployment feedback
We’ve ran into migration issues with directory database (subscribers table) which locked the whole table for 5hrs. After waiting for the migration to finish, we’ve tried to do the same on the slave clusters (replicas) which turned out to be much faster, thus strongening the suspicion that the master cluster is either overloaded or has some issues with performance that need to be addressed.
Second attempt was made with the following changes:
- add a new column to the subscribers table
- add indexes to the new column + some compound indexes with it’s participation
- preliminary tests on the slave clusters showed that the migration should be much faster this time
Takeaways
- each deployment should be careful about any database changes
- all database changes should be tested on the slave clusters first
- RDS performance should be monitored and optimized
deployment render all [[templates/items/deployment]] where page = @page.name