Create VPC, create autoscaling group, create targets, create rds instances, create s3 buckets, create cloudfront, and delete all of the above (and more, aws is complex), are all just scripts. Note: All of the AWS setup is also automated with scripts. The database is an RDS instance so we get some fun things built in there. This was a matter of using webpack to compile the angular application down to production deployable artifacts and than a simple bash script to move those artifacts to the S3 bucket. Our Angular application runs out of S3 buckets with cloudfront caching. CI/CD first runs tests, second, builds the latest docker container, third, places the docker container into a repo, fourth, deploy out the container to the correct ECS environment, five, profit! This is all automated and works the same every time with checks and balances along the way. We use ECS (Elastic Container Service) to maintain our environments. The rails application is bundled into a docker container. Things like "Can you log in?", "Can you select a record of type X, Y, Z, A, B, and C and do those records have the data you would expect in the right places?" You can have a human do this, or you can automate it. Define what helps us to know it works, and automate tests to do that. But it is important to be able to "know" the application works. Step one: We have a lot of tests and we believe in a good test suite. The application in this case is a Ruby on Rails backend, PostgreSQL database, Angular front end, with file storage and a few other smaller services. Honestly, none of that took very long to set up at all. We currently have 4 full time devs, a QA, a DBA consultant, and a Designer on the team. If anyone has any questions about these ideas, feel free to reach out. If it has to do with your infrastructure, script it. Why would you EVER do anything manually? Really. Computers are so good at repetitive tasks. You can do all of this with any of those. My CI/CD is always Gitlab CI at this point. My follow up question is, how much does downtime cost you? It's not as bad as running a second production environment. People always ask me this, so I will answer it first. It's a rather simple process with a ton of gain. I am sure you can guess at this point how we do that? CI/CD. People often forget this but those buckets that get hold all of your file storage in the cloud can be destroyed so easily (sad, sad experience taught me this). It runs the test suite against that for integrity as well. Finally, it will swap from the point in time recovered database to the warm backup. It will test the integrity of that recovery and then test the application against it. The application test will spin up with a point in time recovery of the database backup. We have never had to use the warm standby in an emergency but we have a test to make sure we can cut that over as well.įor point in time backup testing this goes back to our application test above. This leaves us with a warm standby ready to be switched over to in case of the main database server going down. When a transaction log backup is made, CI/CD picks up this change and applies it to the full backup restore and runs a set of tests for integrity check. At this time, the recovery from the day before is destroyed. Again, in our CI/CD when a full backup is taken it is pulled, loaded, and a test routine is run against it to check integrity. We use a combination of daily full backup, incremental transaction log like backup, and point in time backup. This morning it took about about 6 minutes total before tests ran. In this way we know exactly how long it would take to redeploy the entire application from scratch on a new infrastructure and get it up and running. It reports on each portion of the process. It runs a test suite which tests functionality across the application. We have a CI/CD jon which literally makes a new environment, from scratch, and deploys our application to it in a production configuration. This is by far the easiest for me to test. Each environment within my infrastructure takes a bit of a different approach. I do test my backups and restores on a rather constant basis.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |