Recently, Andreas Grabner (@grabnerandi on twitter) of Dynatrace demoed their Unbreakable Pipeline to me and Donovan. I immediately leaned over to Donovan and said, “Dude, that is sick. We can use Dynatrace for Azure Monitoring and we can totally build Dynatrace’s Unbreakable Pipeline using VSTS and Azure!”.
Then I really started geeking out. “We can go totally serverless using VSTS Deployment Gates and Azure functions!!!”
To which Donovan replied, “Dude. Chill. Go do it. Make it happen.”
So I did. I worked closely with Andi, and by work closely I mean Andi graciously worked with me answering all of my questions day after day and at all hours on what they were doing and how they were doing it. He even shared their source code with me. Which, of course, made me pester him even more with question
Andi was SUPER helpful. And I’m kind of surprised he didn’t do this to me when I was bothering him all the time
Dynatrace’s Unbreakable Pipeline
What is the Dynatrace Unbreakable Pipeline? In a nutshell, it’s a build and release pipeline using Dynatrace monitoring to automatically determine if releases should propagate to the next environment. It also has the ability to monitor your production environment and if bad things are happening, it can automatically self heal and roll the release back to a previous working release.
Ok, sounds cool, but how is this different than what people are already doing? Well, traditionally, the way most people use performance monitoring in their DevOps pipelines has been pretty manual. You deploy into a staging environment, run a bunch of load tests and then manually look at the performance metrics gathered to determine if this is a good release or not. Maybe you’ll even manually compare performance results between the new code in staging vs the old code running in production. If it is good, then you manually approve the release to deploy into production. And while the app is running in production, if bad things are happening and alerts are being created, a human will manually look at those alerts and manually trigger a release rolling production back to a previous environment.
Useful as this is, guess what? We can now shift left and automate this! In the Dynatrace Unbreakable Pipeline, after deploying into the staging environment and running load tests, it then sends a monspec file to Dynatrace. A monspec file is a file which describes which metrics you want to pay attention to as well as thresholds and what other environment you want to compare against. And based on this monspec file and whether dynatrace finds any violations, the pipeline automatically passes or fails the release. And also, if the production environment is throwing Dynatrace alerts, the system will self heal and automatically roll back to a previous release.
Whoa… Cool. Dynatrace is using three concepts that I REALLY love.
- monspec files or Monitoring as Code, stored right next to the source code, db schema and infrastructure. Love it.
- Using automated monitoring to determine if a release should continue or not. Love this too.
- And finally, the auto self healing in production is super useful!
Dynatrace Unbreakable Pipeline in VSTS and Azure
So million dollar question. Can we make this Unbreakable Pipeline in VSTS and Azure? Of course! And here it is.
Here is my release pipeline. We have a Staging environment and a Production environment. Both environments are being monitored by Dynatrace.
In the release steps, I deploy my app to to the environment, run load tests and then send a Dynatrace Deployment event to Dynatrace using a custom task.
This custom build/release task that takes parameters so VSTS sends the deployment event to the correct Dynatrace tenant and identifies the correct app that is being released.
Here is the deployment event in Dynatrace
After deploying, a custom post deployment release gate kicks in
This gate sends the monspec file to Dynatrace and based on if there are any violations, automatically either passes or fails. Here is the gate passing
And here is an example where based on the monspec file, Dynatrace reported violations so the gate automatically failed.
Self Healing Production Environment with Dynatrace, Azure and VSTS
What about self healing? You can set up Dynatrace to call a webhook when alerts occur so I wrote an Azure function which acts as that web hook. It takes the release information and rolls back the release to a previous working version.
And here is self healing in action. I deployed Release 276 to production. 276 is a good release which doesn’t generate any errors
And in Dynatrace, you can see my deployment and the load test caused no failures.
Next I kick off a build/release (Release 277) which generates a lot of errors during the load test. Here you can see the release pipeline is in the process of deploying Release 277 to production
And here in the environment overview, you can clearly see that Staging has Release 277, and Production is in the process of getting Release 277
Here you can see that it deployed successfully all the way into Production.
And in Dynatrace, you can see all the failed requests happening.
All these failed requests caused the Dynatrace AI to trigger a Dynatrace Alert, which called my Azure function web hook, which used the VSTS REST api to re-released the previous successful release into the production environment, which was Release 276.
And here, you can see the final result. Even though we release 277 successfully all the way to production, because of the Dynatrace alerts, we self healed and rolled production back to release 276.
So what’s next?
What I created with the help of Andi for VSTS is really cool stuff!!!! I’m going to hand this over to Andi and Dynatrace where they can flesh everything out and turn this into a “real” Dynatrace branded extension for VSTS. But all this is real foks. If you are curious about the source code, here it is. If you use Dynatrace and Azure, you now have a really cool tool in your DevOps tool belt. If you are not using Dynatrace in Azure, maybe you should take a look. And if you are not using Azure? Come on man… time to dive in and create your own unbreakable pipeline in Azure!
The Dynatrace Unbreakable Pipeline in VSTS and Azure
Recently, Andreas Grabner (@grabnerandi on twitter) of Dynatrace demoed their Unbreakable Pipeline to me and Donovan. I immediately leaned over to Donovan and said, “Dude, that is sick. We can use Dynatrace for Azure Monitoring and we can totally build Dynatrace’s Unbreakable Pipeline using VSTS and Azure!”.
Then I really started geeking out. “We can go totally serverless using VSTS Deployment Gates and Azure functions!!!”
To which Donovan replied, “Dude. Chill. Go do it. Make it happen.”
So I did. I worked closely with Andi, and by work closely I mean Andi graciously worked with me answering all of my questions day after day and at all hours on what they were doing and how they were doing it. He even shared their source code with me. Which, of course, made me pester him even more with question
Andi was SUPER helpful. And I’m kind of surprised he didn’t do this to me when I was bothering him all the time
Dynatrace’s Unbreakable Pipeline
What is the Dynatrace Unbreakable Pipeline? In a nutshell, it’s a build and release pipeline using Dynatrace monitoring to automatically determine if releases should propagate to the next environment. It also has the ability to monitor your production environment and if bad things are happening, it can automatically self heal and roll the release back to a previous working release.
Ok, sounds cool, but how is this different than what people are already doing? Well, traditionally, the way most people use performance monitoring in their DevOps pipelines has been pretty manual. You deploy into a staging environment, run a bunch of load tests and then manually look at the performance metrics gathered to determine if this is a good release or not. Maybe you’ll even manually compare performance results between the new code in staging vs the old code running in production. If it is good, then you manually approve the release to deploy into production. And while the app is running in production, if bad things are happening and alerts are being created, a human will manually look at those alerts and manually trigger a release rolling production back to a previous environment.
Useful as this is, guess what? We can now shift left and automate this! In the Dynatrace Unbreakable Pipeline, after deploying into the staging environment and running load tests, it then sends a monspec file to Dynatrace. A monspec file is a file which describes which metrics you want to pay attention to as well as thresholds and what other environment you want to compare against. And based on this monspec file and whether dynatrace finds any violations, the pipeline automatically passes or fails the release. And also, if the production environment is throwing Dynatrace alerts, the system will self heal and automatically roll back to a previous release.
Whoa… Cool. Dynatrace is using three concepts that I REALLY love.
Dynatrace Unbreakable Pipeline in VSTS and Azure
So million dollar question. Can we make this Unbreakable Pipeline in VSTS and Azure? Of course! And here it is.
Here is my release pipeline. We have a Staging environment and a Production environment. Both environments are being monitored by Dynatrace.
In the release steps, I deploy my app to to the environment, run load tests and then send a Dynatrace Deployment event to Dynatrace using a custom task.
This custom build/release task that takes parameters so VSTS sends the deployment event to the correct Dynatrace tenant and identifies the correct app that is being released.
Here is the deployment event in Dynatrace
After deploying, a custom post deployment release gate kicks in
This gate sends the monspec file to Dynatrace and based on if there are any violations, automatically either passes or fails. Here is the gate passing
And here is an example where based on the monspec file, Dynatrace reported violations so the gate automatically failed.
Self Healing Production Environment with Dynatrace, Azure and VSTS
What about self healing? You can set up Dynatrace to call a webhook when alerts occur so I wrote an Azure function which acts as that web hook. It takes the release information and rolls back the release to a previous working version.
And here is self healing in action. I deployed Release 276 to production. 276 is a good release which doesn’t generate any errors
And in Dynatrace, you can see my deployment and the load test caused no failures.
Next I kick off a build/release (Release 277) which generates a lot of errors during the load test. Here you can see the release pipeline is in the process of deploying Release 277 to production
And here in the environment overview, you can clearly see that Staging has Release 277, and Production is in the process of getting Release 277
Here you can see that it deployed successfully all the way into Production.
And in Dynatrace, you can see all the failed requests happening.
All these failed requests caused the Dynatrace AI to trigger a Dynatrace Alert, which called my Azure function web hook, which used the VSTS REST api to re-released the previous successful release into the production environment, which was Release 276.
And here, you can see the final result. Even though we release 277 successfully all the way to production, because of the Dynatrace alerts, we self healed and rolled production back to release 276.
So what’s next?
What I created with the help of Andi for VSTS is really cool stuff!!!! I’m going to hand this over to Andi and Dynatrace where they can flesh everything out and turn this into a “real” Dynatrace branded extension for VSTS. But all this is real foks. If you are curious about the source code, here it is. If you use Dynatrace and Azure, you now have a really cool tool in your DevOps tool belt. If you are not using Dynatrace in Azure, maybe you should take a look. And if you are not using Azure? Come on man… time to dive in and create your own unbreakable pipeline in Azure!