Why didn't Microsoft learn from the mistakes of history?

[AnharHussainMiah]

Some context:

Out of morbid curiosity I wanted to know what the Azure world had to offer in terms of Infrastructure as Code IaaC, coming from Amazon there is of course CDK (and much like Pulumi) the eventual and predictable outcome of evolution of DevOps tooling.

Let me expand what I mean, before CDK as we all know we had Terraform and before that we had other things like Ansible, and before that probably greasy Bash or PowerShell scripts.

We can see the evolution timeline clearly, and while Terraform was a "nice" step in the right direction, it was and is still a DSL, and that creates some problems.

A DSL is not like a full blown Turing complete general purpose programming language, it is as we know a very limited and specialised subset.

And so it is not surprising that as the complexity of infrastructure grows, at some point organisations reach an "inflection" point where the limits of DSL are hit, and it becomes very frustrating and cumbersome to use said DSL.

And of course inevitably we have to "rip the band aid" off and just use a full programming language, and so we end up things like CDK and Pulumi (and there may well be others).

So in light of this, when I searched what Azure/Microsoft has in the IaaC space?

I landed here on "Bicep", at first I thought that was some C# library, I was excited. Then to be very disappointed!

Bicep is a DSL, I slumped into my chair. Microsoft is still at the primitive "DSL" stage of evolution.

C'mon Microsoft, learn from history please.

[anthony-c-martin]

My 2 cents:
There is value in "declarative" solutions for managing infrastructure as code (Bicep, Terraform), and there is also value in more "imperative" solutions (CDK, Pulumi). Different users of the platform have different needs, and I personally don't see a way to design a solution which fully meets the needs of both without compromising somewhere.

Benefits of declarative IaC:

• The conceptual model of a deployment is simpler - you can think of it like an atomic state change from "previous state" to "new state". By constraining yourself to this model, you are likely to have more success with things like static analysis, predicting the outcome of a change before making it, validating a change, or rolling back a bad change.

Downsides of declarative IaC:

• Stuff that would be easy to accomplish with an imperative language often requires more effort, because there's less of a concept of ordering.

Benefits of imperative IaC:

• You typically have the full power of whatever programming langauge you've chosen, and don't have to compromise on learning a new tool. If you're a more advanced user, you likely can accomplish what you need with less effort.

Downsides of imperative IaC:

• You have to think more about your deployment as a series of ordered changes, and potentially have to reason about what happens if a particular change fails, or a runtime expression doesn't evaluate to what you thought it might when authoring. This is fine for more advanced users, but can create significant footguns for less advanced users.

If you've got specific feedback on Bicep, I'd be very happy to talk that over with you.

[slavizh]

I would add that with deployment scripts BIcep also supports imperative via Az PS and CLI. Also nothing stops you from using those additionally to Bicep in CI/CD pipelines.

[AnharHussainMiah]

My 2 cents: There is value in "declarative" solutions for managing infrastructure as code (Bicep, Terraform), and there is also value in more "imperative" solutions (CDK, Pulumi). Different users of the platform have different needs, and I personally don't see a way to design a solution which fully meets the needs of both without compromising somewhere.

Benefits of declarative IaC:

* The conceptual model of a deployment is simpler - you can think of it like an atomic state change from "previous state" to "new state". By constraining yourself to this model, you are likely to have more success with things like static analysis, predicting the outcome of a change before making it, validating a change, or rolling back a bad change.

Downsides of declarative IaC:

* Stuff that would be easy to accomplish with an imperative language often requires more effort, because there's less of a concept of ordering.

Benefits of imperative IaC:

* You typically have the full power of whatever programming langauge you've chosen, and don't have to compromise on learning a new tool. If you're a more advanced user, you likely can accomplish what you need with less effort.

Downsides of imperative IaC:

* You have to think more about your deployment as a series of ordered changes, and potentially have to reason about what happens if a particular change fails, or a runtime expression doesn't evaluate to what you thought it might when authoring. This is fine for more advanced users, but can create significant footguns for less advanced users.

If you've got specific feedback on Bicep, I'd be very happy to talk that over with you.

Thank you for a detailed response, there are some truths to what you say, however I believe with a full programming language everything can be abstracted away, even he issue you mentioned around the ordering of resources. All these can be represented using various programming paradigms, for example builder patterns, as well as type enforcement and seal types etc, there is a lot of rules that can be encoded into the types so that it's enforced by the compiler, and finally there is the limitless ability to do things like pull in external information and interact with multiple different APIs etc, allowing intelligent infrastructure and not just static prescriptive infrastructure.

Want to wire up some kind external compliance validator? you can do that, you can even make your infrastructure change dynamically based on external information and even triggers to events etc. Anything is possible because it can be expressed in a full programming language. That's simply not possible in a DSL.

Finally a full language can embed a DSL, that means a DSL is always a subset of full language anyway.

[OranguTech]

@AnharHussainMiah - Good high-level discussion, I'm by no means a Bicep apologist (was here looking for answers to a vexing issue) but if you want to use a "Full Programming Language", then nothing is stopping you from doing that?

[AnharHussainMiah]

Thanks @OranguTech while it is possible to use the raw APIs the issue with that approach is, it would require to build an entire state diff and orchestration engine that would be akin to rolling your own CDK from scratch (a huge undertaking), which is really something Microsoft should be doing!

Now one could argue companies such as Pulumi have done exactly that, as in build their own cloud agnostic state diff and orchestration engine, the issue however is:

• They're aiming for cloud agnostic, that means it's the "lowest common denominator", and not optimised for Azure
• It's a paid product, which is fine if you have cross cloud requirements.

Basically in a nutshell, Microsoft as of right now has nothing that is equivalent to CDK, and the best is legacy style Terraform clone.

So my question remains, and Microsoft really hasn't learnt from history and isn't even keeping up with it's the cloud's number 1 provider...