How to get around Azure Data Factory ARM Template

Question

I have created an Data Factory in Azure with a about 10-15 pipelines in it and they all work fine and I can debug and run them without issue on the Test Data Factory that I have setup. I have also setup Azure DevOps integration with the Data Factory with a CI/CD pipeline.

Initially this pipeline worked fine as well and I was able to use the generated ARM template to recreate the Pipelines/Triggers/etc on a production Data Factory instance.

Now I have added a few more things to the Data Factory and when it generates the ARM template now it creates a template that with 293 parameters, which when it is run through the CI/CD pipeline fails because only 256 parameters are allowed with an ARM template.

A lot of the items that it put in the parameter file I do not need to be parameterized, like filename and file paths. All I really want to have parameterized is the various connection information.

I tried to create a second ARM Template parameter file and remove the parameters that I didn't want because I read some documentation about being able to do this to add some additional parameters, but this doesn't work because it doesn't remove the parameters out of the ARM Template itself.

So my question is, is there a way to handle this? The things that I can think of doing are creating multiple Data Factories and only having a couple of pipelines in each one. I don't really like this idea because it would become very large and cumbersome over time. I could manually remove the templates out of the parameter file and also out of the template itself, but I really don't like this solution as it is manual and really error prone.

What I would ideally like to be able to do is to define which items I want to be parameterized in the Pipeline, like I mentioned 90% of the things that it parameterized I don't need to be parameterized, but I can't find any way to do this (short of manually doing it).

Answer 1

2 ways:

1. obvious - remove unwanted parameters
2. less obvious - use objects instead of strings.

let me explain, imagine you have 5 input parameters called param1,2,3,4,5. you can "compress" them into a single parameter like this:

"param": {
    "type": object,
    "defaultValue": {
        "param1": "something",
        xxx
        "param5": "otherthing"
    }
}

seeing you have 290 parameters, I'd probably go the object route, because that way you can logically group them.

as for not parameterizing thing - you can define variable of default values for them (but if you define default values they still count for parameters), so variables allow you to do that.