Jenkins Build Pipeline - Restart At Stage

Question

I have the following build pipeline set up as a job:

Stage 1 - verify all dependencies exist
Stage 2 - build the new jar
Stage 3 - Run integration tests
Stage 4 - Deploy to staging environment (manual step)
Stage 5 - Deploy to production environment (manual step)

I am looking for a way to start the build pipeline from a particular stage in case of a transient failure. For example, let's say there was a network issue when the user clicked to deploy to production. I don't think it makes sense to start the pipeline from stage 1... I'd like to try that step again and continue on from there in the pipeline. I don't see any functionality like this in the Build Pipeline Plugin.

Thanks!!

Answer 1

I think checkpoint is what you are looking for. Unfortunately it is only available in the CloudBees Jenkins Enterprise suite, not in the free version.

Let's hope it makes it into the open-source version as it seems to be a very common use case.

Answer 2

You can wrap your code in a retry step:

stage "Deployment"
retry(3) {
  sh "deploy.."
}

EDIT: This might help in the free version of Jenkins. Users of CloudBees Enterprise, please see @tarantoga's answer.

Answer 3

Here's another sketch to run stages conditionally without breaking the Stage View Plugin history.

As they say:

Dynamic stages: in general, if you want to visualize dynamically changing stages, make it conditional to execute the stage contents, not conditional to include the stage

Here's what I've come up with so far. Seems to work mostly: (Just ignore the other dummy steps)

We define a little conditionalStage helper function that neatly wraps up the stage name checking from the JP_STAGE Jenkins Job parameter.

Notice how conditionalStage first opens the stage and then checks stageIsActive within the stage, just skipping all steps. This way, the Stage View Plugin sees all stages and soesn't mess up, but the stages' steps are still skipped.

def stageSelect = JP_STAGE.toLowerCase()

// test if stage or any of sub-stages is active
def stageIsActive(theStage, theStages) { 
    // echo "pass: $theStages"
    // ARGL: https://issues.jenkins-ci.org/browse/JENKINS-26481
    // def lcStages = theStages.collect {it.toLowerCase()}
    def lcStages = []
    for (def s : theStages) { lcStages += s.toLowerCase() }
    def lcAllStages = lcStages + ['all']
    // echo "check: $lcAllStages"
    // echo JP_STAGE.toLowerCase()
    if (JP_STAGE.toLowerCase() in lcAllStages) {
        echo "Run: Stage '$theStage' is active through '$JP_STAGE'."
        return true
    } else {
        echo "Skip: Stage '$theStage' is NOT active through '$JP_STAGE'."
        return false
    }
}

// 1st element should be the stage, optionally followed by all sub-stages
def conditionalStage(names, stageBody) {
  stage(names[0]) { if (stageIsActive(names[0], names)) {
    stageBody()
  }}  
}

timestamps {
// --S--

conditionalStage(['Intro']) { 
    echo 'Outside Node'

    build job: 'FreeX', wait: true
    sleep 3
}

// --S--
conditionalStage(['AtNode', 'Hello', 'Done']) {
    node {
        // Cloudbees Enterprise Only: checkpoint 'Now'
        conditionalStage(['Hello']) {
            echo 'Hello World @ Node'
            sleep 4
        }
        conditionalStage(['Done']) {
            dir('C:/local') {
                echo pwd()
            }
        }
    }
}

}//timestamps

Answer 4

What you could do is to put the single steps into groovy scripts. Then you can make a "runAll"-job that loads all of the scripts in the correct order, and single jobs for the different steps.

Although this is a way that should work, I do not think that this is the ideal solution, as it means that you have to take care how the different steps exchange information, so that the steps can run independently.

A built-in solution would be much better.

Answer 5

A bit old topic but since Jenkins still (!) doesn't support this I'm sending another solution for scripted pipeline implementations. It's based on building stages list dynamically when running pipeline.

1. step - stages definition enum

enum Steps {
  PREPARE(0, "prepare"), 
    BUILD(1, "build"), 
    ANALYSE(2, "analyse"), 
    CHECKQG(3, "checkQG"), 
    PROVISION(4, "provision"), 
    DEPLOY(5, "deploy"), 
    ACTIVATE(6, "activate"), 
    VERIFY(7, "verify"), 
    CLEANUP(8, "cleanup")

  Steps(int id, String name) {
      this.id = id
          this.name = name
  }

  private final int id
    private final String name

  int getId() {
      id
  }

      String getName() {
      name
  }

    public static Steps getByName(String name) {
        println "getting by name " + name
        for(Steps step : Steps.values()) {
          if(step.name.equalsIgnoreCase(name)) { 
              return step 
          }
        }
        throw new IllegalArgumentException()
    }
}

2. method creating the final steps list

  def prepareStages(def startPoint){
        println "preparing build steps starting from " + startPoint
        Set steps = new LinkedHashSet()
        steps.add(Steps.PREPARE)
        steps.add(Steps.BUILD)
        steps.add(Steps.ANALYSE)
        steps.add(Steps.CHECKQG)
        steps.add(Steps.PROVISION)
        steps.add(Steps.DEPLOY)
        steps.add(Steps.ACTIVATE)
        steps.add(Steps.VERIFY)
        steps.add(Steps.CLEANUP)
        List finalSteps = new ArrayList()
        steps.each{
            step ->
                if (step.id >= startPoint.id) {
                    finalSteps.add(step)
                }
        }
        return finalSteps
    }

3. and u can use it like this

def stages = prepareStages(Steps.getByName("${startStage}"))

node {
    try {
        //pipelineTriggers([pollSCM('${settings.scmPoolInterval}')])  //this can be used in future to get rid build hooks 

        sh "echo building " + buildVersionNumber(${settings.isTagDriven})
        tool name: 'mvn_339_jenkins', type: 'maven'

        script {             
            println "running: " + stages
        }

        stage('Prepare') {
            if (stages.contains(Steps.PREPARE)) {
                script { currentStage = 'Prepare' }
               //.....
            }
        } //...

the "startStage" is a build parameter defined as follows

parameters { choiceParam('startStage', [ 'prepare', 'build', 'analyse', 'checkQG', 'provision', 'deploy', 'activate', 'verify', 'cleanup' ], 'Pick up the stage you want to start from') }

This allows me to pick up the stage I want to start the pipeline from (prepare stage is set by default)

Answer 6

Meanwhile, all of the other answers are obsolete, as Jenkins provides a built-in solution that allows you to restart a job from any stage: https://jenkins.io/doc/book/pipeline/running-pipelines/