We use dynamic pipelines (buildkite-agent pipeline upload) and noticed that each of the steps being run is a separate job, as in every hook defined Buildkite Agent Hooks v3 | Buildkite Documentation executes on every step.
That’s fine… a little interesting to wrap your head around, but that’s okay. But that means I can’t use pre-command and post-command, or even pre-exit because they’ll run 15+ times.
Is there a pre-pipeline hook, and a post-pipeline hook? Would love to run something before/after my builds exactly once.
Hi @chomey that’s right hooks run for each job. To achieve what you wish, could you use a step itself instead of a hook?
The first step in the pipeline (could even define it before the pipeline upload step with a wait step between) could perform your logic. And you could define the last step in the pipeline in a similar way.
Dynamic pipelines all upload their steps after the step that is uploading them, so if you define the final step in the web UI, it won’t have steps added after it by future uploads.
And by allowing steps to continue on failure you can ensure that the last step will always still run, even if previous steps fail.
I suppose I could use a step… but the intent was to have a default set of pre/post pipeline hook functions that ran, regardless of job, so that we didn’t have to define them in every pipeline.
Think notifications to Slack, metrics, or annotating gifs into build outputs =)
Oh yeah that makes sense. Its quite a difficult problem because no code runs on our server so the agent would need to know its running the first/last step. But the last one is difficult because that could change during the execution of that step (if it uploaded more). I’ll raise a feature request for it though to see what the team thinks
Please can you share more details on what you have setup in pre-exit hook. I assume you are using buildkite-agent step get "outcome" --step "STEPKEY" in your pre-exit hook but you are observing null value for it
Thank you for confirming. I was able to reproduce the behavior like you said but looking at the documentation and following that where I created pipeline like below with a second step then I could see the pre-exit output of the second step showing proper outcome value
Yes @chomey. Sorry, we do not have a better answer for your usecase at the moment. We raised feature request with our team on having ability to define a hook which runs as pre-start or pre-exit step for a pipeline instead of running it for every pipeline step.
Our product team reviewed the feature request and consider that using plugins for this which you can invoke across pipelines that need it would be better way to handle it. As plugin approach can help with this team considers this should be enough for this usecase.
Here is documentation which explains about how to use plugins in a pipeline
Please let us know if you have any questions. Thank you
I don’t believe plugins really solve the OP’s problem, since a plugin is just a number of hooks, therefore they all run on each step.
If definitionally a hook only runs on the job level, then I feel like Buildkite needs something else to solve the problem of “run this command once, when the build finishes (whether pass or fail)”. Pipeline-level notify is almost that. It would be really nice if Buildkite could add something like finally, which could act like a group of commands, and only run after there are no more steps in the pipeline to execute. I have the same problem as OP.
