I am new to developing pipelines on Buildkite, although our company has been using Buildkite for a while now. I have a daunting task to update a pipeline written by someone who has left the company. So bear with me if my question sounds a bit elementary. I don’t do this everyday.
What is the typical pipeline development workflow?
Modify scripts locally
Commit changes to my Git remote
Buildkite jobs are triggered for that commit
See if the pipeline ran successfully or repeat step 1-4 until it does
This feels a little crazy and almost like I am doing something wrong. Is there a better way to test my pipeline without committing things to Git? Because most of the time I end up committing tiny fixes just to see if Buildkite accepts my syntax. This is walking at a snail’s pace.
I didn’t find any guidance in the tutorial and documentation on this. I can understand that changes have to be in Git so agents can check them out. Is there at least a way to verify that the pipeline configuration + shell scripts + plugins + dynamically generated steps, etc. are all configured right before pushing it to Git? I do use shellcheck.
I have miles to go to update my pipelines and I am stuck in the first gear. Please help!
Thanks!
To documentation maintainers: it would be useful if there is a page in the documentation that explains what a typical pipeline development workflow looks like, so newbies like myself can work on a project effectively. The current tutorials at https://buildkite.com/docs/tutorials/* explain only a part of that setup. Or have I missed something?
Welcome to the exciting world of using Buildkite! We totally understand the challenge of taking on someone else’s project, especially within a short time frame, and we are here to help!
For a typical workflow of a pipeline - it can be drastically simplified, especially for testing purposes. The cool thing is, your pipeline changes don’t have to be in git for agents to be able to check them out.
All you need to do is to create a pipeline via the Buildkite UI (or API using GraphQL or REST) and add steps you’d like to execute in that pipeline. At this point, to test things and the syntax, there is no need to upload a pipeline YAML, you can use the built-in pipeline steps editor in https://buildkite.com/your-org-slug/pipeline-name/settings/steps.
We have some fantiastic documentation you may want to check out to help give you a better understanding of how creating and modifying your pipelines works:
When it comes to using git, if you want to host a pipeline.yml file in your repository, then you can update the UI to do a buildkite agent pipeline upload and go back to the “git” way of updating your pipeline. This would be later on of course, once you’re done testing your workflow via what I shared earlier.
I hope this answers your question. If you have more questions, we’ll be happy to help!
Will do it that way.