# Run multiple repositories in a single pipeline **URL:** \ **Category:** General\ **Created:** [May 27, 2022, 1:48am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058 "2022-05-27T01:48:02Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![ebanster](https://avatars.discourse-cdn.com/v4/letter/e/41988e/32.png) [@ebanster](https://forum.buildkite.community/u/ebanster)\ **Post date:** [May 27, 2022, 1:48am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/1 "2022-05-27T01:48:03Z")
I have a repository to run automated tests that is separate from the source code repository. On the source code repository buildkite pipeline.yaml, I would like to include a step to trigger this other repository to run the tests and check if tests are successful before it gets to the next step. Bitbucket pipeline has a similar extension called Pipe [Bitbucket Pipes integrations | Bitbucket](https://bitbucket.org/product/features/pipelines/integrations?p=atlassian/trigger-pipeline) ---
**Author:** ![paula](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/paula/32/954_2.png) [@paula](https://forum.buildkite.community/u/paula)\ **Post date:** [May 27, 2022, 2:20am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/2 "2022-05-27T02:20:01Z")
Hey @ebanster šŸ‘‹ It looks like you could use the `trigger` step to start another pipeline against a different repository. The docs on how to use that step are at [Trigger step | Buildkite Documentation](https://buildkite.com/docs/pipelines/trigger-step#navicon). You can run it synchronously so the current pipeline will wait for the triggered one to finish and continue on success or stop on failure. You can also run it async, but that means it wonā€™t wait for it to finish. Hope that helps ---
**Author:** ![ebanster](https://avatars.discourse-cdn.com/v4/letter/e/41988e/32.png) [@ebanster](https://forum.buildkite.community/u/ebanster)\ **Post date:** [June 15, 2022, 6:20am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/3 "2022-06-15T06:20:17Z")
Thanks for this @paula . Sorry if Iā€™m missing it from the documentation you provided but I canā€™t find the reference to the other repository (I can only find a `branch` but not the URL of the repo). Or is it one of these? ```auto env: BUILDKITE_PULL_REQUEST: "${BUILDKITE_PULL_REQUEST}" BUILDKITE_PULL_REQUEST_BASE_BRANCH: "${BUILDKITE_PULL_REQUEST_BASE_BRANCH}" BUILDKITE_PULL_REQUEST_REPO: "${BUILDKITE_PULL_REQUEST_REPO}" ``` ---
**Author:** ![paula](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/paula/32/954_2.png) [@paula](https://forum.buildkite.community/u/paula)\ **Post date:** [June 16, 2022, 3:32am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/4 "2022-06-16T03:32:24Z")
No worries šŸ‘ A trigger will create a new build on a pipeline. So you would need to create another pipeline and configure that one against the repo you are wanting to build. Then in your pipeline that is doing the triggering you wonā€™t have to pass the repository because its configured at the pipeline already. So, you would have `pipelineA` that is building `repoA` and `pipelineB` that is building `repoB`. Then the steps in `pipelineA` have a trigger step targeting `pipelineB`. Eg: here you can see it trigger `pipelineB` on the same branch and commit (which if it is a different repo, might not match `repoA` but hopefully you can get the gist) ```yaml - trigger: pipelineB build: message: "${BUILDKITE_MESSAGE}" commit: "${BUILDKITE_COMMIT}" branch: "${BUILDKITE_BRANCH}" ``` ---
**Author:** ![ebanster](https://avatars.discourse-cdn.com/v4/letter/e/41988e/32.png) [@ebanster](https://forum.buildkite.community/u/ebanster)\ **Post date:** [June 16, 2022, 4:26am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/5 "2022-06-16T04:26:28Z")
Is this the correct illustration? ![Screen Shot 2022-06-16 at 2.21.08 pm](https://us1.discourse-cdn.com/flex016/uploads/buildkite1/original/2X/9/9118cc3eda59f1496fcac7cf9ea85449474a1145.png) ā€¦ and then inside the `test-pipeline.yml`, I will have the following? ```auto env: BUILDKITE_PULL_REQUEST: "${BUILDKITE_PULL_REQUEST}" ??? BUILDKITE_PULL_REQUEST_BASE_BRANCH: "master" BUILDKITE_PULL_REQUEST_REPO: "...test-repo.git" ``` ---
**Author:** ![paula](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/paula/32/954_2.png) [@paula](https://forum.buildkite.community/u/paula)\ **Post date:** [June 16, 2022, 4:57am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/6 "2022-06-16T04:57:04Z")
Yeah I think that makes sense šŸ‘ Iā€™m not sure those env vars would be appropriate here. Are you building on PRs? And do those PRs on each separate repository line up each time? Ie do you create a PR on ā€œbuild repoā€ with #x and then another one on ā€œtest repoā€ with #x or would it be a different number? My gut says to not pass these in this example since your first question said they are different repos. ---
**Author:** ![ebanster](https://avatars.discourse-cdn.com/v4/letter/e/41988e/32.png) [@ebanster](https://forum.buildkite.community/u/ebanster)\ **Post date:** [June 16, 2022, 5:37am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/7 "2022-06-16T05:37:29Z")
I guess my question is if I donā€™t have the `BUILDKITE_PULL_REQUEST_REPO` then how do I know which repo (in this case, `test-repo` or `repoB`) that I need to trigger? If both the build and test pipelines are inside the `build-repo` or `repoA` then what is the variable that I need to point to the other repoā€™s URL? ---
**Author:** ![paula](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/paula/32/954_2.png) [@paula](https://forum.buildkite.community/u/paula)\ **Post date:** [June 16, 2022, 6:21am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/8 "2022-06-16T06:21:34Z")
Oh right okay. I was mislead by your original question. This is still possible. You can have multiple pipelines per repository, ie if its a monorepo. You can set up a webhook to trigger only one from GitHub, then within that one (pipelineA) you trigger pipelineB like weā€™ve said above. But both pipelineA and B both have configured repoA as their repository, but only pipelineA is directly built from push events on GitHub and pipelineB is triggered from pipelineA. In that case you could pass those variables along because its the same repo. You might also want to check out this plugin to handle monorepo pipelines: [GitHub - adikari/monorepo-diff-buildkite-plugin: Run separate pipelines for each folder in your monorepo](https://github.com/chronotc/monorepo-diff-buildkite-plugin). This might be a nicer way to set up what you want. Have a play and see how you go ---
**Author:** ![ebanster](https://avatars.discourse-cdn.com/v4/letter/e/41988e/32.png) [@ebanster](https://forum.buildkite.community/u/ebanster)\ **Post date:** [June 16, 2022, 11:48am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/9 "2022-06-16T11:48:46Z")
Hi @paula sorry for confusing you further and thanks for being patient so far. We actually donā€™t have a monorepo but 2 separate repositories for our microservice / component and our e2e tests. First off, the reason my thread title is like that is because that is how the Bitbucket pipeline works. So for example, I have a `bitbucket-pipeline.yaml` on my microservice repository. On that single pipeline, it triggers the other e2e test repo to run and wait first with the following: ```auto script: - pipe: atlassian/trigger-pipeline:5.0.0 variables: BITBUCKET_USERNAME: '' BITBUCKET_APP_PASSWORD: '' REPOSITORY: ' **Author:** ![paula](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/paula/32/954_2.png) [@paula](https://forum.buildkite.community/u/paula)\ **Post date:** [June 16, 2022, 11:49pm UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/10 "2022-06-16T23:49:17Z") Oh okay šŸ‘ That goes back to this answer then to just trigger another pipeline: [Run multiple repositories in a single pipeline - #4 by paula](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/4). You donā€™t need to tell the trigger what repo because its configured against the pipeline you are triggering already. And in each repo you will need a `.buildkite/pipeline.yml` file which each respective pipeline the steps it needs to run. The steps in `repoA` (microservice) would have the trigger step in it as above. The steps in `repoB` (e2e tests) would run the test suite you have. ---
**Author:** ![ebanster](https://avatars.discourse-cdn.com/v4/letter/e/41988e/32.png) [@ebanster](https://forum.buildkite.community/u/ebanster)\ **Post date:** [June 16, 2022, 11:59pm UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/11 "2022-06-16T23:59:36Z")
Thanks @paula but Iā€™m still confused. ![Screen Shot 2022-06-16 at 2.21.08 pm](https://us1.discourse-cdn.com/flex016/uploads/buildkite1/original/2X/9/9118cc3eda59f1496fcac7cf9ea85449474a1145.png) Going back to my illustration above, how would the test pipeline know that I need to run the test repoā€™s `.buildkite/pipleine.yml` file? If my understanding is correct from your explanation, it should be something like this? ![Screen Shot 2022-06-17 at 9.57.29 am](https://us1.discourse-cdn.com/flex016/uploads/buildkite1/original/2X/1/1246c2bfe17b2d34b5356433f9025c60ad35ebaa.png) ā€¦ but again, my question is how would my build pipeline know that it needs to trigger the test pipeline sitting on another repo? Or is buildkite smart enough to know which pipelines to run regardless of which repo it sits against? ---
**Author:** ![paula](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/paula/32/954_2.png) [@paula](https://forum.buildkite.community/u/paula)\ **Post date:** [June 17, 2022, 7:58am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/12 "2022-06-17T07:58:18Z")
No probs. On each pipeline, you configured a specific repository that may have a .buildkite/pipeline.yml file (you can find more info on how the pipeline upload works here [buildkite-agent pipeline v3 | Buildkite Documentation](https://buildkite.com/docs/agent/v3/cli-pipeline#uploading-pipelines)) When you do a [trigger step](https://buildkite.com/docs/pipelines/trigger-step#navicon) (ie: create a new build on another pipeline), the build is going to check the configured repository from that ā€œtriggeredā€ pipeline, and, therefore its corresponding yml file. So you have 2 of everything here: - 1 repository that has the tests in it - 1 pipeline for this repository - this repository has a .buildkite/pipeline.yml file with steps that run those tests - 1 repository for the source code (that is being tested against) - 1 pipeline for this repository - this repository has a .buildkite/pipeline.yml file in it which uses a [trigger step](https://buildkite.com/docs/pipelines/trigger-step#navicon) to trigger the pipeline above which goes off and runs the steps for that pipeline Iā€™m not sure if you have given this a test try yet but I believe if you have a play around it will become apparent how it works. You can sign up for a free account and have 30 days trial to kick the tyres ---
**Author:** ![ebanster](https://avatars.discourse-cdn.com/v4/letter/e/41988e/32.png) [@ebanster](https://forum.buildkite.community/u/ebanster)\ **Post date:** [June 20, 2022, 10:08am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/13 "2022-06-20T10:08:37Z")
My mistake @paula . I got it now and its working as you explained. It initially got me confused as it works differently from the bitbucket pipeline but I think buildkite is smarter in this case. Off the top of your head, I was just wondering if conditional statement like this would work (not sure if its a correct yaml syntax with the `trigger` command): ```nohighlight - label: 'šŸ§Ŗ Run tests' if: build.branch == 'master' trigger: 'test-pipeline' ``` I wanted to run it only against `master` branch as right now, it is running with every branch. ---
**Author:** ![paula](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/paula/32/954_2.png) [@paula](https://forum.buildkite.community/u/paula)\ **Post date:** [June 20, 2022, 11:36pm UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/14 "2022-06-20T23:36:39Z")
Hey @ebanster! Yup, that should work just fine. You can test it out šŸ˜Š Best! ---
**Author:** ![ebanster](https://avatars.discourse-cdn.com/v4/letter/e/41988e/32.png) [@ebanster](https://forum.buildkite.community/u/ebanster)\ **Post date:** [June 21, 2022, 4:33am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/15 "2022-06-21T04:33:30Z")
Thanks @paula I will try it out. Sorry while weā€™re on this same thread, can I just verify this scenario: ```auto - label: "šŸ§Ŗ End to End Tests" if: build.branch == "master" trigger: "test-pipeline" build: env: EMAIL: "${EMAIL}" PASSWORD: "${PASSWORD}" ``` I have declared `EMAIL` and `PASSWORD` as CI variables in the build pipelineā€™s Build steps. If my test pipeline contains these variables and have different values, will it override those values with the one I set on the build pipeline? Example below: `build pipeline` EMAIL: [hello@test.net](mailto:hello@test.net) PASSWORD: 123456 `test pipeline` EMAIL: [hello@test.com](mailto:hello@test.com) PASSWORD: abc123 When the test pipeline is triggered, it will run against `hello@test.net` / `123456` instead. ---
**Author:** ![paula](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/paula/32/954_2.png) [@paula](https://forum.buildkite.community/u/paula)\ **Post date:** [June 21, 2022, 7:02am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/16 "2022-06-21T07:02:35Z")
Thatā€™s right, the env you specify on the trigger step will override the defaults you have set in the `test-pipeline` ---
**Author:** ![ebanster](https://avatars.discourse-cdn.com/v4/letter/e/41988e/32.png) [@ebanster](https://forum.buildkite.community/u/ebanster)\ **Post date:** [June 22, 2022, 12:18am UTC](https://forum.buildkite.community/t/run-multiple-repositories-in-a-single-pipeline/2058/17 "2022-06-22T00:18:38Z")
Thanks @paula and @paula you both are stars! Worked like a charm!