Define Value of Environment Variable as Regex

Hi!

My team has a pipeline that is run whenever we tag a commit. The tags come in a couple different formats (for example, we have different tags for releases versus release candidates) and some of the steps in the pipeline only need to run for certain types of tags. Currently, we’re using if conditionals with regular expressions to specify if a given step should run for a given tag.

Everything is working fine, but as we’ve added more steps we’ve had to copy-paste the same regular expressions into different steps so we thought it would be cleaner if we could define the regular expressions once as environment variables and then reference them in the if conditionals. It seems that the conditional parser doesn’t support this use case though as the type returned by the build.env function is a string but the parser is expecting a regular expression. So even though the value of the environment variable may be a valid regular expression, the parser doesn’t recognize it as such.

The following toy pipeline demonstrates the problem we’re running into.

env:
  ENV_REGEX: "/^foo\$/"
steps:
  - label: "Testing Regular Expressions: Env Var"
    if: 'build.env("ENV_KEY") =~ build.env("ENV_REGEX")'
    command: echo "env var"
  - label: "Testing Regular Expressions: Static"
    if: 'build.env("ENV_KEY") =~ /^foo\$/'
    command: echo "static"

The second step works fine and is the current approach we’re following, but becomes a little cumbersome if we want to refer to the same regular expression in multiple steps. The first step is considered invalid though and the following error is returned:

One of the steps you provided was invalid: Error parsing if expression: Unexpected type: expected regular expression but build.env returns string

Sorry @jerome, as you’ve discovered I don’t think we support coercing strings into regexps. But could you do:

env:
  ENV_REGEX: "/^foo\$/"
steps:
  - label: "Testing Regular Expressions: Env Var"
    if: 'build.env("ENV_KEY") =~ $ENV_REGEX'
    command: echo "env var"
  - label: "Testing Regular Expressions: Static"
    if: 'build.env("ENV_KEY") =~ /^foo\$/'
    command: echo "static"

I think the pipeline upload will interpolate the environment variable before upload, and so the conditional will see the literal regexp.

Hi @sj26! Thanks for the suggestion! Unfortunately, it seems that approach also runs into the same issue. Specifically, when I try to save the pipeline the following error is returned:

One of the steps you provided was invalid: Error parsing if expression: Unexpected type: expected regular expression but found string (line 1, column 25-34)

I also changed the environment variable to use single quotes instead of double quotes to work around a YAML escaping issue, i.e. '/^foo\$/'. Not sure if that information is really relevant but figured I would mention it just in case.

Another thing I tried was to move the forward slashes of the regex into the if condition and leave the rest of the regular expression in the environment variable, i.e.

env:
  ENV_REGEX: '^foo\$'

steps:
  - label: "Testing Regular Expressions: Env Var"
    if: 'build.env("ENV_KEY") =~ /${ENV_REGEX}/'
    command: echo "env var"

This approach didn’t work either, but it did fail differently! In this case, when I tried to run the pipeline, the state of the resulting build was not_run and the state of the only job in the build was broken, which makes me think the environment variable is being substituted into the regex, but that the resulting regex isn’t valid for some reason.

Sorry @jerome, I didn’t test the last solution rigorously, and was assuming it was in a yaml file. When adding directly to a pipeline on the Edit Steps page the pipeline will try to parse the conditionals at save time and choke on the variable interpolation. Here’s an example which I’ve observed working which interpolates the environment variable during the pipeline upload, but it is a mess of escaping:

env:
  ENV_REGEX: "/^foo\\\\\\\\$/"
steps:
  - command: |
      buildkite-agent pipeline upload <<YAML
      steps:
        - label: "Testing Regular Expressions: Env Var"
          if: build.env("ENV_KEY") =~ $ENV_REGEX
          command: echo "env var"
        - label: "Testing Regular Expressions: Static"
          if: build.env("ENV_KEY") =~ /^foo\\\\\$/
          command: echo "static"
      YAML

and when ENV_KEY=foo:

It’s a pretty gnarly workaround, but it works.

I’d also love us to be able to coerce strings into regexps, to make "..." =~ "..." work, but I can’t promise when we can make that happen sorry.

Hi @sj26! Thanks for the suggestion! Our plan is to keep the pipeline definition in a YAML file, I was just using the “Edit Steps” page in my previous post since I found it easier to work with. The YAML escaping above is quite gnarly :grinning_face_with_smiling_eyes:, but since we only need to define it once might not be so bad. Thanks for all your help!