Incorrect escaping of string values with special characters


#1

Hi there,

I activated the new yaml pipeline steps and noticed something odd with environment variables defined like this:

env:
  MY_VAR: 'abc\ndef'

steps:
...

The build script gets the value as ‘abcndef’ (notice the missing backslash).
I expected the value to be the literal string I entered ‘abc\ndef’.
I checked and did some local tests with yaml parsing and it behaves as I expect it.

irb(main):001:0> require 'yaml'
=> true
irb(main):002:0> content = `cat abc.yml`
=> "env:\n  MY_VAR: 'abc\\ndef'\n\n"
irb(main):003:0> loaded = YAML.load(File.read("abc.yml"))
=> {"env"=>{"MY_VAR"=>"abc\\ndef"}}
irb(main):004:0> ENV['MY_VAR'] = loaded['env']['MY_VAR']
=> "abc\\ndef"
irb(main):005:0> test_env = `echo $MY_VAR`
=> "abc\ndef\n"
irb(main):006:0> test_env = `echo $MY_VAR`.chomp
=> "abc\ndef"

What’s happening on buildkite side?

Thanks


#2

Hey @JeanRegisser, I just did some local testing with your example and can confirm the bug!

If I change it too…

env:
  MY_VAR: "abc\ndef"

…and use double quotes instead - I get the desired behaviour.

It looks like when: "this\nline" is parsed by YAML, it’s turned into "this\nline" (the exact same thing) - which we handle correctly…

However 'this\nline' gets parsed as "here\\ngreat" - which our interpreter seems to do something silly with (it looks like we strip the \\ - which we probably shouldn’t do). I think defining a newline with \n isn’t something you can do in YAML single quotes, you’ll always need to use double quotes.

So as a first step, if you change your code to use double quotes - I think that’d work. I’ll look at our parser today and see if I can stop it from stripping those \\.


#3

Glad you’re confirming this.

I discovered this while trying to store a whole json data in an env var (I know this isn’t common but a tool I’m using needed it).

Something like this but with a bigger json data:

env:
  SOME_JSON: |-
    {
      "xxx": "value\nmore_stuff"
    } 

Manually adding more escape would have been a bit too cumbersome and risky to workaround the parser bug in buildkite.
So for now I’ve set that var directly in the agent environment hook.

Hope you can find a sane fix for this.
So nobody gets tripped by this later on.

Thanks,


#4

@JeanRegisser I fixed the escape sequence bug you found!

This is the best way I’ve found to get what you’re after:

env:
  SOME_JSON: '{ "xxx": "value\nmore_stuff" }'

steps:
  - command: "echo \"$$SOME_JSON\" | jq"

Using single quotes in the SOME_JSON definition ensures our YAML parser treats “\n” as a literal - and not an actual new line.