Odd behavior using bash here-string syntax in plugins

General
1

Hi Buildkite,

I wrote a plugin with this line

RESULT=$( echo ${!ENTRY_DETAIL} | jq -R )

or

RESULT=$( jq -R <<<"${!ENTRY_DETAIL}")

it works locally on my machine running with bk local run

but when its running on the remote agent
this was the output when I turn on bash debugging

++++ jq -R
jq - commandline JSON processor [version 1.5]
Usage: jq [options] <jq filter> [file...]
 
	jq is a tool for processing JSON inputs, applying the
	given filter to its JSON text inputs and producing the
	filter's results as JSON on standard output.
	The simplest filter is ., which is the identity filter,
	copying jq's input to its output unmodified (except for
	formatting).
	For more advanced filters see the jq(1) manpage ("man jq")
	and/or https://stedolan.github.io/jq
 
	Some of the options include:
	 -c		compact instead of pretty-printed output;
	 -n		use `null` as the single input value;
	 -e		set the exit status code based on the output;
	 -s		read (slurp) all inputs into an array; apply filter to it;
	 -r		output raw strings, not JSON texts;
	 -R		read raw strings, not JSON texts;
	 -C		colorize JSON;
	 -M		monochrome (don't colorize JSON);
	 -S		sort keys of objects on output;
	 --tab	use tabs for indentation;
	 --arg a v	set variable $a to value <v>;
	 --argjson a v	set variable $a to JSON value <v>;
	 --slurpfile a f	set variable $a to an array of JSON texts read from <f>;
	See the manpage for more options.

any help would be appreciated!

Thanks!!!

2

Hey @lewis!

Welcome to the community :blush:

It sounds like it could be something with your code :thinking: just running jq -R throws that error you are seeing.
Running some tests with your code, I got the following result:


$ unset ENTRY_DETAIL
$ echo "${!ENTRY_DETAIL}" | jq -R
bash: ENTRY_DETAIL: invalid indirect expansion
$ export ENTRY_DETAIL_1=1
$ echo "${!ENTRY_DETAIL}" | jq -R
bash: ENTRY_DETAIL: invalid indirect expansion
$ export ENTRY_DETAIL=1
$ echo "${!ENTRY_DETAIL}" | jq -R
""

If you can share a bit more of your code we can take a deeper look :slight_smile: (you can send an email to support@buildkite.com in case you have sensitive information)

Best!

3

Hi @paula !

the short example of this will look something like

export HELLO='{"data":{"name":"authn","path":"authn/d9ffa304dcd2ee153a8473d7121c2ebc"}}'
RESULT=$( jq -R <<<"${HELLO}")
echo $RESULT

The long answer will look something like

# user of the plugin will set metadata using agent
export HELLO='{"data":{"name":"authn","path":"authn/d9ffa304dcd2ee153a8473d7121c2ebc"}}'

buildkite-agent meta-data set "HELLO" $$HELLO
# the HELLO key will be passed as a argument in the plugin
# plugins:
#   - my-plugin#1.2.0:
#        KEY: 'HELLO'

# on the plugin side
# post-command

ENTRY_DETAIL=$(buildkite-agent meta-data get "HELLO")
RESULT=$( jq -R <<<"${ENTRY_DETAIL}")

echo $RESULT

the full code is quite complex and long but it can be found here

Thanks!

4

@paula

You can try running the plugin yourself as well if you want

steps:
  - label: ':amazon-eventbridge: Publishing Event!'
    commands:
      - echo "bye!!!"
      - ENTRY_DETAIL='{"Hello":"myman"}'
      - buildkite-agent meta-data set 'ENTRY_DETAIL' $$ENTRY_DETAIL
    plugins:
      - tapendium/aws-put-event#v1.0.4a:
          entries:
            - source: 'authn'
              resources: 'resource1,resource2'
              detail-type: 'frontend-build-completed'
              event-bus-name: 'EventBusArn'
              detail-env: 'ENTRY_DETAIL'
5

Thanks! That was very helpful! :raised_hands:

This was executed as shown in the logs:

aws events put-events '--entries Detail="{\"Hello\":\"myman\"}",Source=authn,DetailType=frontend-build-completed,EventBusName=EventBusArn,Resources=resource1,resource2' --output json

Unfortunately that means that it worked as expected, ENTRY_DETAIL is set so this line gets executed with no error to build the data on the Details parameter :thinking:

Not sure what could be your issue. Can you compare the bash and jq version you are running on your local computer with the one you run on the agent? It could be some difference in configuration

Best!

6

@paula

sorry for asking this but could you let me know the version of bash and jq
your agent is using perhaps?

thanks

7

Hey @lewis! That’s something that you control, you need to ensure that jq is installed in your agents and the version you need. The agent it’s a small build runner, and you need to make sure to have everything in it to be able to run builds.

best!