Collect artifacts when timeout

General
1

Hi BK Community,

We have a soft-fail step where we would like to collect all artifacts when it finished (or failed). Right now, we are using artifact_paths.

However, when a timeout occurred on that step, it seems that artifacts are not being collected. Is there a good solution to collect them even when timeout happened?

2

Hi @dyrock :wave:

Welcome to the community! The artifact_paths parameter in the command step is designed to automatically upload artifacts in the paths configured when the command step finishes running - if there is a timeout mid-execution, the artifacts won’t get uploaded.

You could have your build scripts upload your artifacts using the buildkite-agent artifact upload command so that the artifacts are uploaded to your artifact storage as soon as they are written, or at the very least give you more control in the script when the artifact is being uploaded.

Hope that helps!

Cheers,
Jeremy

3

Hey @jeremy,

I took a swing at this in our context, and to my surprise (which I couldn’t suss-out from the processing order of hooks outlined here) repository-local hooks don’t get executed on build-step timeouts. Is there a repository-local way to do this? Or does it require changes to our agent hooks and a new agent pool rolled out?

Is there any documentation you can point me to for the hook-execution order for when steps timeout?

Cheers,
Tim

4

Hey @timf,

welcome back to the Buildkite Community.

It depends on the specific setup and configuration of your Buildkite environment.

There is a timeout/cancel grace period set on the agent, which by default is 10 seconds. You can find more details about this in the Buildkite documentation here. When the agent receives the cancel signal from Buildkite, this grace period begins. During this time, the agent will attempt to run the pre-exit hook and perform any necessary cleanup. You can increase the BUILDKITE_CANCEL_GRACE_PERIOD env variable and make sure you are using the v3.73.1 agent version, which contains fix related to this.

Thanks,

5

Thanks for the prompt reply @ivanna!

I’m using a v3.73.0 agent, however the code-changes look to be configuration only and tweaking our configuration to align was easy enough (however to no-avail). Once thing I noticed, was how the hook chain gets executed (agent/internal/job/artifacts.go at main · buildkite/agent · GitHub), and our global pre-artifact hook has a non-zero exit code (so the local hook isn’t running). However, our pre-artifact hook is trivial (exports only).

I’m curious if you have other ideas you could throw my way. I did spot:

		// although error is an exit error, it's not returned. (seems like a bug)
		// TODO: investigate phasing this out under a experiment
		return nil, nil

Which raises some alarms – as this seems like a BK agent bug, but I’m happy to keep trying what I can!

Please advise,
Tim

6

Hey @timf,

To debug this issue, I need a few more details from your setup. Could you please provide the following information?

1.	Contents of the Global Pre-Artifact Hook: Please share the script or code that is currently being used in your global pre-artifact hook. 
2.	Additional Logging: To get more insight into what’s happening, could you add some logging statements to your global pre-artifact hook?  Once you have done that, please provide the output logs generated by these logging statements.
3.	Pipeline Link and Error Details: If possible, share the link to the specific pipeline that is encountering these errors along with the exact error messages being thrown.

For security reason, I would recommend to share the existing links and code via email. Buildkite Support email is support@buildkite.com

Thanks,

7

Email sent. For the community, our pre-artifact hook is really simple:

#!/bin/bash
set -euo pipefail

export BUILDKITE_ARTIFACT_UPLOAD_DESTINATION="s3://XXXXXXXXXX/${BUILDKITE_JOB_ID:-1}"
export BUILDKITE_S3_ACL='private'
export BUILDKITE_S3_DEFAULT_REGION='XXXXXXXXXX'

And the error looks like:

Running global pre-artifact hook
$ /etc/buildkite-agent/hooks/pre-artifact
🚨 Error: The global pre-artifact hook exited with status -1

There really isn’t anything to log in our pre-artifact hook. :sweat_smile: