Buildkite Agent with spawn=2 will randomly fail/create new agents

2019-05-24 06:58:46 DEBUG  CICDM016-1 GET https://agent.buildkite.com/v3/jobs/a12e2a3d-be99-4080-9bef-d1a4d3c9505f 
2019-05-24 06:58:46 DEBUG  CICDM016-1 ↳ GET https://agent.buildkite.com/v3/jobs/a12e2a3d-be99-4080-9bef-d1a4d3c9505f proto=HTTP/2.0 status=200 Δ=27.460123ms

  _           _ _     _ _    _ _                                _
 | |         (_) |   | | |  (_) |                              | |
 | |__  _   _ _| | __| | | ___| |_ ___    __ _  __ _  ___ _ __ | |_
 | '_ \| | | | | |/ _` | |/ / | __/ _ \  / _` |/ _` |/ _ \ '_ \| __|
 | |_) | |_| | | | (_| |   <| | ||  __/ | (_| | (_| |  __/ | | | |_
 |_.__/ \__,_|_|_|\__,_|_|\_\_|\__\___|  \__,_|\__, |\___|_| |_|\__|
                                                __/ |
 http://buildkite.com/agent                    |___/

2019-05-24 07:03:17 NOTICE Starting buildkite-agent v3.11.2 with PID: 772
2019-05-24 07:03:17 NOTICE The agent source code can be found here: https://github.com/buildkite/agent
2019-05-24 07:03:17 NOTICE For questions and support, email us at: hello@buildkite.com
2019-05-24 07:03:17 INFO   Configuration loaded path=/usr/local/etc/buildkite-agent/buildkite-agent.cfg
2019-05-24 07:03:17 DEBUG  Bootstrap command: /usr/local/bin/buildkite-agent bootstrap
2019-05-24 07:03:17 DEBUG  Build path: /Users/cicd/.buildkite-agent/builds
2019-05-24 07:03:17 DEBUG  Hooks directory: /usr/local/etc/buildkite-agent/hooks
2019-05-24 07:03:17 DEBUG  Plugins directory: /Users/cicd/.buildkite-agent/plugins
2019-05-24 07:03:17 INFO   Registering agent 1 of 2 with Buildkite...
2019-05-24 07:03:17 DEBUG  POST https://agent.buildkite.com/v3/register
2019-05-24 07:03:18 DEBUG  ↳ POST https://agent.buildkite.com/v3/register proto=HTTP/2.0 status=201 Δ=266.298053ms
2019-05-24 07:03:18 INFO   Registering agent 2 of 2 with Buildkite...
2019-05-24 07:03:18 DEBUG  POST https://agent.buildkite.com/v3/register
2019-05-24 07:03:18 DEBUG  ↳ POST https://agent.buildkite.com/v3/register proto=HTTP/2.0 status=201 Δ=37.774678ms
2019-05-24 07:03:18 INFO   CICDM016-3 Connecting to Buildkite... 
2019-05-24 07:03:18 INFO   CICDM016-4 Connecting to Buildkite... 
2019-05-24 07:03:18 DEBUG  CICDM016-3 POST https://agent.buildkite.com/v3/connect 
2019-05-24 07:03:18 INFO   Started 2 Agent(s)
2019-05-24 07:03:18 INFO   You can press Ctrl-C to stop the agents
2019-05-24 07:03:18 DEBUG  CICDM016-4 POST https://agent.buildkite.com/v3/connect 
2019-05-24 07:03:18 DEBUG  CICDM016-4 ↳ POST https://agent.buildkite.com/v3/connect proto=HTTP/2.0 status=200 Δ=129.825443ms
2019-05-24 07:03:18 INFO   CICDM016-4 Waiting for work... 
2019-05-24 07:03:18 DEBUG  CICDM016-4 GET https://agent.buildkite.com/v3/ping 
2019-05-24 07:03:18 DEBUG  CICDM016-3 ↳ POST https://agent.buildkite.com/v3/connect proto=HTTP/2.0 status=200 Δ=135.57234ms
2019-05-24 07:03:18 INFO   CICDM016-3 Waiting for work... 
2019-05-24 07:03:18 DEBUG  CICDM016-3 GET https://agent.buildkite.com/v3/ping 
2019-05-24 07:03:18 DEBUG  CICDM016-3 ↳ GET https://agent.buildkite.com/v3/ping proto=HTTP/2.0 status=200 Δ=38.426374ms

From the logs above, I don’t see any reason this would happen. It caused some of our jobs to get “Agent Lost”. Other nodes running buildkite agents from the same network/switch were fine (I looked at their logs).

What do you suggest I check?

@lox can you spot anything awry in the log here?

I’d upgrade to the latest, there have been several fixes that might be relevant.

Thank you gentlemen, I will notify you once we’re upgraded and if we see anything