[shard 3/4] [2026-10-28 23:20:37] Awaiting shard 0's workspace module cache preparation outcome before creating this follower's droplet
[shard 3/4] [2026-10-28 23:21:33] Shard 0 failed before its workspace module cache preparation began (ProvisioningFailureException: Provisioning failed for runId=bd2844d6 shardIndex=0 method=createBuildDroplet attempts=1/3 elapsedMs=56475 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-bd2844d6-a00854c' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: DropletServiceCallStalledException: Droplet service call createDropletAsync stalled for 30000ms while creating 'buildtest-bd2844d6-a00854c' (region=sfo3, size=s-2vcpu-4gb, image=245388985).); this follower's droplet was not created and its runner will not start.
[shard 3/4] [2026-10-28 23:21:33] ERROR: Provisioning failed for runId=bd2844d6 shardIndex=0 method=createBuildDroplet attempts=1/3 elapsedMs=56475 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-bd2844d6-a00854c' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: DropletServiceCallStalledException: Droplet service call createDropletAsync stalled for 30000ms while creating 'buildtest-bd2844d6-a00854c' (region=sfo3, size=s-2vcpu-4gb, image=245388985).
[shard 3/4] [2026-10-29 02:31:17] Awaiting shard 0's workspace module cache preparation outcome before creating this follower's droplet
[shard 3/4] [2026-10-29 02:44:31] Shard 0's workspace module cache is ready; creating this follower's droplet
[shard 3/4] [2026-10-29 02:47:32] Droplet created: id=613722764, name=buildtest-bd2844d6-s3-a14b37a
[shard 3/4] [2026-10-29 02:47:32] Droplet active at 164.90.153.151
[shard 3/4] [2026-10-29 02:47:35] SSH access granted to droplet 613722764
[shard 3/4] [2026-10-29 02:47:35] Waiting for guest readiness on droplet 613722764
[shard 3/4] [2026-10-29 02:47:53] Guest readiness complete on droplet 613722764
[shard 3/4] [2026-10-29 02:47:54] Dynamic memory admission: droplet 3 has 4096 MiB RAM, 3221225472 bytes available to resident tests, and runner parallelism 4
[shard 3/4] [2026-10-29 02:47:55] Connecting via SSH and installing dependencies...
[shard 3/4] [2026-10-29 02:47:58] SSH connected
[shard 3/4] Stopped, disabled, and masked unattended-upgrades.service, apt-daily.service, apt-daily-upgrade.service, apt-daily.timer, and apt-daily-upgrade.timer; waited 0 seconds for apt/dpkg locks to clear
[shard 3/4] Dependencies already present (snapshot image), skipping installation
[shard 3/4] Java: openjdk version "21.0.11" 2026-04-21
[shard 3/4] Node.js: v18.19.1
[shard 3/4] Chromium: /usr/bin/chromium-browser
[shard 3/4] Coursier: /root/tools/coursier
[shard 3/4] Docker already present: Docker version 29.1.3, build 29.1.3-0ubuntu3~24.04.2
[shard 3/4] haveged already present
[shard 3/4] securerandom.source=file:/dev/urandom already set in /usr/lib/jvm/java-21-openjdk-amd64/conf/security/java.security
[shard 3/4] Dependencies installed successfully
[shard 3/4] Install image id: 245388985; base dependency fast path taken: true
[shard 3/4] [BuildDriver] Memory before swap: MemoryInfo(totalMemoryKb=4009860, availableMemoryKb=3500752, totalSwapKb=0, freeSwapKb=0)
[shard 3/4] mkswap: /swapfile: warning: wiping old swap signature.
[shard 3/4] Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
[shard 3/4] no label, UUID=800b59c8-3f30-4e70-8de8-d5f4186ebd61
[shard 3/4] Swap file created and enabled
[shard 3/4] [BuildDriver] Memory after swap: MemoryInfo(totalMemoryKb=4009860, availableMemoryKb=3496208, totalSwapKb=4194300, freeSwapKb=4194300)
[shard 3/4] [BuildDriver] Swap verified: 4194300KB total swap (was 0KB)
[shard 3/4] [2026-10-29 02:48:07] Dependencies installed
[shard 3/4] [2026-10-29 02:48:08] Uploading project archive...
[shard 3/4] [2026-10-29 02:48:08] Waiting for the shared project archive source...
[shard 3/4] [2026-10-29 02:48:08] Fetching project archive directly from source shard 0...
[shard 3/4] Extracted projects:
[shard 3/4] [2026-10-29 02:48:10] Projects on remote: CodexCoder21Organization-HandoffServiceServer-82480ed9f504cca2c4811c534c1d17934c10cbfa
[shard 3/4] [2026-10-29 02:48:12] Projects from archive: handoff:serviceserver
[shard 3/4] [2026-10-29 02:48:13] Fetching workspace module cache directly from shard 0...
[shard 3/4] [2026-10-29 02:48:55] Installed workspace module cache fetched directly from shard 0
[shard 3/4] [2026-10-29 02:48:55] Building and testing workspace...
[shard 3/4] [2026-10-29 02:48:55] Shard test specs (0): 
[shard 3/4] System memory: 3915MB, JVM heap: 1957MB
[shard 3/4] Downloading https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.124/buildtest-runner-0.0.124.pom
[shard 3/4] Downloaded https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.124/buildtest-runner-0.0.124.pom
[shard 3/4] Downloading https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.124/buildtest-runner-0.0.124.pom.sha1
[shard 3/4] Downloaded https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.124/buildtest-runner-0.0.124.pom.sha1
[shard 3/4] Downloading https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.pom
[shard 3/4] Downloading https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.pom
[shard 3/4] Downloading https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.pom
[shard 3/4] Downloading https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.pom
[shard 3/4] Downloading https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.pom
[shard 3/4] Downloading https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.pom
[shard 3/4] Downloaded https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.pom
[shard 3/4] Downloading https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.pom
[shard 3/4] Downloaded https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.pom
[shard 3/4] Downloading https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.pom.sha1
[shard 3/4] Downloaded https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.pom
[shard 3/4] Downloading https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.pom.sha1
[shard 3/4] Downloaded https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.pom
[shard 3/4] Downloading https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.pom.sha1
[shard 3/4] Failed to download https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.pom.sha1
[shard 3/4] Downloading https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.pom.sha1
[shard 3/4] Downloaded https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.pom
[shard 3/4] Downloading https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.pom.sha1
[shard 3/4] Downloaded https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.pom
[shard 3/4] Downloading https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.pom.sha1
[shard 3/4] Downloaded https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.pom
[shard 3/4] Downloading https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.pom.sha1
[shard 3/4] Downloaded https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.pom.sha1
[shard 3/4] Downloaded https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.pom.sha1
[shard 3/4] Downloaded https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.pom.sha1
[shard 3/4] Downloaded https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.pom.sha1
[shard 3/4] Downloaded https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.pom.sha1
[shard 3/4] Downloaded https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.pom.sha1
[shard 3/4] Downloading https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.jar
[shard 3/4] Downloading https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.jar
[shard 3/4] Downloading https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.jar
[shard 3/4] Downloading https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.jar
[shard 3/4] Downloading https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.jar
[shard 3/4] Downloading https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.jar
[shard 3/4] Downloaded https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.jar
[shard 3/4] Downloading https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.jar
[shard 3/4] Downloaded https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.jar
[shard 3/4] Downloading https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.124/buildtest-runner-0.0.124.jar
[shard 3/4] Downloaded https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.jar
[shard 3/4] Downloading https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.jar.sha1
[shard 3/4] Downloaded https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.jar.sha1
[shard 3/4] Downloading https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.jar.sha1
[shard 3/4] Downloaded https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.jar
[shard 3/4] Downloading https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.jar.sha1
[shard 3/4] Downloaded https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.jar.sha1
[shard 3/4] Downloading https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.jar.sha1
[shard 3/4] Downloaded https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.jar
[shard 3/4] Downloading https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.jar.sha1
[shard 3/4] Downloaded https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.124/buildtest-runner-0.0.124.jar
[shard 3/4] Downloading https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.124/buildtest-runner-0.0.124.jar.sha1
[shard 3/4] Downloaded https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.jar.sha1
[shard 3/4] Downloaded https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.jar
[shard 3/4] Downloading https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.jar.sha1
[shard 3/4] Downloaded https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.jar.sha1
[shard 3/4] Failed to download https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.jar.sha1
[shard 3/4] Downloaded https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.124/buildtest-runner-0.0.124.jar.sha1
[shard 3/4] Downloaded https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.jar.sha1
[shard 3/4] Downloaded https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.jar
[shard 3/4] Downloading https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.jar.sha1
[shard 3/4] Downloaded https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.jar.sha1
[shard 3/4] [2026-10-29 06:22:17] provisioning_failure runId=bd2844d6 shardIndex=3 shardCount=4 method=createBuildDroplet attempt=1 maxAttempts=3 elapsedMs=130962 classification=INFRASTRUCTURE/PROVISIONING action=exhausted error=DropletServiceCallStalledException: Droplet creation outcome for 'buildtest-bd2844d6-s3-a9d3a9a' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: DropletServiceCallStalledException: Droplet service call getDropletCreationStatus stalled for 30000ms while waiting for droplet 'buildtest-bd2844d6-s3-a9d3a9a' (requestId=645fea6e-62c9-4d2d-838d-a62ceabdc17b).
[shard 3/4] [2026-10-29 06:22:17] ERROR: Provisioning failed for runId=bd2844d6 shardIndex=3 method=createBuildDroplet attempts=1/3 elapsedMs=130962 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-bd2844d6-s3-a9d3a9a' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: DropletServiceCallStalledException: Droplet service call getDropletCreationStatus stalled for 30000ms while waiting for droplet 'buildtest-bd2844d6-s3-a9d3a9a' (requestId=645fea6e-62c9-4d2d-838d-a62ceabdc17b).