← Back to all runs

Build Run: b6699ad4

CodexCoder21Organization/UrlResolver

Stored run summary updated . Data age: 1365m 2s.

Build Rules

Test Results

[buildtest-coordinator] [2026-10-06 12:03:17] Coordinator work: waiting for archive analysis, queue position 3 of 3
[2026-10-06 12:04:44] Admission accounting: reserved=314 unreserved=0 requested=10 limit=200 observedFleet=77 fleetObservationValid=true
[2026-10-06 12:04:44] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=77 unreservedObserved=0 reservationsSum=314 reservationHolders=44 reservationHoldersWithNoObservedDroplet=16 otherGrantedRemainder=122 dropletLimit=200 queueDepth=75 queuePos=53 fleetObservationValid=true fleetObservationAge=16s fleetObservationUsable=true reservedProjection=209/200 freshProjection=209/200]
[2026-10-06 12:04:44] Waiting for droplet capacity; retrying admission in 30s
[2026-10-06 12:05:15] Admission accounting: reserved=304 unreserved=0 requested=10 limit=200 observedFleet=77 fleetObservationValid=true
[2026-10-06 12:05:15] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=77 unreservedObserved=0 reservationsSum=304 reservationHolders=43 reservationHoldersWithNoObservedDroplet=15 otherGrantedRemainder=122 dropletLimit=200 queueDepth=74 queuePos=52 fleetObservationValid=true fleetObservationAge=47s fleetObservationUsable=true reservedProjection=209/200 freshProjection=209/200]
[2026-10-06 12:05:15] Waiting for droplet capacity; retrying admission in 60s
[2026-10-06 12:06:16] Admission accounting: reserved=304 unreserved=0 requested=10 limit=200 observedFleet=80 fleetObservationValid=true
[2026-10-06 12:06:16] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=80 unreservedObserved=0 reservationsSum=304 reservationHolders=43 reservationHoldersWithNoObservedDroplet=15 otherGrantedRemainder=119 dropletLimit=200 queueDepth=74 queuePos=52 fleetObservationValid=true fleetObservationAge=24s fleetObservationUsable=true reservedProjection=209/200 freshProjection=209/200]
[2026-10-06 12:06:16] Waiting for droplet capacity; retrying admission in 120s
[2026-10-06 12:08:16] Admission accounting: reserved=304 unreserved=0 requested=10 limit=200 observedFleet=71 fleetObservationValid=true
[2026-10-06 12:08:16] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=71 unreservedObserved=0 reservationsSum=304 reservationHolders=43 reservationHoldersWithNoObservedDroplet=15 otherGrantedRemainder=128 dropletLimit=200 queueDepth=79 queuePos=52 fleetObservationValid=true fleetObservationAge=18s fleetObservationUsable=true reservedProjection=209/200 freshProjection=209/200]
[2026-10-06 12:08:16] Waiting for droplet capacity; retrying admission in 240s
[2026-10-06 12:12:17] Admission accounting: reserved=304 unreserved=0 requested=10 limit=200 observedFleet=66 fleetObservationValid=true
[2026-10-06 12:12:17] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=66 unreservedObserved=0 reservationsSum=304 reservationHolders=43 reservationHoldersWithNoObservedDroplet=16 otherGrantedRemainder=131 dropletLimit=200 queueDepth=81 queuePos=52 fleetObservationValid=true fleetObservationAge=25s fleetObservationUsable=true reservedProjection=207/200 freshProjection=207/200]
[2026-10-06 12:12:17] Waiting for droplet capacity; retrying admission in 300s
[2026-10-06 12:17:19] Admission accounting: reserved=302 unreserved=0 requested=10 limit=200 observedFleet=58 fleetObservationValid=true
[2026-10-06 12:17:19] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=58 unreservedObserved=0 reservationsSum=302 reservationHolders=42 reservationHoldersWithNoObservedDroplet=16 otherGrantedRemainder=138 dropletLimit=200 queueDepth=81 queuePos=49 fleetObservationValid=true fleetObservationAge=36s fleetObservationUsable=true reservedProjection=206/200 freshProjection=206/200]
[2026-10-06 12:17:19] Waiting for droplet capacity; retrying admission in 300s
[2026-10-06 12:22:30] Admission accounting: reserved=300 unreserved=0 requested=10 limit=200 observedFleet=53 fleetObservationValid=true
[2026-10-06 12:22:30] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=53 unreservedObserved=0 reservationsSum=300 reservationHolders=41 reservationHoldersWithNoObservedDroplet=15 otherGrantedRemainder=143 dropletLimit=200 queueDepth=85 queuePos=48 fleetObservationValid=true fleetObservationAge=38s fleetObservationUsable=true reservedProjection=206/200 freshProjection=206/200]
[2026-10-06 12:22:30] Waiting for droplet capacity; retrying admission in 300s
[2026-10-06 12:27:31] Admission accounting: reserved=300 unreserved=0 requested=10 limit=200 observedFleet=52 fleetObservationValid=true
[2026-10-06 12:27:31] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=52 unreservedObserved=0 reservationsSum=300 reservationHolders=41 reservationHoldersWithNoObservedDroplet=15 otherGrantedRemainder=144 dropletLimit=200 queueDepth=87 queuePos=48 fleetObservationValid=true fleetObservationAge=49s fleetObservationUsable=true reservedProjection=206/200 freshProjection=206/200]
[2026-10-06 12:27:31] Waiting for droplet capacity; retrying admission in 300s
[2026-10-06 12:32:33] Admission accounting: reserved=295 unreserved=0 requested=10 limit=200 observedFleet=47 fleetObservationValid=true
[2026-10-06 12:32:33] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=47 unreservedObserved=0 reservationsSum=295 reservationHolders=40 reservationHoldersWithNoObservedDroplet=15 otherGrantedRemainder=153 dropletLimit=200 queueDepth=85 queuePos=46 fleetObservationValid=true fleetObservationAge=50s fleetObservationUsable=true reservedProjection=210/200 freshProjection=210/200]
[2026-10-06 12:32:33] Waiting for droplet capacity; retrying admission in 300s
[2026-10-06 12:37:34] Admission accounting: reserved=255 unreserved=10 requested=10 limit=200 observedFleet=80 fleetObservationValid=true
[2026-10-06 12:37:34] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=80 unreservedObserved=10 reservationsSum=255 reservationHolders=35 reservationHoldersWithNoObservedDroplet=11 otherGrantedRemainder=120 dropletLimit=200 queueDepth=83 queuePos=42 fleetObservationValid=true fleetObservationAge=1s fleetObservationUsable=true reservedProjection=210/200 freshProjection=210/200]
[2026-10-06 12:37:34] Waiting for droplet capacity; retrying admission in 300s
[2026-10-06 12:42:35] Admission accounting: reserved=253 unreserved=0 requested=10 limit=200 observedFleet=79 fleetObservationValid=true
[2026-10-06 12:42:35] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=79 unreservedObserved=0 reservationsSum=253 reservationHolders=34 reservationHoldersWithNoObservedDroplet=10 otherGrantedRemainder=119 dropletLimit=200 queueDepth=85 queuePos=40 fleetObservationValid=true fleetObservationAge=27s fleetObservationUsable=true reservedProjection=208/200 freshProjection=208/200]
[2026-10-06 12:42:35] Waiting for droplet capacity; retrying admission in 300s
[2026-10-06 12:47:42] Admission accounting: reserved=209 unreserved=4 requested=10 limit=200 observedFleet=108 fleetObservationValid=true
[2026-10-06 12:47:42] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=108 unreservedObserved=4 reservationsSum=209 reservationHolders=24 reservationHoldersWithNoObservedDroplet=8 otherGrantedRemainder=85 dropletLimit=200 queueDepth=76 queuePos=30 fleetObservationValid=true fleetObservationAge=9s fleetObservationUsable=true reservedProjection=203/200 freshProjection=203/200]
[2026-10-06 12:47:42] Waiting for droplet capacity; retrying admission in 300s
[2026-10-06 12:52:43] Admission accounting: reserved=201 unreserved=0 requested=10 limit=200 observedFleet=97 fleetObservationValid=true
[2026-10-06 12:52:43] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=97 unreservedObserved=0 reservationsSum=201 reservationHolders=26 reservationHoldersWithNoObservedDroplet=4 otherGrantedRemainder=94 dropletLimit=200 queueDepth=71 queuePos=19 fleetObservationValid=true fleetObservationAge=8s fleetObservationUsable=true reservedProjection=201/200 freshProjection=201/200]
[2026-10-06 12:52:43] Waiting for droplet capacity; retrying admission in 300s
[2026-10-06 12:57:43] Admission accounting: reserved=203 unreserved=0 requested=10 limit=200 observedFleet=72 fleetObservationValid=true
[2026-10-06 12:57:43] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=72 unreservedObserved=0 reservationsSum=203 reservationHolders=26 reservationHoldersWithNoObservedDroplet=1 otherGrantedRemainder=121 dropletLimit=200 queueDepth=70 queuePos=12 fleetObservationValid=true fleetObservationAge=47s fleetObservationUsable=true reservedProjection=203/200 freshProjection=203/200]
[2026-10-06 12:57:43] Waiting for droplet capacity; retrying admission in 300s
[2026-10-06 13:02:45] Admission accounting: reserved=203 unreserved=0 requested=10 limit=200 observedFleet=67 fleetObservationValid=true
[2026-10-06 13:02:45] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=67 unreservedObserved=0 reservationsSum=203 reservationHolders=24 reservationHoldersWithNoObservedDroplet=1 otherGrantedRemainder=126 dropletLimit=200 queueDepth=70 queuePos=8 fleetObservationValid=true fleetObservationAge=53s fleetObservationUsable=true reservedProjection=203/200 freshProjection=203/200]
[2026-10-06 13:02:45] Waiting for droplet capacity; retrying admission in 300s
[2026-10-06 13:07:45] Admission accounting: reserved=208 unreserved=0 requested=10 limit=200 observedFleet=107 fleetObservationValid=true
[2026-10-06 13:07:45] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=107 unreservedObserved=0 reservationsSum=208 reservationHolders=24 reservationHoldersWithNoObservedDroplet=2 otherGrantedRemainder=91 dropletLimit=200 queueDepth=71 queuePos=7 fleetObservationValid=true fleetObservationAge=59s fleetObservationUsable=true reservedProjection=208/200 freshProjection=208/200]
[2026-10-06 13:07:45] Waiting for droplet capacity; retrying admission in 300s
[2026-10-06 13:12:45] Admission accounting: reserved=208 unreserved=0 requested=10 limit=200 observedFleet=162 fleetObservationValid=true
[2026-10-06 13:12:45] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=162 unreservedObserved=0 reservationsSum=208 reservationHolders=24 reservationHoldersWithNoObservedDroplet=1 otherGrantedRemainder=36 dropletLimit=200 queueDepth=72 queuePos=7 fleetObservationValid=true fleetObservationAge=57s fleetObservationUsable=true reservedProjection=208/200 freshProjection=208/200]
[2026-10-06 13:12:45] Waiting for droplet capacity; retrying admission in 300s
[2026-10-06 13:17:46] Admission accounting: reserved=208 unreserved=0 requested=10 limit=200 observedFleet=165 fleetObservationValid=true
[2026-10-06 13:17:46] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=165 unreservedObserved=0 reservationsSum=208 reservationHolders=24 reservationHoldersWithNoObservedDroplet=1 otherGrantedRemainder=33 dropletLimit=200 queueDepth=73 queuePos=7 fleetObservationValid=true fleetObservationAge=0s fleetObservationUsable=true reservedProjection=208/200 freshProjection=208/200]
[2026-10-06 13:17:46] Waiting for droplet capacity; retrying admission in 300s
[2026-10-06 13:22:47] Admission accounting: reserved=178 unreserved=23 requested=10 limit=200 observedFleet=139 fleetObservationValid=true
[2026-10-06 13:22:47] admissionDecision[run=b6699ad4 path=fresh requested=10 myReservation=0 observedFleet=139 unreservedObserved=23 reservationsSum=178 reservationHolders=20 reservationHoldersWithNoObservedDroplet=4 otherGrantedRemainder=52 dropletLimit=200 queueDepth=73 queuePos=4 fleetObservationValid=true fleetObservationAge=2s fleetObservationUsable=true reservedProjection=201/200 freshProjection=201/200]
[2026-10-06 13:22:47] Waiting for droplet capacity; retrying admission in 300s
[buildtest-coordinator] [2026-10-06 13:43:35] Resuming sharded build after service restart (was PENDING, 10 shards)
[2026-10-06 13:43:35] Reconciled dynamic restart recovery against 1916 durable test_discovered identity/identities: restored 0 omitted identity/identities and ignored 0 recovery-only identity/identities from fresh archive analysis
[2026-10-06 13:43:35] Rebuilding dynamic dispatch after service restart from 1916 discovered test(s) and 0 durable terminal result(s)
[2026-10-06 13:44:32] Dynamic dispatch: seeded 1909 work unit(s) (1916 discovered test(s), 0 terminal before seeding) into the shared lease queue
[dispatch] Settled restart ownership before dynamic registration: registered 0 durable dynamic queue(s); 0 spec(s) remain exclusively owned by shard collection or static fallback
[2026-10-06 13:44:34] Admission accounting: reserved=327 unreserved=0 requested=10 limit=200 observedFleet=75 fleetObservationValid=true
[2026-10-06 13:44:34] admissionDecision[run=b6699ad4 path=reserved-head requested=10 myReservation=10 observedFleet=75 unreservedObserved=0 reservationsSum=327 reservationHolders=43 reservationHoldersWithNoObservedDroplet=27 otherGrantedRemainder=124 dropletLimit=200 queueDepth=70 queuePos=66 fleetObservationValid=true fleetObservationAge=29s fleetObservationUsable=true reservedProjection=209/200 freshProjection=209/200]
[2026-10-06 13:44:34] Waiting for droplet capacity; retrying admission in 30s
[2026-10-06 13:45:10] Admission accounting: reserved=769 unreserved=0 requested=10 limit=200 observedFleet=75 fleetObservationValid=true
[2026-10-06 13:45:10] admissionDecision[run=b6699ad4 path=reserved-head requested=10 myReservation=10 observedFleet=75 unreservedObserved=0 reservationsSum=769 reservationHolders=89 reservationHoldersWithNoObservedDroplet=73 otherGrantedRemainder=121 dropletLimit=200 queueDepth=82 queuePos=20 fleetObservationValid=true fleetObservationAge=3s fleetObservationUsable=true reservedProjection=206/200 freshProjection=206/200]
[2026-10-06 13:45:10] Waiting for droplet capacity; retrying admission in 60s
[shard 1/10] [2026-10-06 13:46:55] Droplet created: id=606633228, name=buildtest-b6699ad4-s1-a72da7e
[shard 1/10] [2026-10-06 13:46:55] Droplet active at 64.23.208.214
[shard 2/10] [2026-10-06 13:46:55] Droplet created: id=606633218, name=buildtest-b6699ad4-s2-a72da7d
[shard 2/10] [2026-10-06 13:46:56] Droplet active at 64.227.97.12
[shard 1/10] [2026-10-06 13:46:57] ssh_key_grant_retry runId=b6699ad4 shardIndex=1 dropletId=606633228 failedAttempt=1 maxAttempts=3 action=retry-same-droplet error=SandboxException: Cannot invoke method on closed sandbox connection for service 'digitalocean-droplets'.
[shard 2/10] [2026-10-06 13:47:26] ssh_key_grant_retry runId=b6699ad4 shardIndex=2 dropletId=606633218 failedAttempt=1 maxAttempts=3 action=retry-same-droplet error=DropletServiceCallStalledException: Droplet service call addSshKeyToDroplet stalled for 30000ms while granting SSH access to droplet 606633218.
[shard 3/10] [2026-10-06 13:47:29] provisioning_failure runId=b6699ad4 shardIndex=3 shardCount=10 method=createBuildDroplet attempt=1 maxAttempts=3 elapsedMs=128985 classification=INFRASTRUCTURE/PROVISIONING action=exhausted error=DropletServiceCallStalledException: Droplet creation outcome for 'buildtest-b6699ad4-s3-a72da72' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: buildtest.embedded.DropletServiceCallStalledException: Droplet service call listDroplets stalled for 30000ms during failed-creation recovery for 'buildtest-b6699ad4-s3-a72da72'.
[shard 1/10] [2026-10-06 13:47:27] ssh_key_grant_retry runId=b6699ad4 shardIndex=1 dropletId=606633228 failedAttempt=2 maxAttempts=3 action=retry-same-droplet error=DropletServiceCallStalledException: Droplet service call addSshKeyToDroplet stalled for 30000ms while granting SSH access to droplet 606633228.
[shard 9/10] [2026-10-06 13:47:31] provisioning_failure runId=b6699ad4 shardIndex=9 shardCount=10 method=createBuildDroplet attempt=1 maxAttempts=3 elapsedMs=129110 classification=INFRASTRUCTURE/PROVISIONING action=exhausted error=DropletServiceCallStalledException: Droplet creation outcome for 'buildtest-b6699ad4-s9-a72daae' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: buildtest.embedded.DropletServiceCallStalledException: Droplet service call listDroplets stalled for 30000ms during failed-creation recovery for 'buildtest-b6699ad4-s9-a72daae'.
[shard 7/10] [2026-10-06 13:47:33] provisioning_failure runId=b6699ad4 shardIndex=7 shardCount=10 method=createBuildDroplet attempt=1 maxAttempts=3 elapsedMs=130716 classification=INFRASTRUCTURE/PROVISIONING action=exhausted error=DropletServiceCallStalledException: Droplet creation outcome for 'buildtest-b6699ad4-s7-a72da9e' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: buildtest.embedded.DropletServiceCallStalledException: Droplet service call listDroplets stalled for 30000ms during failed-creation recovery for 'buildtest-b6699ad4-s7-a72da9e'.
[shard 5/10] [2026-10-06 13:47:34] provisioning_failure runId=b6699ad4 shardIndex=5 shardCount=10 method=createBuildDroplet attempt=1 maxAttempts=3 elapsedMs=132731 classification=INFRASTRUCTURE/PROVISIONING action=exhausted error=DropletServiceCallStalledException: Droplet creation outcome for 'buildtest-b6699ad4-s5-a72da9d' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: buildtest.embedded.DropletServiceCallStalledException: Droplet service call listDroplets stalled for 30000ms during failed-creation recovery for 'buildtest-b6699ad4-s5-a72da9d'.
[shard 0/10] [2026-10-06 13:47:35] provisioning_failure runId=b6699ad4 shardIndex=0 shardCount=10 method=createBuildDroplet attempt=1 maxAttempts=3 elapsedMs=134340 classification=INFRASTRUCTURE/PROVISIONING action=exhausted error=DropletServiceCallStalledException: Droplet creation outcome for 'buildtest-b6699ad4-a72da6e' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: buildtest.embedded.DropletServiceCallStalledException: Droplet service call listDroplets stalled for 30000ms during failed-creation recovery for 'buildtest-b6699ad4-a72da6e'.
[shard 4/10] [2026-10-06 13:47:36] provisioning_failure runId=b6699ad4 shardIndex=4 shardCount=10 method=createBuildDroplet attempt=1 maxAttempts=3 elapsedMs=135059 classification=INFRASTRUCTURE/PROVISIONING action=exhausted error=DropletServiceCallStalledException: Droplet creation outcome for 'buildtest-b6699ad4-s4-a72da88' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: buildtest.embedded.DropletServiceCallStalledException: Droplet service call listDroplets stalled for 30000ms during failed-creation recovery for 'buildtest-b6699ad4-s4-a72da88'.
[shard 8/10] [2026-10-06 13:47:37] provisioning_failure runId=b6699ad4 shardIndex=8 shardCount=10 method=createBuildDroplet attempt=1 maxAttempts=3 elapsedMs=135004 classification=INFRASTRUCTURE/PROVISIONING action=exhausted error=DropletServiceCallStalledException: Droplet creation outcome for 'buildtest-b6699ad4-s8-a72daaf' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: DropletServiceCallStalledException: Droplet service call getDropletCreationStatus stalled for 30000ms while waiting for droplet 'buildtest-b6699ad4-s8-a72daaf' (requestId=6991ec4d-3082-49c1-91c5-1526069fc845).
[shard 3/10] [2026-10-06 13:47:37] ERROR: Provisioning failed for runId=b6699ad4 shardIndex=3 method=createBuildDroplet attempts=1/3 elapsedMs=128985 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-b6699ad4-s3-a72da72' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: buildtest.embedded.DropletServiceCallStalledException: Droplet service call listDroplets stalled for 30000ms during failed-creation recovery for 'buildtest-b6699ad4-s3-a72da72'.
[shard 7/10] [2026-10-06 13:47:37] ERROR: Provisioning failed for runId=b6699ad4 shardIndex=7 method=createBuildDroplet attempts=1/3 elapsedMs=130716 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-b6699ad4-s7-a72da9e' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: buildtest.embedded.DropletServiceCallStalledException: Droplet service call listDroplets stalled for 30000ms during failed-creation recovery for 'buildtest-b6699ad4-s7-a72da9e'.
[shard 0/10] [2026-10-06 13:47:37] ERROR: Provisioning failed for runId=b6699ad4 shardIndex=0 method=createBuildDroplet attempts=1/3 elapsedMs=134340 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-b6699ad4-a72da6e' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: buildtest.embedded.DropletServiceCallStalledException: Droplet service call listDroplets stalled for 30000ms during failed-creation recovery for 'buildtest-b6699ad4-a72da6e'.
[shard 8/10] [2026-10-06 13:47:37] ERROR: Provisioning failed for runId=b6699ad4 shardIndex=8 method=createBuildDroplet attempts=1/3 elapsedMs=135004 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-b6699ad4-s8-a72daaf' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: DropletServiceCallStalledException: Droplet service call getDropletCreationStatus stalled for 30000ms while waiting for droplet 'buildtest-b6699ad4-s8-a72daaf' (requestId=6991ec4d-3082-49c1-91c5-1526069fc845).
[shard 5/10] [2026-10-06 13:47:37] ERROR: Provisioning failed for runId=b6699ad4 shardIndex=5 method=createBuildDroplet attempts=1/3 elapsedMs=132731 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-b6699ad4-s5-a72da9d' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: buildtest.embedded.DropletServiceCallStalledException: Droplet service call listDroplets stalled for 30000ms during failed-creation recovery for 'buildtest-b6699ad4-s5-a72da9d'.
[shard 9/10] [2026-10-06 13:47:37] ERROR: Provisioning failed for runId=b6699ad4 shardIndex=9 method=createBuildDroplet attempts=1/3 elapsedMs=129110 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-b6699ad4-s9-a72daae' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: buildtest.embedded.DropletServiceCallStalledException: Droplet service call listDroplets stalled for 30000ms during failed-creation recovery for 'buildtest-b6699ad4-s9-a72daae'.
[shard 4/10] [2026-10-06 13:47:37] ERROR: Provisioning failed for runId=b6699ad4 shardIndex=4 method=createBuildDroplet attempts=1/3 elapsedMs=135059 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-b6699ad4-s4-a72da88' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: buildtest.embedded.DropletServiceCallStalledException: Droplet service call listDroplets stalled for 30000ms during failed-creation recovery for 'buildtest-b6699ad4-s4-a72da88'.
[shard 6/10] [2026-10-06 13:47:37] provisioning_failure runId=b6699ad4 shardIndex=6 shardCount=10 method=createBuildDroplet attempt=1 maxAttempts=3 elapsedMs=136840 classification=INFRASTRUCTURE/PROVISIONING action=exhausted error=DropletServiceCallStalledException: Droplet creation outcome for 'buildtest-b6699ad4-s6-a72da93' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: DropletServiceCallStalledException: Droplet service call getDropletCreationStatus stalled for 30000ms while waiting for droplet 'buildtest-b6699ad4-s6-a72da93' (requestId=639446af-b9ee-445a-8bf3-6cfcd39bbded).
[shard 6/10] [2026-10-06 13:47:37] ERROR: Provisioning failed for runId=b6699ad4 shardIndex=6 method=createBuildDroplet attempts=1/3 elapsedMs=136840 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-b6699ad4-s6-a72da93' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: DropletServiceCallStalledException: Droplet service call getDropletCreationStatus stalled for 30000ms while waiting for droplet 'buildtest-b6699ad4-s6-a72da93' (requestId=639446af-b9ee-445a-8bf3-6cfcd39bbded).
[shard 2/10] [2026-10-06 13:47:56] ssh_key_grant_retry runId=b6699ad4 shardIndex=2 dropletId=606633218 failedAttempt=2 maxAttempts=3 action=retry-same-droplet error=DropletServiceCallStalledException: Droplet service call addSshKeyToDroplet stalled for 30000ms while granting SSH access to droplet 606633218.
[shard 1/10] [2026-10-06 13:48:02] provisioning_failure runId=b6699ad4 shardIndex=1 shardCount=10 method=addSshKeyToDroplet attempt=1 maxAttempts=3 elapsedMs=162076 classification=INFRASTRUCTURE/PROVISIONING action=retrying error=DropletServiceCallStalledException: Droplet service call addSshKeyToDroplet stalled for 30000ms while granting SSH access to droplet 606633228.
[shard 1/10] [2026-10-06 13:48:04] Waiting for failed droplet 606633228 deletion before retrying provisioning
[shard 2/10] [2026-10-06 13:48:27] provisioning_failure runId=b6699ad4 shardIndex=2 shardCount=10 method=addSshKeyToDroplet attempt=1 maxAttempts=3 elapsedMs=187177 classification=INFRASTRUCTURE/PROVISIONING action=retrying error=DropletServiceCallStalledException: Droplet service call addSshKeyToDroplet stalled for 30000ms while granting SSH access to droplet 606633218.
[shard 2/10] [2026-10-06 13:48:33] Waiting for failed droplet 606633218 deletion before retrying provisioning
[shard 1/10] [2026-10-06 13:49:01] Waiting for droplet capacity before retrying failed boot; refreshing admission in 30s
[shard 1/10] [2026-10-06 13:49:15] provisioning_retry_backoff runId=b6699ad4 shardIndex=1 method=addSshKeyToDroplet completedAttempt=1 backoffMs=1000 classification=INFRASTRUCTURE/PROVISIONING
[shard 2/10] [2026-10-06 13:49:19] Waiting for droplet capacity before retrying failed boot; refreshing admission in 30s
[shard 2/10] [2026-10-06 13:49:45] provisioning_retry_backoff runId=b6699ad4 shardIndex=2 method=addSshKeyToDroplet completedAttempt=1 backoffMs=1000 classification=INFRASTRUCTURE/PROVISIONING
[shard 1/10] [2026-10-06 13:50:25] Droplet created: id=606634291, name=buildtest-b6699ad4-s1-a72dad1
[shard 1/10] [2026-10-06 13:50:25] Droplet active at 64.23.228.41
[shard 1/10] [2026-10-06 13:50:41] SSH access granted to droplet 606634291
[shard 1/10] [2026-10-06 13:50:41] Waiting for guest readiness on droplet 606634291
[shard 1/10] [2026-10-06 13:51:27] Guest readiness complete on droplet 606634291
[shard 1/10] [2026-10-06 13:51:28] Dynamic memory admission: droplet 1 has 4096 MiB RAM, 3221225472 bytes available to resident tests, and runner parallelism 4
[shard 1/10] [2026-10-06 13:51:29] Connecting via SSH and installing dependencies...
[shard 1/10] [2026-10-06 13:51:29] SSH connected
[shard 1/10] Stopped, disabled, and masked unattended-upgrades.service, apt-daily.service, apt-daily-upgrade.service, apt-daily.timer, and apt-daily-upgrade.timer; waited 0 seconds for apt/dpkg locks to clear
[shard 1/10] Dependencies already present (snapshot image), skipping installation
[shard 1/10] Java: openjdk version "21.0.11" 2026-04-21
[shard 1/10] Node.js: v18.19.1
[shard 1/10] Chromium: /usr/bin/chromium-browser
[shard 1/10] Coursier: /root/tools/coursier
[shard 1/10] Docker already present: Docker version 29.1.3, build 29.1.3-0ubuntu3~24.04.2
[shard 1/10] haveged already present
[shard 1/10] securerandom.source=file:/dev/urandom already set in /usr/lib/jvm/java-21-openjdk-amd64/conf/security/java.security
[shard 1/10] Dependencies installed successfully
[shard 1/10] Install image id: 245388985; base dependency fast path taken: true
[shard 1/10] [BuildDriver] Memory before swap: MemoryInfo(totalMemoryKb=4009872, availableMemoryKb=3462860, totalSwapKb=0, freeSwapKb=0)
[shard 1/10] mkswap: /swapfile: warning: wiping old swap signature.
[shard 1/10] Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
[shard 1/10] no label, UUID=0846276c-12f3-4486-8b63-5fdb1b115448
[shard 1/10] Swap file created and enabled
[shard 1/10] [BuildDriver] Memory after swap: MemoryInfo(totalMemoryKb=4009872, availableMemoryKb=3463980, totalSwapKb=4194300, freeSwapKb=4194300)
[shard 1/10] [BuildDriver] Swap verified: 4194300KB total swap (was 0KB)
[shard 1/10] [2026-10-06 13:51:42] Dependencies installed
[shard 1/10] [2026-10-06 13:51:43] Uploading project archive...
[shard 1/10] Extracted projects:
CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8

[shard 1/10] [2026-10-06 13:51:46] Projects on remote: CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8
[shard 1/10] [2026-10-06 13:51:46] Projects from archive: foundation.url:resolver
[shard 1/10] [2026-10-06 13:51:49] Building and testing workspace...
[shard 1/10] [2026-10-06 13:51:49] Shard test specs (0): 
[2026-10-06 13:51:49] [dispatch] droplet 1 ready
[2026-10-06 13:51:50] [dispatch] cpu_skip_advisory: CPU admission would have skipped test [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testRetainedWithdrawalToDeadMultiAddressBootstrapIsRateBounded.kts] on droplet 1: its reservation of 190 millicores would raise the worst-case concurrent CPU of the droplet's 4 lanes above its 2000 millicores (already 3188 millicores reserved by resident and selected tests); it was admitted because CPU admission is in OBSERVE mode
[2026-10-06 13:51:50] [dispatch] cpu_skip_advisory: CPU admission would have skipped test [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testDeadConfiguredBootstrapDialsBackOffExponentially.kts] on droplet 1: its reservation of 175 millicores would raise the worst-case concurrent CPU of the droplet's 4 lanes above its 2000 millicores (already 3378 millicores reserved by resident and selected tests); it was admitted because CPU admission is in OBSERVE mode
[2026-10-06 13:51:50] [dispatch] cpu_skip_advisory: CPU admission would have skipped test [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testRecoveredConfiguredBootstrapReconnectsAfterSustainedBackoff.kts] on droplet 1: its reservation of 328 millicores would raise the worst-case concurrent CPU of the droplet's 4 lanes above its 2000 millicores (already 3553 millicores reserved by resident and selected tests); it was admitted because CPU admission is in OBSERVE mode
[2026-10-06 13:51:50] [dispatch] cpu_skip_advisory: CPU admission would have skipped test [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/stressTestResolverNodeSurvivesMidNegotiationChurnDropWithConstrainedHeap.kts] on droplet 1: its reservation of 9372 millicores would raise the worst-case concurrent CPU of the droplet's 4 lanes above its 2000 millicores (already 3881 millicores reserved by resident and selected tests); it was admitted because CPU admission is in OBSERVE mode
[2026-10-06 13:51:50] [dispatch] cpu_skip_advisory: CPU admission would have skipped test [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testNewServiceRevisionRestartsDeadBootstrapDialLadder.kts] on droplet 1: its reservation of 242 millicores would raise the worst-case concurrent CPU of the droplet's 4 lanes above its 2000 millicores (already 13078 millicores reserved by resident and selected tests); it was admitted because CPU admission is in OBSERVE mode
[2026-10-06 13:51:50] [dispatch] cpu_skip_advisory: CPU admission would have skipped test [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/stressTestFreshJoinWithdrawalDeliveryUnderConcurrentReannounce.kts] on droplet 1: its reservation of 14345 millicores would raise the worst-case concurrent CPU of the droplet's 4 lanes above its 2000 millicores (already 13130 millicores reserved by resident and selected tests); it was admitted because CPU admission is in OBSERVE mode
[2026-10-06 13:51:50] [dispatch] cpu_skip_advisory: CPU admission would have skipped test [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/perfClusterJoinTime.kts] on droplet 1: its reservation of 4316 millicores would raise the worst-case concurrent CPU of the droplet's 4 lanes above its 2000 millicores (already 27233 millicores reserved by resident and selected tests); it was admitted because CPU admission is in OBSERVE mode
[2026-10-06 13:51:50] [dispatch] lease_issued: Issued lease cb1d4998-026a-4eb0-b141-1e6f046a7501-b6699ad4-c39ddda7-14c9-4edc-a1a2-341d6f983bf3-L0 with 8 test(s) [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/stressTestSandboxedConnectionColdRetriesAlwaysReachTheService.kts, CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testRetainedWithdrawalToDeadMultiAddressBootstrapIsRateBounded.kts, CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testDeadConfiguredBootstrapDialsBackOffExponentially.kts, CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testRecoveredConfiguredBootstrapReconnectsAfterSustainedBackoff.kts, CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/stressTestResolverNodeSurvivesMidNegotiationChurnDropWithConstrainedHeap.kts, CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testNewServiceRevisionRestartsDeadBootstrapDialLadder.kts, CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/stressTestFreshJoinWithdrawalDeliveryUnderConcurrentReannounce.kts, CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/perfClusterJoinTime.kts] to droplet 1 (pull-based handout from shared queue; 1901 unit(s) remaining). Memory admission: predicted peak 1827704832 bytes; resident predicted peak 1827704832 of 3221225472 bytes; 1393520640 bytes remaining budget
[shard 1/10] System memory: 3915MB, JVM heap: 1957MB
[shard 1/10] Downloading https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.pom
[shard 1/10] Downloaded https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.pom
[shard 1/10] Downloading https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.pom.sha1
[shard 1/10] Downloaded https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.pom.sha1
[shard 1/10] Downloading https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.pom
[shard 1/10] Downloading https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.pom
[shard 1/10] Downloading https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.pom
[shard 1/10] Downloading https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.pom
[shard 1/10] Downloading https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.pom
[shard 1/10] Downloading https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.pom
[shard 1/10] Downloaded https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.pom
[shard 1/10] Downloading https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.pom
[shard 1/10] Downloaded https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.pom
[shard 1/10] Downloading https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.pom.sha1
[shard 1/10] Downloaded https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.pom
[shard 1/10] Downloading https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.pom.sha1
[shard 1/10] Downloaded https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.pom
[shard 1/10] Downloading https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.pom.sha1
[shard 1/10] Downloaded https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.pom
[shard 1/10] Downloading https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.pom.sha1
[shard 1/10] Downloaded https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.pom
[shard 1/10] Downloading https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.pom.sha1
[shard 1/10] Downloaded https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.pom
[shard 1/10] Downloading https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.pom.sha1
[shard 1/10] Failed to download https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.pom.sha1
[shard 1/10] Downloaded https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.pom.sha1
[shard 1/10] Downloading https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.pom.sha1
[shard 1/10] Downloaded https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.pom.sha1
[shard 1/10] Downloaded https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.pom.sha1
[shard 1/10] Downloaded https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.pom.sha1
[shard 1/10] Downloaded https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.pom.sha1
[shard 1/10] Downloaded https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.pom.sha1
[shard 1/10] Downloading https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.jar
[shard 1/10] Downloading https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.jar
[shard 1/10] Downloading https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.jar
[shard 1/10] Downloading https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.jar
[shard 1/10] Downloading https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.jar
[shard 1/10] Downloading https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.jar
[shard 1/10] Downloaded https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.jar
[shard 1/10] Downloading https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.jar
[shard 1/10] Downloaded https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.jar
[shard 1/10] Downloading https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.jar
[shard 1/10] Downloaded https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.jar
[shard 1/10] Downloaded https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.jar
[shard 1/10] Downloading https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.jar.sha1
[shard 1/10] Downloading https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.jar.sha1
[shard 1/10] Downloaded https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.jar
[shard 1/10] Downloading https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.jar.sha1
[shard 1/10] Downloaded https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.jar.sha1
[shard 1/10] Downloading https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.jar.sha1
[shard 1/10] Downloaded https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.jar.sha1
[shard 1/10] Downloading https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.jar.sha1
[shard 1/10] Downloaded https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.jar.sha1
[shard 1/10] Downloaded https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.jar
[shard 1/10] Downloading https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.jar.sha1
[shard 1/10] Downloaded https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.jar
[shard 1/10] Downloading https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.jar.sha1
[shard 1/10] Failed to download https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.jar.sha1
[shard 1/10] Downloaded https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.jar.sha1
[shard 1/10] Downloaded https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.jar.sha1
[shard 1/10] Downloaded https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.jar.sha1
[shard 1/10] Downloaded https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.jar
[shard 1/10] Downloading https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.jar.sha1
[shard 1/10] Downloaded https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.jar.sha1
[shard 2/10] [2026-10-06 13:51:53] Droplet created: id=606634364, name=buildtest-b6699ad4-s2-a72dad4
[shard 2/10] [2026-10-06 13:51:53] Droplet active at 137.184.13.87
[shard 2/10] [2026-10-06 13:51:58] SSH access granted to droplet 606634364
[shard 2/10] [2026-10-06 13:51:58] Waiting for guest readiness on droplet 606634364
[shard 2/10] [2026-10-06 13:52:23] Guest readiness complete on droplet 606634364
[shard 2/10] [2026-10-06 13:52:23] Dynamic memory admission: droplet 2 has 4096 MiB RAM, 3221225472 bytes available to resident tests, and runner parallelism 4
[shard 2/10] [2026-10-06 13:52:24] Connecting via SSH and installing dependencies...
[shard 2/10] [2026-10-06 13:52:25] SSH connected
[shard 2/10] Stopped, disabled, and masked unattended-upgrades.service, apt-daily.service, apt-daily-upgrade.service, apt-daily.timer, and apt-daily-upgrade.timer; waited 0 seconds for apt/dpkg locks to clear
[shard 2/10] Dependencies already present (snapshot image), skipping installation
[shard 2/10] Java: openjdk version "21.0.11" 2026-04-21
[shard 2/10] Node.js: v18.19.1
[shard 2/10] Chromium: /usr/bin/chromium-browser
[shard 2/10] Coursier: /root/tools/coursier
[shard 2/10] Docker already present: Docker version 29.1.3, build 29.1.3-0ubuntu3~24.04.2
[shard 2/10] haveged already present
[shard 2/10] securerandom.source=file:/dev/urandom already set in /usr/lib/jvm/java-21-openjdk-amd64/conf/security/java.security
[shard 2/10] Dependencies installed successfully
[shard 2/10] Install image id: 245388985; base dependency fast path taken: true
[shard 2/10] [BuildDriver] Memory before swap: MemoryInfo(totalMemoryKb=4009856, availableMemoryKb=3476964, totalSwapKb=0, freeSwapKb=0)
[shard 2/10] mkswap: /swapfile: warning: wiping old swap signature.
[shard 2/10] Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
[shard 2/10] no label, UUID=dfd6463c-87ff-403d-8bd1-50a3828342b1
[shard 2/10] Swap file created and enabled
[shard 2/10] [BuildDriver] Memory after swap: MemoryInfo(totalMemoryKb=4009856, availableMemoryKb=3472588, totalSwapKb=4194300, freeSwapKb=4194300)
[shard 2/10] [BuildDriver] Swap verified: 4194300KB total swap (was 0KB)
[shard 2/10] [2026-10-06 13:52:42] Dependencies installed
[shard 2/10] [2026-10-06 13:52:43] Uploading project archive...
[shard 2/10] Extracted projects:
CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8

[shard 2/10] [2026-10-06 13:52:47] Projects on remote: CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8
[shard 2/10] [2026-10-06 13:52:47] Projects from archive: foundation.url:resolver
[shard 2/10] [2026-10-06 13:52:48] Building and testing workspace...
[shard 2/10] [2026-10-06 13:52:48] Shard test specs (0): 
[2026-10-06 13:52:49] [dispatch] droplet 2 ready
[2026-10-06 13:52:50] [dispatch] cpu_skip_advisory: CPU admission would have skipped test [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/stressTestEffectAwareHandlerTimeoutSendsError.kts] on droplet 2: its reservation of 727 millicores would raise the worst-case concurrent CPU of the droplet's 4 lanes above its 2000 millicores (already 6260 millicores reserved by resident and selected tests); it was admitted because CPU admission is in OBSERVE mode
[2026-10-06 13:52:50] [dispatch] cpu_skip_advisory: CPU admission would have skipped test [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testLargeRpcRequestSurvivesOrdinaryTrafficOnItsOwnConnection.kts] on droplet 2: its reservation of 971 millicores would raise the worst-case concurrent CPU of the droplet's 4 lanes above its 2000 millicores (already 6987 millicores reserved by resident and selected tests); it was admitted because CPU admission is in OBSERVE mode
[2026-10-06 13:52:50] [dispatch] cpu_skip_advisory: CPU admission would have skipped test [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testNetlabDynamicFailureRecovery.kts] on droplet 2: its reservation of 1210 millicores would raise the worst-case concurrent CPU of the droplet's 4 lanes above its 2000 millicores (already 7958 millicores reserved by resident and selected tests); it was admitted because CPU admission is in OBSERVE mode
[2026-10-06 13:52:50] [dispatch] cpu_skip_advisory: CPU admission would have skipped test [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testDialEntryCancellationIsNotInheritedByLaterCallers.kts] on droplet 2: its reservation of 265 millicores would raise the worst-case concurrent CPU of the droplet's 4 lanes above its 2000 millicores (already 9168 millicores reserved by resident and selected tests); it was admitted because CPU admission is in OBSERVE mode
[2026-10-06 13:52:50] [dispatch] cpu_skip_advisory: CPU admission would have skipped test [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testWithdrawalOutranksLateJoinResend.kts] on droplet 2: its reservation of 2320 millicores would raise the worst-case concurrent CPU of the droplet's 4 lanes above its 2000 millicores (already 9168 millicores reserved by resident and selected tests); it was admitted because CPU admission is in OBSERVE mode
[2026-10-06 13:52:50] [dispatch] cpu_skip_advisory: CPU admission would have skipped test [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/stressTestBootstrapServiceResyncDoesNotGiveUpUnderSaturation.kts] on droplet 2: its reservation of 15800 millicores would raise the worst-case concurrent CPU of the droplet's 4 lanes above its 2000 millicores (already 10761 millicores reserved by resident and selected tests); it was admitted because CPU admission is in OBSERVE mode
[2026-10-06 13:52:50] [dispatch] cpu_skip_advisory: CPU admission would have skipped test [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testNetlabNatToNatRelayOutageRecovery.kts] on droplet 2: its reservation of 3187 millicores would raise the worst-case concurrent CPU of the droplet's 4 lanes above its 2000 millicores (already 25590 millicores reserved by resident and selected tests); it was admitted because CPU admission is in OBSERVE mode
[2026-10-06 13:52:50] [dispatch] lease_issued: Issued lease cb1d4998-026a-4eb0-b141-1e6f046a7501-b6699ad4-c39ddda7-14c9-4edc-a1a2-341d6f983bf3-L1 with 8 test(s) [CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/perfGossipPropagationTime.kts, CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/stressTestEffectAwareHandlerTimeoutSendsError.kts, CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testLargeRpcRequestSurvivesOrdinaryTrafficOnItsOwnConnection.kts, CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testNetlabDynamicFailureRecovery.kts, CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testDialEntryCancellationIsNotInheritedByLaterCallers.kts, CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testWithdrawalOutranksLateJoinResend.kts, CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/stressTestBootstrapServiceResyncDoesNotGiveUpUnderSaturation.kts, CodexCoder21Organization-UrlResolver-7b62e742aafdc9216e78448ddf8d015bf40cc9c8/tests/testNetlabNatToNatRelayOutageRecovery.kts] to droplet 2 (pull-based handout from shared queue; 1893 unit(s) remaining). Memory admission: predicted peak 1802874880 bytes; resident predicted peak 1802874880 of 3221225472 bytes; 1418350592 bytes remaining budget
[shard 2/10] System memory: 3915MB, JVM heap: 1957MB
[shard 2/10] Downloading https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.pom
[shard 2/10] Downloaded https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.pom
[shard 2/10] Downloading https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.pom.sha1
[shard 2/10] Downloaded https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.pom.sha1
[shard 2/10] Downloading https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.pom
[shard 2/10] Downloading https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.pom
[shard 2/10] Downloading https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.pom
[shard 2/10] Downloading https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.pom
[shard 2/10] Downloading https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.pom
[shard 2/10] Downloading https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.pom
[shard 2/10] Downloaded https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.pom
[shard 2/10] Downloading https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.pom
[shard 2/10] Downloaded https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.pom
[shard 2/10] Downloading https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.pom.sha1
[shard 2/10] Downloaded https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.pom
[shard 2/10] Downloading https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.pom.sha1
[shard 2/10] Downloaded https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.pom
[shard 2/10] Downloading https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.pom.sha1
[shard 2/10] Downloaded https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.pom
[shard 2/10] Downloading https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.pom.sha1
[shard 2/10] Downloaded https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.pom
[shard 2/10] Downloading https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.pom.sha1
[shard 2/10] Downloaded https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.pom
[shard 2/10] Downloading https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.pom.sha1
[shard 2/10] Downloaded https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.pom.sha1
[shard 2/10] Failed to download https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.pom.sha1
[shard 2/10] Downloading https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.pom.sha1
[shard 2/10] Downloaded https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.pom.sha1
[shard 2/10] Downloaded https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.pom.sha1
[shard 2/10] Downloaded https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.pom.sha1
[shard 2/10] Downloaded https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.pom.sha1
[shard 2/10] Downloaded https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.pom.sha1
[shard 2/10] Downloading https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.jar
[shard 2/10] Downloading https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.jar
[shard 2/10] Downloading https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.jar
[shard 2/10] Downloading https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.jar
[shard 2/10] Downloading https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.jar
[shard 2/10] Downloading https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.jar
[shard 2/10] Downloaded https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.jar
[shard 2/10] Downloading https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.jar
[shard 2/10] Downloaded https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.jar
[shard 2/10] Downloading https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.jar
[shard 2/10] Downloaded https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.jar
[shard 2/10] Downloading https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.jar.sha1
[shard 2/10] Downloaded https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.jar
[shard 2/10] Downloading https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.jar.sha1
[shard 2/10] Downloaded https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.jar
[shard 2/10] Downloading https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.jar.sha1
[shard 2/10] Downloaded https://kotlin.directory/tools/kotlin/build/compile/interfaces/executionenvironment/tools-kotlin-build-compile-interfaces-executionenvironment/0.0.2/tools-kotlin-build-compile-interfaces-executionenvironment-0.0.2.jar.sha1
[shard 2/10] Downloading https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.jar.sha1
[shard 2/10] Downloaded https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.jar
[shard 2/10] Downloading https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.jar.sha1
[shard 2/10] Downloaded https://kotlin.directory/community/kotlin/kompile/testrunner/bootstrap/community-kotlin-kompile-testrunner-bootstrap/0.0.16/community-kotlin-kompile-testrunner-bootstrap-0.0.16.jar.sha1
[shard 2/10] Failed to download https://kotlin.directory/io/get-coursier/coursier-cache_2.13/2.1.31-buffer2/coursier-cache_2.13-2.1.31-buffer2.jar.sha1
[shard 2/10] Downloading https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.jar.sha1
[shard 2/10] Downloaded https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.jar
[shard 2/10] Downloading https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.jar.sha1
[shard 2/10] Downloaded https://kotlin.directory/util/cache/hashtofile/hash-to-file-cache/0.0.11/hash-to-file-cache-0.0.11.jar.sha1
[shard 2/10] Downloaded https://kotlin.directory/kompile/executionenvironment/kompile-executionenvironment/0.0.44/kompile-executionenvironment-0.0.44.jar.sha1
[shard 2/10] Downloaded https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.123/buildtest-runner-0.0.123.jar.sha1
[shard 2/10] Downloaded https://kotlin.directory/kompile/buildscript/kompile-buildscript/0.0.43/kompile-buildscript-0.0.43.jar.sha1
[shard 2/10] Downloaded https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.jar
[shard 2/10] Downloading https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.jar.sha1
[shard 2/10] Downloaded https://kotlin.directory/kompile/manager/0.0.205/manager-0.0.205.jar.sha1