← Back to all runs

Build Run: a386d9f5

CodexCoder21Organization/UrlResolver

Stored run summary updated . Data age: 377m 13s.

Test Results

[buildtest-coordinator] [2026-10-07 18:59:47] Coordinator work: waiting for archive analysis, queue position 5 of 5
[2026-10-07 19:01:43] Admission accounting: reserved=299 unreserved=1 requested=10 limit=200 observedFleet=58 fleetObservationValid=true
[2026-10-07 19:01:43] admissionDecision[run=a386d9f5 path=fresh requested=10 myReservation=0 observedFleet=58 unreservedObserved=1 reservationsSum=299 reservationHolders=34 reservationHoldersWithNoObservedDroplet=12 otherGrantedRemainder=140 dropletLimit=200 queueDepth=26 queuePos=25 fleetObservationValid=true fleetObservationAge=12s fleetObservationUsable=true reservedProjection=208/200 freshProjection=208/200]
[2026-10-07 19:01:43] Waiting for droplet capacity; retrying admission in 30s
[2026-10-07 19:02:14] Admission accounting: reserved=299 unreserved=1 requested=10 limit=200 observedFleet=58 fleetObservationValid=true
[2026-10-07 19:02:14] admissionDecision[run=a386d9f5 path=fresh requested=10 myReservation=0 observedFleet=58 unreservedObserved=1 reservationsSum=299 reservationHolders=34 reservationHoldersWithNoObservedDroplet=12 otherGrantedRemainder=140 dropletLimit=200 queueDepth=27 queuePos=25 fleetObservationValid=true fleetObservationAge=16s fleetObservationUsable=true reservedProjection=208/200 freshProjection=208/200]
[2026-10-07 19:02:14] Waiting for droplet capacity; retrying admission in 60s
[2026-10-07 19:03:14] Admission accounting: reserved=295 unreserved=3 requested=10 limit=200 observedFleet=55 fleetObservationValid=true
[2026-10-07 19:03:14] admissionDecision[run=a386d9f5 path=fresh requested=10 myReservation=0 observedFleet=55 unreservedObserved=3 reservationsSum=295 reservationHolders=34 reservationHoldersWithNoObservedDroplet=12 otherGrantedRemainder=141 dropletLimit=200 queueDepth=26 queuePos=25 fleetObservationValid=true fleetObservationAge=21s fleetObservationUsable=true reservedProjection=206/200 freshProjection=206/200]
[2026-10-07 19:03:14] Waiting for droplet capacity; retrying admission in 120s
[2026-10-07 19:05:15] Admission accounting: reserved=288 unreserved=10 requested=10 limit=200 observedFleet=63 fleetObservationValid=true
[2026-10-07 19:05:15] admissionDecision[run=a386d9f5 path=fresh requested=10 myReservation=0 observedFleet=63 unreservedObserved=10 reservationsSum=288 reservationHolders=32 reservationHoldersWithNoObservedDroplet=11 otherGrantedRemainder=135 dropletLimit=200 queueDepth=23 queuePos=22 fleetObservationValid=true fleetObservationAge=4s fleetObservationUsable=true reservedProjection=208/200 freshProjection=208/200]
[2026-10-07 19:05:15] Waiting for droplet capacity; retrying admission in 240s
[2026-10-07 19:09:16] Admission accounting: reserved=285 unreserved=0 requested=10 limit=200 observedFleet=68 fleetObservationValid=true
[2026-10-07 19:09:16] admissionDecision[run=a386d9f5 path=fresh requested=10 myReservation=0 observedFleet=68 unreservedObserved=0 reservationsSum=285 reservationHolders=31 reservationHoldersWithNoObservedDroplet=11 otherGrantedRemainder=127 dropletLimit=200 queueDepth=21 queuePos=19 fleetObservationValid=true fleetObservationAge=10s fleetObservationUsable=true reservedProjection=205/200 freshProjection=205/200]
[2026-10-07 19:09:16] Waiting for droplet capacity; retrying admission in 300s
[2026-10-07 19:14:17] Admission accounting: reserved=263 unreserved=0 requested=10 limit=200 observedFleet=77 fleetObservationValid=true
[2026-10-07 19:14:17] admissionDecision[run=a386d9f5 path=fresh requested=10 myReservation=0 observedFleet=77 unreservedObserved=0 reservationsSum=263 reservationHolders=30 reservationHoldersWithNoObservedDroplet=8 otherGrantedRemainder=116 dropletLimit=200 queueDepth=19 queuePos=17 fleetObservationValid=true fleetObservationAge=1s fleetObservationUsable=true reservedProjection=203/200 freshProjection=203/200]
[2026-10-07 19:14:17] Waiting for droplet capacity; retrying admission in 300s
[2026-10-07 19:19:18] Admission accounting: reserved=263 unreserved=0 requested=10 limit=200 observedFleet=71 fleetObservationValid=true
[2026-10-07 19:19:18] admissionDecision[run=a386d9f5 path=fresh requested=10 myReservation=0 observedFleet=71 unreservedObserved=0 reservationsSum=263 reservationHolders=30 reservationHoldersWithNoObservedDroplet=8 otherGrantedRemainder=122 dropletLimit=200 queueDepth=18 queuePos=17 fleetObservationValid=true fleetObservationAge=4s fleetObservationUsable=true reservedProjection=203/200 freshProjection=203/200]
[2026-10-07 19:19:18] Waiting for droplet capacity; retrying admission in 300s
[buildtest-coordinator] [2026-10-07 19:33:01] Resuming sharded build after service restart (was PENDING, 10 shards)
[2026-10-07 19:33:01] Reconciled dynamic restart recovery against 1973 durable test_discovered identity/identities: restored 0 omitted identity/identities and ignored 0 recovery-only identity/identities from fresh archive analysis
[2026-10-07 19:33:01] Rebuilding dynamic dispatch after service restart from 1973 discovered test(s) and 0 durable terminal result(s)
[2026-10-07 19:33:29] Dynamic dispatch: seeded 1966 work unit(s) (1973 discovered test(s), 0 terminal before seeding) into the shared lease queue
[dispatch] Settled restart ownership before dynamic registration: registered 0 durable dynamic queue(s); 0 spec(s) remain exclusively owned by shard collection or static fallback
[2026-10-07 19:33:31] Admission accounting: reserved=234 unreserved=0 requested=10 limit=200 observedFleet=63 fleetObservationValid=true
[2026-10-07 19:33:31] admissionDecision[run=a386d9f5 path=reserved-head requested=10 myReservation=10 observedFleet=63 unreservedObserved=0 reservationsSum=234 reservationHolders=39 reservationHoldersWithNoObservedDroplet=18 otherGrantedRemainder=131 dropletLimit=200 queueDepth=5 queuePos=2 fleetObservationValid=true fleetObservationAge=27s fleetObservationUsable=true reservedProjection=204/200 freshProjection=204/200]
[2026-10-07 19:33:31] Waiting for droplet capacity; retrying admission in 30s
[shard 9/10] [2026-10-07 19:35:18] provisioning_failure runId=a386d9f5 shardIndex=9 shardCount=10 method=createBuildDroplet attempt=1 maxAttempts=3 elapsedMs=94805 classification=INFRASTRUCTURE/PROVISIONING action=exhausted error=DropletServiceCallStalledException: Droplet creation outcome for 'buildtest-a386d9f5-s9-ab2f168' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: java.util.concurrent.CancellationException: Job was cancelled
[shard 8/10] [2026-10-07 19:35:18] provisioning_failure runId=a386d9f5 shardIndex=8 shardCount=10 method=createBuildDroplet attempt=1 maxAttempts=3 elapsedMs=96591 classification=INFRASTRUCTURE/PROVISIONING action=exhausted error=DropletServiceCallStalledException: Droplet creation outcome for 'buildtest-a386d9f5-s8-ab2f161' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: SandboxException: Cannot invoke 'dropletserviceserver/DropletCreationStatusImpl.getRequestId()Ljava/lang/String;': the sandboxed connection was invalidated (a prior call 'dropletserviceserver/DropletServiceClientImpl.addSshKeyToDroplet(JLjava/lang/String;)V' timed out after 30s). Reopen the connection to use a fresh SJVM sandbox.
[shard 7/10] [2026-10-07 19:35:20] provisioning_failure runId=a386d9f5 shardIndex=7 shardCount=10 method=createBuildDroplet attempt=1 maxAttempts=3 elapsedMs=98375 classification=INFRASTRUCTURE/PROVISIONING action=exhausted error=DropletServiceCallStalledException: Droplet creation outcome for 'buildtest-a386d9f5-s7-ab2f15c' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: SandboxException: Cannot invoke 'dropletserviceserver/DropletCreationStatusImpl.getRequestId()Ljava/lang/String;': the sandboxed connection was invalidated (a prior call 'dropletserviceserver/DropletServiceClientImpl.addSshKeyToDroplet(JLjava/lang/String;)V' timed out after 30s). Reopen the connection to use a fresh SJVM sandbox.
[shard 1/10] [2026-10-07 19:35:20] provisioning_failure runId=a386d9f5 shardIndex=1 shardCount=10 method=createBuildDroplet attempt=1 maxAttempts=3 elapsedMs=99739 classification=INFRASTRUCTURE/PROVISIONING action=exhausted error=DropletServiceCallStalledException: Droplet creation outcome for 'buildtest-a386d9f5-s1-ab2f166' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: SandboxException: Cannot invoke 'dropletserviceserver/DropletCreationStatusImpl.getRequestId()Ljava/lang/String;': the sandboxed connection was invalidated (a prior call 'dropletserviceserver/DropletServiceClientImpl.addSshKeyToDroplet(JLjava/lang/String;)V' timed out after 30s). Reopen the connection to use a fresh SJVM sandbox.
[shard 1/10] [2026-10-07 19:35:20] ERROR: Provisioning failed for runId=a386d9f5 shardIndex=1 method=createBuildDroplet attempts=1/3 elapsedMs=99739 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-a386d9f5-s1-ab2f166' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: SandboxException: Cannot invoke 'dropletserviceserver/DropletCreationStatusImpl.getRequestId()Ljava/lang/String;': the sandboxed connection was invalidated (a prior call 'dropletserviceserver/DropletServiceClientImpl.addSshKeyToDroplet(JLjava/lang/String;)V' timed out after 30s). Reopen the connection to use a fresh SJVM sandbox.
[shard 9/10] [2026-10-07 19:35:20] ERROR: Provisioning failed for runId=a386d9f5 shardIndex=9 method=createBuildDroplet attempts=1/3 elapsedMs=94805 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-a386d9f5-s9-ab2f168' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: java.util.concurrent.CancellationException: Job was cancelled
[shard 8/10] [2026-10-07 19:35:20] ERROR: Provisioning failed for runId=a386d9f5 shardIndex=8 method=createBuildDroplet attempts=1/3 elapsedMs=96591 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-a386d9f5-s8-ab2f161' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: SandboxException: Cannot invoke 'dropletserviceserver/DropletCreationStatusImpl.getRequestId()Ljava/lang/String;': the sandboxed connection was invalidated (a prior call 'dropletserviceserver/DropletServiceClientImpl.addSshKeyToDroplet(JLjava/lang/String;)V' timed out after 30s). Reopen the connection to use a fresh SJVM sandbox.
[shard 7/10] [2026-10-07 19:35:20] ERROR: Provisioning failed for runId=a386d9f5 shardIndex=7 method=createBuildDroplet attempts=1/3 elapsedMs=98375 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-a386d9f5-s7-ab2f15c' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: SandboxException: Cannot invoke 'dropletserviceserver/DropletCreationStatusImpl.getRequestId()Ljava/lang/String;': the sandboxed connection was invalidated (a prior call 'dropletserviceserver/DropletServiceClientImpl.addSshKeyToDroplet(JLjava/lang/String;)V' timed out after 30s). Reopen the connection to use a fresh SJVM sandbox.
[shard 4/10] [2026-10-07 19:35:22] provisioning_failure runId=a386d9f5 shardIndex=4 shardCount=10 method=createBuildDroplet attempt=1 maxAttempts=3 elapsedMs=100752 classification=INFRASTRUCTURE/PROVISIONING action=exhausted error=DropletAllocationException: Failed to create droplet in all regions (sfo3, nyc3). Last failure: CancellationException: Job was cancelled
[2026-10-07 19:35:22] Shard execution failed without a surviving dynamic owner; stopping the other provisioning workers so the resumed run can publish its failure
[shard 4/10] [2026-10-07 19:35:22] ERROR: Provisioning failed for runId=a386d9f5 shardIndex=4 method=createBuildDroplet attempts=1/3 elapsedMs=100752 classification=INFRASTRUCTURE/PROVISIONING: Failed to create droplet in all regions (sfo3, nyc3). Last failure: CancellationException: Job was cancelled
[2026-10-07 19:35:22] Shard execution failed without a surviving dynamic owner; stopping the other provisioning workers so the resumed run can publish its failure
[shard 0/10] [2026-10-07 19:35:22] ERROR: Provisioning stopped for runId=a386d9f5 shardIndex=0 method=createBuildDroplet attempts=1/3 elapsedMs=100893 classification=INFRASTRUCTURE/PROVISIONING dropletId=0 reason=terminal-or-sibling-failure status=PENDING threadInterrupted=true cause=ProvisioningStoppedForTerminalRunException: Provisioning stopped for runId=a386d9f5 because the run is terminal or a sibling shard failed; status=PENDING
[2026-10-07 19:35:22] Shard execution failed without a surviving dynamic owner; stopping the other provisioning workers so the resumed run can publish its failure
[shard 6/10] [2026-10-07 19:35:22] ERROR: Provisioning stopped for runId=a386d9f5 shardIndex=6 method=createBuildDroplet attempts=1/3 elapsedMs=100676 classification=INFRASTRUCTURE/PROVISIONING dropletId=0 reason=terminal-or-sibling-failure status=PENDING threadInterrupted=true cause=ProvisioningStoppedForTerminalRunException: Provisioning stopped for runId=a386d9f5 because the run is terminal or a sibling shard failed; status=PENDING
[2026-10-07 19:35:22] Shard execution failed without a surviving dynamic owner; stopping the other provisioning workers so the resumed run can publish its failure
[shard 3/10] [2026-10-07 19:35:22] ERROR: Provisioning stopped for runId=a386d9f5 shardIndex=3 method=createBuildDroplet attempts=1/3 elapsedMs=100786 classification=INFRASTRUCTURE/PROVISIONING dropletId=0 reason=terminal-or-sibling-failure status=PENDING threadInterrupted=true cause=ProvisioningStoppedForTerminalRunException: Provisioning stopped for runId=a386d9f5 because the run is terminal or a sibling shard failed; status=PENDING
[2026-10-07 19:35:22] Shard execution failed without a surviving dynamic owner; stopping the other provisioning workers so the resumed run can publish its failure
[shard 5/10] [2026-10-07 19:35:22] ERROR: Provisioning stopped for runId=a386d9f5 shardIndex=5 method=createBuildDroplet attempts=1/3 elapsedMs=100703 classification=INFRASTRUCTURE/PROVISIONING dropletId=0 reason=terminal-or-sibling-failure status=PENDING threadInterrupted=true cause=ProvisioningStoppedForTerminalRunException: Provisioning stopped for runId=a386d9f5 because the run is terminal or a sibling shard failed; status=PENDING
[2026-10-07 19:35:22] Shard execution failed without a surviving dynamic owner; stopping the other provisioning workers so the resumed run can publish its failure
[shard 2/10] [2026-10-07 19:35:22] ERROR: Provisioning stopped for runId=a386d9f5 shardIndex=2 method=createBuildDroplet attempts=1/3 elapsedMs=100860 classification=INFRASTRUCTURE/PROVISIONING dropletId=0 reason=terminal-or-sibling-failure status=PENDING threadInterrupted=true cause=ProvisioningStoppedForTerminalRunException: Provisioning stopped for runId=a386d9f5 because the run is terminal or a sibling shard failed; status=PENDING
[2026-10-07 19:35:22] ERROR during resume: 10 of 10 shard(s) failed during resume: shard 0: Provisioning stopped for runId=a386d9f5 shardIndex=0 method=createBuildDroplet attempts=1/3 elapsedMs=100893 classification=INFRASTRUCTURE/PROVISIONING dropletId=0 reason=terminal-or-sibling-failure status=PENDING threadInterrupted=true cause=ProvisioningStoppedForTerminalRunException: Provisioning stopped for runId=a386d9f5 because the run is terminal or a sibling shard failed; status=PENDING; shard 1: Provisioning failed for runId=a386d9f5 shardIndex=1 method=createBuildDroplet attempts=1/3 elapsedMs=99739 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-a386d9f5-s1-ab2f166' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: SandboxException: Cannot invoke 'dropletserviceserver/DropletCreationStatusImpl.getRequestId()Ljava/lang/String;': the sandboxed connection was invalidated (a prior call 'dropletserviceserver/DropletServiceClientImpl.addSshKeyToDroplet(JLjava/lang/String;)V' timed out after 30s). Reopen the connection to use a fresh SJVM sandbox.; shard 2: Provisioning stopped for runId=a386d9f5 shardIndex=2 method=createBuildDroplet attempts=1/3 elapsedMs=100860 classification=INFRASTRUCTURE/PROVISIONING dropletId=0 reason=terminal-or-sibling-failure status=PENDING threadInterrupted=true cause=ProvisioningStoppedForTerminalRunException: Provisioning stopped for runId=a386d9f5 because the run is terminal or a sibling shard failed; status=PENDING; shard 3: Provisioning stopped for runId=a386d9f5 shardIndex=3 method=createBuildDroplet attempts=1/3 elapsedMs=100786 classification=INFRASTRUCTURE/PROVISIONING dropletId=0 reason=terminal-or-sibling-failure status=PENDING threadInterrupted=true cause=ProvisioningStoppedForTerminalRunException: Provisioning stopped for runId=a386d9f5 because the run is terminal or a sibling shard failed; status=PENDING; shard 4: Provisioning failed for runId=a386d9f5 shardIndex=4 method=createBuildDroplet attempts=1/3 elapsedMs=100752 classification=INFRASTRUCTURE/PROVISIONING: Failed to create droplet in all regions (sfo3, nyc3). Last failure: CancellationException: Job was cancelled; shard 5: Provisioning stopped for runId=a386d9f5 shardIndex=5 method=createBuildDroplet attempts=1/3 elapsedMs=100703 classification=INFRASTRUCTURE/PROVISIONING dropletId=0 reason=terminal-or-sibling-failure status=PENDING threadInterrupted=true cause=ProvisioningStoppedForTerminalRunException: Provisioning stopped for runId=a386d9f5 because the run is terminal or a sibling shard failed; status=PENDING; shard 6: Provisioning stopped for runId=a386d9f5 shardIndex=6 method=createBuildDroplet attempts=1/3 elapsedMs=100676 classification=INFRASTRUCTURE/PROVISIONING dropletId=0 reason=terminal-or-sibling-failure status=PENDING threadInterrupted=true cause=ProvisioningStoppedForTerminalRunException: Provisioning stopped for runId=a386d9f5 because the run is terminal or a sibling shard failed; status=PENDING; shard 7: Provisioning failed for runId=a386d9f5 shardIndex=7 method=createBuildDroplet attempts=1/3 elapsedMs=98375 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-a386d9f5-s7-ab2f15c' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: SandboxException: Cannot invoke 'dropletserviceserver/DropletCreationStatusImpl.getRequestId()Ljava/lang/String;': the sandboxed connection was invalidated (a prior call 'dropletserviceserver/DropletServiceClientImpl.addSshKeyToDroplet(JLjava/lang/String;)V' timed out after 30s). Reopen the connection to use a fresh SJVM sandbox.; shard 8: Provisioning failed for runId=a386d9f5 shardIndex=8 method=createBuildDroplet attempts=1/3 elapsedMs=96591 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-a386d9f5-s8-ab2f161' is not safe to retry inline because the async request result was not delivered conclusively. No per-droplet fleet polling was started; the coordinator must use its fleet-wide cleanup-before-requeue path. Last failure: SandboxException: Cannot invoke 'dropletserviceserver/DropletCreationStatusImpl.getRequestId()Ljava/lang/String;': the sandboxed connection was invalidated (a prior call 'dropletserviceserver/DropletServiceClientImpl.addSshKeyToDroplet(JLjava/lang/String;)V' timed out after 30s). Reopen the connection to use a fresh SJVM sandbox.; shard 9: Provisioning failed for runId=a386d9f5 shardIndex=9 method=createBuildDroplet attempts=1/3 elapsedMs=94805 classification=INFRASTRUCTURE/PROVISIONING: Droplet creation outcome for 'buildtest-a386d9f5-s9-ab2f168' is unknown because its fleet observation failed during failed-creation recovery. The allocation must be requeued without another create attempt. Observation failure: java.util.concurrent.CancellationException: Job was cancelled