BuildTest health
Seven days of build-fleet activity, sampled every 15 minutes.
Data: /api/health JSON · Plain liveness check: /health (returns OK).
Live updates are paused: the last attempt to read /api/health failed, so the fleet numbers and timestamps on this page are frozen at their last known values.
The latest background refresh failed; the charts below show the last available snapshot.
The live projection feed is disconnected, has an unapplied gap, or has a reset that has not finished; this dashboard shows the last committed stored view.
Running81
Starting28
Tearing down—
Unclassified0
Available capacity91
Limit200
Active runs19
Pending runs1
Fleet observed
Snapshot generated . Data age: 21m 2s.
503 runs could not be assigned to lifecycle states because legacy phase timestamps were unavailable.
Build runs
Unfinished build runs only. Showing 20 of 20 runs that have not finished. Active runs are shown before pending runs; newest first within each group.
Snapshot takenUtilization sampled
| Build ID | Project | Status | Droplets | Started | Pending time | Execution time | Utilization |
|---|---|---|---|---|---|---|---|
| 66c8155a | kotlinbuild.ci:kotlin-build-ci | BUILDING | + | ||||
| 8d0c3415 | kotlinbuild.ci:kotlin-build-ci | BUILDING | + | ||||
| 45a7d8c4 | kotlinbuild.ci:kotlin-build-ci | BUILDING | + | ||||
| 401dd610 | foundation.url:resolver | BUILDING | + | ||||
| f8694a5a | foundation.url:resolver | BUILDING | + | ||||
| 97477eda | kotlinbuild.ci:kotlin-build-ci | BUILDING | + | ||||
| 1d116f91 | foundation.url:protocol | BUILDING | + | ||||
| ffb93f01 | foundation.url:resolver | TESTING | + | ||||
| 4d756ed4 | foundation.url:resolver | TESTING | + | ||||
| 5671febd | foundation.url:protocol | TESTING | + | ||||
| fe646414 | foundation.url:resolver | TESTING | + | ||||
| b7a4a5ab | foundation.url:protocol | TESTING | + | ||||
| 52366186 | foundation.url:protocol | TESTING | + | ||||
| df820230 | foundation.url:protocol | TESTING | + | ||||
| 361da0d3 | foundation.url:resolver | TESTING | + | ||||
| 6b39ac85 | foundation.url:resolver | TESTING | + | ||||
| b2e4fd37 | foundation.url:protocol | TESTING | + | ||||
| f307e954 | foundation.url:resolver | TESTING | + | ||||
| 5873b123 | foundation.url:resolver | TESTING | + | ||||
| 2d8b96e8 | - | PENDING | — | + | not started |
Build → droplet assignments
Every queued and running build, plus terminal records missing their completion time, and every droplet it holds — one row per shard, not just the primary. Builds still holding machines after finishing are listed first, then the rest sorted by longest time in state.
Run projection read
Fleet observed by coordinator
This table is older than it should be: the stored run projection behind it has not been refreshed recently, so a build may have been given a droplet, finished, or failed since these rows were written. Treat every row below as a past reading, not a current one.
The coordinator's own fleet observation has stopped advancing, so the active-droplet count this table reconciles against is frozen at an old reading. A frozen count looks exactly like a steady one: do not read it as evidence that the fleet is healthy.
Visible unfinished builds20
Builds with no droplet1
Queued builds holding a droplet0
Active droplets held by an unfinished buildunknown
Active droplets held by a finished buildunknown
Active droplets no build accounts forunknown
At least one build's per-run droplet records could not be read, so the droplet counts above cannot be reconciled against the 82 droplets the coordinator reports active: that build may be holding more machines than it can show.
The coordinator's per-run droplet records could not be read for at least one build below, so that build shows only the single droplet its run record names and may be holding more. Every fleet-wide droplet count on this page therefore reads unknown rather than a number that would look exact.
| Droplet | Role | State | Created | In state since | Size |
|---|---|---|---|---|---|
| b2e4fd37TESTINGin this state 7 dropletsCodexCoder21Organization/UrlProtocolBranch: review/rvUP657bt-394c8286 | |||||
| buildtest-b2e4fd37-ab2f396 id 607125043 · 143.244.184.140 | primary | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-b2e4fd37-s5-ab2f3b6 id 607127113 · 64.23.143.176 | shard 5 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-b2e4fd37-s9-ab2f3b8 id 607127117 · 137.184.39.36 | shard 9 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-b2e4fd37-s1-ab2f3bc id 607127126 · 143.110.145.180 | shard 1 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-b2e4fd37-s4-ab2f3b9 id 607127123 · 164.92.92.37 | shard 4 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-b2e4fd37-s7-ab2f3bb id 607127128 · 137.184.115.111 | shard 7 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-b2e4fd37-s3-ab2f3bd id 607127133 · 64.23.244.21 | shard 3 | Testing | s-2vcpu-4gb sfo3 | ||
| 5873b123TESTINGin this state 2 dropletsCodexCoder21Organization/UrlResolverBranch: work/fx1205w-round12 | |||||
| buildtest-5873b123-ab2f394 id 607124975 · 64.23.177.249 | primary | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-5873b123-s8-ab2f3c2 id 607127313 · 146.190.118.28 | shard 8 | Testing | s-2vcpu-4gb sfo3 | ||
| 6b39ac85TESTINGin this state 8 dropletsCodexCoder21Organization/UrlResolverBranch: work/fx1205x-evidence | |||||
| buildtest-6b39ac85-ab2f397 id 607125055 · 147.182.249.159 | primary | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-6b39ac85-s1-ab2f3c8 id 607127489 · 137.184.238.250 | shard 1 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-6b39ac85-s3-ab2f3c9 id 607127494 · 137.184.190.123 | shard 3 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-6b39ac85-s9-ab2f3cb id 607127501 · 143.110.230.145 | shard 9 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-6b39ac85-s5-ab2f3ce id 607127507 · 147.182.232.254 | shard 5 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-6b39ac85-s8-ab2f3d0 id 607127518 · 147.182.192.84 | shard 8 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-6b39ac85-s2-ab2f3cf id 607127520 · 137.184.10.163 | shard 2 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-6b39ac85-s4-ab2f3cd id 607127509 · 143.198.96.131 | shard 4 | Testing | s-2vcpu-4gb sfo3 | ||
| 361da0d3TESTINGin this state 5 dropletsCodexCoder21Organization/UrlResolverBranch: work/fx1205x-evidence | |||||
| buildtest-361da0d3-ab2f398 id 607125070 · 24.199.112.107 | primary | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-361da0d3-s8-ab2f3d1 id 607127675 · 146.190.164.121 | shard 8 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-361da0d3-s5-ab2f3d5 id 607127713 · 64.227.104.85 | shard 5 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-361da0d3-s6-ab2f3d4 id 607127712 · 146.190.115.11 | shard 6 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-361da0d3-s1-ab2f3d9 id 607127726 · 137.184.13.240 | shard 1 | Testing | s-2vcpu-4gb sfo3 | ||
| f307e954TESTINGin this state 5 dropletsCodexCoder21Organization/UrlResolverBranch: work/fx1205x-evidence | |||||
| buildtest-f307e954-s2-ab2f3db id 607127774 · 146.190.126.164 | shard 2 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-f307e954-s4-ab2f3dc id 607127786 · 209.38.68.30 | shard 4 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-f307e954-s1-ab2f3e0 id 607127793 · 146.190.113.95 | shard 1 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-f307e954-s7-ab2f3e1 id 607127797 · 64.23.167.94 | shard 7 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-f307e954-s3-ab2f3e2 id 607127799 · 146.190.127.7 | shard 3 | Testing | s-2vcpu-4gb sfo3 | ||
| 52366186TESTINGin this state 3 dropletsCodexCoder21Organization/UrlProtocolPR #650 | |||||
| buildtest-52366186-s3-ab2f3e8 id 607129891 · 143.198.141.26 | shard 3 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-52366186-s2-ab2f3f2 id 607129917 · 64.23.209.60 | shard 2 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-52366186-s8-ab2f3f4 id 607129918 · 209.38.77.63 | shard 8 | Testing | s-2vcpu-4gb sfo3 | ||
| df820230TESTINGin this state 4 dropletsCodexCoder21Organization/UrlProtocolPR #645 | |||||
| buildtest-df820230-s3-ab2f3ec id 607129902 · 143.198.56.109 | shard 3 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-df820230-s8-ab2f3ef id 607129906 · 209.38.135.172 | shard 8 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-df820230-s2-ab2f3f3 id 607129920 · 24.144.82.92 | shard 2 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-df820230-s5-ab2f3f7 id 607129921 · 164.92.65.117 | shard 5 | Testing | s-2vcpu-4gb sfo3 | ||
| b7a4a5abTESTINGin this state 4 dropletsCodexCoder21Organization/UrlProtocolPR #645 | |||||
| buildtest-b7a4a5ab-s8-ab2f400 id 607131106 · 144.126.217.110 | shard 8 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-b7a4a5ab-s7-ab2f3fe id 607131104 · 164.92.89.74 | shard 7 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-b7a4a5ab-s4-ab2f402 id 607131120 · 147.182.207.194 | shard 4 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-b7a4a5ab-s1-ab2f404 id 607131113 · 146.190.155.248 | shard 1 | Testing | s-2vcpu-4gb sfo3 | ||
| fe646414TESTINGin this state 10 dropletsCodexCoder21Organization/UrlResolverPR #1169 | |||||
| buildtest-fe646414-ab2f3fa id 607130582 · 137.184.4.44 | primary | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-fe646414-s4-ab2f40e id 607132861 · 146.190.54.25 | shard 4 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-fe646414-s2-ab2f40c id 607132854 · 137.184.42.13 | shard 2 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-fe646414-s8-ab2f40d id 607132857 · 143.198.129.68 | shard 8 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-fe646414-s9-ab2f40f id 607132864 · 165.232.158.41 | shard 9 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-fe646414-s7-ab2f410 id 607132872 · 137.184.121.77 | shard 7 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-fe646414-s1-ab2f411 id 607132870 · 24.144.81.35 | shard 1 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-fe646414-s5-ab2f412 id 607132874 · 143.198.155.39 | shard 5 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-fe646414-s6-ab2f413 id 607132881 · 143.198.59.12 | shard 6 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-fe646414-s3-ab2f414 id 607132878 · 143.244.189.220 | shard 3 | Testing | s-2vcpu-4gb sfo3 | ||
| 5671febdTESTINGin this state 7 dropletsCodexCoder21Organization/UrlProtocolPR #616 | |||||
| buildtest-5671febd-ab2f408 id 607131811 · 164.92.108.235 | primary | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-5671febd-s5-ab2f417 id 607134190 · 143.244.188.82 | shard 5 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-5671febd-s7-ab2f419 id 607134196 · 143.198.76.234 | shard 7 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-5671febd-s1-ab2f415 id 607134191 · 143.198.108.153 | shard 1 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-5671febd-s3-ab2f416 id 607134193 · 165.232.150.203 | shard 3 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-5671febd-s4-ab2f41b id 607134209 · 147.182.244.121 | shard 4 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-5671febd-s9-ab2f41c id 607134211 · 147.182.200.95 | shard 9 | Testing | s-2vcpu-4gb sfo3 | ||
| 1d116f91BUILDINGin this state 1 dropletCodexCoder21Organization/UrlProtocolBranch: review/656-tests-fxUPhalfclose4 | |||||
| buildtest-1d116f91-ab2f41e id 607134496 · 134.199.230.16 | primary | Building | s-2vcpu-4gb sfo3 | ||
| ffb93f01TESTINGin this state 10 dropletsCodexCoder21Organization/UrlResolverPR #1201 | |||||
| buildtest-ffb93f01-ab2f40a id 607131923 · 137.184.228.221 | primary | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-ffb93f01-s9-ab2f425 id 607135225 · 24.199.112.228 | shard 9 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-ffb93f01-s8-ab2f423 id 607135219 · 137.184.236.70 | shard 8 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-ffb93f01-s6-ab2f424 id 607135221 · 146.190.145.34 | shard 6 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-ffb93f01-s2-ab2f429 id 607135332 · 134.199.213.186 | shard 2 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-ffb93f01-s3-ab2f428 id 607135260 · 165.232.134.43 | shard 3 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-ffb93f01-s7-ab2f426 id 607135255 · 64.227.103.63 | shard 7 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-ffb93f01-s1-ab2f42b id 607135336 · 143.198.227.174 | shard 1 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-ffb93f01-s4-ab2f427 id 607135284 · 147.182.195.191 | shard 4 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-ffb93f01-s5-ab2f42a id 607135330 · 164.92.83.21 | shard 5 | Testing | s-2vcpu-4gb sfo3 | ||
| 97477edaBUILDINGin this state 1 dropletopus-fix301-akotlin-build-ci-upload-request=797b0b55-ca44-4cb0-ab0a-f4d1d8018f39 | |||||
| buildtest-97477eda-ab2f422 id 607134975 · 147.182.197.42 | primary | Building | s-2vcpu-4gb sfo3 | ||
| 4d756ed4TESTINGin this state 10 dropletsCodexCoder21Organization/UrlResolverPR #1155 | |||||
| buildtest-4d756ed4-ab2f409 id 607131887 · 64.23.231.194 | primary | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-4d756ed4-s3-ab2f42e id 607136342 · 143.198.238.106 | shard 3 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-4d756ed4-s7-ab2f42c id 607136338 · 165.232.133.53 | shard 7 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-4d756ed4-s5-ab2f42d id 607136343 · 64.23.203.251 | shard 5 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-4d756ed4-s4-ab2f431 id 607136369 · 165.232.152.104 | shard 4 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-4d756ed4-s9-ab2f42f id 607136357 · 164.92.83.70 | shard 9 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-4d756ed4-s6-ab2f430 id 607136364 · 146.190.169.12 | shard 6 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-4d756ed4-s1-ab2f432 id 607136374 · 64.23.169.86 | shard 1 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-4d756ed4-s8-ab2f434 id 607136393 · 64.23.227.12 | shard 8 | Testing | s-2vcpu-4gb sfo3 | ||
| buildtest-4d756ed4-s2-ab2f433 id 607136379 · 64.23.168.95 | shard 2 | Testing | s-2vcpu-4gb sfo3 | ||
| f8694a5aBUILDINGin this state 1 dropletCodexCoder21Organization/UrlResolverPR #1127 | |||||
| buildtest-f8694a5a-ab2f435 id 607137114 · 64.23.172.170 | primary | Building | s-2vcpu-4gb sfo3 | ||
| 401dd610BUILDINGin this state 1 dropletCodexCoder21Organization/UrlResolverBranch: review/fx1126c-host-lifecycle-proof-wip | |||||
| buildtest-401dd610-ab2f436 id 607137175 · 64.227.103.100 | primary | Building | s-2vcpu-4gb sfo3 | ||
| 45a7d8c4BUILDINGin this state 1 dropletopus-fix301-mut-oldkotlin-build-ci-upload-request=220c1238-fbc0-466d-ab7d-a1913867dbea | |||||
| buildtest-45a7d8c4-ab2f437 id 607138666 · 143.198.60.253 | primary | Building | s-2vcpu-4gb sfo3 | ||
| 2d8b96e8PENDINGin this state no dropletCodexCoder21Organization/UrlResolverBranch: work/fx1212f-round-five-wip | |||||
| 8d0c3415BUILDINGin this state 1 dropletopus-fix301-mut-newkotlin-build-ci-upload-request=958a1b86-2f7a-4ece-882c-b07892781ebf | |||||
| buildtest-8d0c3415-ab2f438 id 607138703 · 146.190.174.33 | primary | Building | s-2vcpu-4gb sfo3 | ||
| 66c8155aBUILDINGin this state 1 dropletopus-fix301-oldorderkotlin-build-ci-upload-request=cd3ed9fe-7503-4cff-b508-00ae0db566f7 | |||||
| buildtest-66c8155a-ab2f439 id 607138765 · 137.184.3.180 | primary | Building | s-2vcpu-4gb sfo3 | ||
Assignments read from the stored run projection . Reload to re-read it.
Droplet ages
How long each droplet the fleet is holding right now has been up, in five-minute bands, with each column stacked by what its droplets are doing. Ages are measured from when the backend recorded the droplet as created, against the snapshot time above. The last band is open-ended. A tall column on the right made of early stages — provisioning, installing — is droplets that have been stuck coming up rather than droplets doing work.
82 droplets: 7 building, 75 testing.