Upgrade notes tell you what changed. They do not show you your next plan, or the step at which a removed feature fails. So we tested it. We applied configurations with OpenTofu 1.12.7, ran 1.13.1 against the same state, and recorded what each command printed. This post goes through the results and ends with a checklist you can run before you change the version in CI.

TLDR

• base64gzip in 1.13.1 returns a different string for the same input. OpenTofu moved to Go 1.27, and Go changed its DEFLATE encoder. Both outputs decompress to identical bytes, but providers compare the string. With no config change, our plan was 1 to add, 1 to change, 1 to destroy.
• On aws_instance, a changed user_data_base64 means a stop/start, or a replacement if you set user_data_replace_on_change. On azurerm_linux_virtual_machine, a changed custom_data forces a new VM.
• ignore_changes hides the diff. In our test it also hid a real edit to the cloud-init file. Gzip done inside the cloudinit provider did not change at all.
• A winrm provisioner passes tofu validate and tofu plan on 1.13.1, with the old "will be removed in a future version" warning. It fails at apply, after the resource exists, and leaves the resource tainted.
• assumenotnull fixed a classic Invalid count argument error. assumestringprefix moved a mis-wired module input from a mid-apply failure to a plan error. Older versions reject these functions, and since 1.12 a required_version in a .tf file does not stop them.
• -lint=all is a useful experiment, but it only adds warnings. The exit code stays 0.

Prerequisites

• An OpenTofu 1.12.x codebase, or a Terraform codebase that you plan to move to OpenTofu
• A CI job or shell that can run tofu plan with read access to your real state
• jq, for the plan JSON recipe
• The 1.13.1 zip for your platform from the GitHub release page, checked against its SHA256SUMS file

How we tested

We ran everything on a Raspberry Pi with a 64-bit (arm64) OS: OpenTofu 1.12.7 and 1.13.1 for linux_arm64, the 1.13.1 linux_arm (32-bit) build, and Terraform 1.16.5 for comparison. Every zip matched its published SHA256 sum. In the outputs below, tofu-1.12.7 and tofu-1.13.1 are the two release binaries side by side.

We did not use a cloud account. The built-in terraform_data resource stands in for provider attributes. Its input argument updates in place when the value changes, and its triggers_replace argument forces a replacement. Those are the two ways real providers treat user data, which we checked in the provider docs. Every output in this post comes from these runs. Where we cut lines from an output, the block shows ... or the grep/tail we used, and long base64 strings are shortened with ....

base64gzip: same input, different string

base64gzip compresses a string with gzip and base64-encodes the result. Its most common job is cloud-init user data, because EC2 limits user data to 16 KB and compression gives you more room. Here is the same call on both versions, plus Terraform 1.16.5, and then a check of what a real payload (a 1,208-byte cloud-init file that installs nginx and writes a systemd unit) decompresses to:

base64gzip across versions

# same input, three binaries
$ echo 'base64gzip("hello")' | ./tofu-1.12.7 console
"H4sIAAAAAAAA/8pIzcnJBwAAAP//AQAA//+GphA2BQAAAA=="
$ echo 'base64gzip("hello")' | ./tofu-1.13.1 console
"H4sIAAAAAAAA/wAFAPr/aGVsbG8AAAD//wMAhqYQNgUAAAA="
$ echo 'base64gzip("hello")' | ./terraform-1.16.5 console
"H4sIAAAAAAAA/8pIzcnJBwAAAP//AQAA//+GphA2BQAAAA=="
# decompress the cloud-init payload from each version
$ echo 'base64gzip(file("cloud-init.yaml"))' | ./tofu-1.12.7 console | tr -d '"' | base64 -d | gunzip | sha256sum
3a62ee85006428492dc1aeccc789867735496026ada3bfaef3d0cf00dcc1bcd1  -
$ echo 'base64gzip(file("cloud-init.yaml"))' | ./tofu-1.13.1 console | tr -d '"' | base64 -d | gunzip | sha256sum
3a62ee85006428492dc1aeccc789867735496026ada3bfaef3d0cf00dcc1bcd1  -
$ sha256sum < cloud-init.yaml
3a62ee85006428492dc1aeccc789867735496026ada3bfaef3d0cf00dcc1bcd1  -

The encoded strings differ. The content does not: both outputs decompress to the exact bytes of the source file.

The cause is upstream. OpenTofu 1.12.7 is built with Go 1.26.6 and 1.13.1 with Go 1.27.1 (each binary records its Go version). The Go 1.27 release notes say that "the exact encoded output from Writer may be different from Go 1.26 as a result of the encoder implementation change", and that this carries through to compress/gzip. The OpenTofu 1.13 changelog calls the new output "equivalent to but not equal to" the output of earlier releases. The new output is stable: three runs of 1.13.1 gave the same string.

Terraform 1.16.5 is built with Go 1.26.8 and produced the same bytes as OpenTofu 1.12.7. So you also get this diff when you move from Terraform 1.16 to OpenTofu 1.13, not only when you upgrade OpenTofu.

What the plan shows

1. cloud-init.yaml unchanged
2. base64gzip() Go 1.27 DEFLATE
3. New string same bytes after gunzip
4. Provider diff string != state
5. Plan update or replace

This is the config we applied with 1.12.7:

locals {
  user_data = base64gzip(file("${path.module}/cloud-init.yaml"))
}

# Stands in for an attribute that a provider updates in place,
# such as user_data_base64 on aws_instance.
resource "terraform_data" "web_in_place" {
  input = local.user_data
}

# Stands in for an attribute that forces replacement,
# such as custom_data on azurerm_linux_virtual_machine.
resource "terraform_data" "web_replace" {
  triggers_replace = local.user_data
}

After tofu-1.12.7 apply, a 1.12.7 plan reported no changes. Then we planned with 1.13.1 against the same state and the same config:

$ tofu-1.13.1 plan
terraform_data.web_replace: Refreshing state... [id=4437d213-ef2e-bd8b-9c81-b5831d1b46c8]
terraform_data.web_in_place: Refreshing state... [id=b0ec5936-31c0-4653-0fb6-e353122675fa]

OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place (current -> planned)
-/+ destroy and then create replacement

OpenTofu will perform the following actions:

  # terraform_data.web_in_place will be updated in-place
  ~ resource "terraform_data" "web_in_place" {
        id     = "b0ec5936-31c0-4653-0fb6-e353122675fa"
      ~ input  = "H4sIAAAAAAAA/5RU3U7zRhC9z..." -> "H4sIAAAAAAAA/5RTXW/jNhB89..."
      ~ output = "H4sIAAAAAAAA/5RU3U7zRhC9z..." -> (known after apply)
    }

  # terraform_data.web_replace must be replaced
-/+ resource "terraform_data" "web_replace" {
      ~ id               = "4437d213-ef2e-bd8b-9c81-b5831d1b46c8" -> (known after apply)
      ~ triggers_replace = "H4sIAAAAAAAA/5RU3U7zRhC9z..." -> "H4sIAAAAAAAA/5RTXW/jNhB89..."
    }

Plan: 1 to add, 1 to change, 1 to destroy.

Two resources, no edits, one update and one replacement. On real resources, the result depends on the provider:

• aws_instance: for user_data_base64, where gzip output belongs, "Updates to this field will trigger a stop/start of the EC2 instance by default. If the user_data_replace_on_change is set then updates to this field will trigger a destroy and recreate of the EC2 instance."
• azurerm_linux_virtual_machine: for custom_data, "Changing this forces a new resource to be created."
• aws_launch_template: a changed user_data creates a new template version. Instances get it the next time your Auto Scaling group launches or refreshes from that version.

None of this is a disaster if you plan for it. But a stop/start of a single production box at 2 p.m. is the kind of surprise you want to catch in plan review, not after someone approves an apply because "it is only a version bump".

Find it before the upgrade

Start with a search. Run it after tofu init, so that it also covers registry and git modules in .terraform/modules:

grep -rnE --include='*.tf' --include='*.tofu' 'base64gzip\(|"winrm"' .

On our test folder it found both problems this post covers:

main.tf:2:  user_data = base64gzip(file("${path.module}/cloud-init.yaml"))
winrm/main.tf:6:      type     = "winrm"

A search only finds literal calls. A plan is the reliable check. Run a 1.13.1 plan in a throwaway job, save it, and list each changed attribute with this jq filter (save it as changed-attrs.jq):

.resource_changes[]
| select(.change.actions != ["no-op"])
| . as $r
| [ ($r.change.before // {}) | keys[]
    | select($r.change.before[.] != $r.change.after[.]
             and ($r.change.after_unknown[.] | not)) ]
| "\($r.change.actions | join("+"))  \($r.address)  changed: \(join(", "))"

$ tofu-1.13.1 plan -lock=false -out=upgrade.tfplan > /dev/null
$ tofu-1.13.1 show -json upgrade.tfplan | jq -r -f changed-attrs.jq
update  terraform_data.web_in_place  changed: input
delete+create  terraform_data.web_replace  changed: triggers_replace

A plan does not write state, and -lock=false stops a dry-run job from blocking a real apply. Remove the flag if you prefer to wait for the lock. You want a list in which every changed attribute is user data. Anything else in the list is real drift or a different upgrade change, so examine it separately.

Three ways to handle it

Handling the base64gzip diff

Accept it

# Plan with 1.13.1, review, and apply in a window you choose.
# The content is identical, so the only effect is the
# stop/start or replacement itself.
tofu plan -out=upgrade.tfplan
tofu show -json upgrade.tfplan | jq -r -f changed-attrs.jq
tofu apply upgrade.tfplan

Hide it (temporary)

resource "aws_instance" "web" {
  # ...
  user_data_base64 = base64gzip(file("${path.module}/cloud-init.yaml"))

  lifecycle {
    # TEMPORARY: OpenTofu 1.13 re-encodes base64gzip output.
    # This also hides real cloud-init edits. Remove it on the next change.
    ignore_changes = [user_data_base64]
  }
}

Gzip in the provider

data "cloudinit_config" "web" {
  gzip          = true
  base64_encode = true

  part {
    content_type = "text/cloud-config"
    content      = file("${path.module}/cloud-init.yaml")
  }
}

resource "aws_instance" "web" {
  # ...
  user_data_base64 = data.cloudinit_config.web.rendered
}

Accept it. This is usually the correct choice. Do the upgrade apply in a maintenance window, one environment at a time, and use the jq list as the change record.

Hide it. The release notes suggest ignore_changes as a temporary fix, and it works: with ignore_changes on both test resources, the 1.13.1 plan said No changes. Then we added a line to cloud-init.yaml and planned again. It still said No changes.

Warning

ignore_changes cannot tell the encoder change from a real edit. While it is in place, changes to your cloud-init file do not reach your instances, and the plan does not tell you. If you use it, open a ticket to remove it, and remove it on the next intentional user data change.

Move the gzip into the provider. With cloudinit_config and gzip = true, the compression runs in the provider binary, not in OpenTofu. We rendered the same file with hashicorp/cloudinit v2.4.1 under both 1.12.7 and 1.13.1, and the SHA256 of rendered was identical. This does not make you immune. It moves the dependency: v2.4.1 is built with Go 1.26.8, and a future provider release built with Go 1.27 can cause the same one-time diff. So pin the provider version and read its changelog. The switch itself also changes your user data once, because cloudinit_config wraps parts in a MIME multi-part document. Do it in the same window as the upgrade.

WinRM: passes validate and plan, fails at apply

OpenTofu 1.12 deprecated the winrm connection type, and 1.13 removed it (#4012) because "some of the upstream libraries OpenTofu was using to implement these features are no longer maintained". We expected tofu validate to report it. It does not. This is the test config (the host is a closed local port with a short timeout, so that the run fails fast):

resource "terraform_data" "bootstrap" {
  provisioner "remote-exec" {
    inline = ["powershell -Command Install-WindowsFeature Web-Server"]

    connection {
      type     = "winrm"
      host     = "127.0.0.1"
      user     = "Administrator"
      password = "example-only"
      https    = true
      timeout  = "10s"
    }
  }
}

$ tofu-1.13.1 validate
Warning: WinRM connection type is deprecated

  on main.tf line 6, in resource "terraform_data" "bootstrap":
   5:     connection {
   6:       type     = "winrm"

The winrm connection type is deprecated and will be removed in a future
version of OpenTofu.
...
Success! The configuration is valid, but there were some validation warnings
as shown above.

$ tofu-1.13.1 plan | grep 'Plan:'
Plan: 1 to add, 0 to change, 0 to destroy.

$ time tofu-1.13.1 apply -auto-approve
...
Error: remote-exec provisioner error

  with terraform_data.bootstrap,
  on main.tf line 2, in resource "terraform_data" "bootstrap":
   2:   provisioner "remote-exec" {

'winrm' connections are not supported in OpenTofu v1.13 or later

Error: Provisioners no longer support WinRM
...
real    0m0.153s

$ tofu-1.13.1 show | head -2
# terraform_data.bootstrap: (tainted)
resource "terraform_data" "bootstrap" {

1.13.1 still prints the 1.12 deprecation warning at validate and plan. It says the feature "will be removed in a future version", but the feature is already gone in this version. The error comes in under a second at apply. For comparison, 1.12.7 tried to connect to port 5986 and failed at the 10-second timeout, as it should.

The order matters. Provisioners run after the resource is created, and the OpenTofu docs say "if a creation-time provisioner fails, the resource is marked as tainted" and "will be planned for destruction and recreation upon the next tofu apply". With a real Windows VM, the VM is created and billed, then marked for replacement, and each apply after that recreates it and fails again until you remove the provisioner. Existing VMs whose provisioners ran long ago are not affected until something replaces them. Be careful with that last point: if a Windows VM uses gzipped custom_data, the base64gzip change above is exactly the kind of thing that replaces it, and then its WinRM provisioner runs again and fails.

To fix it, move to SSH or remove the provisioner:

• Windows Server 2019 and later can run OpenSSH Server. Set type = "ssh" and target_platform = "windows" in the connection block. If the SSH default shell is PowerShell, the connection docs also tell you to set script_path to a .ps1 path.
• Better, if you can: bake the configuration into the image, or run it from custom_data or user_data, so that no provisioner has to connect at all.

The grep above finds literal "winrm" strings. It does not find type = var.connection_type, so also search for connection blocks whose type comes from a variable.

32-bit builds: last series

The 1.13 changelog says this is "the final release series that will include official builds for 32-bit CPU architectures" (*_386 and *_arm). The Pi's 64-bit kernel can run 32-bit ARM binaries, so we ran the linux_arm build:

$ uname -m
aarch64
$ tofu-1.13.1-linux-arm version
OpenTofu v1.13.1
on linux_arm
$ tofu-1.13.1-linux-arm init

Warning: Support for 32-bit CPU architectures is ending soon

OpenTofu v1.13 is the last release series that will include official release
packages for 32-bit CPU architectures.

We recommend planning to migrate to a 64-bit CPU architecture instead.
Alternatively, you could build OpenTofu for linux_arm from source code
yourself, ...

Look at the second line of tofu version on each runner. If it says on linux_arm or on linux_386, that runner has to move. Common cases are 32-bit Raspberry Pi OS runners, old ARMv7 boards, and i386 container images. As the test shows, a 64-bit kernel does not help if the image or the binary you install is 32-bit. You have time: per the changelogs, the 1.13 series is supported until August 1, 2027, and 1.12 until February 1, 2027. The 1.11 series lost support on August 1, 2026.

The new functions: hints for unknown values

This is the main feature of the release. During a plan, any value that an API assigns at creation time is unknown, and OpenTofu cannot use an unknown value to decide how many instances to create. Here is the classic case: a network module creates a VPC, and a second module creates flow logs only when it gets a VPC ID.

# modules/network/main.tf
# terraform_data stands in for aws_vpc: its output is unknown until apply,
# the same way a VPC id is decided by the AWS API at creation time.
resource "terraform_data" "vpc" {
  input = "vpc-0a1b2c3d4e5f60718"
}

output "vpc_id" {
  value = terraform_data.vpc.output
}

# modules/flow_logs/main.tf
variable "vpc_id" {
  type    = string
  default = null
}

resource "terraform_data" "flow_log" {
  count = var.vpc_id != null ? 1 : 0
  input = var.vpc_id
}

On a first plan, 1.12.7 and 1.13.1 fail the same way:

Error: Invalid count argument

  on modules/flow_logs/main.tf line 7, in resource "terraform_data" "flow_log":
   7:   count = var.vpc_id != null ? 1 : 0

The "count" value depends on resource attributes that cannot be determined
until apply, so OpenTofu cannot predict how many instances will be created.
...

The assume... functions let the module author state what is true about the value even before it exists. Our first attempt failed:

Error: Invalid function argument

  on modules/network/main.tf line 8, in output "vpc_id":
   8:   value = assumenotnull(terraform_data.vpc.output)

Invalid value for "value" parameter: given value must have a known type;
consider using the \"convert\" function to specify a type to assume.

terraform_data.output takes on the type of input, so its type is also unknown during the plan. Most provider attributes, such as aws_vpc.id, are typed strings and do not have this problem. For a value like this, the docs tell you to combine the hint with the new convert function:

output "vpc_id" {
  value = assumenotnull(convert(terraform_data.vpc.output, string))
}

$ tofu-1.13.1 plan | grep -E 'will be created|Plan:'
  # module.flow_logs.terraform_data.flow_log[0] will be created
  # module.network.terraform_data.vpc will be created
Plan: 2 to add, 0 to change, 0 to destroy.

$ tofu-1.13.1 apply -auto-approve | tail -1
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

The usual workaround for this error is a two-step apply with -exclude (the error message itself suggests it). Here, one plan is enough.

A hint is a promise, and OpenTofu checks it. The docs say that if the value turns out to be null, "the function raises an error", so the apply fails. Use a hint only where the provider really guarantees it, for example that an ID is never null after create.

Catch wiring mistakes at plan time

assumestringprefix is useful together with variable validation. Here the caller connects the subnet output to an input that expects a VPC ID, a mistake that is easy to miss in review:

# main.tf
module "flow_logs" {
  source = "./modules/flow_logs"
  vpc_id = module.network.subnet_id # wrong output wired in
}

# modules/flow_logs/main.tf
variable "vpc_id" {
  type = string

  validation {
    condition     = startswith(var.vpc_id, "vpc-")
    error_message = "vpc_id must be a VPC id (vpc-...)."
  }
}

Without hints, the plan passes because the value is unknown, and validation waits for apply. The apply created the VPC and the subnet, and then failed. Both outputs are filtered to the key lines, and the IDs are shortened:

$ tofu-1.13.1 plan
Plan: 3 to add, 0 to change, 0 to destroy.
$ tofu-1.13.1 apply -auto-approve
module.network.terraform_data.subnet: Creation complete after 0s [id=...]
module.network.terraform_data.vpc: Creation complete after 0s [id=...]
Error: Invalid value for variable
vpc_id must be a VPC id (vpc-...).
$ tofu-1.13.1 state list
module.network.terraform_data.subnet
module.network.terraform_data.vpc

Then we added prefix hints to the network module outputs:

output "vpc_id" {
  value = assumenotnull(assumestringprefix(convert(terraform_data.vpc.output, string), "vpc-"))
}

output "subnet_id" {
  value = assumenotnull(assumestringprefix(convert(terraform_data.subnet.output, string), "subnet-"))
}

Now the same mistake fails the plan, before anything is created:

$ tofu-1.13.1 plan
Error: Invalid value for variable

  on main.tf line 7, in module "flow_logs":
   7:   vpc_id = module.network.subnet_id # wrong output wired in
    ├────────────────
    │ var.vpc_id is a string

vpc_id must be a VPC id (vpc-...).

This was checked by the validation rule at modules/flow_logs/main.tf:4,3-13.

If you maintain shared modules, the outputs of your network, IAM and DNS modules are the best places for these hints. Callers get the benefit without changing their own code. assumeequal goes further: the docs show it with the AWS provider's arn_build function to make an IAM role ARN fully known at plan time, so that policy checks can see the real policy document.

Guard the module version correctly

A module that uses these functions does not work on older versions, and the errors are not clear. On 1.12.7, assumenotnull(...) alone gives Call to unknown function. With convert(..., string), the error is Invalid reference, because 1.12 reads string as a resource address. Terraform 1.16.5 also gives Call to unknown function.

You would usually add required_version = ">= 1.13.0" to a terraform block. That does not work here. Since 1.12, OpenTofu ignores required_version in .tf files and honors it only in .tofu files (see the RFC tracking issue and the settings docs). We tested this with a constraint that no version can meet:

So put the guard in a .tofu file:

# versions.tofu (the language block needs OpenTofu 1.12 or later)
language {
  compatible_with {
    opentofu = ">= 1.13"
  }
}

On 1.12.7 this gives This module is not compatible with OpenTofu v1.12.7, which is the clear message you want. If a module must also work with Terraform, use the .tofu precedence rule: when outputs.tf and outputs.tofu are both present, OpenTofu loads only the .tofu file. We put a hinted output in outputs.tofu and a plain one in outputs.tf. tofu validate (1.13.1) and terraform validate (1.16.5) both passed.

The lint experiment

validate, plan, apply and refresh accept a new -lint flag. 1.13 has four rules, all for the root module only: core:no-type-variable, core:unused-variable, core:unused-local, and core:count-instead-enabled. The last one suggests the enabled lifecycle argument instead of count = cond ? 1 : 0. We ran the experiment on a small file that breaks all four rules:

$ tofu-1.13.1 validate -lint=all -json | jq -r '.diagnostics[] | select(.severity == "warning") | .summary'
Experimental linting enabled
Input variable not used (core:unused-variable)
Local value not used (core:unused-local)
Could use enabled instead of count (core:count-instead-enabled)
Variable with no type (core:no-type-variable)

The exit code was 0. Lint results are warnings, and OpenTofu always adds an "Experimental linting enabled" warning. You can turn off one rule with !, for example -lint='all,!core:unused-variable'. The linting docs say that rules can change even in minor releases. For now, make it a non-blocking CI step and read the output. Do not gate merges on it yet.

Smaller changes worth a line

• Plan text. After No changes. Your infrastructure matches the configuration., 1.12.7 printed one more paragraph ("OpenTofu has compared your real infrastructure ... no changes are needed."). 1.13.1 does not. If a script greps for that paragraph, change it to use tofu plan -detailed-exitcode, which returns 0 for no changes, 1 for errors, and 2 for changes. Our upgrade plan returned 2.
• Platforms. Windows on ARM64 is now an official platform, and macOS builds require macOS 13 Ventura or later.
• State encryption. The aws_kms key provider accepts an encryption_context. gcp_kms accepts additional_authenticated_data, and openbao accepts associated_data.
• Crash recovery. A Go panic now writes a partial errored.tfstate to help you recover.
• Saved plans include the provider schemas, so tofu show on a plan file usually does not need to start providers.
• If you stay on 1.12 for now, take 1.12.7. It fixes a deadlock that an attacker-controlled SSH server could cause in remote-exec and file provisioners (CVE-2026-78662).

Upgrade checklist

1. After tofu init, search your code and .terraform/modules for base64gzip( and "winrm", and look for connection blocks whose type comes from a variable.
2. Remove WinRM provisioners before you upgrade. Validate and plan will not stop you, and a failure at apply taints the resource.
3. Run a 1.13.1 plan against real state in a throwaway job, save it, and run the jq filter. Every changed attribute should be user data.
4. For each user data change, decide: accept the stop/start or replacement in a window, or use a temporary ignore_changes with a ticket to remove it.
5. Check tofu version on every runner and workstation image, and move anything on linux_arm or linux_386 to 64-bit before August 1, 2027.
6. Change scripts that read plan text to use -detailed-exitcode.
7. In shared modules, add assume... hints to outputs whose IDs callers use in count, for_each or validation. Put a language block in a .tofu file to guard them.
8. Add -lint=all to CI as a non-blocking step and see what it finds.

The upgrade is not hard, but two of these changes are not visible until a specific step: the base64gzip diff shows only in a plan against real state, and the WinRM removal shows only at apply. If you run steps 1 to 3 first, you see both before they affect a real server.

Shopify's engineering blog describes the rehearsal in unusual detail: load tests at 150% of last year's peak, capacity planned with the cloud providers months ahead, a throttle at the edge that queues buyers when checkout is full, and inventory reserved during payment so two buyers cannot claim the last unit. This post walks through each from Shopify's own write-ups, then rebuilds the checkout half on a Raspberry Pi with k6 to measure what the last two steps buy.

The short version: checking stock before payment oversold 41 to 47 units of 500 in three runs, and reserving the unit first sold exactly 500. When buyers arrived twice as fast as payment could take them, a plain queue charged more shoppers after they had given up than it confirmed purchases for. A simple waiting room kept checkout under 400 ms at p95 instead, at the cost of turning the overflow into shoppers who gave up waiting.

TL;DR

• Shopify rehearses all year: its load generator, Genghis, runs checkout flows in production with flash-sale bursts on top, at 150% of last year's load, and capacity is planned months ahead.
• When checkout is full, an edge throttle queues buyers. The first version was a lottery that left some waiting 40 minutes; a signed first-attempt timestamp made it fair.
• Inventory is reserved when payment starts and claimed when it succeeds.
• In our runs the oversell from check-then-pay was roughly arrival rate times payment time: lower traffic shrank it but did not remove the race. At twice the payment capacity, the question was not how many orders got through but who got them.
• Our first batch of runs quietly skipped up to a fifth of its shoppers. Fail arrival-rate load tests on dropped_iterations.

Prerequisites

• Comfort with HTTP, SQL transactions and the idea of a load test
• To run the demo: Node.js 22.13 or newer (for the built-in node:sqlite) and the k6 binary; no Docker, no database server
• About 25 minutes of machine time for the full set of recorded runs

A flash sale is an overload you can schedule

Bart de Water, who worked on Shopify Payments, defined the term in his QCon talk on Shopify's flash-sale architecture: "A flash sale is a sale for a limited amount of time, often with limited stock. It's over in a flash because the product can sell out in seconds, even if there are thousands of items in inventory." He also draws the line that matters here: "Storefront is mostly about read traffic, while our checkout does most of the writing and has to interact with external systems as well."

Reads cache well. Writes that call a payment provider do not, and that is where Shopify got hurt. The first of two posts on its checkout throttle describes Kylie Cosmetics running sales that sold out quickly, roughly every week, and one in February 2016 that "took down not just her store, but all others on the database shard where her store was allocated." What followed was a set of habits Shopify still writes about, and de Water's talk has the one-line reason they keep paying off: "today's flash sale will be tomorrow's base load."

Those habits answer three questions. Will the platform hold at the peak? What happens to buyers who arrive when checkout is full? Does the one write that matters stay correct under contention?

Load testing as a discipline, not a launch task

Shopify's 2025 BFCM readiness post opens with "Bimonthly fire drills all year, simulating 150% of last year's BFCM load." The tool is in-house:

Our load testing tool Genghis runs scripted workflows that mimic user behavior like browsing, cart adds, and checkout flows. We gradually ramp traffic to find breaking points. Tests run on production infrastructure simultaneously from three GCP regions (us-central, us-east, and europe-west4) to simulate global traffic patterns. We inject flash sale bursts on top of baseline load to test peak capacity.

Five major scale tests ran from April to October 2025. The fourth reached 146 million requests per minute and more than 80,000 checkouts per minute; the last went to the p99 forecast of 200 million requests per minute. Early tests found that "core operations threw errors and checkout queues backed up", and adding authenticated checkout "exposed rate-limit paths that anonymous browsing never touches."

Three details worth copying at any scale:

• Production-shaped targets. De Water's talk describes Genghis hitting benchmark stores in production, at least one per pod, "at least weekly", paying through a benchmark gateway that "can respond with both successful and failed payments with a realistic distribution of response time latencies that we see in production." The demo copies that idea.
• The burst, not just the volume. A 2023 post lists a flash-sale flow in which "simulated users purchase a single product", plus an "abort switch" that stops every test at once.
• Written-down failure. Toxiproxy, Shopify's open source tool for simulating network conditions, injects failures during load tests, and findings go into a Resiliency Matrix of failure scenarios, recovery objectives and runbooks.

Pick the load model before the tool

What decides whether a load test can see a flash sale at all is the workload model. k6's page on open and closed models states the trap: "When the target system is stressed and starts to respond more slowly, a closed model load test will wait, resulting in increased iteration durations and a tapering off of the arrival rate of new VU iterations." A closed test slows down exactly when the system does and reports a calm result. Buyers at a drop do not wait for the previous buyer's page to load, so you need an open model, where arrivals follow a schedule whatever the server is doing.

All three common open source tools can drive an open model and fail a CI job; k6 and Gatling also have explicit closed-model options:

On merit: k6 is a single Go binary whose thresholds work on custom metrics, which made "fail if any unit was sold twice" a one-line rule in the demo. Gatling separates open and closed injection in its API, so the model is explicit in every script, and suits JVM teams. Artillery's YAML scenarios are the quickest to write for plain HTTP, and its Playwright engine drives real browsers when the checkout page's JavaScript matters. Any of them could run the demo.

Pre-scale for the peak you know is coming

Shopify's 2025 preparation started in March with capacity planning, including "submitting our estimates to our cloud providers so they don't run out of cloud." In the uncertain year of 2020, Capacity Planning at Scale records the choice: "We decided to scale to our more aggressive growth scenarios to ensure our platform is stable regardless of what happens." Before scale tests, components are brought up to a "BFCM profile" ahead of time.

Change is managed the same way. A 2018 post describes a feature freeze that "starts several weeks before BFCM" and a code freeze a few days before. The 2025 post puts it as a rule: "We don't use BFCM as a release deadline." Architectural changes and migrations land months earlier.

Why not let autoscaling handle it? Shopify's posts do not say they turn it off, and we found no primary source that does. But in the demo below the stock is gone about six seconds into the sale, while a Kubernetes Horizontal Pod Autoscaler checks metrics every 15 seconds by default, before a new pod is even scheduled. Autoscaling suits the slow tides of a long weekend; for the first minute of a drop, the capacity has to exist already. Our Kubernetes HPA lab and horizontal vs vertical scaling simulator let you feel that lag safely.

Some capacity cannot be pre-scaled at all because it belongs to someone else, like a payment provider. The second experiment is about what to do when arrivals exceed it.

Put the queue at the edge, not inside checkout

After the Kylie outage, Shopify had a week before the next sale. Part I explains why a plain rate limit would not do: "For customers anything that looked like the website crashing would be interpreted as such." The team built a throttle into its Nginx and OpenResty edge tier:

The solution we landed on was to throttle users using a leaky bucket algorithm built into our edge tier. [...] The number of requests served would be reset every period (in our case, 5 seconds), and it was up to us to inform the client when to retry a rejected request.

Buyers over the limit saw a queue page that polled /checkout; those who got through received a signed cookie to skip the throttle for the session. The platform held, and then buyers complained of waiting up to 40 minutes for a 40-minute sale. The post admits that "in reality they were randomly polling the throttle", a lottery.

Part II fixed fairness without adding state to the edge. A buyer's first checkout attempt is stamped with a timestamp in a signed cookie, and each load balancer computes a threshold, "the virtual version of the 'Now Serving: 42' counters at delis", that lets the earliest timestamps through to the leaky bucket. A feedback controller moves the threshold; after simulations, the team kept only its proportional term.

1. Buyer clicks checkout
2. Edge throttle Nginx + Lua
3. Checkout writes + payment

Outcomes:

• under capacity: signed cookie, straight through
• over capacity: queue page, poll, earliest timestamp first

Five years later, de Water's 10 tips for resilient payment systems (2022) still describes "scriptable load balancers to throttle the amount of checkouts happening at any given time", with a waiting queue when demand exceeds capacity. It explains why with Little's Law and a sentence worth pinning above any capacity plan: "your application can't out scale the world." At the edge, a waiting buyer costs a cached page and a poll, not a worker, a database connection and a payment slot.

Protect the one write that must not go wrong

The 2025 post calls checkout, payment processing, order creation and fulfillment "critical journeys". Older parts of the architecture exist to keep them alive:

• Pods. A pod "consists of a set of shops that live on a fully isolated set of datastores", which limits how far one shop's sale can reach. De Water adds that some extra-large merchants get a pod to themselves, and other systems stop a big merchant's flash sale from trying to "monopolize all the capacity" of a shared pod.
• Semian. Shopify's circuit breaker and bulkhead library for Ruby starts from the fact that slow resources fail slowly, and while threads wait on one, "the slow resource has caused a cascading failure by occupying workers and therefore losing capacity."
• Inventory reservations, the part the demo rebuilds.

Shopify's May 2026 post, We replaced Redis with MySQL for inventory reservations, says oversell protection works "by reserving inventory during payment processing", which it calls "a short hold that prevents two concurrent checkouts from claiming the same unit." Reserve holds items when payment starts; claim deducts them from the ledger when payment succeeds. The reservation used to be a Redis DECR, but then the claim step had to update the MySQL ledger and clean up Redis, two operations that could not be wrapped in one atomic step. In MySQL, the obvious schema failed: "A single row with a quantity column couldn't handle the contention." The shipped design uses one row per sellable unit, taken with SELECT ... FOR UPDATE SKIP LOCKED so concurrent checkouts take different rows, from a pool capped at 1,000 rows per item and location that a replenishment process refills. The post also ties this article's two halves together: "Slow reservations trigger throttling and a worse buyer experience." Remember the single-row detail; the demo's fix is the single-row version.

Rebuilding the checkout half on a Raspberry Pi

The demo is a small checkout service and one k6 scenario, built to answer two questions a skeptical reader can check: does reserving before payment matter at modest traffic, and what does a queue in front of checkout change when arrivals exceed what payment can process?

The-DevOps-Daily/flash-sale-playbook on GitHub

Everything ran on a 4-core Raspberry Pi 4 that was also doing other work, with k6 2.3.0 and the service on the same machine, so absolute numbers are small. Every variant ran the same scenario on the same machine, interleaved with the others, and each run records the load average before and after it.

• The service is one Node.js 24 process, one product, and an SQLite database in memory-backed storage. One process stands in for a fleet: the await between the read and the write lets other requests run in between, as requests on separate servers would.
• The payment provider is a stand-in that takes 100 to 400 ms (uniform, seeded per run) and declines 5%. In the second experiment it also has a fixed number of slots.
• The scenario is a k6 ramping-arrival-rate executor with every VU created before the sale. Each iteration is one shopper. After the sale, teardown() asks the server what it sold and reports it as metrics, so thresholds fail the run on correctness, not just latency.

// k6/flash-sale.js, trimmed
export const options = {
  scenarios: {
    flash_sale: {
      executor: 'ramping-arrival-rate',
      startRate: 0,
      timeUnit: '1s',
      preAllocatedVUs: profile.vus,
      maxVUs: profile.vus,
      stages: profile.stages, // drop: 0 -> 300/s over 10s, hold 20s, down over 5s
    },
  },
  thresholds: {
    dropped_iterations: ['count==0'], // k6 kept its own schedule
    oversold_units: ['count==0'], // reported by teardown() from the server
    orders_to_clients_that_left: ['count==0'],
    'http_req_duration{name:checkout}': ['p(95)<2000'],
  },
};

Before the recorded runs, scripts/predict.mjs simulated every variant 400 times with an idealized copy of k6's arrival schedule (it has one extra arrival at the very end) and the same payment model, ignoring server CPU time. Its output from the start of the first batch, kept with the discarded runs, is identical to the one used here. Every recorded order count below fell inside the simulated 5th to 95th percentile range; a few latency figures landed just outside it, which is unsurprising for a model with no CPU time.

Experiment 1: check, pay, then write

The naive checkout is the order most people write first (simplified from server.mjs):

async function checkoutNaive(shopper) {
  const { stock } = q.readStock.get(SKU); // SELECT stock ...
  if (stock <= 0) return { status: 409 }; // sold out
  const payment = await authorizePayment(); // 100-400 ms; other requests run here
  if (payment === 'declined') return { status: 402 };
  q.decrement.run(SKU); // UPDATE inventory SET stock = stock - 1
  recordOrder(shopper);
  return { status: 200 };
}

The fix moves a conditional decrement in front of payment, so the database decides who gets each unit:

async function checkoutReserve(shopper) {
  // UPDATE inventory SET stock = stock - 1 WHERE sku = ? AND stock > 0
  const { changes } = q.reserve.run(SKU);
  if (changes === 0) return { status: 409 }; // sold out, no payment call
  const payment = await authorizePayment();
  if (payment !== 'paid') {
    q.release.run(SKU); // give the unit back
    return { status: 402 };
  }
  recordOrder(shopper);
  return { status: 200 };
}

The drop profile ramps from zero to 300 new shoppers a second over 10 seconds, holds for 20 and ramps down over 5: 8,249 shoppers for 500 units. Each mode ran three times, interleaved to spread background load across both:

Every run started all its scheduled shoppers and used the same code, stamped by hash in its run.json. The p95 covers every checkout request, including the quick 409 sold-out answers most shoppers got.

Units sold beyond the 500 in stock

Recorded k6 runs on a Raspberry Pi 4. Tick marks show the mean of 400 simulated runs from scripts/predict.mjs. Reserve sold exactly 500 every time.

flash-sale-playbook

# two of the recorded runs, as scripts/run-all.sh started them
$ scripts/run.sh drop-naive-1 drop MODE=naive SEED=1
drop-naive-1: k6 exit 99, results in results/drop-naive-1
$ scripts/run.sh drop-reserve-1 drop MODE=reserve SEED=1
drop-reserve-1: k6 exit 0, results in results/drop-reserve-1
# what the server says it sold
$ jq -c '{mode: .config.mode, stockInitial, orders, oversold, stockNow}' results/drop-naive-1/server-stats.json
{"mode":"naive","stockInitial":500,"orders":544,"oversold":44,"stockNow":-44}
$ jq -c '{mode: .config.mode, stockInitial, orders, oversold, stockNow}' results/drop-reserve-1/server-stats.json
{"mode":"reserve","stockInitial":500,"orders":500,"oversold":0,"stockNow":0}
# why the naive run exited 99
$ grep -A14 THRESHOLDS results/drop-naive-1/k6-output.txt
  █ THRESHOLDS 

    dropped_iterations
    ✓ 'count==0' count=0

    http_req_duration{name:checkout}
    ✓ 'p(95)<2000' p(95)=189.78ms

    orders_to_clients_that_left
    ✓ 'count==0' count=0

    oversold_units
    ✗ 'count==0' count=44

Every naive drop run sold units that did not exist, 8 to 9% more than the stock, and left the counter negative, so nothing downstream would have stopped those orders. Every reservation run sold exactly 500. Latency could not tell them apart (most shoppers arrived after the sellout and got a quick 409), so a load test that only checked p95 and errors would have passed both.

The size of the oversell is predictable. Every shopper who passes the stock check before the counter reaches zero gets an order, and the counter only moves when a payment completes. So at the moment it hits zero, about (arrival rate × average payment time × share of payments approved) shoppers are still in flight. The 500th naive order landed about 6.2 seconds after the sale opened, with arrivals near 185 a second: 185 × 0.25 s × 0.95 ≈ 44. The simulation predicted a mean of 42.5, with 5th to 95th percentiles of 37 and 48; the runs landed at 44, 41 and 47.

Lower traffic shrinks it but does not remove the race: at a tenth of the arrival rate, the prediction was 4 to 9 and the run oversold 8 (1.6% of the stock). A slower payment step makes it larger. The bug is about how many payments are in flight when the last unit goes, not about Black Friday traffic.

Experiment 2: more buyers than payment can take

The second scenario is the BFCM shape: plenty of stock, arrivals beyond what checkout can process. Payment has 8 slots and averages 250 ms, so it completes about 32 checkouts a second (Little's Law: 8 / 0.25 s). The peak profile ramps to 64 shoppers a second, twice that, holds for 30 seconds and ramps down over 10: 2,559 shoppers. Each will spend at most 15 seconds trying to buy, in one slow request or several tries.

Three variants, one change each, on the reservation code:

• Queue: no protection. Requests wait for a payment slot as long as it takes.
• Deadline: the baseline a hostile reader would ask for. k6 sends the time the shopper gives up, and the server skips the charge (504) when less than the stand-in's maximum payment time, 400 ms, is left.
• Waiting room: at most 8 checkouts in progress; everyone else gets 503 with Retry-After: 1 and retries after a jittered half to one and a half seconds, giving up when the next wait would take them past 15 seconds. This is Shopify's random-polling first version, the lottery, not the timestamp fix.

The first five rows add up to all 2,559 shoppers in every run. The charges in the sixth row overlap with the timeouts: in the queue variant, most shoppers who saw a timeout were charged anyway.

2,559 shoppers, payment capacity about 32 checkouts a second

Mean of three recorded k6 runs per variant on a Raspberry Pi 4. Tick marks show the mean of 400 simulated runs from scripts/predict.mjs.

Through the 45 seconds of overload, every variant completed 29 to 31 orders a second, by the server's own timestamps. Payment was the limit and none of the three changed it. What changed was who got those orders, and how long everyone waited.

The queue turned the overload into charges nobody saw. The line for payment grew past 1,100 requests; once the wait passed 15 seconds, nearly every request was served after its shopper had given up. Of about 2,440 orders per run, roughly 1,390 went to shoppers whose browser had shown a timeout, more than the 1,050 or so who saw a confirmation. In a real shop, each is a card charged for a purchase the buyer thinks failed.

The deadline removed those charges and sold the most, about 1,810. The cost is time: the queue still settles near the patience limit, so the median buyer waited 12.5 to 13.1 seconds, with up to 949 requests open at once. In a thread-per-request server each would also hold a worker, the failure Semian exists to prevent.

The waiting room kept checkout fast: 8 in progress at most, under 400 ms at p95, and a median time to purchase of 4.3 to 5.0 seconds including the waiting. Between 826 and 847 shoppers per run gave up after close to 15 seconds of "please wait", and the server answered about 22,000 cheap 503s, around 8.6 per shopper. That is the trade Shopify made by moving the wait to a cached page at the edge.

It confirmed about 10% fewer purchases than the deadline queue, and the timestamps show where most of that gap built up. Both sold about 30 a second through the overload. After arrivals stopped, 50 seconds into the sale, the deadline queue kept serving its newest arrivals, who still had patience left, until about 62 seconds; the waiting room ran dry at about 58. Server-recorded orders after the 50-second mark account for about 154 of the roughly 175-order gap. The cause is this particular polling policy rather than waiting rooms in general: admission is random, so shoppers who had waited longest gave up first; a shopper gives up as soon as the next random wait would cross the 15-second limit, up to 1.5 seconds early; and a freed slot sits idle until someone's retry arrives. It is Shopify's Part I lesson in miniature: random retries spend the patience of the people who came first. The 10 to 14 late charges are shoppers admitted with less patience left than their payment took. Adding the deadline check to the waiting room should remove most of them; we did not run that combination.

The bug in our own load test

Our first full batch of runs, kept under results/discarded/, started with 50 pre-allocated VUs and let k6 create more on demand. In the peak runs every waiting shopper holds a VU; k6 could not create them fast enough and skipped the shoppers it could not start on time: 541, 542 and 440 of 2,559. The runs completed, the summaries looked plausible, and the queue variant looked far better than it should, because it had faced a lighter sale.

k6 reports this as dropped_iterations, and our summary script flagged it. The fix was to create every VU up front and fail the run on dropped_iterations: ['count==0']. Later, a burst of unrelated work on the Pi pushed the load average near 9 on four cores. One peak run in that window skipped 253 shoppers, and another started 2,557 of 2,559 without reporting any as dropped, which only surfaced when the summary script compared runs. Both were re-run once the machine was quiet and kept with a note. The rule for throwing a run away was incomplete delivery of the schedule, not load: peak-queue-3 started under a load average of 8.9, delivered all 2,559 shoppers, landed inside its predicted ranges and was kept. Before you believe anything an arrival-rate test says, check that it delivered its schedule.

What this does not show

• Scale. A few hundred requests a second on one small machine, with k6 and the server sharing four cores. The direction of each result should hold; the absolute numbers do not transfer.
• Hot-row contention. SQLite serializes writes, so the single-row conditional UPDATE never fought for a row lock the way it would in MySQL or Postgres. Shopify says that design "couldn't handle the contention" at its scale. The demo shows that reserving before payment is necessary, not that one row is enough.
• The cost of a held request. Node.js holds a waiting connection cheaply. In a thread-per-request server, the queue and deadline variants would look worse.
• A fair waiting room. Ours is the lottery Shopify replaced. A timestamp-ordered queue should change who gets served and narrow the spread of waits; we did not build one.
• Real payment latency. The stand-in and the predictor use a uniform 100 to 400 ms with no long tail. Real providers have tails, which make every queue worse.
• Rare cases. Three runs per variant (one for low) agree with each other, and their order counts sit inside the simulated ranges, but three runs cannot rule out an occasional bad one.

The playbook, condensed

1. Rehearse the shape, not just the volume. Model the drop as a burst on baseline traffic with an open-model load test, and run it on a schedule.
2. Make the load test check correctness. After the run, ask the system what it sold. Fail on oversells, on charges to buyers who had left and on dropped arrivals, not only on p95.
3. Have the capacity before the minute you need it. Forecast, agree capacity with providers, scale up ahead of the peak and stop risky changes well before it.
4. Bound the work that reaches checkout. Admit what payment and inventory can finish and park everyone else somewhere cheap, ideally at the edge and in arrival order.
5. Reserve before you charge, and release on failure. The database decides who gets the last unit, not the application's memory of an earlier read.

Shopify's own posts are worth reading in full, starting with the 2025 readiness program, the checkout throttle story and the inventory reservations post. For the neighbouring problems, our post on how Stripe avoids double-charging covers retries around the payment call, and the rate limit simulator shows how leaky and token buckets behave under bursts.

It is the second disclosure of this kind in five weeks. On September 1, Manifold Security published GitSpawn: eight findings across seven coding agents (Claude Code, Codex, Cursor, goose, Qwen Code, Grok Build and Hermes Agent) where repository-controlled git settings ran commands outside the agent's sandbox. Most of the detailed cases used a core.fsmonitor line in the repo's own .git/config; in Claude Code's case, the start-up git status ran it while the workspace-trust prompt was still waiting for an answer.

Both reports are about agents, but the mechanism is plain git, and git also runs in your CI jobs, bots and editor plugins. So we tested git itself: which everyday commands run programs that a repo's .git/config or hooks point at, whether the usual hardening flags stop them, and which ways of moving a repo between machines carry that config along.

TLDR

• A bare git status ran repo-supplied programs: core.fsmonitor, a post-index-change hook from .git/hooks or core.hooksPath, and on git 2.55 a hook defined in config.
• git diff --no-ext-diff --no-textconv still ran the clean filter (and a filter.<name>.process helper). Overrides only stopped a filter when we named its driver.
• On git 2.55, -c core.hooksPath=/dev/null did not stop a hook defined in config (hook.<name>.command, new in git 2.54). Adding -c hook.<event>.enabled=false did, in our fixture.
• git clone and cloning from a bundle never brought the config or hooks. A tar of the working copy, which is what a workspace cache or artifact is, always did.
• git's ownership check (safe.directory) refused a copy owned by another user on our Raspberry Pi. On GitHub-hosted runners it never fired, because the images set safe.directory = *.
• The scripts, the recorded results and a read-only audit are in the repo below.

Prerequisites

• git and bash, to run the scripts
• A machine where you can create throwaway repos: every "payload" in the tests only appends a line to a log file

Why a repo can make git run programs at all

Some git settings are commands, not values. core.fsmonitor names a helper that tells git which files changed, so big repos do not stat every file. A filter driver (filter.<name>.clean, smudge or the long-running process) rewrites file content on the way into and out of the index; that is how Git LFS works. diff.external and diff.<name>.textconv replace or preprocess diffs. Hooks are executables in .git/hooks or in the directory core.hooksPath points to, and since git 2.54 a hook can also be a command defined in config (hook.<name>.command plus hook.<name>.event, see git-hook).

Git reads all of these from the repository's own .git/config and hooks directory. That is documented, intended behaviour for a repo you created. The trouble starts when .git came from someone else.

A normal clone protects you: git does not transfer .git/config or hooks over a fetch. .gitattributes does travel, because it is a tracked file, but a filter named there does nothing unless config defines it. So the question for a DevOps team is not "can a repo do this", it is "where do we copy a .git directory instead of cloning it". GitLab's report lists the answers: "an archive, a synced folder, a CI cache, or a devcontainer build", plus a compromised agent that writes the config into a repo you cloned normally.

The test

The repo builds small throwaway repositories. Each has one committed file and the same file with a line appended, so there is always a change to show, and a .git/config (or hooks directory) that points one setting at a script whose only job is to log that it ran and pass content through:

The-DevOps-Daily/git-config-exec-check on GitHub

We tested eleven settings: core.fsmonitor; clean, smudge and process filters; a textconv driver; diff.external; hooks in .git/hooks; hooks via core.hooksPath; a hook defined in config; core.sshCommand; and core.pager. For each one, scripts/matrix.sh copies the repo fresh, runs 15 commands that tools and people run all the time, and records the exit code and what ran.

We ran it on a Raspberry Pi with git 2.39.5, and on GitHub-hosted ubuntu-latest and macos-latest runners with git 2.55.0. Ubuntu and macOS were identical. Git 2.39.5 ran the same programs except for the config-hook column, which it does not support. This is the git 2.55.0 table:

How to read it:

• A blank cell means "not observed with this fixture", not "can never run". Because our modified file is longer than the committed one, git can see the change from the file size without filtering it. With a same-size edit or a touched file, git status may need to compare content, and then filters can run too.
• hooks dir was identical for .git/hooks and core.hooksPath, so they share a column. core.pager never ran, because none of the commands had a terminal; agents and CI jobs usually do not either.
• Our process helper does not speak git's filter protocol. Git started it either way; git 2.39.5 then exited with 128, and git 2.55.0 carried on and exited 0.
• Only the sshCommand fixture has a remote, and its fetch fails on purpose. A successful fetch can fire more hooks than this row shows.

Three rows deserve a second look.

git status is not read-only. It refreshes the index, and refreshing the index is when git asks the fsmonitor helper what changed. When the refresh writes the index back, git fires the post-index-change hook, from the hooks directory and from config. All of that happened on a repo with one modified file.

git fetch ran fsmonitor before it contacted the remote. In the fsmonitor fixture there is no remote at all. scripts/trace-fetch.sh shows git starting the helper twice (asking for protocol version 2, then falling back to version 1 after our helper exited non-zero) and only then failing to find origin:

git 2.39.5 on Linux aarch64
run-command.c:655       trace: run_command: cd <tmp>/r; '<tmp>/hook.sh fsmonitor' 2 1791269076974045040
run-command.c:655       trace: run_command: cd <tmp>/r; '<tmp>/hook.sh fsmonitor' 1 1791269076974045040
run-command.c:655       trace: run_command: unset GIT_PREFIX; GIT_PROTOCOL=version=2 'git-upload-pack '\''origin'\'''
fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

The diff flags that hardened tools add are aimed at the wrong half. --no-ext-diff and --no-textconv stop the programs that render a diff. The clean filter runs earlier, when git turns the working-tree file into a blob to compare. That is the GitLab finding, and plain git 2.55 behaves the same way. Diffs between two commits (git log -p, git show) never ran a filter, because no working-tree file is involved.

Do the usual flags help?

scripts/overrides.sh builds one repo with fsmonitor, a hook in .git/hooks, a hook defined in config, diff.external, a textconv driver and a clean filter. Each row starts from a fresh copy, runs git status and git diff, and adds more protection. On git 2.55.0:

git 2.55.0 on Linux x86_64
plain                                                    exit 0/0  ran: diff-external,filter-clean,fsmonitor,hook:config,hook:dir
diff --no-ext-diff --no-textconv                         exit 0/0  ran: filter-clean,fsmonitor,hook:config,hook:dir
+ -c core.fsmonitor=false -c core.hooksPath=/dev/null    exit 0/0  ran: filter-clean,hook:config
+ -c filter.lab.clean= (needs the driver name)           exit 0/0  ran: hook:config
+ -c hook.post-index-change.enabled=false (per event)    exit 0/0  ran:

Read it row by row:

1. With no protection, five programs ran: fsmonitor, the hooks-directory hook, the config hook, the clean filter and diff.external. The textconv driver did not, because diff.external replaces the whole diff; the matrix above is where textconv shows up, and where --no-textconv stops it.
2. The diff flags stopped diff.external, and nothing else.
3. core.fsmonitor=false and core.hooksPath=/dev/null are what a careful wrapper adds. They stopped fsmonitor and the hooks-directory hook, but not the hook defined in config, which is not looked up in a hooks directory. The clean filter also still ran.
4. Clearing the clean filter worked only because we knew the driver was called lab. An attacker picks the name, and .gitattributes can name a different driver for each file pattern. The same goes for smudge and process, which a wrapper has to clear too.
5. hook.<event>.enabled=false, which the git 2.55 git-hook documentation describes as a switch for every hook of one event, stopped the config hook in this fixture without knowing its name. We only tested it together with core.hooksPath=/dev/null, so keep both. It is also per event, so a wrapper has to list every event its commands can fire.

On git 2.39.5 the same script gave the same first four rows, minus the config hook, which that version does not know about.

Which copies bring the config along

scripts/delivery.sh builds one repo with core.fsmonitor, a clean filter and a post-index-change hook. All three call a helper stored inside .git by relative path, so the payload travels with any copy that includes .git. It copies the repo four ways, records anything that ran during the copy, and then runs git status and git diff in each copy. On the Pi:

git 2.39.5 on Linux aarch64
safe.directory already set on this machine: no
original repo                          copy ran: -        then ran: filter-clean,fsmonitor,hook:post-index-change  exit 0/0
git clone                              copy ran: nothing  then ran: nothing                                        exit 0/0
git clone from a bundle                copy ran: nothing  then ran: nothing                                        exit 0/0
tar of the working copy                copy ran: nothing  then ran: filter-clean,fsmonitor,hook:post-index-change  exit 0/0
same tar, owned by another user        copy ran: -        then ran: nothing                                        exit 128/129 fatal: detected dubious ownership in repository at '/tmp/tmp.S42grY0m9P/other-owner'
  ...with no system or global config   copy ran: -        then ran: nothing                                        exit 128/129 fatal: detected dubious ownership in repository at '/tmp/tmp.S42grY0m9P/other-owner'
  ...with -c safe.directory=*          copy ran: -        then ran: filter-clean,fsmonitor,hook:post-index-change  exit 0/0

The clone and the bundle were clean, both during the copy and afterwards. The tar was not, and a tar of the working copy is what most "cache the workspace" setups produce: an actions/cache or GitLab cache: entry that includes .git, a build artifact someone zipped from the job directory, a devcontainer volume, a folder synced between machines. Once that copy lands, the next tool to run git status in it runs the repo's programs.

Here is that as a terminal session, from scripts/demo-restored-workspace.sh: a workspace restored from a tarball, two git commands, and our audit script at the end. The helper sits inside .git, so it arrived with the tarball, and the script empties ran.log before each git command.

restored workspace, git 2.39.5

# a workspace restored from a cache tarball, not cloned
$ git status --short
 M notes.txt
$ cat ../ran.log
ran: fsmonitor
ran: fsmonitor
$ git diff --no-ext-diff --no-textconv --stat
 notes.txt | 1 +
 1 file changed, 1 insertion(+)
$ cat ../ran.log
ran: fsmonitor
ran: fsmonitor
ran: filter-clean
ran: filter-clean
$ audit.sh .; echo "exit $?"
Repo config in . can run programs:
file:.git/config  core.fsmonitor    .git/lab-hook.sh fsmonitor
file:.git/config  filter.lab.clean  .git/lab-hook.sh filter-clean
exit 1

The ownership check, and the runners that turn it off

Since git 2.35.2 (the fix for CVE-2022-24765), git refuses to work in a repository owned by another user unless that path is allowed by safe.directory. On the Pi, that check stopped the poisoned tar as soon as another user owned it.

On GitHub-hosted runners, the same step ran everything. The script prints where safe.directory comes from, and on the Ubuntu runner (image ubuntu24 20260927.320.1) it said:

git 2.55.0 on Linux x86_64
safe.directory already set on this machine: system file:/etc/gitconfig *
original repo                          copy ran: -        then ran: filter-clean,fsmonitor,hook:post-index-change  exit 0/0
git clone                              copy ran: nothing  then ran: nothing                                        exit 0/0
git clone from a bundle                copy ran: nothing  then ran: nothing                                        exit 0/0
tar of the working copy                copy ran: nothing  then ran: filter-clean,fsmonitor,hook:post-index-change  exit 0/0
same tar, owned by another user        copy ran: -        then ran: filter-clean,fsmonitor,hook:post-index-change  exit 0/0
  ...with no system or global config   copy ran: -        then ran: nothing                                        exit 128/129 fatal: detected dubious ownership in repository at '/tmp/tmp.Y1kIjtRPwM/other-owner'
  ...with -c safe.directory=*          copy ran: -        then ran: filter-clean,fsmonitor,hook:post-index-change  exit 0/0

The Ubuntu image writes directory = * into /etc/gitconfig, and the macOS image (macos26 20260907.0351.1 in our run) adds it to the global config; you can see both in the Ubuntu and macOS image scripts. The Ubuntu script's comment says why: git 2.35.2 "introduces security fix that breaks action\checkout". With system and global config switched off, git 2.55 refused the copy exactly like git 2.39 on the Pi, so the behaviour is git's and the wildcard is the image's.

Two caveats keep this in proportion. The ownership check only helps when the files belong to a different user; a cache that the job restores as itself passes the check with or without a wildcard. And a refusal stops git, it does not make the repository safe. Still, on those runners the ownership check is not part of your defence, so it comes down to whether you restore .git at all.

To see what your own runners and build containers do, print every entry with its scope and file. Git only honours safe.directory from system, global and command-line config, so a repo cannot allow itself. A * in any of those means the check is off for all paths, unless a later empty entry resets the list, and removing a global * does nothing if the system config still has one:

git --no-pager config --show-scope --show-origin --get-all safe.directory

What to do

Do not restore .git from a cache or artifact

Cache what is expensive to rebuild, not the repository. .git rarely needs caching: actions/checkout fetches a single commit by default, and a fetch never carries config or hooks. If you need full history for speed, cache a bundle (git bundle create) and clone from it; in our test, a bundle clone ran nothing.

This closes one route, not cache poisoning in general. A restored node_modules or provider directory can also contain code that runs, and GitHub's own cache security guidance says to treat restored caches as untrusted input. Keep caches separated by trust level, so a pull request job cannot write what the main branch job restores.

Treat artifacts the same way. If a later job downloads an artifact that contains a .git directory and runs any git command inside it, that job trusts whoever produced the artifact.

Audit before the first git command

When you cannot avoid a copied .git (a synced folder, a devcontainer volume, a repo an agent has had write access to), read its config before you run git in it. git --no-pager config --list reads config files without running any of the programs above; keep --no-pager, because a configured pager can run when the output goes to a terminal.

Our scripts/audit.sh wraps that. It lists repo-level keys whose value is a command, a hooks directory or an include (including hook.<name>.command, process filters, shell aliases and pager.<cmd>), plus executable hooks in the repo's hooks directory, and exits 0 (none), 1 (found) or 2 (it could not inspect the repo, for example because git refused it). scripts/audit-selftest.sh checks it against every fixture plus odd layouts, such as a driver name containing =, a symlinked hook, a linked worktree and an include outside .git, and confirms that auditing ran nothing. It is a detector for the settings it knows, not a safety certificate.

Run a copy you trust, kept outside the directory you are checking; a poisoned workspace can replace any script inside it. In CI, that means fetching the audit at a pinned commit before you restore anything:

# before restoring the cache: fetch the audit from a pinned commit, outside the workspace
curl -fsSL -o "$RUNNER_TEMP/audit.sh" \
  "https://raw.githubusercontent.com/The-DevOps-Daily/git-config-exec-check/<commit-sha>/scripts/audit.sh" || exit 1
# after restoring it
bash "$RUNNER_TEMP/audit.sh" "$GITHUB_WORKSPACE" || { echo "restored repo can run programs"; exit 1; }

Expect some findings on developer machines. git lfs install --local writes a filter driver into the repo config, and many teams use hooks on purpose. That is fine: the point is to see them before git runs them, in a place where you did not put them yourself.

If you write tools that shell out to git

GitLab's advice is to override every relevant key on every call, or avoid git's filter and textconv machinery. From our results, for the commands a tool runs to read state (status, diff, blame, log), that means:

• Pass -c core.fsmonitor=false -c core.hooksPath=/dev/null on every call, including git status.
• On git 2.55, also pass -c hook.<event>.enabled=false for each event your commands can fire. Our table saw pre-commit, post-commit, post-checkout, post-index-change and reference-transaction.
• Add --no-ext-diff --no-textconv to diffs, and --no-pager to anything that might reach a terminal.
• For filters, read the config first and clear clean, smudge and process for every driver it defines, or use commit-to-commit diffs when they are enough.
• For fetch and push, the remote URL and core.sshCommand both come from the copied config. Set the transport yourself (for example -c core.sshCommand=ssh and an explicit URL) or do not let the tool talk to remotes from that copy.
• When the repo came from outside, run git as a user that does not own it, with no safe.directory wildcard in system or global config.

These cover what we tested. They are not a complete list of every setting git can turn into a command, which is why the audit and the "do not copy .git" advice come first.

If you use one of the agents named in the two reports, update it. The DeepSeek-Reasonix fix is Studio 2.21.0 and npm 1.39.3. Manifold's post has a table of fixed versions; four of its eight findings were still unpatched when it was published.

What we did not test

• Windows, credential helpers and editors. Each of those needs a different trigger.
• A fetch that succeeds, and a process filter that speaks the protocol. Both can run more than our rows show.
• Any specific agent. These scripts test git. How a given agent calls git decides which rows of the table apply to it, and the two reports above are the place to look for that.
• Every command and every state. We picked 15 common commands and one kind of change. git rev-parse and git log --oneline ran nothing here; that is not a promise about other commands or other repo states, so run matrix.sh with the commands your tools use.

If you have read our posts on pre-commit hook security or the MCP design flaw, this is the same lesson from another side. The dangerous input is not always code you run on purpose. Sometimes it is the configuration of the tool you run, read from the directory you are standing in.

We wanted to know how far off the obvious guesses are, so we measured. Three features on one key, two models on DigitalOcean Serverless Inference, 151 recorded calls, and every number below comes from the usage block the provider returned. The feature we expected to be expensive was not the problem. The one-word classifier was.

TLDR

• On Kimi K2.6, an incident answer the engineer read was about 100 tokens long. The request billed about 2,900 tokens, and 928 of them were reasoning tokens that nobody reads.
• A one-word alert label used a median of 566.5 reasoning tokens on Kimi K2.6 and 184.5 on DeepSeek V4.1 Flash, with single calls as high as 1,590 and 1,783. On mean costs, one label cost between 32 and 48 percent of a full incident investigation.
• Turning reasoning off for the label (reasoning_effort: "none" on Kimi K2.6) cut the cost per 1,000 labels 41 times at list price. It also changed 6 of the 20 labels.
• The agent loop was not the multiplier we expected. In 19 of 20 runs the model asked for all four tools in one step, so an investigation took two calls, not the ten we had feared.
• The same prompt used anywhere from 551 to 1,652 reasoning tokens across ten runs. Per-request cost is a distribution, not a number.
• The fix is one wrapper: every call carries the feature and tenant that caused it, and writes the provider's usage in OpenTelemetry GenAI attribute names.

Prerequisites

• Node.js 20 or later
• An API key for an OpenAI-compatible endpoint that returns a usage block (we used DigitalOcean Serverless Inference)
• A rough idea of per-token pricing: you pay one rate for input tokens and a higher rate for output tokens

The scripts, the fixtures, every raw response's usage and the report are in the companion repo:

The-DevOps-Daily/ai-feature-cost-ledger on GitHub

The invoice has the wrong shape

A cloud bill has the same problem, and the cloud answer is cost allocation tags. Model spend has no tags. The provider sees an API key and a model name, so that is what it bills by. Everything you would want to group by, such as the feature, the customer and the environment, exists only in your code at the moment you make the call.

There is a second problem, and it is the one that surprised us. Even per request, what a user sees has little to do with what you pay for. A request bills four things:

• Input tokens: the system prompt, the conversation so far, tool definitions and tool results, every time
• Cached input tokens: the part of the input the provider served from its prompt cache, at a lower rate
• Output tokens: the answer
• Reasoning tokens: thinking the model does before it answers, billed as output and usually never shown

The first and last are the ones people underestimate. So we built three small features that share one key and measured each.

The three features

All three run on the same key, which is the point: the provider sees one stream of calls.

1. Incident assistant. An engineer asks why checkout-api is returning 502s. The assistant has four tools: list pods, read logs, list recent deploys and read metrics. The tools return fixed text describing pods that are OOM-killed after a deploy raised a cache size, so every run sees the same incident. We ran it two ways: as a tool-calling agent, and single-shot with all four tool outputs pasted into one prompt. The system prompt asks for at most three short sentences.
2. Ticket summary. Two sentences for an account manager from a nine-message support thread.
3. Alert classifier. One word, page, ticket or ignore, for 20 different alert lines.

Each incident condition ran 10 times on each of two models, DeepSeek V4.1 Flash and Kimi K2.6. The classifier ran all 20 alerts under four settings. Both models report reasoning tokens and cached tokens separately in usage, which is why we picked them. All 40 incident answers identified the intended root cause (a keyword check for the memory exhaustion and the cache change behind it), so the cost comparisons below are between answers that found the right cause.

Prices are DigitalOcean's published Standard rates per million tokens, read from the pricing page on 5 October 2026:

What one answer actually billed

Here is the single-shot incident answer on Kimi K2.6, straight from the report file:

ai-feature-cost-ledger

# one incident answer on Kimi K2.6, median of 10 runs
$ jq '.incident["kimi-k2.6 single"] | {medianInput, medianOutput, medianReasoning, medianVisibleAnswer, medianBilledPerVisible}' data/report.json
{
  "medianInput": 1893,
  "medianOutput": 1023.5,
  "medianReasoning": 927.5,
  "medianVisibleAnswer": 102,
  "medianBilledPerVisible": 30.6
}

The engineer read a 102-token answer. The request billed a median of 1,893 input tokens and 1,024 output tokens, and 928 of the output tokens were reasoning. For every token the engineer read, the request billed about 30 (the median of the ten per-run ratios). On DeepSeek V4.1 Flash the same question billed about 17 tokens per token read, because it reasoned less: a median of 126 reasoning tokens against Kimi's 928.

One incident answer: what was billed vs what was read

Kimi K2.6, single-shot, median of 10 runs. Each bar is its own median, so the bars need not add up. Source: data/report.json in the companion repo.

Warning

Reasoning tokens are in completion_tokens, and you pay the output rate for them. If you estimate cost by counting the tokens in the answer text, you miss most of the output bill on a reasoning model. Read usage.completion_tokens_details.reasoning_tokens where the provider reports it.

The agent loop was not the problem

We expected the agent to be the expensive version. An agent resends the whole conversation on every step: system prompt, tool definitions, every earlier tool result. Ten sequential steps means paying for the first prompt ten times.

That is not what happened. In 19 of the 20 agent runs, the model asked for all four tools in its first step, received the results, and answered in the second. The one exception, a Kimi run, asked for three tools, then the fourth, then answered: three calls. With two calls, the resend costs one extra copy of a short prompt.

Those are list prices with no cache discount, which is the fair comparison here (the cache section explains why). On Kimi the agent was no dearer at all, because it reasoned less once the tool results were in front of it.

This does not mean agent loops are cheap. It means the step count decides, and the step count belongs to the model and the task, not to you. A model that calls one tool per step, or a task where each tool result decides the next call, turns two calls into eight. Measure steps per request as its own number, and alert on it.

The one-word answer that cost the most

The classifier is the feature nobody worries about. The prompt is roughly 50 to 90 tokens and the answer is one word. At list price, it cost $0.48 per 1,000 labels on DeepSeek V4.1 Flash with default settings, and $2.45 on Kimi K2.6.

Almost all of that is reasoning. The answer was 2 to 4 tokens every time. The reasoning was not:

Reasoning tokens spent on a one-word alert label

20 alerts per setting, one call each. The visible answer was 2 to 4 tokens every time. Source: data/feature-runs.jsonl.

The tail is where the money goes. A DNS failure rate of 14 percent for three minutes drew 1,783 reasoning tokens from DeepSeek before it said ticket. A node memory-pressure alert drew 1,218. On DeepSeek at default settings, 14 of the 20 alerts used between 50 and 250. Nothing in the request tells you in advance which call will be the expensive one.

Put next to the incident assistant, this is the number that changes priorities. Comparing mean costs, one label cost 32 to 48 percent of a full agent investigation: 32 percent on DeepSeek at list price, 48 percent on Kimi with the cache discounts both features got. A feature that runs on every alert is billed at a third to a half of a feature that runs when an engineer is paged.

Turning reasoning down, and what it changes

Both models accept a reasoning_effort parameter on this endpoint, but not the same values, and the values do not mean the same thing. DeepSeek V4.1 Flash rejected none and minimal (reasoning_effort must be one of [low high xhigh max] for this model) and accepted low. Kimi K2.6 accepted all three, but on the probe alert low and minimal used more reasoning than the default (1,210 and 1,021 tokens against 656), and only none turned it off. That is one call each, recorded in data/effort-probe.json, so read it as "check what the setting does", not as a rule.

ai-feature-cost-ledger

# the alert classifier on Kimi K2.6, default vs reasoning_effort none
$ jq '.classifier["kimi default"] | {runs, medianOutput, medianReasoning, maxOutput, medianVisible, costPer1000}' data/report.json
{
  "runs": 20,
  "medianOutput": 569.5,
  "medianReasoning": 566.5,
  "maxOutput": 1593,
  "medianVisible": 3,
  "costPer1000": 2.4195
}
$ jq '.classifier["kimi effort=none"] | {runs, medianOutput, medianReasoning, costPer1000, agreesWithDefault}' data/report.json
{
  "runs": 20,
  "medianOutput": 2,
  "medianReasoning": 0,
  "costPer1000": 0.0381,
  "agreesWithDefault": 14
}

Cost per 1,000 alert labels

Mean cost of 20 real calls per setting, scaled to 1,000, at DigitalOcean list prices on 5 Oct 2026 with no cache discount (costPer1000NoCache). Source: data/report.json.

With reasoning off, Kimi produced 2 output tokens per label. At list price, the cost per 1,000 labels fell from $2.45 to $0.06, 41 times less. As billed, with the cache discounts some of those calls got, it fell from $2.42 to $0.04, 63 times less. DeepSeek at low saved 1.8 times either way, and its tail did not go away: one call still used 1,138 reasoning tokens.

The catch is in the last field. Without reasoning, Kimi gave a different label on 6 of the 20 alerts. Four moved up to page: the disk at 91 percent, a node under memory pressure, a CI queue delay and a single OOM-killed pod. The other two moved from ignore to ticket. DeepSeek at low changed 2 labels. Each alert ran once per setting, so these are observed differences, not a measured rate, and we have no ground truth for the alerts, so we cannot say which setting was right. We can say that "turn reasoning off" is a product decision about who gets woken up, not a free saving. Run your own labelled set through both settings before you flip it.

Caching is a discount you cannot schedule

Prompt caching is the other big lever. DeepSeek V4.1 Flash bills cached input at $0.006 per million tokens against $0.30 uncached, 50 times less. On the nine single-shot runs that hit the cache, the median cost was $0.000342, against $0.000925 for the same tokens at list price.

Two things stop you from budgeting on that number:

• The first call always paid full price. The first call of every incident condition had zero cached tokens, and so did the first call of the second Kimi agent run, even though the run before it had just sent the same prefix.
• What repeats in our test does not repeat in production. We sent the identical incident ten times, so later runs found almost the entire single-shot prompt in cache (1,984 of 2,019 tokens on DeepSeek). A real incident brings new logs and new metrics. Only the shared prefix (system prompt and tool definitions, 576 tokens on DeepSeek here) can repeat across different incidents, which is why the agent comparison above uses list prices.

Record cache_read tokens per call and watch the hit rate as its own metric. Put the stable part of every prompt first and the per-request part last, so the prefix the cache can match is as long as possible. Then budget at list price and treat the cache as a discount, not a plan.

Tag every call with the feature that caused it

None of the numbers above helps finance unless each call says which feature it belongs to. The fix is boring, and that is the point. Every model call goes through one function. The function knows the feature and the tenant, reads the usage block from the response, prices it, and writes one row:

// src/ledger.mjs, trimmed
import { appendFileSync } from 'node:fs';
import { chat } from './client.mjs';
import { costUsd, splitUsage } from './cost.mjs';

export async function meteredChat({ feature, tenant, ledger }, body) {
  const res = await chat(body);
  const s = splitUsage(res.data.usage);
  const row = {
    ts: new Date().toISOString(),
    'app.feature': feature, // who caused this call
    'app.tenant': tenant, // who you would bill or rate limit
    'gen_ai.operation.name': 'chat',
    'gen_ai.provider.name': 'digitalocean',
    'gen_ai.request.model': body.model,
    'gen_ai.response.model': res.data.model,
    'gen_ai.usage.input_tokens': s.input, // includes cached tokens
    'gen_ai.usage.cache_read.input_tokens': s.cached,
    'gen_ai.usage.output_tokens': s.output, // includes reasoning tokens
    'gen_ai.usage.reasoning.output_tokens': s.reasoning,
    'app.cost_usd': costUsd(body.model, res.data.usage),
  };
  appendFileSync(ledger, JSON.stringify(row) + '\n');
  return res;
}

And the pricing function it calls, which is where most homemade cost trackers go wrong:

// src/cost.mjs, trimmed
export function splitUsage(u = {}) {
  const input = u.prompt_tokens ?? 0;
  const cached = u.prompt_tokens_details?.cached_tokens ?? u.cache_read_input_tokens ?? 0;
  const output = u.completion_tokens ?? 0;
  const reasoning = u.completion_tokens_details?.reasoning_tokens ?? 0;
  return { input, cached, uncached: input - cached, output, reasoning };
}

export function costUsd(model, usage) {
  const p = pricing.models[model]; // USD per million tokens, with source and date
  const s = splitUsage(usage);
  const cachedRate = p.cachedInput ?? p.input;
  return (s.uncached * p.input + s.cached * cachedRate + s.output * p.output) / 1e6;
}

Three details matter more than the rest.

• Use the provider's numbers, not your own estimate. The usage block is what you are billed for. A tokenizer on your side counts the text you sent; it does not see reasoning tokens or what the cache served.
• Split before you price. prompt_tokens already includes cached tokens and completion_tokens already includes reasoning tokens. Add them on top and you bill yourself twice for the same tokens.
• Name the fields the way OpenTelemetry does. The GenAI semantic conventions define gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens and gen_ai.usage.reasoning.output_tokens, and state that the cached and reasoning counts are included in the totals, the same rule as above. The conventions are still marked Development, so pin the version you emit. The names still mean a JSON line today can become a span attribute later without a rename.

Once every row carries app.feature, the roll-up is a GROUP BY. The 151 calls in this experiment cost $0.19 as billed and $0.22 at list price. As billed, the 80 classifier calls cost more than the 41 agent calls; at list price, the agent calls were slightly ahead:

The mix of calls here is our test plan, not a production workload, so the shares mean nothing on their own. The shape is the lesson: the feature with the shortest answers spent the most reasoning tokens.

Where a gateway fits

You can stop at the wrapper. It is about 30 lines, it lives in your code, and it adds no one who sees your prompts beyond the inference provider. Keeping it consistent gets harder when the calls come from many services in several languages, and it reports spend without enforcing a budget. That is the problem the AI gateway and LLM observability vendors sell into, and they solve the tagging part in similar ways:

• Portkey takes an x-portkey-metadata header with any keys you choose, such as feature and _user, and has budget limits in front of the provider.
• Helicone uses one header per property, Helicone-Property-<Name>, and lets you segment requests and cost by them.
• Langfuse records usage and cost per generation. It infers cost from model definitions it ships for OpenAI, Anthropic and Google models, so for a model like DeepSeek V4.1 Flash on DigitalOcean you add your own definition with the prices, or send usage and cost yourself. Ingested values take priority over inferred ones, and you should send them.
• Braintrust logs traces with token counts, cost and metadata alongside its evaluations, which fits the question this post leaves open: whether a cheaper reasoning setting still labels alerts correctly.

The trade-offs are the usual ones. A proxy gateway adds a network hop to every model call, and a hosted one is another company that sees your prompts. Helicone, Langfuse and Portkey's gateway all publish source you can run yourself, which answers the second point and makes the first your problem. Whichever you pick, check that it records the provider's reported usage, including reasoning and cached tokens, rather than estimating from the text. Counting only what the user saw would have put the incident assistant's token bill far too low in this test: the median billed-to-read ratio per condition ran from 17 to 40, and single investigations from 14 to 50.

What we could not conclude

• Whether failed attempts are billed. The 151 recorded calls all succeeded on the first attempt, so the client's retries for 429 and 5xx never fired in the record. A request that fails or times out on the client side may still reach the provider and be billed, and nothing we recorded can show whether it was.
• Which classifier labels were right. We have no ground truth for the 20 alerts, so the 6 changed labels are a difference, not an error rate.
• Whether reasoning helps the incident answers. All 40 identified the intended root cause by our keyword check, which is not a full accuracy review, so we cannot price the accuracy reasoning buys on this task. A harder incident might change that.
• Anything about other models. Two models, one endpoint, one day. Others reason more or less, and the OpenAI GPT-5 models on the same pricing page were not available on our account tier, so we could not include them.
• Steady-state cache hit rates. Our repeated prompts overstate hits for anything that changes per request.

Summary

• Price every request from the provider's usage block. The answer text is not the bill.
• On reasoning models, reasoning tokens can be most of the output bill, even for a one-word answer. Track reasoning_tokens as its own metric and look at the tail, not the median.
• Steps per request decides what an agent costs. The median was 2 here: 19 investigations took two calls and one took three. Measure it and alert when it climbs.
• reasoning_effort is a large lever and a product decision. Test the labels it changes before you ship it.
• Budget at list price. Treat the prompt cache as a discount, put the stable prefix first, and watch the hit rate.
• Wrap every model call once, tag it with the feature and tenant, and use the OpenTelemetry GenAI names. Then finance gets one line per feature, and you get it before they ask.

For two other ways a model bill grows quietly, see why most LLM calls are classification and what a semantic cache really saves.

Most of them missed. The latest gitleaks release, with its default rules, found nothing. The common hand-written patterns hid only the first 40 to 46 characters, and in our sample that part never changed from one token to the next, so the "redacted" log line still held everything an attacker needs.

TLDR

• The Actions token we measured is 377 characters, not the "about 520" in GitHub's changelog. All 40 samples had the same length and shape: ghs_15368_ plus a three-part JWT signed with ES256.
• The first 46 characters were identical in all 40 tokens. They are the app ID of GitHub Actions and a base64url JWT header that never changed. A rule that redacts only that part hides nothing secret.
• Five common patterns redacted the whole token in 0 of 40 cases. Three never matched at all, so the full token would stay in the log. Two matched only the 40 or 46 public characters.
• GitHub's own first recommended regex left the hyphen out. The May 15 version, ghs_[A-Za-z0-9\._]{36,}, fully matched 8 of 40 tokens, because 32 had a - in the JWT. GitHub corrected it on May 26, and the fixed regex matched all 40.
• gitleaks 8.30.1 (latest release, March 21) reported 0 findings. trufflehog 3.97.9 found a 46-character fragment and marked it unverified. detect-secrets 1.5.0 flagged the line. Fixes for gitleaks and trufflehog have been open pull requests since July.
• Storage fails loudly, except one case. Postgres and default MySQL 8.4 reject the token in a VARCHAR(255) column. MySQL with strict mode off stores the first 255 characters with only a warning.
• Two custom rules, gitleaks and trufflehog, catch the full 377-character token today. Both are below and in the companion repo.

Prerequisites

• Code, logs, or a database that handles GitHub App installation tokens or the Actions GITHUB_TOKEN
• ripgrep (rg) to search a codebase for old patterns
• Optional: a GitHub account to fork the test repo and run it against your own rules

What changed, and when

GitHub announced the change on April 24. The staged rollout started on April 27 with the Actions GITHUB_TOKEN and first-party integrations, then moved to all App installation tokens from mid-May to late June. The October 2 post marks it as complete.

What changed:

• Format: ghs_APPID_JWT. The ghs_ prefix stays.
• Length: "~520 characters" in GitHub's words, and it "will vary based on the data stored within it."
• Contents: GitHub says the JWT "contains details about the token such as the target installation, the application, and basic validation details," and that clients must not validate it or depend on its contents.

What did not change: permissions, repository scoping, the one-hour expiration, and the REST endpoint that mints the token. GitHub Enterprise Server is not affected.

One date still matters. GitHub added a temporary X-GitHub-Stateless-S2S-Token request header in May so apps could force either format. Apps that send disabled to keep getting old tokens lose that option on November 30, 2026, when GitHub stops respecting the header.

How we tested without leaking a token

We did not have a third-party GitHub App to mint tokens from, so we used the token every Actions job already has. GitHub's April notice says the new format covers "GitHub App installation server-to-server tokens, including Actions GITHUB_TOKEN." A workflow in a private test repository read secrets.GITHUB_TOKEN and passed it to small Python scripts that print only structure: lengths, separator positions, the decoded JWT header, the claim names, and how many characters each regex leaves visible. No step printed the payload values or the signature, and we checked the downloaded logs for any token value before making the repository public.

Two workflows ran on October 5, 2026, on ubuntu-latest:

• check.yml: one token through the shape script, the regex list, gitleaks, trufflehog, detect-secrets, and Postgres and MySQL service containers.
• sample.yml: 40 matrix jobs, one token each, to see whether the results hold across tokens.

This is the shape script's output from the recorded check run:

check.yml, Token shape step

# TOKEN is the job's GITHUB_TOKEN; the script prints structure only
$ python3 scripts/shape.py
length: 377
prefix: ghs_
separators (index, char): [(3, '_'), (9, '_'), (46, '.'), (290, '.'), (321, '-'), (360, '-')]
app id segment: 15368 (public: the app's numeric id)
segment lengths header/payload/signature: 36 243 86
chars used outside [A-Za-z0-9]: ['-']
header: {"alg":"ES256","typ":"JWT"}
payload claim names: ['aud', 'ctx', 'exp', 'iat', 'iss', 'jti', 'ver']
exp - iat (s): 3600
signature bytes: 64

Put together, a current GITHUB_TOKEN looks like this:

ghs_15368_eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.<243-character payload>.<86-character signature>
|____________________________________________|
  46 characters, the same in all 40 samples:
  ghs_ + 15368 (the app ID of GitHub Actions) + _ +
  the base64url of {"alg":"ES256","typ":"JWT"}

A few things we did not expect:

• 377, not 520. All 40 Actions tokens were exactly 377 characters. GitHub's "about 520" probably describes tokens for other apps, which carry more claims. We could not mint one, so we cannot confirm that. Plan for at least 520, as GitHub asks, and do not assume a fixed length either way.
• The prefix is public. The 46-character head (ghs_15368_ plus the header segment) had one SHA-256 value across all 40 samples. It decodes to {"alg":"ES256","typ":"JWT"}, which anyone can encode.
• The JWT uses base64url, so - and _ appear inside it. 32 of the 40 tokens had a -, 27 had a _, and only 3 had neither. This detail breaks the most regexes.
• Actions masks the token in its own logs, but nobody else does. GitHub masks secret values in workflow logs and says that this redaction is not guaranteed. Your application logs, proxies, error trackers, and log pipelines do not get that masking at all.

The redaction test

For each pattern, the scripts ran re.search against the token and counted the characters outside the first match. Every pattern here starts with ghs_, which appears once per token, so that is also what a re.sub replacement would leave visible. The patterns are ones you find in real redaction code: GitHub's old documented shape, the regexes from the current gitleaks and trufflehog releases, the two regexes GitHub recommended in May, and the regexes from the open scanner fixes.

Tokens fully redacted, out of 40 real GITHUB_TOKENs

Each pattern run with Python re against 40 Actions tokens minted on 2026-10-05 (sample.yml in The-DevOps-Daily/ghs-token-check). 'Fully redacted' means re.sub would replace all 377 characters.

The failures fall into three groups.

Never matches, so the full token stays in the log. The legacy shape and the gitleaks rule expect 36 letters or digits right after ghs_. The new token has five digits (15368) and then an underscore, so the match fails at character 10. The word-bounded variant allows underscores, but its 36 characters end in the middle of the header, where no word boundary exists.

Matches the public prefix only. Allow underscores and the pattern matches ghs_15368_ plus 30 header characters. Make it open-ended ({36,255}, which is what trufflehog uses) and it runs to the first . and stops at 46 characters. Either way, the payload and the full signature stay in the log line. In our sample the part that disappeared was the same in every token, so anyone who reads the redacted line can paste the prefix back. The result is a working token for as long as it lives, which for an installation token is up to an hour.

Stops at the first hyphen. GitHub's May 15 changelog recommended ghs_[A-Za-z0-9\._]{36,} "to match both new and current format tokens." A snapshot from May 19 shows that version. An editor's note dated May 26 updated it to ghs_[A-Za-z0-9\.\-_]{36,}. The first version has no -, and base64url uses -. In our sample, the match stopped somewhere in the signature on 32 of 40 tokens and left 8 to 86 characters visible. For redaction this is the least bad failure, because the hidden part always includes the payload, and in most of those 32 tokens part of the signature too. In one sample the whole signature stayed visible. For validation it is worse: code that checks a token with fullmatch against the May 15 regex rejects every token that contains a hyphen. In our sample that was 80% of real tokens.

If you copied GitHub's regex in the second half of May, check which version you have.

What the scanners found

The check workflow wrote the token into a file as GITHUB_TOKEN=<token> and pointed the latest release of each scanner at it:

gitleaks v8.30.1
gitleaks findings: 0
trufflehog 3.97.9
trufflehog findings: 1
  Github verified False raw_len 46
detect-secrets 1.5.0
detect-secrets findings: 2
  JSON Web Token
  GitHub Token

• gitleaks 8.30.1 found nothing. Its github-app-token rule is (?:ghu|ghs)_[0-9a-zA-Z]{36}, the first "never matches" case above. Someone reported the format change in issue #2192 on July 15, and PR #2193 with a fix has been open since July 16. The latest release is from March 21. If gitleaks 8.30.1 with its default rules guards your pre-commit hook or CI, its rule cannot match this token shape, and it did not flag our test file.
• trufflehog 3.97.9 found a fragment. Its detector matched the 46-character prefix and reported that fragment as an unverified finding. Most examples in trufflehog's README run with --results=verified, and that filter drops this finding. PR #5156, open since July 27, adds a pattern for the new format.
• detect-secrets 1.5.0 flagged the line twice, once as a JSON Web Token and once as a GitHub Token. Its GitHub pattern also matches only a prefix, but a scanner only needs to flag the line, and it did.

That last point explains why the same partial match is fine in one tool and bad in another. A scanner that matches 46 characters still points a person at the right line. A redactor that matches 46 characters prints the rest of the token.

Storing the token

GitHub asks for columns that "accept at least 520 characters." We inserted the 377-character token into narrow columns to see how each database reacts:

postgres t40:
ERROR:  value too long for type character varying(40)
postgres t255:
ERROR:  value too long for type character varying(255)
mysql 8.4 default sql_mode: ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
mysql strict t255:
ERROR 1406 (22001) at line 1: Data too long for column 'tok' at row 1
mysql non-strict t255:
stored_len
255

Postgres and MySQL 8.4 with its default strict mode fail on insert, which is the good outcome: the error is in your logs the first time a new token arrives. MySQL with sql_mode cleared, which some older applications set on purpose, stores the first 255 characters and keeps going. The insert succeeds with a warning that is easy to miss, and the failure shows up later as a 401 from the GitHub API.

Installation tokens last an hour, so most apps cache them rather than store them for long. Look anyway: caches with fixed-size keys or values, cookie-backed sessions, and old migrations from when the token was always 40 characters.

What to change

1. Find the old patterns. Search code and config for anything that encodes the old shape:

# regexes that expect the old 36-character body, and anything mentioning ghs_
rg -n --hidden --glob '!.git' --glob '!node_modules' -e 'ghs_' -e 'gh\[[a-z]+\]_' -e '\{36\}' .

# columns that might hold a token and are too short
rg -n -i -e 'varchar\((40|64|100|128|255)\)' --glob '*.sql' --glob '*migration*' .

If your log pipeline or error tracker has its own scrubbing rules, check those too. A rule written for the old shape lives wherever someone pasted it.

2. Redact with a pattern that covers the whole JWT. GitHub's corrected regex works:

import re

# GitHub's recommendation as of May 26, 2026; matches old and new ghs_ tokens
GHS = re.compile(r"ghs_[A-Za-z0-9\.\-_]{36,}")

def redact(line: str) -> str:
    return GHS.sub("ghs_[REDACTED]", line)

It is greedy, so it also eats a trailing . or - that ends a sentence. For redaction that is the right tradeoff.

3. Do not validate the structure. GitHub says clients "must not take a dependency on the contents of this JWT." If you check tokens at all, check the ghs_ prefix and a generous maximum length, and let the GitHub API decide if the token is valid.

4. Patch your scanners until the upstream fixes ship. Both of these caught the full 377-character token in the recorded run (gitleaks with config/gitleaks.toml findings: 1, secret_len 377 and CustomRegex verified False raw_len 377):

Custom rules for the new installation token format

gitleaks

# .gitleaks.toml: keep the default rules, add one
[extend]
useDefault = true

[[rules]]
id = "github-app-installation-token-jwt"
description = "GitHub App installation token, stateless ghs_<app id>_<JWT> format"
regex = '''ghs_[0-9]+_eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+'''
keywords = ["ghs_"]

trufflehog

# trufflehog --config trufflehog.yaml
detectors:
  - name: GitHubInstallationTokenJWT
    keywords:
      - ghs_
    regex:
      token: 'ghs_[0-9]+_eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+'

A trufflehog custom detector without a verification endpoint reports its findings as unverified. A run with --results=verified drops them too, so let unverified results through for this detector.

5. Size storage for the documented length, not the one you measured. Use text in Postgres, or at least VARCHAR(1024) where you need a bound, and make sure MySQL runs in strict mode so a short column fails on insert.

6. Remove the override header. If your app sends X-GitHub-Stateless-S2S-Token: disabled to keep old-format tokens, that stops working on November 30, 2026. Fix whatever needed it before then.

Test your own rules

The workflows, scripts, and recorded results are public. Fork the repository, add your redaction or validation regex to patterns.txt, and run the workflows. The test uses the fork's own GITHUB_TOKEN, prints no token, and reports how much of a live token your rule leaves visible.

The-DevOps-Daily/ghs-token-check on GitHub

What we could not test

• Tokens from other GitHub Apps. We measured only the Actions GITHUB_TOKEN. GitHub says other installation tokens are about 520 characters, and their payloads may differ. Every regex that covers the full JWT structure should still match, but we did not run one.
• GitHub's own secret scanning and push protection. Testing them means pushing a live token to a repository, which we chose not to do, and it would say little about your own pipeline.
• Commercial log scrubbers. We tested regexes and three open source scanners, not the built-in rules in Datadog, Splunk, or Sentry. Run your own provider's rule through the test repo if you can export it as a regex.
• How long the prefix stays fixed. In our 40 samples the header segment never changed. GitHub could change the signing algorithm or add a header field at any time, so treat "the first 46 characters are public" as a property of today's tokens, not a guarantee.

Summary

GitHub warned about this change for five months and lists "logging and secret redaction rules" in its own checklist. The tools many teams use for that job have not caught up: the latest gitleaks release does not detect the new tokens, trufflehog sees only a fragment that it cannot verify, and the common hand-written patterns redact a prefix that, in our tests, was identical in every token. The fixes are small. Use a pattern that covers the whole JWT, add one custom rule to each scanner, give the column room for 520 characters or more, and test the result against a real token instead of a 40-character example in a unit test.

If you are reviewing the rest of your GitHub Actions setup, our posts on self-hosted runner version enforcement and on finding secrets before you have to rotate them cover two related changes.

📌 Handpicked by DevOps Daily - Your weekly dose of curated DevOps news and updates!

⚓ Kubernetes

📄 KubeCon + CloudNativeCon North America 2026: Build your infrastructure engineer journey

Infrastructure engineers sit at one of the busiest intersections in cloud native. Kubernetes clusters need to scale. Networks need to connect them. Storage needs to follow workloads. Platforms need to

📅 Oct 2, 2026 • 📰 CNCF Blog

🔗 Read more

📄 RHCOS 10 - Red Hat’s new worker node OS for OpenShift

ContextRed Hat OpenShift uses a purpose-built, immutable operating system for cluster nodes, commonly known as Red Hat Enterprise Linux CoreOS (RHCOS). This operating system, which uses the same kerne

📅 Oct 2, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 The Rise of Agentic AI on Kubernetes: Unleashing the New Infrastructure Layer

AI is changing expectations around infrastructure and operations, including Kubernetes management. When models run close to the data they use, it tends to push deployment, scaling and governance respo

📅 Oct 1, 2026 • 📰 SUSE Blog

🔗 Read more

📄 Building Kubernetes PR Previews with Shared Pulumi Components

My team develops a microservices application on Kubernetes, with hundreds of PRs opened each day. To let engineers test and review those changes in isolation before they’re merged, we give every pull

📅 Oct 1, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 How athenahealth modernized healthcare workloads with Amazon EKS Hybrid Nodes

Learn how athenahealth used Amazon EKS Hybrid Nodes to modernize latency-sensitive healthcare workloads on premises, cutting response times in half, reducing hardware and operational costs by 50%, and

📅 Sep 30, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Red Hat delivers peak performance on Kubernetes and CPUs in MLPerf Inference v6.1

Red Hat is proud to announce our results from the industry-standard MLPerf Inference v6.1 benchmark. This submission builds on our track record across recent rounds: In v5.1, we demonstrated cost-effe

📅 Sep 29, 2026 • 📰 OpenShift Blog

🔗 Read more

☁️ Cloud Native

📄 Amazon ECS adds Amazon VPC Lattice support for blue/green, linear, and canary deployments

Amazon Elastic Container Service (Amazon ECS) now supports built-in blue/green, linear, and canary deployment strategies for ECS services using Amazon VPC Lattice. Applications that use VPC Lattice fo

📅 Oct 2, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 KubeCon + CloudNativeCon North America 2026: Join the cloud native community at OpenTofu Day

Most of KubeCon + CloudNativeCon assumes the cluster already exists. OpenTofu Day covers everything that has to be provisioned before and around it: cloud accounts, networking, managed services, and t

📅 Oct 2, 2026 • 📰 CNCF Blog

🔗 Read more

📄 KubeCon + CloudNativeCon North America 2026: From user to contributor to maintainer

You do not need “maintainer” in your job title to start following the maintainer journey at KubeCon + CloudNativeCon North America 2026. You might be an SRE who has spent years operating a CNCF projec

📅 Oct 2, 2026 • 📰 CNCF Blog

🔗 Read more

📄 KubeCon + CloudNativeCon North America 2026: Build your application developer journey

Cloud native development increasingly means thinking beyond the application itself. How will it be built? How will it be deployed? How much infrastructure should developers need to understand? Where s

📅 Oct 1, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Trust Docker for the agents you don’t

At WeAreDevelopers, Docker introduced Cloud Sandboxes, the open Sandbox Kit specification, and a commitment to bring Kits to the CNCF for neutral governance.

📅 Oct 1, 2026 • 📰 Docker Blog

🔗 Read more

📄 Implementing feature flags in container environments with AWS AppConfig

Learn how to implement dynamic feature flags in Amazon ECS and Amazon EKS using AWS AppConfig with the sidecar pattern. You set up an AWS AppConfig feature flag, deploy the AWS AppConfig Agent as a si

📅 Oct 1, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 AI-powered EKS migration assessment with Amazon Bedrock AgentCore

Learn how to build an AI-powered migration assessment agent using Amazon Bedrock AgentCore and the Strands Agents SDK. The agent reads application source code and container artifacts, scores Amazon EK

📅 Sep 30, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 How to Deploy and Scale Docker Containers on Railway in 2026

Deploy a Docker container on Railway from a Dockerfile or a pre-built image, then learn how Railway builds and caches images, how to scale with replica limits and regions, what Docker workloads cost,

📅 Sep 30, 2026 • 📰 Railway Blog

🔗 Read more

📄 Protect your OpenFaaS functions with built-in OAuth - Part 1

You can now add OAuth and OpenID Connect (OIDC) login to your functions with the OpenFaaS watchdog, without adding logic to your function code. The watchdog acts as a proxy to handle sign-in and valid

📅 Sep 30, 2026 • 📰 OpenFaaS Blog

🔗 Read more

🔄 CI/CD

📄 ReviewBench: An open benchmark for AI code review

We’re launching ReviewBench, a benchmark for code review agents built on representative GitHub pull requests, multi-source ground truth, calibrated evaluation, and production-aligned metrics. The post

📅 Oct 5, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Knowledge Graphs for Software Delivery: Architecture & Guide

Discover how software delivery knowledge graphs unify fragmented SDLC data, enable schema-as-code, and power deterministic AI reasoning. | Blog

📅 Oct 5, 2026 • 📰 Harness Blog

🔗 Read more

📄 Blog: flux9s GA: Flux cluster state, from the terminal

flux9s recently reached 1.0. It is a K9s-inspired terminal UI for Flux: real-time state for every Flux resource in your cluster, with the operations you reach for most a single keystroke away. This po

📅 Oct 5, 2026 • 📰 Flux CD Blog

🔗 Read more

📄 Two front doors: Module-level access in a Django GRC app

GitLab's engineering team builds a lot of our own internal tooling, but one platform in particular forced us to rethink how we handle authorization: our internal GRC tool that serves two very differen

📅 Oct 5, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Judgment, not generation rebuilding AI API Classifier on Jev

Rebuild AI API classification with Jev to cut costs, reduce latency, improve calibration, and make production decisions more efficient without sacrificing accur | Blog

📅 Oct 5, 2026 • 📰 Harness Blog

🔗 Read more

📄 AI governance frameworks

AI governance frameworks turn policy into runtime controls for models in production. Learn the seven operational pillars and AI governance best practices.

📅 Oct 3, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 AI Model Management in Production: A Practical Workflow

Manage AI models, prompts, evaluations, rollouts, and production performance with a practical AI model management workflow.

📅 Oct 3, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Feature Engineering in Machine Learning: Concepts & Workflow

Understand feature engineering in machine learning, including data transformation, feature selection, and pipeline design for reliable model performance.

📅 Oct 3, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 AI is changing developer work. Here are three skills to strengthen.

Learn to direct AI agents, critically review their output, and keep technical judgment at the center of your workflow. The post AI is changing developer work. Here are three skills to strengthen. appe

📅 Oct 2, 2026 • 📰 GitHub Blog

🔗 Read more

📄 DeepSeek-Reasonix: How a poisoned config can hijack an AI coding agent

GitLab's Threat Research Group discovered a command execution vulnerability (GHSA-grg2-7gc6-36m6, CVE-2026-102437) in DeepSeek-Reasonix Studio, a desktop git client designed for developers pairing wit

📅 Oct 2, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Bazel Q3 2026 Community Update

Announcements BazelCon 2026 With BazelCon fast approaching, we are excited to bring you a sneak peek of what’s to come, along with a few event announcements. Join the #bazelcon channel on Bazel Slack

📅 Oct 2, 2026 • 📰 Bazel Blog

🔗 Read more

📄 10 technical talks I’m excited about at GitHub Universe 2026

From verifying AI-written code to securing npm dependencies, these are the sessions I’m building my Universe agenda around. The post 10 technical talks I’m excited about at GitHub Universe 2026 appear

📅 Oct 1, 2026 • 📰 GitHub Blog

🔗 Read more

🏗️ IaC

📄 Building zero trust networks with Red Hat Ansible

With artificial intelligence (AI) models now capable of discovering thousands of vulnerabilities in days and developing exploits in hours, organizations are moving from preventing to containing the br

📅 Oct 5, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Red Hat is named a Leader in IDC MarketScape: Worldwide Private and Hybrid Cloud Management with Automation

Red Hat has been named a Leader in the IDC MarketScape: Worldwide Private and Hybrid Cloud Management with Automation 2026 Vendor Assessment (Doc #US54644626e, June 2026).The IDC MarketScape noted, “A

📅 Oct 5, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Amazon Aurora DSQL now supports partial indexes

Amazon Aurora DSQL now lets you build an index over a specific subset of a table, storing only qualifying rows rather than every row in the entire table, which improves query performance and lowers in

📅 Oct 2, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Get Ship Done: Everything Harness Shipped in September 2026

102 product updates in 30 days: merge queues, a Homebrew CLI, secret scanning in infrastructure as code, and ChatGPT control of Harness pipelines. | Blog

📅 Oct 1, 2026 • 📰 Harness Blog

🔗 Read more

📊 Observability

📄 Restart EC2 and on-premises fleets faster with AWS CodeDeploy RESTART deployment mode

AWS CodeDeploy now offers a purpose-built RESTART deployment mode that reapplies the last successful revision to EC2 and on-premises fleets. It keeps restarts inside CodeDeploy with the same batch siz

📅 Oct 5, 2026 • 📰 AWS DevOps Blog

🔗 Read more

📄 XCOR launches to trace outages in minutes. It still pages engineers.

Palo Alto Networks introduced a new approach to AI-driven observability last week, signaling a shift from dashboards and manual incident The post XCOR launches to trace outages in minutes. It still pa

📅 Oct 5, 2026 • 📰 The New Stack

🔗 Read more

📄 AI Agent Observability: The 8 Best Tools for Production Agents

Compare the top AI agent observability tools for production — tracing, evaluation, and cost — and see which type of platform actually fits the gap you have.

📅 Oct 5, 2026 • 📰 New Relic Blog

🔗 Read more

📄 9 Incident Management Tools Behind Faster MTTR

Compare the top incident management tools for engineering and DevOps teams — coordination, escalation, and the detection layer that determines how fast they work.

📅 Oct 5, 2026 • 📰 New Relic Blog

🔗 Read more

📄 AWS Private CA now provides detailed certificate issuance logs

AWS Private CA announces detailed certificate issuance logs, a new AWS CloudTrail service event that records the complete certificate content, issuing CA information, requester identity, and signing s

📅 Oct 5, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Open Source Mod Brings Rate Limits, Costs and CI Status Into View for Claude Code Users

Claude Statuspane gives developers real-time visibility into Claude Code context use, rate limits, costs and CI status, highlighting a growing need for agent observability.

📅 Oct 5, 2026 • 📰 DevOps.com

🔗 Read more

📄 Machine Learning Pipeline Architecture: Components and Control Points

Learn how machine learning pipeline architecture connects data validation, feature engineering, training, deployment, guarded rollouts, and monitoring.

📅 Oct 3, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 How to Move AI SRE Agents From Demo to Production

An AI agent that works on an engineer’s laptop can feel like a breakthrough. It can read logs, query observability tools, inspect cloud resources and connect a failed deployment to a bad configuration

📅 Oct 2, 2026 • 📰 DevOps.com

🔗 Read more

📄 8 major updates to Cloudflare Observability

Cloudflare is launching eight major updates that bring logs, traces, analytics, alerts, dashboards, querying, and telemetry export into one observability platform, with simpler and more predictable pr

📅 Oct 2, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Closed-loop incident response: connect AWS DevOps Agent to OpenSearch

Connect AWS DevOps Agent to your Amazon OpenSearch Service observability data using the Model Context Protocol (MCP), so an alert that fires at 2 AM triggers autonomous root cause investigation. Cover

📅 Oct 2, 2026 • 📰 AWS DevOps Blog

🔗 Read more

📄 7 PagerDuty Alternatives for Modern Incident Response Teams

Comparing PagerDuty alternatives for on-call and incident coordination? See the top contenders, what to evaluate, and why alert quality matters as much as the tool.

📅 Oct 2, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Seer, the Sentry MCP and CLI, or your own coding agent: where each one fits

Seer, the Sentry agent plugin, the CLI, and coding agents aren't competing options. Here's what each one does that the others don't.

📅 Oct 2, 2026 • 📰 Sentry Blog

🔗 Read more

🔐 Security

📄 Threats Making WAVs - Incident Response to a Cryptomining Attack

Guardicore security researchers describe and uncover a full analysis of a cryptomining attack, which hid a cryptominer inside WAV files. The report includes the full attack vectors, from detection, in

📅 Oct 5, 2026 • 📰 Linode Blog

🔗 Read more

📄 Everything we launched during Birthday Week 2026

We celebrated our 16th birthday with 46 announcements across open source, post-quantum security, AI agents, and developer platform upgrades. Here’s a day-by-day roundup of everything we shipped.

📅 Oct 5, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 One year later: the power of 1.1.1.1 interns

A year after announcing our goal to hire 1,111 interns, more than 750 early-career builders have shipped real products across 48 teams at Cloudflare. From Birthday Week launches to post-quantum securi

📅 Oct 5, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 TeamCity 2026.2.1 and 2026.1.5 Are Out

Today we’re rolling out two bug-fix updates for the 2026.1 and 2026.2 TeamCity On-Premises versions. Similarly to the previous updates, these focus heavily on security issues, addressing over 40 vulne

📅 Oct 5, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 pg_vault_tde v1.7.2 : Critical crash fixes, new on-disk format, and stability improvements

Version 1.7.2 of pg_vault_tde is now available. This is a binary patch release where extension version stays at 1.7, but users can distinguish the build at runtime using pg_vault_tde_build_version().

📅 Oct 5, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 pgvector 0.8.7 Released

pgvector 0.8.7 is now available. This release fixes a buffer overflow with IVFFlat index builds (CVE-2026-103484), which can lead to arbitrary code execution. Users are encouraged to upgrade when poss

📅 Oct 5, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 AI is speeding up exploits. Vulnerability spreadsheets can’t keep up.

Artificial intelligence has changed almost every aspect of software development and cybersecurity. But perhaps one of the most profound changes The post AI is speeding up exploits. Vulnerability sprea

📅 Oct 3, 2026 • 📰 The New Stack

🔗 Read more

📄 Why AI Coding Agents Keep Writing Broken Access Control

AI coding agents can produce authorization logic that compiles and passes review while exposing one tenant’s data to another. Learn why broken access control is difficult to detect and how to prevent

📅 Oct 1, 2026 • 📰 Snyk Blog

🔗 Read more

📄 What Is Agentic AppSec?

Learn how Agentic AppSec uses grounded, bounded, and independently verified AI agents to run the application security loop.

📅 Sep 30, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Evo ADS Govern Agent Behavior Goes GA: Bringing MCP Usage Under Control

Evo ADS Govern Agent Behavior is now generally available, starting with MCP Governance. Discover, approve, monitor, log, and block MCP server usage across leading AI coding agents.

📅 Sep 30, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Fluentd v1.19.4 has been released

Hi users! We have released v1.19.4 on 2026-09-29. ChangeLog is here. This release is a maintenance release of v1.19 series. This release will be bundled for fluent-package LTS version v6.0.5! Security

📅 Sep 29, 2026 • 📰 Fluentd Blog

🔗 Read more

💾 Databases

📄 PLEASE_READ_ME: The Opportunistic Ransomware Devastating MySQL Servers

Guardicore Labs uncovers a Ransomware detection campaign targeting MySQL servers. Attackers use Double Extortion and publish data to pressure victims.

📅 Oct 5, 2026 • 📰 Linode Blog

🔗 Read more

📄 pg_ivm 1.16 released

IVM Development Group is pleased to announce the release of pg_ivm 1.16. Changes since the v1.15 release include: What's Changed New features Add support for PostgreSQL 19 ( Devrim Gündüz ) Bug fixes

📅 Oct 5, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 dbForge 2026.2 Adds a PostgreSQL Debugger, Visual Object Editors and Broader Context for dbForge AI Assistant

The release adds a PostgreSQL Debugger, visual editors for event triggers, functions and procedures, as well as file attachments and model choice in dbForge AI Assistant. Devart, a leading developer o

📅 Oct 5, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 Announcing Spanner queues: Transactional messaging for agentic workloads and beyond

AI agents don't just answer queries — they can autonomously issue refunds, manage inventory, execute multi-step handoffs, and orchestrate sub-agents, to name but a few complex agentic workflows. That

📅 Oct 2, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 The Database Is the Product: What Breaks When Memory Devices Scale

Editor’s note: This post originally appeared on The New Stack and is republished with permission. The original version is available here. Imagine you just finished a two-hour meeting. You were wearing

📅 Oct 1, 2026 • 📰 TiDB Blog

🔗 Read more

📄 How Jev Uses Meko to Make Code Review Feedback Stick

When a Jev-powered check gets something wrong, and a person corrects it, where does that correction go, and how can you ensure the correction is implemented in future results? This blog explains why a

📅 Oct 1, 2026 • 📰 Yugabyte Blog

🔗 Read more

📄 Vector and Full-Text Search Explained: Why AI Agents Need Both

Key Takeaways Ask an AI support agent about error E-4012 and it may come back with a confident answer about E-4021. The embeddings for the two codes sit almost on top of each other, so the agent pulle

📅 Sep 30, 2026 • 📰 TiDB Blog

🔗 Read more

📄 Redis expands strategic collaboration agreement with AWS to accelerate real-time data and AI apps

Expanded collaboration will help customers modernize apps, build faster AI experiences, and simplify adoption of Redis Cloud on AWS Redis has announced an expansion of its strategic collaboration agre

📅 Sep 29, 2026 • 📰 Redis Blog

🔗 Read more

🌐 Platforms

📄 The Oracle of Delphi Will Steal Your Credentials

Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic

📅 Oct 5, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger

In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum

📅 Oct 5, 2026 • 📰 Linode Blog

🔗 Read more

📄 Introducing Google Cloud Modernize, transforming for (and with) AI

Today, we’re announcing Google Cloud Modernize, an end-to-end transformation portfolio to help enterprises collapse multi-year roadmaps with the power of AI. Google Cloud Modernize brings together Goo

📅 Oct 5, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Your first 10 minutes on SUSE Rancher for AWS

You subscribed on AWS Marketplace. The 30-day trial clock is running. You log in expecting a dashboard, and instead the product asks you for something most SaaS never asks for: an IAM role inside your

📅 Oct 5, 2026 • 📰 SUSE Blog

🔗 Read more

📄 The energy grid and the larger story of Europe’s digital sovereignty

In June, the European Commission proposed the Cloud and AI Development Act (CADA). One passage in it captures what is at stake for Europe more clearly than most. Recital 50 warns that depending on a s

📅 Oct 5, 2026 • 📰 SUSE Blog

🔗 Read more

📄 Red Hat named a "Leader" in 2026 IDC MarketScape

We are thrilled to announce that Red Hat has been named a Leader in the 2026 IDC MarketScape for Worldwide Private and Hybrid Cloud Management with Automation.We believe this recognition from IDC Mark

📅 Oct 5, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 AI21 achieves an 83% reduction in time-to-start for AI workloads with AI Hypercomputer

Editor’s note: AI21 Labs is a leading global AI lab with a long track record of building foundation models, most notably the Jamba family, and today focuses on specialized LLMs and agent optimization

📅 Oct 2, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 AWS Health introduces the version catalog for software lifecycle management

Today, AWS Health introduces the version catalog which provides a centralized source of lifecycle information for software versions across AWS services. The version catalog helps customers move from r

📅 Oct 2, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Streamline: custom video pipelines with Cloudflare Stream and Workers

Streamline demonstrates how to build long-running, continuous video processing pipelines by pairing Cloudflare Workers and Durable Objects with a containerized media engine.

📅 Oct 2, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Base44 Unveils AI Platform to Transform Application Development

Base44 this week unfurled a cloud-based platform that leverages artificial intelligence (AI) to foster greater collaboration between software engineers, application designers and product management te

📅 Oct 2, 2026 • 📰 DevOps.com

🔗 Read more

📄 What’s new with Google Cloud

Want to know the latest from Google Cloud? Find it here in one handy location. Check back regularly for our newest updates, announcements, resources, events, learning opportunities, and more. Tip: Not

📅 Oct 2, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Integrate AWS DevOps Agent with third-party tools using Amazon EventBridge

Learn how to connect AWS DevOps Agent investigation events to third-party tools such as Jira using Amazon EventBridge and AWS Lambda. This post walks through a deployable AWS CDK solution that automat

📅 Oct 2, 2026 • 📰 AWS DevOps Blog

🔗 Read more

📰 Misc

📄 Visual Studio Code 1.141 (Insiders)

Learn what is new in Visual Studio Code 1.141 (Insiders). Read the full article

📅 Oct 7, 2026 • 📰 VS Code Blog

🔗 Read more

📄 dotInsights | October 2026

Did you know? The System.Threading.Interlocked class provides atomic operations on shared variables, so multiple threads can update them safely without a lock. Welcome to dotInsights by JetBrains! Thi

📅 Oct 5, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Ten Great DevOps Job Opportunities

DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted as part of an effort to better serve our audience. Our goal in these ch

📅 Oct 5, 2026 • 📰 DevOps.com

🔗 Read more

📄 Java Annotated Monthly – October 2026

September gave us plenty to talk about, with major releases and thought-provoking reads. Java 27 arrived, and JetBrains opened a new chapter in agentic development with the announcement of JetBrains A

📅 Oct 5, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Discontinuing Swift Language IDE Support in the Kotlin Multiplatform Plugin

Starting with IntelliJ IDEA 2026.3 and Android Studio Rabbit 2 | 2026.2.2, we’re discontinuing Swift language IDE features in the Kotlin Multiplatform (KMP) plugin. This includes editing, syntax highl

📅 Oct 5, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 MLOps for edge AI: Preparing AI models for deployment at the edge

When I started playing with AI models for edge computing use cases, everything went smoothly. But once I reached the "it works" point and started thinking about how to operationalize it, the whole thi

📅 Oct 5, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Agents have made CI the bottleneck. Faster pipelines are the wrong fix.

Three posts landed in September that I think engineering leaders should read together. Anthropic’s engineering team wrote that their continuous The post Agents have made CI the bottleneck. Faster pipe

📅 Oct 4, 2026 • 📰 The New Stack

🔗 Read more

📄 Anthropic’s answer to Dots and Muse is already inside Claude

I’m Matt Burns, Chief Content Officer at Insight Media Group. Each week, I round up the most important AI developments, The post Anthropic’s answer to Dots and Muse is already inside Claude appeared f

📅 Oct 3, 2026 • 📰 The New Stack

🔗 Read more

📄 Canonical named a Challenger in the 2026 GartnerⓇ Magic Quadrant™ for Server Virtualization Platforms

Canonical has been recognized as a Challenger in the 2026 GartnerⓇ Magic Quadrant™ for Server Virtualization Platforms. The inaugural edition of the report provides an independent evaluation of server

📅 Oct 1, 2026 • 📰 Canonical Blog

🔗 Read more

📄 Avoiding Vendor Lock-In Through an Open Source Approach: A Developer’s Perspective

Every infrastructure team makes decisions that are difficult to reverse. Most of the time, that works out. Sometimes it does not. Vendor lock-in usually begins as a reasonable choice, made under press

📅 Oct 1, 2026 • 📰 SUSE Blog

🔗 Read more

📄 Visual Studio Code 1.140

Learn what's new in Visual Studio Code 1.140 Read the full article

📅 Sep 30, 2026 • 📰 VS Code Blog

🔗 Read more

📄 Realtime transaction fraud detection – with an LLM?

Conversational AI with a chatbot is great for drafting emails or debugging code, but it’s less ideal for real-time application middleware. If you’re trying to inspect a financial transaction for poten

📅 Sep 30, 2026 • 📰 Ubuntu Blog

🔗 Read more

On October 1, 2026, Cloudflare announced new capabilities for Artifacts, its Git-compatible store that you drive from Cloudflare Workers, together with a contest to build what comes next on top of it. Artifacts is in open beta on the Workers Paid plan. Its core idea is easy to say and has big consequences: stop sharing one repository between workers, and give each unit of work its own. This post explains what Artifacts is, measures why a shared branch stops working as agents multiply, shows the repo-per-agent pattern in code, and works out what it costs.

TL;DR

• Artifacts is a Git-compatible repository store you create and control from a Worker. Any Git client can clone and push with a short-lived bearer token.
• Cloudflare's own guidance is one repo per unit of autonomous work: 10,000 agents, 10,000 repos. Repos are cheap to create and fork.
• In our local experiment, failed pushes grew roughly with the square of the number of agents sharing one branch: 40 agents produced about 700 failed push attempts.
• A repo per agent removes the collisions but moves the hard part to merging the work back. Cloudflare left that layer open and is running a contest for it.
• Pricing: 10,000 operations and 1 GB free per month, then $0.15 per 1,000 operations and $0.50 per GB-month. Each repo is capped at 1 GB.

Prerequisites

To follow the code:

• A Cloudflare account on the Workers Paid plan (Artifacts is not available on the free plan)
• Wrangler 4.145.0 or later, so wrangler types knows the Artifacts binding
• Git 2.x on the machine that clones and pushes
• Basic familiarity with Workers bindings and wrangler.toml

The push experiment below needs Git, Bash and standard Unix tools (bc, paste, sort).

What Artifacts is

An Artifacts namespace holds repositories. Each repository is a real Git remote: you get an HTTPS URL of the form https://<ACCOUNT_ID>.artifacts.cloudflare.net/git/<namespace>/<repo>.git, and any Git client can talk to it. Authentication is a bearer token passed as an extra HTTP header, so the token never lands in .git/config:

# Clone and push with a repo-scoped token; nothing is written to the remote URL
git -c http.extraHeader="Authorization: Bearer $ARTIFACTS_TOKEN" clone "$ARTIFACTS_REMOTE" work
cd work
# ...edit and commit...
git -c http.extraHeader="Authorization: Bearer $ARTIFACTS_TOKEN" push origin main

What makes it different from a hosted Git server is the control plane. A Worker gets a binding with methods to create, import, fork, list and delete repositories, mint read or write tokens with a lifetime in seconds, and read history and files without cloning:

A few more pieces make it a platform rather than storage:

• Workers Builds can deploy a Worker from an Artifacts repo. Pushes to main run the deploy command, and once you enable builds for preview branches, pushes to other branches produce preview URLs. Only main can be the production branch for now.
• Repository events (creates, forks, pushes, clones and so on) can be delivered to Workers Queues, so a push can start a test run or a review agent.
• Jurisdictions keep a namespace's data in the US or the EU.
• Metrics per repository: operations, pulls, pushes and error rates.

Under the hood, Cloudflare describes each repo as a single logical instance that it can route to from any region, with data replicated synchronously across data centers and copied to object storage in the background. It has not published how concurrent pushes to the same ref are ordered, which matters for the next section.

Why one shared branch falls apart

Git protects a branch with a compare-and-swap. A push says "move main from commit A to commit B". If someone else moved main first, the server refuses, because B was built on a history that is no longer the tip. Two agents are enough to see it:

two agents, one branch

# both agents cloned the same baseline and made one commit each
$ git push origin main   # in agent-a
To ../shared.git
   35bfd50..11799a8  main -> main
$ git push origin main   # in agent-b
To ../shared.git
 ! [rejected]        main -> main (fetch first)
error: failed to push some refs to '../shared.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

Agent B now has to fetch, rebase and try again. That is fine for two humans. To see what happens with more writers, we ran a small experiment: N agents each commit one file (no two agents touch the same file, so the work never conflicts), are launched concurrently, and push to main of one shared repository, retrying immediately with git pull --rebase until the push lands.

#!/usr/bin/env bash
# N agents each commit one file and push to the same branch of one shared repo,
# retrying with fetch + rebase until the push lands. Prints total attempts.
set -u
N=$1; W=$(mktemp -d); cd "$W"
git init -q --bare -b main shared.git
git clone -q shared.git seed 2>/dev/null
(cd seed && git -c user.email=s@x -c user.name=seed commit -q --allow-empty -m baseline && git push -q origin main)
for i in $(seq 1 "$N"); do git clone -q shared.git "a$i"; done
agent() {
  cd "a$1"
  echo "$1" > "agent-$1.txt"; git add .; git -c user.email=a@x -c user.name="agent-$1" commit -q -m "agent-$1"
  tries=0
  until git push -q origin main 2>/dev/null; do
    tries=$((tries+1))
    git -c user.email=a@x -c user.name="agent-$1" pull -q --rebase origin main 2>/dev/null
  done
  echo "$tries" > ../fails-$1
}
for i in $(seq 1 "$N"); do agent "$i" & done; wait
total=$(cat fails-* | paste -sd+ | bc); max=$(cat fails-* | sort -n | tail -1)
commits=$(git --git-dir=shared.git rev-list --count main)
echo "agents=$N failed_pushes=$total worst_agent_retries=$max commits_on_main=$commits"
rm -rf "$W"

We ran it three times for each size on a 4-core Raspberry Pi 4 with Git 2.39 and a bare repository on local disk. The counter is failed push attempts; the script does not record why each one failed.

race.sh, three runs per size

$ for n in 5 10 20 40; do for run in 1 2 3; do ./race.sh $n; done; done
agents=5 failed_pushes=10 worst_agent_retries=4 commits_on_main=6
agents=5 failed_pushes=10 worst_agent_retries=4 commits_on_main=6
agents=5 failed_pushes=10 worst_agent_retries=4 commits_on_main=6
agents=10 failed_pushes=45 worst_agent_retries=9 commits_on_main=11
agents=10 failed_pushes=45 worst_agent_retries=9 commits_on_main=11
agents=10 failed_pushes=45 worst_agent_retries=9 commits_on_main=11
agents=20 failed_pushes=190 worst_agent_retries=19 commits_on_main=21
agents=20 failed_pushes=186 worst_agent_retries=19 commits_on_main=21
agents=20 failed_pushes=190 worst_agent_retries=19 commits_on_main=21
agents=40 failed_pushes=711 worst_agent_retries=36 commits_on_main=41
agents=40 failed_pushes=716 worst_agent_retries=37 commits_on_main=41
agents=40 failed_pushes=702 worst_agent_retries=38 commits_on_main=41

commits_on_main is the baseline plus one commit per agent, so every agent's work landed in the end. The cost is in the retries.

Failed push attempts when N agents share one branch

race.sh on a Raspberry Pi 4, Git 2.39, local bare repository. No agents edited the same file, so every rebase succeeded; real conflicts make it worse.

A simple model explains the shape. If the agents move in rounds and only one push wins each round, every other pending push fails, and the total is N(N-1)/2. Our results sit close to that model. They fall below it at 40 agents, which is what you expect when an agent fetches several new commits in one rebase and skips some rounds. The model gives 499,500 failed pushes at 1,000 agents; we did not measure that scale.

Keep the limits of this test in mind. It is a conflict-free workload on local disk with immediate retries; backoff and spread-out arrival times would lower the numbers, and a hosted server adds a network round trip to every attempt. Real agents editing the same code would also hit merge conflicts that a retry loop cannot fix. And the work around Git has its own budgets: GitHub generally allows up to 80 content-generating requests per minute and 500 per hour across its API and web interface, and REST and GraphQL share a limit of 100 concurrent requests. Those limits apply to things like creating branches, pull requests and comments, not to the Git pushes counted here.

The usual answer is to give each agent its own branch in the shared repo. That removes the race on main, but the repository is still one shared object for permissions, clone size and blast radius. A token that can push to the repo can usually push to every unprotected branch in it.

The repo-per-agent pattern

Artifacts makes the next step cheap: a fork per session. A reviewed baseline repo holds the code you trust. Every agent session gets its own fork, a write token that only works on that fork and expires in minutes, and nothing else.

1. baseline reviewed main
2. agent-a-s41 fork + 15 min token
3. agent-b-s42 fork + 15 min token
4. agent-c-s43 fork + 15 min token
5. Orchestrator tests, review, merge

Connections:

• baseline -> agent-a-s41 (fork)
• baseline -> agent-b-s42 (fork)
• baseline -> agent-c-s43 (fork)
• agent-a-s41 -> Orchestrator (push event)
• agent-b-s42 -> Orchestrator (push event)
• agent-c-s43 -> Orchestrator (push event)

The Worker that hands out sessions is short. Configure the binding:

name = "agent-sessions"
main = "src/index.ts"
compatibility_date = "2026-10-02"

[[artifacts]]
binding = "ARTIFACTS"
namespace = "agents"

Then fork the baseline per session and return a scoped, short-lived token. This assumes a repo named baseline already exists in the namespace with your reviewed code on main (create it with create and push, or import it from GitHub), and that you ran wrangler types so Env includes the binding and the secret:

// src/index.ts: one fork and one 15-minute write token per agent session
export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    if (request.method !== 'POST') return new Response('POST only', { status: 405 });
    // This endpoint hands out write tokens: only your orchestrator may call it.
    // ORCHESTRATOR_SECRET is a Worker secret (wrangler secret put ORCHESTRATOR_SECRET).
    if (request.headers.get('Authorization') !== `Bearer ${env.ORCHESTRATOR_SECRET}`) {
      return new Response('Unauthorized', { status: 401 });
    }
    const { agent, session } = (await request.json()) as { agent: string; session: string };

    // Keep names readable but unique: validate the inputs and add a random suffix
    if (!/^[a-z0-9]{1,24}$/.test(agent) || !/^[a-z0-9]{1,24}$/.test(session)) {
      return new Response('agent and session must be lowercase letters and digits', {
        status: 400,
      });
    }
    const name = `${agent}-${session}-${crypto.randomUUID().slice(0, 8)}`;

    // `using` disposes the repo handle when the block ends, as the binding requires
    using baseline = await env.ARTIFACTS.get('baseline');
    const fork = await baseline.fork(name, { defaultBranchOnly: true });

    using repo = await env.ARTIFACTS.get(name);
    const token = await repo.createToken('write', 900); // 900 s = 15 minutes

    // plaintext is the Git token string; expiresAt tells the agent when to stop
    return Response.json({
      name,
      remote: fork.remote,
      token: token.plaintext,
      expiresAt: token.expiresAt,
    });
  },
};

The agent clones the remote, works, and pushes with the token. When it is done, the orchestrator does not need to clone anything to see what happened:

// Inspect a finished session without cloning it
async function summarize(env: Env, name: string) {
  using repo = await env.ARTIFACTS.get(name);
  const commits = await repo.log({ ref: 'main', limit: 20 }); // newest first
  const plan = await repo.readFile({ ref: 'main', path: 'PLAN.md' });
  return { commits, plan: plan ? await plan.text() : null };
}

Cloudflare's best-practice notes add three habits worth copying:

1. Give each token the least it needs, for as little time as possible. Read tokens for indexing and review, write tokens only for the agent doing the work.
2. Fork from a reviewed baseline instead of copying files into each new repo, so every session starts from the same known state.
3. Keep run metadata out of the tree. Attach prompts, model output and run IDs with git notes, so the commit holds the work and the notes hold the context.

And one of our own: delete forks once their work is merged or rejected. Storage is billed, and a pile of abandoned agent repos is the 2026 version of a pile of abandoned feature branches.

Getting the work back is the hard part

A repo per agent does not make conflicts go away. It moves them from push time, where they cost retries, to integration time, where something has to decide what lands. That is a better place for them, because you can be deliberate there, but someone still has to build it. Cloudflare says so directly: Artifacts is the storage primitive, and its contest asks developers to build the coordination, review and merge layer on top.

The options teams use today, from simplest to most ambitious:

• Orchestrator merges in order. One process fetches finished forks, rebases each onto the baseline, runs the tests and pushes. It is a merge queue with exactly one writer to the baseline, so the race from the experiment above cannot happen.
• Human review gate. Each fork's push event creates a review item. With Workers Builds, a branch push can also produce a preview URL to look at before anything merges.
• Agent reviewers. A second agent with a read-only token reviews the diff and either approves it into the queue or sends it back. Keep the merge itself with the orchestrator, so no reviewing agent ever holds a write token to the baseline.
• Best-of-N. Fork several sessions from the same baseline for the same task, test them all, and merge only the winner. Forks are cheap enough that this stops being wasteful.

None of these are new ideas. What changes is that the per-session repository, its credentials and its cleanup are now an API call instead of something you script around a hosted Git service.

What it costs

Artifacts bills two things. The numbers below are from the pricing page. It lists October 14, 2026 as the day billing starts; the announcement says October 15:

To size it, assume one agent session costs seven operations: a fork, a clone, three pushes, one fetch by the reviewer and a delete. That is an assumption, not a published figure; count your own workflow before you budget.

Estimated monthly operations cost by agent sessions per day

Assumes 7 billable operations per session, 30 days, and the 10,000 free operations a month. Excludes storage and other Cloudflare charges.

Storage depends on your repos and how quickly you delete forks. It is billed on the average of each day's peak, so deleting a fork stops it from adding up over the month but does not remove that day's peak. Cloudflare has not documented whether a fork shares objects with its parent or counts its full size, so measure that in the beta before you plan around hundreds of long-lived forks.

Limits and open questions

Know these before you commit:

• 1 GB per repository and 32 MB per file or blob. A repository above 1 GB, or one with a single file above 32 MB, does not fit. Account storage is 1 TB by default and can be raised.
• Rate limits of 2,000 requests per 10 seconds per namespace for the control plane, and 2,000 Git requests per 10 seconds per repository. Split busy workloads across namespaces.
• Open beta. The API and limits can still change, and it requires the Workers Paid plan.
• Production deploys from main only in the Workers Builds integration.
• Undocumented so far: how concurrent pushes to one ref are ordered, whether forks share storage, and what Artifacts guarantees about publishing events. Queues itself delivers at least once and without ordering guarantees, so make event handlers idempotent. Test the rest before you rely on it.

Note

The contest runs until October 14, 2026. Entries need a 5 to 10 minute demo video, open source code under MIT, Apache or BSD, and instructions to run it. Up to two members of each of the three winning teams are flown to Cloudflare Connect in San Francisco, and first place also gets $25,000 in Cloudflare credits. Details are on Cloudflare's announcement.

Summary

Shared branches assume few writers. Our small experiment shows what happens when that assumption fails: with immediate retries, failed pushes grew roughly with the square of the number of agents, before a single real conflict appeared. The fix is not a faster retry loop. It is isolation, and Artifacts makes isolation an API call: fork a reviewed baseline per session, hand the agent a 15-minute token that works on nothing else, and read its work back without cloning.

What Artifacts does not give you is the decision about what merges. Plan that layer first. A single-writer merge queue plus tests is enough to start, and it is the part where your team's judgment matters most. If you want to start small, move one agent workflow, such as dependency updates or test generation, to forks of a baseline and watch the error rate and the bill for a month. The Artifacts documentation covers the binding, tokens and Workers Builds setup.

The vendors' answer is reassuring: the data plane keeps working. We wanted to know what that means in practice, so we ran an open-source control server (Headscale) with real Tailscale clients, stopped it, and probed the network about once a minute for 30 minutes. Existing tunnels survived the whole outage. Joining, revoking and renewing did not, and what happens when a client restarts during the outage is the result most worth adding to your runbook.

TLDR

Prerequisites

• A Linux machine with tailscale and tailscaled installed (we used 1.102.4)
• Basic familiarity with Tailscale or another mesh VPN

Everything runs as a normal user on one host, and it does not touch an existing Tailscale install. The scripts are in the companion repo:

The-DevOps-Daily/tailnet-control-outage on GitHub

The setup

One Headscale process plays the control server. Five tailscaled processes play devices, each in userspace-networking mode (no TUN device and no root, so they can share one host). Each node serves hello from <name> on a local port, and a probe fetches that page through another node's SOCKS5 proxy. A probe only succeeds if traffic really crossed the tailnet.

Each node has one job:

1. a SOCKS5 probe
2. WireGuard direct path
3. b hello from b

• Control plane stopped during the test
  • Headscale 0.29.4 keys, peers, policy
• Devices tailscaled 1.102.4, userspace mode
  • a, b long-lived pair
  • c joins mid-outage
  • d restarted mid-outage
  • e key expires mid-outage

The outage script stops Headscale and then works through a timeline: probe every node, start c (its join attempt runs from about 25 seconds to 70 seconds), try an admin command, let e's key expire at about two minutes, and restart d at five to six minutes, probing roughly once a minute throughout. Another script then brings the control server back and times the recovery.

What kept working: the tunnels you already had

The long-lived pair never noticed. In the 30-minute run, a reached b and b reached a on every probe, 27 probes in each direction, the last one 1,793 seconds after the control server stopped. (In one of the shorter runs, a single probe from b to a failed once and the next one succeeded.) Here is the start and the end of that run, from runs/nocache-30min/02-outage.txt:

16:34:16 t=0s control server stopped (e's key expires at 2026-10-02T16:36:10Z)
16:34:16 t=0s a->b ok (hello from b) | b->a ok (hello from a) | a->d ok (hello from d) | a->e ok (hello from e)
16:34:41 t=25s a's own status line: offline
...
17:03:09 t=1723s a->b ok (hello from b) | b->a ok (hello from a) | a->d FAIL | a->e FAIL
17:04:19 t=1793s a->b ok (hello from b) | b->a ok (hello from a) | a->d FAIL | a->e FAIL

This matches what Tailscale documents: every node keeps its peers, their endpoints and the packet filter, and traffic never goes through the coordination server in the first place. Each probe is a fresh HTTP connection, so this shows that new connections between existing peers keep working; we did not hold one long-lived session open.

Note the third line. While a was moving traffic, tailscale status showed a itself as offline, because a could not reach the control server. If your monitoring alerts on that, it will tell you the mesh is down while it is working. Probe real traffic between real nodes instead.

What stopped working

New devices cannot join

Node c started with a valid, reusable pre-auth key and never got past NeedsLogin:

t=71s new node c tries to join: timeout waiting for Tailscale service to enter a Running state; check health with "tailscale status" (state NeedsLogin)

It kept retrying on its own, and got an address 14 to 22 seconds after the control server came back (three runs). Nobody had to touch it. During the outage, though, a replacement laptop, a new CI runner or an autoscaled node simply cannot get on the network.

Nobody can revoke anything

The admin interface is the control server. The headscale nodes list command we ran to find a node to remove failed with context deadline exceeded. With Tailscale's hosted service the equivalent is the admin console and API, and they are part of the same control plane.

This is the security cost of the design. Tailscale's own documentation lists it: during an outage, "existing users cannot have their keys revoked." If you need to cut off a stolen laptop or a departing employee during a control plane incident, the mesh cannot do it for you. The device keeps every peer and every rule it had when the outage started.

Expiring keys still expire

We set e's key to expire about two minutes into the outage (114 seconds in the 30-minute run). The first probe after that time failed, and every probe after it. The client does this itself: it marks peers whose key has expired and stops talking to them (ipn/ipnlocal/expiry.go in the client source).

So key expiry is enforced by the peers themselves, without the control server. That is good for security, and it means a control outage can become a data plane outage: every device whose key expires during the outage drops off, and cannot renew until control is back. After recovery, e stayed Logged out until we cleared its expiry on the server, and then it came back within 1 to 2 seconds. In real life an admin extends or clears the expiry, or the user re-authenticates.

If your tailnet uses short key expiry for servers, a long control outage will take them off the network one by one. Tailscale lets you disable or temporarily extend expiry per device, and a device that is tagged when it first authenticates has key expiry disabled by default (key expiry docs).

The restart trap

Five minutes into the outage we restarted d's tailscaled, keeping its state directory, the way an unattended upgrade, a reboot or a crashed process does.

Without the netmap cache, d came back with no address and no peers:

t=317s restarted d's tailscaled while control is down: state NoState, ip , netmap cache files: 0

Its health check said why (from d's log, in log-excerpts.txt in the repo):

You are logged out. The last login error was: fetch control key: Get "http://127.0.0.1:18080/key?v=142": dial tcp 127.0.0.1:18080: connect: connection refused

d stayed off the network for the rest of the outage. The client keeps its keys on disk, but without the cache it does not keep the network map: the list of peers, their addresses and the packet filter only live in memory. Without the control server, a restarted client knows who it is but not who anyone else is. It kept retrying, and once the control server was back it logged in again by itself, within 3 to 4 seconds according to its own log.

During a control incident, the devices that restart are exactly the ones that disappear. A device that is replaced rather than restarted, such as a new Kubernetes node or a container without persistent state, is in the same position as c: it has never been on the network and cannot join until control is back.

Netmap caching fixes it

Tailscale has a fix: netmap caching. With it, each client writes its network map to disk and loads it at startup. Tailscale's September 22 post says it is a feature flag in the current client and is expected to be on by default from version 1.104, after more testing (mobile clients later). The cache only helps a device that restarts with its state directory intact; a fresh replacement has nothing cached.

In the 1.102 client we tested, the switch is on the server side. The client only writes the cache when the control server grants it the cache-network-maps node attribute. The TS_USE_CACHED_NETMAP environment variable defaults to on and works as an off switch. Headscale 0.29 passes node attributes through from its policy file, so turning it on was one block:

{
  "acls": [{ "action": "accept", "src": ["lab@"], "dst": ["lab@:*"] }],
  "nodeAttrs": [{ "target": ["*"], "attr": ["cache-network-maps"] }]
}

d then kept nine files under its state directory (profile-data/<id>/netmap-cache/), covering itself, its peers, the user, the packet filter, the DERP map and DNS. With the cache on, the same restart came back like this:

Start: loaded netmap from disk cache; 3 peers
t=352s restarted d's tailscaled while control is down: state Running, ip 100.64.0.3, netmap cache files: 9

The first line is from d's log (log-excerpts.txt), the second from the outage script, both from the same 10-minute run.

d had its address immediately, but peers could not reach it straight away. A restarted client gets a new disco key (the key Tailscale uses for path discovery), and peers normally learn it from the control server. With the cache, the client advertises the new key to its peers directly over the tunnel instead; the logs show sending TSMP disco key advertisement and the peers receiving it. We timed how long until a could reach d again, with a retrying constantly (a failed attempt times out after 5 seconds, and the next one starts a second later):

Time until a peer reaches a restarted node, control server down

12 restarts with the cache-network-maps attribute, Tailscale 1.102.4 and Headscale 0.29.4. Measured from when the restarted daemon answered, with a new attempt about every 6 s. Without the cache, the node was still unreachable after 300 s.

Eleven of the twelve restarts were reachable again in 15 to 19 seconds. One took 140 seconds, on a clean setup, so slow recoveries do happen. The restarts run one after another, so only the first in each row waited the full outage time; within that limit, a longer outage made no consistent difference, and neither did keeping the same UDP port.

Does the recovered path last? We restarted d once more and probed every 30 seconds for 10 minutes, once with a fixed port and once with a random port. Both times the first probe, right after the restart, failed, and the next 19, over the following 10 minutes, all reached d. In our 10-minute cache outage run, though, d answered for two minutes after its restart, then failed two probes in a row (at 484 and 559 seconds) and only answered again at 630 seconds, just before control returned. We could not reproduce that, so treat a cached restart as a big improvement, not a guarantee.

Without the cache, the same test (fixed port) gave up after 300 seconds.

The cache has a security side. The packet filter is on disk too, so a restarted node enforces the rules it had cached, not newer ones. That is the same trade-off a running node already makes during an outage, now extended across restarts.

When the control server comes back

Recovery was fast and needed no help, except for the expired key. From the 10-minute run without the cache:

16:21:27 control server back
16:21:48 c (tried to join during the outage) gets an address after 21s
16:21:55 a->c reachable after 7s
16:21:55 a->b ok (hello from b) | b->a ok (hello from a)
16:21:55 a->d (restarted during the outage) reachable after 0s
16:22:00 a->e (key expired during the outage): FAIL; e's state: Logged out.
16:22:01 admin clears e's expiry: a->e reachable after 1s

With the control server up, access changes are fast

The flip side is how quickly the control plane enforces a change when it is up. We changed the policy so that a may no longer reach b, reloaded Headscale with SIGHUP, and probed again and again until the result changed. Then we restored it, and then deleted b:

17:04:59 policy changed to block a->b: blocked after 6s
17:05:00 policy restored: a->b works again after 1s
17:05:06 node b deleted: a->b blocked after 5s

In both blocking cases the first probe after the change already failed. The 5 to 6 seconds is mostly that probe's own 5-second timeout. So with the control server up, a policy change or a removed device takes effect in seconds. With it down, it does not take effect at all. Your access control is only as live as your control plane.

This is also where ACLs as code pays off. A policy file in Git, reviewed and applied by CI, is a common way to run both Tailscale and Headscale. During an outage you cannot apply it, but once control is back, the reviewed version goes out in seconds. Git tells you which policy you intended at any point; it does not prove that every device received it, so check from the devices too.

What we got wrong first

Three mistakes in our first runs are worth sharing, because they are easy to make in your own health checks:

1. We restarted half of the long-lived pair. The first run used b both as a long-lived peer and as the restart target, so after about six minutes the "do existing tunnels survive?" measurement was gone. We added d as a separate restart target and ran it again.
2. Our recovery script read d's address once, at the start. A node restarted without a cache has no address until it logs in again, so every probe went to an empty address, and the script reported that d never recovered. d's own log showed it had logged in again seconds after control returned. If you write tailnet health checks, look up addresses on every attempt.
3. Our first probe accepted any response, and our restart check misread a logged-out client. A review of the scripts pointed out that the probe only checked for a non-empty body; it now requires curl to succeed and the page to name the node we meant to reach. The restart helper used tailscale status, which exits with an error when a node is logged out, so it reported the uncached restart as a failed start instead of measuring it; it now uses tailscale status --json. Every number in this post comes from runs with the fixed scripts.

What this means if you are replacing a VPN

• Count the control plane in your access path. For Tailscale's hosted service, that means their availability. For Headscale, it is one process with one database, so back up the database and treat upgrades like any other change.
• Turn on netmap caching where your client and control server support it, or plan to upgrade to the release where it is the default. In our tests, without it, a restart during the outage took the device off the network until control returned.
• Do not restart clients during a control incident. Pause automatic client updates and node rotation until control is back.
• Choose key expiry on purpose. Expiry limits the damage of a stolen key and also turns a long control outage into lost devices. Decide per device class, and alert before keys expire.
• Keep a way to contain a device that does not need the control plane, such as host firewalls or cloud security groups in front of your most sensitive servers that you can change without the mesh. Revoking sessions at your identity provider helps for applications that check it, but it does not close the network path.
• Monitor real traffic, not the self status. tailscale status called a working node offline.

What we could not test

• Relays. Everything ran on one host, and the client status showed direct paths; we did not force relayed connections. If your traffic goes through DERP relays, and especially if you run the relay on the same host as Headscale, an outage may take the relays down as well. We did not measure that.
• Other products. NetBird, ZeroTier, Twingate and Teleport split control and data differently. The questions in this post (joins, revocation, expiry, restarts) are the right ones to ask of any of them, but our numbers only describe Tailscale clients with a Headscale control server.
• Tailscale's hosted control plane. It may roll out features like netmap caching differently from Headscale.
• Long outages. Our longest was 30 minutes. Anything key-related scales with your expiry settings, not with our test.

Summary

• With the control server down for 30 minutes, existing tunnels kept working in both directions on every probe.
• New devices could not join, and nobody could revoke a device. Revocation is the real security cost of the outage.
• Key expiry is enforced by the peers. A key that expires during an outage takes its device off the network until someone re-authenticates it.
• Without the netmap cache, a client restarted during the outage came back with no peers and stayed off the network until control returned. With it (the cache-network-maps node attribute, expected as the default from Tailscale 1.104), peers reached it again in 15 to 19 seconds in 11 of 12 restarts, and it stayed reachable in two 10-minute checks (one earlier run saw an unexplained two-minute drop).
• With the control server up, a policy change or a deleted device took effect at the first probe, within seconds.

Most coverage calls it "the Ubuntu container escape". That framing is too narrow, and it can make you look in the wrong place. The bug is in upstream Linux code that arrived in 6.10 and was later backported to the 6.1 and 6.6 long-term kernels. Upstream has fixed 6.12, 6.18 and 7.1, but as of today there is no fix in the 6.1 or 6.6 branches at all. Debian 12, for example, is listed as vulnerable.

This post shows how to tell whether a machine has the affected code (in one command that usually works inside a pod too), where fixes exist, and what to do on nodes that cannot be patched yet.

TLDR

Prerequisites

• Shell access to the hosts you want to check, or kubectl access to a cluster
• For the cluster check: permission to run kubectl debug node/...
• Your distribution's security tracker for the final answer on a distro kernel

What happened

Unix domain sockets can carry open file descriptors from one process to another (SCM_RIGHTS). A socket can even be sent over itself, so the kernel needs a garbage collector to find groups of sockets that only reference each other and free them.

Linux 6.10 replaced that garbage collector with a new design that tracks sockets as vertices and edges in a graph and groups them into strongly connected components (SCCs). The fix commit describes a race in that code: while one thread sends a socket and another closes the sockets involved, the collector can free a vertex but leave it linked in a cached SCC list. The next garbage collection run walks that list and touches freed memory. That use-after-free is what the exploit turns into a host root shell.

Two details make this bad for container platforms:

• Ordinary containers can reach it. Creating Unix sockets and passing file descriptors are everyday operations. Docker's default seccomp profile allows them, and so does the RuntimeDefault profile of the common container runtimes, because almost all software needs them.
• The kernel is shared. A container is a process on the host kernel. Namespaces, cgroups and dropped capabilities limit what a process can do through normal interfaces, and a kernel memory bug like this one goes around them.

Upstream fixed it in August (af_unix: Unlink scc_entry in unix_del_edge()), and the kernel CVE team published the CVE on August 26. The fix is one line. Getting it into every distribution kernel is the slow part.

Who is affected

According to the kernel CVE record, the vulnerable code is in:

• every kernel from 6.10 onward, until the fixed releases below
• 6.1.141 and later in the 6.1 series, and 6.6.93 and later in the 6.6 series, because the new garbage collector was backported to those long-term branches

The fix is in 6.12.111, 6.18.53, 7.1.10 and 7.2. The CVE record lists no fixed 6.1 or 6.6 release, and a search of the 6.1.y and 6.6.y stable branch history on October 2 found no commit with the fix's title (the same search finds it in 6.12.y and 6.18.y). The newest releases there (6.1.188 and 6.6.157, both from September 14) still have the bug.

What the main distributions say, checked on October 2:

Sources: the Ubuntu CVE page, the Debian security tracker and the Amazon Linux advisory. We did not check every vendor. If you run RHEL and its rebuilds, Azure Linux, Container-Optimized OS, Bottlerocket or Talos, check that vendor's tracker.

Note

Some early reports listed Ubuntu 22.04 as vulnerable. Ubuntu's own tracker marks the 22.04 linux kernel (5.15) as not affected, which matches the upstream record: 5.15 never received the new garbage collector. The 22.04 HWE kernels are newer and are tracked separately, so check them on the tracker.

Why the version number is not enough

The natural first step is to compare uname -r against the list above. That works for upstream kernels and fails for distribution kernels, in both directions.

Ubuntu 24.04 ships a 6.8 kernel. Upstream 6.8 never had the new garbage collector, so a version check says "not affected". Ubuntu's tracker says vulnerable: distribution kernels pull in fixes from the stable branches, and the backport that brought the new code into 6.6 can bring it into a distro's 6.8 too. The reverse also happens. A distro can backport the one-line fix and keep the old version number.

So check for the code itself. The new garbage collector adds functions named unix_add_edges and unix_del_edges, and the kernel lists its function names in /proc/kallsyms. That file is world-readable by default, and unprivileged users usually see the names with the addresses zeroed. An ordinary container sees the host kernel's list, because there is only one kernel (a gVisor or Kata sandbox does not).

grep -cE ' unix_(add|del)_edges$' /proc/kallsyms

• 0: the new garbage collector was not found. On a standard distribution kernel, that means this CVE does not apply. On a custom or heavily modified kernel, confirm with whoever builds it.
• 1 or more: the new garbage collector is present. Now the version and your vendor decide whether the fix is in.

The presence check cannot see the fix itself, because the fix adds one line to a different function, unix_del_edge(). That is why the script below combines it with the version, and sends you to your vendor when the kernel is a distro build. Treat it as a fast first filter, not a certificate.

A check script

#!/usr/bin/env bash
# Heuristic check for CVE-2026-80521 (AF_UNIX garbage collector use-after-free).
# Usually works inside a container too: /proc/kallsyms lists the host kernel's symbols.
# Exit codes: 0 not detected or fixed, 1 affected, 2 check your vendor, 3 cannot tell.
set -u

release=${KERNEL_RELEASE:-$(uname -r)}   # KERNEL_RELEASE only for testing the logic
if [[ ! $release =~ ^([0-9]+)\.([0-9]+)(\.([0-9]+))?(.*)$ ]]; then
  echo "verdict:     CANNOT TELL (unrecognised kernel release '$release')"; exit 3
fi
major=${BASH_REMATCH[1]} minor=${BASH_REMATCH[2]} patch=${BASH_REMATCH[4]:-0}
suffix=${BASH_REMATCH[5]}   # anything after x.y.z: a distro, custom or -rc build

# The bug is in the garbage collector that Linux 6.10 rewrote (backported to
# 6.1.141+ and 6.6.93+). unix_add_edges and unix_del_edges exist only in that code.
if ! head -n 1 /proc/kallsyms 2>/dev/null | grep -q .; then
  new_gc=unknown
else
  grep -qE ' unix_(add|del)_edges$' /proc/kallsyms
  case $? in 0) new_gc=yes ;; 1) new_gc=no ;; *) new_gc=unknown ;; esac
fi

# Upstream releases that contain the fix (kernel CVE record, 2 October 2026).
fixed_upstream=no
case "$major.$minor" in
  6.12) (( patch >= 111 )) && fixed_upstream=yes ;;
  6.18) (( patch >= 53 )) && fixed_upstream=yes ;;
  7.1)  (( patch >= 10 )) && fixed_upstream=yes ;;
esac
if (( major > 7 || (major == 7 && minor >= 2) )); then fixed_upstream=yes; fi

echo "kernel:      $release"
echo "new AF_UNIX GC found: $new_gc"

case $new_gc in
  unknown) echo "verdict:     CANNOT TELL (could not read /proc/kallsyms); check $major.$minor with your vendor"; exit 3 ;;
  no)      echo "verdict:     NOT DETECTED (no new garbage collector; on a standard distro kernel this means not affected)"; exit 0 ;;
esac
if [[ -n $suffix ]]; then
  if [[ $fixed_upstream == yes && $suffix != -rc* ]]; then
    echo "verdict:     PROBABLY FIXED (base $major.$minor.$patch has the fix upstream); confirm with your vendor"; exit 2
  fi
  echo "verdict:     CHECK YOUR VENDOR (new garbage collector present; distro kernels backport fixes without changing the version)"; exit 2
fi
if [[ $fixed_upstream == yes ]]; then
  echo "verdict:     FIXED (if this is an unmodified upstream $release build)"; exit 0
fi
echo "verdict:     AFFECTED (upstream $release has the bug and no fix)"; exit 1

We ran it as an unprivileged user on two machines: a Raspberry Pi test box on Raspberry Pi OS, and an Ubuntu 22.04 server. The KERNEL_RELEASE variable is only there to test the version logic with made-up strings.

af-unix-gc-check

# Raspberry Pi OS, kernel 6.12.62: new garbage collector present, below the 6.12.111 fix
$ ./af-unix-gc-check.sh; echo "exit=$?"
kernel:      6.12.62+rpt-rpi-v8
new AF_UNIX GC found: yes
verdict:     CHECK YOUR VENDOR (new garbage collector present; distro kernels backport fixes without changing the version)
exit=2
# Ubuntu 22.04, kernel 5.15: old garbage collector
$ ./af-unix-gc-check.sh; echo "exit=$?"
kernel:      5.15.0-191-generic
new AF_UNIX GC found: no
verdict:     NOT DETECTED (no new garbage collector; on a standard distro kernel this means not affected)
exit=0

We also read /proc/kallsyms from inside a running Docker container on the 22.04 server. The symbol names were there (addresses zeroed), so the one-line check worked from inside the container. Hardened setups can mask that file; then the script says it cannot tell.

Check a whole cluster

Start with the kernel and OS image of every node:

kubectl get nodes -o custom-columns=NAME:.metadata.name,OS:.status.nodeInfo.osImage,KERNEL:.status.nodeInfo.kernelVersion

Then run the presence check on every distinct kernel build you see, or simply on every node during a rollout, when old and new images run side by side:

# Starts a debugging pod on the node; /proc/kallsyms is the node's kernel symbol list
kubectl debug node/<node-name> -it --image=busybox:1.36 -- \
  grep -cE ' unix_(add|del)_edges$' /proc/kallsyms

kubectl debug node creates a pod that shares the node's PID, network and IPC namespaces, so it needs the matching RBAC and must be allowed by your admission policy. Delete the debug pod when you are done (kubectl get pods shows it as node-debugger-...).

1. Node kernel uname -r
2. kallsyms check unix_add_edges?
3. Fix in this build? version or vendor tracker

Outcomes:

• 0 matches or fixed build: not exposed
• Code present, no fix: isolate untrusted workloads

What to do now

1. Patch where a fix exists

If your kernel line has a fix (Debian 13, upstream 6.12.111+, 6.18.53+, 7.1.10+ or 7.2+), install it. A new kernel package does nothing until the node boots into it, so plan the reboots: cordon and drain one node at a time, or replace node pools with a new image. On Amazon Linux 2023 with the matching kernel, the livepatch applies without a reboot once kernel livepatching is enabled; verify that it is applied.

2. Find the workloads that run code you did not write

Where there is no fix yet (Ubuntu 24.04 and 26.04, Debian 12, anything on a 6.1 or 6.6 kernel at or above the backport), the question is who can run arbitrary code on those nodes. A container escape needs code execution inside a container first. The usual places where strangers get exactly that:

• CI runners and build agents that run pull or merge request code from forks
• multi-tenant clusters where teams or customers deploy their own images
• notebooks, online code runners, plugin systems and "run your script" features
• any pod with a known remote code execution bug that you have not patched yet

Single-tenant nodes that only run your own images are still worth patching, but they are not where you start.

3. Move untrusted workloads into a sandbox or onto unaffected nodes

For the workloads in step 2, the most reliable option you can use today is a different isolation boundary:

• gVisor runs the container on a user-space kernel (the Sentry) that implements Unix sockets itself, so the sandbox's own Unix sockets do not use the host kernel's AF_UNIX garbage collector. (Access to host Unix sockets is a separate option and is off by default.) On GKE this is GKE Sandbox. Elsewhere, install runsc on the nodes, register it as a containerd runtime handler, and then add a RuntimeClass.
• Kata Containers runs each pod in a lightweight VM, and can use Firecracker as its VM monitor. The guest kernel may have the same bug, but an escape then lands in a VM, not on the host.
• A separate node pool on an unaffected kernel, such as Ubuntu 22.04 with its 5.15 kernel, for the untrusted workloads until your main image is fixed. It is a short-term measure with its own tradeoffs (an older kernel and different hardware support), not a destination.

A RuntimeClass for gVisor, and a pod that uses it:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc # the containerd runtime handler configured on the node
---
apiVersion: v1
kind: Pod
metadata:
  name: untrusted-build
spec:
  runtimeClassName: gvisor
  containers:
    - name: build
      image: registry.example.com/ci/build-runner:2026.10

Test your workloads under gVisor before you switch: it does not implement every system call, and I/O heavy jobs can run slower.

4. Know what does not help

• Default seccomp. The default profiles allow Unix sockets and file descriptor passing.
• Blocking AF_UNIX with a custom seccomp profile, in general. For the Copy Fail AF_ALG bug earlier this year, blocking one rarely used socket family was a clean mitigation. AF_UNIX is different: DNS resolution through NSS, database clients on local sockets, process managers, logging and many language runtimes use it. Blocking it breaks most real applications. A custom profile can still make sense for a specific workload that you have tested without Unix sockets.
• Rootless containers, user namespaces and dropped capabilities. They are good hardening, but they are not a fix here: the vulnerable code is reachable without any special privilege.

5. Watch for the fix and re-check

Subscribe to your distribution's security notices, and keep the check above in your node image pipeline. Once the fixed kernel is out, the version check plus the vendor's fixed package version tells you which nodes still need a reboot. Then move the sandboxed workloads back, or keep them sandboxed. Copy Fail and Fragnesia were container escapes too, earlier this year, so the sandbox may be worth keeping.

Summary

• CVE-2026-80521 is a use-after-free in the Linux AF_UNIX garbage collector. A public exploit escapes default Docker and Kubernetes containers to host root.
• It is not limited to Ubuntu. The vulnerable code is in 6.10 and later and in the 6.1.141+ and 6.6.93+ long-term kernels. Upstream fixed 6.12, 6.18 and 7.1, but not 6.1 or 6.6.
• Ubuntu 24.04 and 26.04 and Debian 12 had no fixed kernel on October 2. Ubuntu 22.04 (5.15), Debian 11 and other kernels without the new garbage collector are not affected.
• Version numbers mislead on distro kernels. grep -cE ' unix_(add|del)_edges$' /proc/kallsyms tells you whether the new garbage collector is there, usually even from inside a container.
• Until you can patch, put workloads that run untrusted code behind gVisor, Kata or a VM, or on nodes with an unaffected kernel. Default seccomp and user namespaces do not stop this one.

That makes the change quiet, which is the problem. This post shows how to list every action your workflows use, read the runtime each one declares, and find the node20 actions hidden behind SHA pins and composite actions. Then it covers what to do about each one, and the self-hosted runner hosts that GitHub no longer supports.

TLDR

• Node 20 is gone from GitHub Actions runners as of September 23, 2026. JavaScript actions run on Node 24, and ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION no longer brings Node 20 back.
• node20 actions are not rejected. The runner forces them onto Node 24 and adds a job warning that names them. Jobs we checked on September 28 passed with that warning.
• The risk is the action that breaks on Node 24, because you can no longer fall back.
• Floating major tags do not save you. On October 1, actions/checkout@v4, actions/cache@v4, actions/setup-node@v4 and actions/upload-artifact@v5 all still declared node20.
• SHA pins inside other people's composite actions are invisible to Dependabot, because the pin lives in a repository you do not own.
• A 100-line Python script lists every uses: reference in a repository, follows composite actions, and prints each one's runs.using.
• Self-hosted runners on macOS 13.4 or older, or on ARM32, are no longer supported.

Prerequisites

• A repository with GitHub Actions workflows, checked out locally
• The GitHub CLI (gh), logged in. The script uses it to read action.yml files from other repositories, including private ones your account can read.
• Python 3.8 or newer
• For the runner section: admin access to list self-hosted runners, or shell access to the runner hosts

What GitHub removed, and when

GitHub announced the deprecation on September 19, 2025, because Node 20 reached end of life in April 2026. The plan moved several times, and the editor's notes on that post record each move. The final timeline was:

1. Runner v2.328.0 added Node 24 next to Node 20, with Node 20 still the default. Setting FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true let you test Node 24 early.
2. June 16, 2026: runners started using Node 24 by default. ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true let you opt back into Node 20.
3. September 23, 2026: Node 20 was removed, and that opt-out stopped working.

The September 23 post says: "This is the final notification that Node 20 is no longer available on GitHub Actions runners. Runners now use Node 24 for JavaScript actions. The temporary ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION opt-out is no longer available." It applies to github.com and GitHub with Data Residency. Maintainers should set runs.using to node24 and release; workflow authors should move to versions that support Node 24.

What happens to a node20 action now

The changelog says runners now use Node 24 for JavaScript actions. The runner source shows what that means for an action that declares node20. In the final phase, the runner picks Node 24 for any action that declares node20, whatever environment variables you set. Actions that declare node12 or node16 are first mapped to node20, so they end up on Node 24 too. At the end of the job, the runner adds a warning that lists them.

We looked at the annotations of three jobs that ran on GitHub-hosted runners on September 28, in an open-source ebook repository that still pins old actions. All three jobs passed. One of them carried this warning:

Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/cache@v4, actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Another job used actions/checkout@v2, which declares node12, and its warning listed actions/checkout@v2 as a Node.js 20 action forced onto Node 24. That matches the mapping in the source.

What the runner does with an old JavaScript action

1. runs.using: node12 or node16 mapped to node20 first
2. runs.using: node20 action.yml
3. Runner picks Node 24 no opt-out since Sept 23

Outcomes:

• Code works on Node 24 step passes, job gets a warning
• Code breaks on Node 24 step fails, no way back to Node 20

What can break? Node 24 is two major versions after Node 20. The Node.js 24.0.0 release notes list removals such as tls.createSecurePair and fs.Dirent's path property, and runtime deprecations such as url.parse(). Whether an action hits one of these depends on its code and its dependencies. Until you have run it on Node 24, treat a node20 action as untested.

Why SHA pins and unmaintained actions are the main risk

Pinning actions to a full commit SHA is good supply-chain practice. It also freezes the runtime. actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 is v4.2.2, and it will declare node20 forever. The pin works as intended. It hides the update.

Floating major tags do not help much either. The first-party actions we checked moved to Node 24 in new major versions, so the old major tags stay on Node 20. We read the action.yml of each tag on October 1:

Note upload-artifact: v5 still declares node20. A newer major does not always mean Node 24, so read the file.

Dependabot helps less than you might expect. Per GitHub's docs, version updates for actions only run when dependabot.yml lists the github-actions ecosystem. Dependabot only supports the owner/repo@ref syntax, ignores actions and reusable workflows referenced by a local path, and does not support docker:// references. It also only edits files in your repository. When a third-party composite action pins a node20 action inside its own action.yml, no pull request in your repository can fix it.

That case is easy to find in the wild. The latest release of dominikh/staticcheck-action, v1.4.1 from March 12, 2026, is a composite action. Inside, it pins actions/cache to the commit for v4.3.0, which declares node20. Your workflow says dominikh/staticcheck-action@v1.4.1, and nothing in it mentions Node.

Then there are unmaintained actions. If an action has not had a release in years, no node24 version is coming. You need a plan for it, not a version bump.

How to find every node20 action

The method is simple:

1. List every uses: value in .github/workflows/*.yml and in any action.yml in the repository.
2. Split each value into owner, repository, optional path, and ref.
3. Read that action's action.yml or action.yaml at that ref, and look at runs.using.
4. If it says composite, repeat for the steps inside it.

You can do step 3 by hand with gh or with raw.githubusercontent.com:

read runs.using by hand

$ gh api "repos/actions/checkout/contents/action.yml?ref=v4" --jq .content | base64 -d | grep using:
  using: node20
$ curl -s https://raw.githubusercontent.com/actions/checkout/v5/action.yml | grep using:
  using: node24

The gh route works for private repositories and accepts any ref: a tag, a branch, or a SHA. For every reference in every workflow, use the script below.

A quick signal from job annotations

If a workflow ran recently, its jobs already carry the runner's warning. You can read it without opening the web UI:

# Job IDs for one run
gh run view RUN_ID --json jobs --jq '.jobs[].databaseId'

# The Node 20 warning for one job, if it has one
gh api repos/OWNER/REPO/check-runs/JOB_ID/annotations \
  --jq '.[] | select(.message | startswith("Node.js 20")) | .message'

This only covers what ran. A release workflow that runs once a quarter will not show up until it fails. The scan below reads what the workflows declare instead.

The script

It reads workflows and local actions from disk and fetches everything else through the GitHub API:

#!/usr/bin/env python3
"""Print the runtime (runs.using) of every action a repository's workflows use.

Usage: find-node20-actions.py [path-to-repo]   (needs python3 and a logged-in gh)
Exit codes: 0 all clear, 1 node12/16/20 actions found, 2 some refs could not be checked.
"""
import base64
import os
import pathlib
import re
import subprocess
import sys

USES = re.compile(r"""^\s*(?:-\s+)?uses:\s*['"]?([^'"\s#]+)""")
USING = re.compile(r"""^\s+using:\s*['"]?([A-Za-z0-9_-]+)""")
OLD = {"node12", "node16", "node20"}  # all of these now run on Node 24
PROBLEMS = {"not found", "unparsed", "unknown"}

root = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else ".").resolve()
seen = {}  # ref -> (runtime, text), so each action is fetched and walked once
found_old = found_problem = False

def uses_in(text):
    return [m.group(1) for line in text.splitlines() if (m := USES.match(line))]

def runtime_of(text):
    for line in text.splitlines():
        if m := USING.match(line):
            return m.group(1)
    return "unknown"

def read_remote(owner, repo, path, ref):
    for name in ("action.yml", "action.yaml"):
        file = f"{path}/{name}" if path else name
        r = subprocess.run(
            ["gh", "api", f"repos/{owner}/{repo}/contents/{file}?ref={ref}", "--jq", ".content"],
            capture_output=True, text=True,
        )
        if r.returncode == 0:
            return base64.b64decode(r.stdout).decode()
    return None

def resolve(ref):
    """Return (runtime, action.yml text or None) for one uses: value."""
    if ref.startswith("docker://"):
        return "docker image", None
    if re.search(r"\.ya?ml(@|$)", ref):
        return "reusable workflow", None  # scan that workflow's repo too
    if ref.startswith("./"):
        for name in ("action.yml", "action.yaml"):
            file = root / ref / name
            if file.is_file():
                text = file.read_text()
                return runtime_of(text), text
        return "not found", None
    m = re.fullmatch(r"([^/@]+)/([^/@]+)(?:/([^@]+))?@(.+)", ref)
    if not m:
        return "unparsed", None
    owner, repo, path, version = m.groups()
    text = read_remote(owner, repo, path or "", version)
    if text is None:
        return "not found", None
    return runtime_of(text), text

def walk(source, text):
    global found_old, found_problem
    for ref in uses_in(text):
        first = ref not in seen
        if first:
            seen[ref] = resolve(ref)
        runtime, action_text = seen[ref]
        found_old |= runtime in OLD
        found_problem |= runtime in PROBLEMS
        flag = "!!" if runtime in OLD else "  "
        print(f"{flag} {source:<38} {ref:<52} {runtime}")
        # A remote composite action can wrap a node20 action. Local ones are scanned as files below.
        if first and runtime == "composite" and action_text and not ref.startswith("./"):
            walk(ref, action_text)

files = sorted(root.glob(".github/workflows/*.y*ml"))
for dirpath, dirnames, filenames in os.walk(root):
    dirnames[:] = sorted(d for d in dirnames if d not in ("node_modules", ".git"))
    files += [pathlib.Path(dirpath, n) for n in sorted(filenames) if n in ("action.yml", "action.yaml")]
for f in files:
    text = f.read_text()
    rel = f.relative_to(root)
    if f.name.startswith("action."):
        runtime = runtime_of(text)
        found_old |= runtime in OLD
        found_problem |= runtime in PROBLEMS
        print(f"{'!!' if runtime in OLD else '  '} {str(rel):<38} {'(this action)':<52} {runtime}")
    walk(str(rel), text)

sys.exit(2 if found_problem else 1 if found_old else 0)

What it handles:

• ./local-action paths are read from disk, relative to the repository root.
• docker:// references are reported as Docker images, which do not use the runner's Node.
• Composite actions in other repositories are followed, so a node20 action two levels down still shows up. Local ones are scanned as files.
• action.yml and action.yaml are both tried, in that order. GitHub's metadata docs allow either name.
• Subdirectory actions such as github/codeql-action/init@v4.38.2 resolve to that path in the repository.
• Reusable workflows are labeled, not followed. Scan their repositories separately.

A reference it cannot read prints not found and makes the exit code 2, so an expired token or a deleted repository never reads as a clean result.

What we ran

We built a test repository whose workflow mixes every case: a SHA-pinned action, two local actions (one composite, one JavaScript with an action.yaml), a docker:// image, two third-party actions, a subdirectory action, and a reusable workflow. The octo-org reference is a made-up name, which the script labels without fetching:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
      - uses: ./.github/actions/setup # composite, uses actions/setup-node@v4
      - uses: ./.github/actions/legacy-js # action.yaml, runs.using: 'node20'
      - uses: docker://alpine:3.20
      - uses: JS-DevTools/npm-publish@v4.1.5
      - uses: dominikh/staticcheck-action@v1.4.1
      - uses: github/codeql-action/init@v4.38.2
  deploy:
    uses: octo-org/shared/.github/workflows/deploy.yml@main

Here is the run, on October 1, 2026:

scan the test repository

$ python3 find-node20-actions.py demo-repo; echo "exit code: $?"
!! .github/workflows/ci.yml               actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 node20
   .github/workflows/ci.yml               ./.github/actions/setup                              composite
!! .github/workflows/ci.yml               ./.github/actions/legacy-js                          node20
   .github/workflows/ci.yml               docker://alpine:3.20                                 docker image
   .github/workflows/ci.yml               JS-DevTools/npm-publish@v4.1.5                       node24
   .github/workflows/ci.yml               dominikh/staticcheck-action@v1.4.1                   composite
   dominikh/staticcheck-action@v1.4.1     actions/setup-go@4b73464bb391d4059bd26b0524d20df3927bd417 node24
!! dominikh/staticcheck-action@v1.4.1     actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 node20
   .github/workflows/ci.yml               github/codeql-action/init@v4.38.2                    node24
   .github/workflows/ci.yml               octo-org/shared/.github/workflows/deploy.yml@main    reusable workflow
!! .github/actions/legacy-js/action.yaml  (this action)                                        node20
   .github/actions/setup/action.yml       (this action)                                        composite
!! .github/actions/setup/action.yml       actions/setup-node@v4                                node20
exit code: 1

Rows marked !! need work. The actions/cache row is the one Dependabot cannot fix: it comes from inside staticcheck-action.

We also ran the script on two real repositories. On this site's own repository it printed 20 rows across six workflows, all node24 or docker, and exited 0. On the ebook repository from earlier, it flagged actions/checkout@v4, actions/checkout@v2 (node12), and the actions/cache@v4 inside a composite build action. Those are the same three actions that GitHub's job annotations listed for its September 28 runs.

To scan a whole organization, run the script in each checked-out repository. Remember that it only sees the branch you have checked out.

Fix each affected action

For each !! row you have three options.

Upgrade to a node24 release. This is the usual fix. Find the newest release, read its action.yml to confirm node24, and update the reference. For SHA pins, update the SHA and the version comment together, so Dependabot and humans can still read it. Check the release notes for breaking changes, because the Node 24 releases were often new major versions with other changes too.

Replace it. If the action is unmaintained, switch to a maintained one, or drop it for a run: step. Many small actions wrap one CLI command, and gh release create or aws s3 sync in a run: step has no Node runtime to go stale.

Fork it and bump runs.using. For an action with no node24 release, fork it, change runs.using to node24, run its tests on Node 24, rebuild any bundled dist/ folder, and pin your fork by SHA. The runner already runs the action on Node 24, so the edit alone only removes the warning. The value is in the testing and in owning the fix. You also own the fork's security updates now.

For composite actions you do not own, such as the staticcheck-action case, the fix belongs upstream. Open an issue or a pull request, and fork in the meantime if the wrapped action breaks.

Internal JavaScript actions

Your own actions need the same change, and nobody else will make it. To find them across an organization, GitHub code search works:

gh search code node20 --owner YOUR_ORG --filename action.yml
gh search code node20 --owner YOUR_ORG --filename action.yaml

Search for the bare word. In our tests, the phrase "using: node20" returned nothing, even in an organization where the bare word found files that contain exactly that line. Code search only covers default branches, and the word can also match comments and test fixtures, so confirm each hit with the script. For each real one:

1. Set runs.using: node24 in its action.yml.
2. Run its tests on Node 24, and set the same version in its own CI.
3. Rebuild any bundled output, such as a dist/ folder built with ncc.
4. Publish a new tag, and move the major tag if your consumers use one.
5. Update the repositories that pin the old SHA. The script above finds them.

Self-hosted runners need a runner version that knows node24. The announcement names v2.328.0 as the release that added it. GitHub now enforces much newer runner versions anyway, as we covered in our post on the self-hosted runner version window.

Self-hosted runner hosts that lose support

The September 23 post also says: "Node 24 is incompatible with macOS 13.4 and earlier, and it doesn't officially support ARM32. Self-hosted runners using these operating systems or architectures are no longer supported." The Node.js 24.0.0 release notes agree: the minimum macOS version went up to 13.5, and armv7 support was downgraded to experimental.

The runner source has a kill switch for this. On Linux ARM32, when GitHub turns it on, a JavaScript action step fails with "Linux ARM32 runners are no longer supported. Please migrate to a supported platform." That message is from the runner source. We have no ARM32 runner and did not see it in a real job.

To find these hosts, start with the API:

gh api repos/OWNER/REPO/actions/runners \
  --jq '.runners[] | [.name, .os, ([.labels[].name] | join(","))] | @tsv'

Use orgs/YOUR_ORG/actions/runners for organization runners. Do not trust the labels alone. GitHub's docs note that when default labels such as x64 are set with the configuration script, GitHub "does not validate that the runner is actually using that operating system or architecture." On the hosts themselves, uname -m shows the architecture (armv7l or armv6l means ARM32), and sw_vers -productVersion shows the macOS version.

There is no Node-only fix for these hosts. Move ARM32 runners to arm64 hardware or an arm64 OS image, and upgrade old Macs or retire them. A workflow with only run: steps and Docker actions needs no JavaScript action, but almost every workflow starts with actions/checkout.

Caveats

• The script is a line scanner, not a YAML parser. It can miss uses: in flow-style YAML, and it can pick up a line that starts with uses: inside a multi-line run: block. It takes the first indented using: line as the runtime.
• It reads one branch. Workflows that exist only on other branches are not scanned.
• It does not test anything. A node20 row means the action declares Node 20. Whether it works on Node 24 is a separate question.
• Reusable workflows from other repositories are not followed.
• Our runtime check is one snapshot. The tag table and the scan output are from October 1, 2026. Tags move, and maintainers ship releases.
• We did not test GitHub Enterprise Server. The September 23 post names github.com and GitHub with Data Residency only.

Summary

Node 20 left GitHub Actions on September 23, 2026, and the opt-out went with it. Actions that declare node20, node16 or node12 now run on Node 24, with a warning that names them. The ones that still work are easy to ignore, and the first one that breaks has no fallback.

Read the job annotations for a quick signal, then scan what your workflows declare: every uses: reference, resolved to its action.yml, with composite actions followed. Watch SHA pins, old major tags, and node20 actions pinned inside third-party composite actions, because no automated update reaches those. Upgrade, replace, or fork each one, move your own JavaScript actions to node24, and retire any self-hosted runner on ARM32 or macOS 13.4 or older.

This post covers what changes, which clients and keys are at risk, how to test them, and how to rotate keys.

TLDR

• Schedule: new RSA keys need at least 3072 bits from October 14, 2026. Brownouts on November 4 and December 9. Removal on January 13, 2027.
• HTTPS remotes are not affected.
• Your RSA key can stay if your client signs with rsa-sha2-256 or rsa-sha2-512.
• The keys most at risk are old. Since March 2022, only RSA keys added before November 2, 2021 can sign with SHA-1 on GitHub.
• Key exchange is the second trap. Besides the one being removed, GitHub offered us only sntrup761, curve25519 and ECDH. A client that supports none of them fails with any key.
• Test with ssh -vT git@github.com and read the kex: lines. Add -vvv to see the signature algorithm for your key.

Prerequisites

• Git remotes that use SSH (git@github.com:... or ssh://)
• Shell access to the machines and images that run Git: CI agents, runners, build containers
• The GitHub CLI (gh), with the read:public_key scope for account keys and admin access for deploy keys
• OpenSSH's ssh and ssh-keygen

What GitHub is changing, and when

The changelog lists four changes:

1. Removal of "RSA keys using SHA-1 in SSH (i.e., the ssh-rsa signature type, including ssh-rsa-cert-v01@openssh.com certificates using SHA-1)".
2. Removal of the key exchange mechanism diffie-hellman-group-exchange-sha256.
3. "All new RSA SSH keys uploaded after October 14, 2026 must be at least 3072 bits in size, both for signing and authentication." That includes keys you use to sign commits.
4. A new post-quantum key exchange, mlkem768x25519-sha256, on github.com and on GitHub Enterprise Cloud with data residency, except for the U.S. region.

GitHub SSH change schedule

1. Sep 22, 2026 changelog published
2. Oct 14, 2026 new RSA keys 3072+ bits, ML-KEM enabled
3. Nov 4, 2026 first brownout
4. Dec 9, 2026 second brownout
5. Jan 13, 2027 ssh-rsa and DH group exchange removed

Both brownouts cover ssh-rsa and diffie-hellman-group-exchange-sha256. The changelog gives no time of day or duration for them. On GitHub Enterprise Server, everything takes effect in version 3.25, except ML-KEM, which comes in 3.24. GitHub Enterprise Server users of the unauthenticated Git protocol are affected too.

ML-KEM needs nothing from you: older clients "should automatically fall back", GitHub says. OpenSSH added mlkem768x25519-sha256 in 9.9 and made it the default in 10.0, according to its release notes.

Who is actually affected

HTTPS users are not. In GitHub's words: "If your Git remotes start with https://, nothing here will affect you." For SSH users, there are two questions.

1. Can your client sign with SHA-2? As GitHub points out, ssh-rsa is both a key type and a signature type. Every RSA key has key type ssh-rsa, but it can sign with SHA-1 (ssh-rsa), SHA-256 (rsa-sha2-256) or SHA-512 (rsa-sha2-512). The client decides which signature it sends, so the key itself is fine.

In 2021 GitHub announced that from March 15, 2022, "RSA keys uploaded after the cut-off point above will work only with SHA-2 signatures", with November 2, 2021 as the cut-off. So if your RSA key was added after that date and works today, your client already signs with SHA-2. The keys at risk are RSA keys added before November 2, 2021, used by a client that still signs with SHA-1.

2. Does your client share a key exchange with GitHub? This does not depend on your key. On October 1, 2026, GitHub's SSH server offered us this list:

sntrup761x25519-sha512, sntrup761x25519-sha512@openssh.com,
curve25519-sha256, curve25519-sha256@libssh.org,
ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521,
diffie-hellman-group-exchange-sha256, kex-strict-s-v00@openssh.com

Plain Diffie-Hellman groups such as diffie-hellman-group14-sha256 are not offered. When group exchange goes, a client needs sntrup761, curve25519, ECDH or, after October 14, ML-KEM. A client that only does classic Diffie-Hellman fails even with an Ed25519 key.

GitHub lists these minimum versions for RSA with SHA-2 in the default configuration:

Where to look:

• Old CI images and build containers. The SSH client is whatever the base image shipped.
• Jenkins agents. Command-line Git uses the agent's ssh. An agent set to JGit uses Java SSH code instead, so test it separately.
• Java tools on the original JSch (com.jcraft:jsch). The fork GitHub names is published as com.github.mwiede:jsch.
• Python tools on Paramiko. It is not in GitHub's table. Its changelog says 2.9.0 (December 2021) added RSA SHA-2 signatures.
• libssh2-based tools, such as curl's SFTP and SCP support and many libgit2-based clients.
• Windows. Git for Windows ships its own ssh.exe, Windows has a separate built-in OpenSSH, and some setups use PuTTY's plink or TortoiseGit's PuTTY-based plink.
• Appliances such as artifact servers and backup tools, which often embed their own SSH library.

Test a client now

Start with ssh -V. Anything older than OpenSSH_7.2 cannot sign with SHA-2, so plan to replace it.

Then connect with verbose output. Key exchange runs before authentication, so no registered key is needed. This is a real run from the Debian 12 machine we wrote this post on, on October 1, 2026:

OpenSSH 9.2p1 against github.com, 2026-10-01

# -F /dev/null ignores ssh_config, no key is offered, the long server-sig-algs line is shortened
$ ssh -F /dev/null -o BatchMode=yes -o PubkeyAuthentication=no -o UserKnownHostsFile=./kh -o StrictHostKeyChecking=accept-new -vT git@github.com 2>&1 | grep -E '^OpenSSH|kex: algorithm|host key algorithm|Server host key|server-sig-algs|Permanently|denied'
OpenSSH_9.2p1 Debian-2+deb12u10, OpenSSL 3.0.22 25 Aug 2026
debug1: kex: algorithm: sntrup761x25519-sha512
debug1: kex: host key algorithm: ssh-ed25519
debug1: Server host key: ssh-ed25519 SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519-cert-v01@openssh.com,...,rsa-sha2-512,rsa-sha2-256,ssh-rsa>
git@github.com: Permission denied (publickey).

Read three lines:

• kex: algorithm: is the agreed key exchange. If it says diffie-hellman-group-exchange-sha256, this client breaks during the brownouts.
• kex: host key algorithm: is how GitHub's host key is verified. The changelog words the SHA-1 removal generally, so treat ssh-rsa here as at risk too.
• server-sig-algs= lists the signature types GitHub accepts for your key. Today it still ends in ssh-rsa.

For your own test, drop -F /dev/null. Git uses your ssh_config, and a stale KexAlgorithms or HostKeyAlgorithms line there can pin you to the old algorithms.

See both offers. With -vv, OpenSSH prints debug2: KEX algorithms: and debug2: host key algorithms: twice: first your client's offer, then GitHub's. The key exchange list above comes from GitHub's.

See the signature for your key. With -vvv and a key GitHub accepts, OpenSSH 9.2p1 logs a line of this form when it signs:

debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:<your key fingerprint>

rsa-sha2-512 or rsa-sha2-256 is safe. ssh-rsa breaks. Our machine has no key registered on GitHub, so we confirmed this line against a local test server.

Test without the old algorithms. Remove both from your client's defaults. If this still authenticates, the brownout will not affect this client:

ssh -o KexAlgorithms=-diffie-hellman-group-exchange-sha256 \
    -o HostKeyAlgorithms=-ssh-rsa \
    -o PubkeyAcceptedAlgorithms=-ssh-rsa \
    -T git@github.com

A leading - removes algorithms from the default list. Before OpenSSH 8.5, PubkeyAcceptedAlgorithms was called PubkeyAcceptedKeyTypes.

Know what failure looks like. We forced a key exchange GitHub does not offer, and OpenSSH printed this (GitHub's offer list trimmed):

Unable to negotiate with 140.82.121.4 port 22: no matching key exchange method found. Their offer: sntrup761x25519-sha512,...

With HostKeyAlgorithms=ssh-dss, it printed no matching host key type found. Search CI logs for these strings on November 4. Libraries word their errors differently.

List what the binary supports. ssh -Q kex lists key exchanges and ssh -Q sig lists signature algorithms (added in OpenSSH 7.9). For the settings your config actually applies to GitHub, run ssh -G github.com and read kexalgorithms, hostkeyalgorithms and pubkeyacceptedalgorithms.

Test the clients that are not your shell's ssh

Which ssh does Git run? core.sshCommand, GIT_SSH and GIT_SSH_COMMAND can point Git at another program. Ask Git:

GIT_TRACE=1 git ls-remote git@github.com:your-org/your-repo.git 2>&1 | grep run_command

On our machine the run_command line contained ssh -o SendEnv=GIT_PROTOCOL git@github.com 'git-upload-pack ...', so Git used the ssh on the path.

Java. Find the original JSch in your dependency tree:

mvn dependency:tree -Dincludes=com.jcraft:jsch
./gradlew dependencyInsight --dependency com.jcraft --configuration runtimeClasspath

The fork's README shows how to exclude com.jcraft:jsch when it arrives as a transitive dependency.

Python. python3 -m pip show paramiko prints the installed version.

libssh2. curl -V shows the libssh2 version curl links against. On our Debian 12 machine it printed libssh2/1.10.0, below GitHub's 1.11.0. We did not test whether it fails against GitHub, and distributions sometimes backport fixes, so test the tool rather than trusting the number.

Go. GitHub does not name the module behind "Go SSH". If it means golang.org/x/crypto, check its version in go.mod.

Windows. In PowerShell, Get-Command ssh shows which ssh.exe comes first on the path. Also check git config --show-origin --get core.sshCommand and GIT_SSH. If they point at plink, compare its version with PuTTY 0.82.

Audit your keys

ssh-keygen -l -f ~/.ssh/id_rsa.pub prints a key's size in bits, its fingerprint and its type, such as (RSA) or (ED25519). With -f - it reads keys from standard input.

Account keys. This lists every authentication key on your account with its upload date, size and type:

# Needs the read:public_key scope: gh auth refresh -h github.com -s read:public_key
gh api /user/keys --paginate --jq '.[] | [.created_at[0:10], .title, .key] | @tsv' |
  while IFS=$'\t' read -r created title key; do
    size_type=$(printf '%s\n' "$key" | ssh-keygen -l -f - | awk '{print $1, $NF}')
    printf '%s  %-14s  %s\n' "$created" "$size_type" "$title"
  done

RSA keys dated before 2021-11-02 are the only ones GitHub still lets sign with SHA-1. Check the clients that use them first.

Deploy keys. The deploy keys API returns created_at and last_used. This loop needs admin access to each repository and silently skips any it cannot read, so check the count:

ORG=your-org
gh repo list "$ORG" --limit 1000 --json nameWithOwner --jq '.[].nameWithOwner' |
  while read -r repo; do
    gh api "repos/$repo/keys" --paginate \
      --jq ".[] | [\"$repo\", .id, .created_at[0:10], (.last_used // \"never\")[0:10], .title, .key] | @tsv" 2>/dev/null
  done |
  while IFS=$'\t' read -r repo id created used title key; do
    size_type=$(printf '%s\n' "$key" | ssh-keygen -l -f - | awk '{print $1, $NF}')
    printf '%s\t%s\tadded %s\tused %s\t%s\t%s\n' "$repo" "$id" "$created" "$used" "$size_type" "$title"
  done

A key that was never used is a candidate for deletion, not rotation.

Machine users. Public authentication keys need no token:

curl -s https://github.com/your-machine-user.keys | ssh-keygen -l -f -

For upload dates, run the account key audit with the machine user's token.

Rotate to Ed25519

GitHub recommends "an Ed25519 key whenever possible". For an account key:

# 1. Create the key. Add -N '' only for unattended CI keys.
ssh-keygen -t ed25519 -C "ci-runner-2026-10" -f ~/.ssh/id_ed25519_github

# 2. Register it. If the token lacks the scope, gh prints the refresh command.
gh ssh-key add ~/.ssh/id_ed25519_github.pub --title "ci-runner 2026-10"

# 3. Prove the new key works on its own.
ssh -i ~/.ssh/id_ed25519_github -o IdentitiesOnly=yes -T git@github.com

Then point Git at it in ~/.ssh/config (our post on using a specific SSH key for Git has other ways):

Host github.com
  IdentityFile ~/.ssh/id_ed25519_github
  IdentitiesOnly yes

When nothing uses the old key, find its ID with gh ssh-key list and remove it with gh ssh-key delete <id>.

For deploy keys, gh repo deploy-key add key.pub --title "deploy 2026-10" -R your-org/your-repo adds a read-only key, and --allow-write grants push access. Its help text warns that the key is "associated with the current authentication token" and is removed if that token is de-authorized. Add long-lived deploy keys in the repository settings instead.

Warning

A new key type does not fix a key exchange problem. If a client's only key exchange in common with GitHub is diffie-hellman-group-exchange-sha256, it fails on January 13 with any key. Upgrade the client or library first.

Where Ed25519 is not supported:

• ECDSA. GitHub says "All Ed25519 and ECDSA keys we support are strong, secure, and will continue to work for the indefinite future." Use ssh-keygen -t ecdsa -b 256.
• RSA with 3072 bits or more, if another service needs RSA: ssh-keygen -t rsa -b 4096. The client must still sign with SHA-2.
• Old key parsers. Since OpenSSH 7.8, ssh-keygen writes its own private key format. Some older libraries only read PEM, which -m PEM produces for RSA and ECDSA keys.

Checklist before November 4

CI images and agents

1. List every image, runner and agent that runs Git over SSH, including container jobs.
2. Run ssh -V in each. Replace anything older than OpenSSH 7.2.
3. Run the test without old algorithms inside each image, with the job's credentials.
4. Check /etc/ssh/ssh_config, /etc/ssh/ssh_config.d/ and baked-in ~/.ssh/config files for KexAlgorithms, HostKeyAlgorithms, PubkeyAcceptedAlgorithms and PubkeyAcceptedKeyTypes.
5. Find JSch, Paramiko, libssh2 and Go SSH in your build tools.

Deploy keys and machine users

1. Run the deploy key audit for each organization and delete unused keys.
2. For RSA keys added before November 2, 2021, test the client that uses them, or switch to Ed25519.
3. Audit each machine user's keys with its own token, and test from the host that uses it.

On the day

1. Watch CI for Unable to negotiate and Permission denied (publickey).
2. Treat any failure as a real finding, even if it stops after the brownout.

What we could not verify

• Brownout timing. The changelog gives dates, not times or durations.
• Existing small RSA keys. The 3072-bit rule covers "new RSA SSH keys uploaded after October 14, 2026". The changelog does not say existing 2048-bit keys stop working. We assume deploy keys and re-uploaded old keys count as new uploads, but it does not say so.
• Libraries. We did not test JSch, Paramiko, libssh2, Go or PuTTY. The table is GitHub's.
• A real SHA-1 signature. We had no RSA key from before November 2021 to test with.
• Server lists. These are what GitHub offered on October 1, 2026. They can change.

Summary

GitHub removes RSA with SHA-1 signatures and one Diffie-Hellman key exchange on January 13, 2027, with brownouts on November 4 and December 9. HTTPS remotes are not affected. Your RSA key can stay, but the client must sign with SHA-2 and share a modern key exchange with GitHub.

Before November 4, test every place that runs Git over SSH with the old algorithms removed, and audit account keys, deploy keys and machine users. Move to Ed25519 where you can, and to ECDSA or 3072-bit RSA where you cannot. For another client-side SSH risk in some of the same libraries, see our post on libssh2 CVE-2026-55200.

This post shows the data, what each runner did, and how to find every runner in your fleet before one of them goes quiet.

TLDR

• We saw it on a free github.com organization. The API reports our test org's plan as free, and both the registration check and the job check fired. GitHub says Enterprise Server is not affected.
• Registration floor: runner 2.328.0 was refused with "The minimum runner version required to register with GitHub Actions is now 2.329.0."
• Job floor: runner 2.335.1 registered fine, then exited with "Runner version v2.335.1 is deprecated and cannot receive messages." The probe job stayed queued.
• The exit looks clean: run.sh exited with code 0. With ACTIONS_RUNNER_RETURN_VERSION_DEPRECATED_EXIT_CODE=1 set, the same runner exited with code 7, which a supervisor can see.
• The scheduled window: for the 18 versions with a runtime deprecation date and a later release, GitHub's API put the date 62 to 70 whole days after that next release (median 64.5). The documentation says 30.
• The floor is not a safe version: 2.329.0, the registration minimum, has an API runtime deprecation date of January 23, 2026, which has passed.
• On September 30, 2026, only 2.336.0 (date November 5) and 2.337.0 (no date yet) were inside their schedule, and both ran our probe job.

Prerequisites

• Self-hosted runners on github.com, on VMs, bare metal, or Kubernetes with Actions Runner Controller (ARC)
• The GitHub CLI (gh), logged in as an admin of at least one repository
• For the fleet check: shell access to your runner hosts, or kubectl access to the cluster that runs them

What GitHub announced

The enforcement has a long history, according to GitHub's changelog. GitHub first set the 2.329.0 minimum for March 16, 2026, then paused it three days before. In June it published a new timeline: July 31 for Enterprise Cloud with data residency, September 25 for Enterprise Cloud. The last changelog post moved that to September 29.

The June post has the part that matters most, and it is easy to miss:

• 2.329.0 is only the registration minimum. It is the oldest runner the new Actions backend will accept.
• Running jobs is a moving target. A runner has to install each new release within 30 days, or "the GitHub Actions service will stop queuing jobs to it."
• Pinned runners are on their own. "A runner pinned to 2.329.0 that never updates again will not pick up jobs."

In early September GitHub also shipped a REST API for runner version deprecations: GET /actions/runners/deprecations/{version} at repository, organization, or enterprise level. It is the source for the numbers below.

What GitHub's API says about every release

We listed all 146 actions/runner release records and asked the repository-level endpoint about each one. Our calls used a gh login with the classic repo scope for a user who administers the repository. The organization-level endpoint answered 403 for the same token, because it needs the admin:org scope or the fine-grained self-hosted runners permission:

gh api /repos/your-org/your-repo/actions/runners/deprecations/2.335.1

{"runner_version":"2.335.1","runtime_deprecates_at":"2026-09-24T15:30:55Z"}

Three things came back that the announcement does not say:

1. The API only knows 19 versions, 2.321.0 to 2.337.0. Anything older returns 404. So does 2.327.0, which was replaced by 2.327.1 three days later.
2. No response had a registration_deprecates_at field. The changelog lists it, but every successful response contained only runner_version and runtime_deprecates_at. The only registration rule we saw is the fixed 2.329.0 floor.
3. The runtime dates follow a pattern. For each version, we counted the whole days, rounded down, between the next non-prerelease release and its runtime_deprecates_at.

Days a runner version keeps running jobs after the next release ships

Documented: 30 days: 30 days

runtime_deprecates_at from GitHub's runner version deprecations API, minus the next release's publish date. 18 versions, 2.321.0 to 2.336.0. Swept 2026-09-30. GitHub's documentation says 30 days.

For every one of the 18 versions, the scheduled date fell 62 to 70 days after its successor was published. That is about nine weeks, twice the 30 days in the documentation. We do not know why. The dates may include a rollout delay, or a buffer GitHub keeps for itself.

One test lines up with the API rather than the 30 days. Thirty days after 2.337.0 was published was September 25. On September 30, a 2.336.0 runner still registered and completed our probe job, and its API date is November 5. That is one observation on one day, not a promise of extra time.

Counted from its own release date, a version's scheduled life is 66 to 138 days (median 104). Here is where recent versions stood on September 30, 2026:

Warning

The registration minimum is not a safe version. 2.329.0 meets the registration floor, but its API runtime deprecation date, January 23, 2026, passed eight months before enforcement started. We did not test 2.329.0 itself. The one version we tested that was past its date, 2.335.1, registered and then exited, as shown below.

What a real runner does

API dates are a schedule, not behavior. To see the behavior, we tried to register one runner per version to a private repository in a free organization, with auto-update turned off (--disableupdate) so each runner stayed on its version. Each runner that registered then got a single workflow_dispatch job. The script downloads the runner, registers it, starts it, dispatches the job, polls the run for about three minutes, and removes the runner. All output below is from runs on September 30, 2026, between 06:54 and 07:34 UTC, on a linux-arm64 host. An earlier pass of the same four tests that morning, with a first version of the script, gave the same four results; both sets of transcripts are in the repo.

Below the registration floor, 2.328.0. The runner is refused before it exists:

runner 2.328.0

$ REPO=The-DevOps-Daily/runner-floor-lab scripts/floor-test.sh 2.328.0
07:16:15Z runner 2.328.0 (arm64), label floor-2-328-0-1790752575
07:20:30Z config.sh --disableupdate
  ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
  │   RUNNER UPDATE REQUIRED                                                                                            │
  ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │                                                                                                                     │
  │   The minimum runner version required to register with GitHub Actions is now 2.329.0.                               │
  │   Please upgrade your runner.                                                                                       │
  │                                                                                                                     │
  │   For more information, see:                                                                                        │
  │   https://github.blog/changelog/2026-02-05-github-actions-self-hosted-runner-minimum-version-enforcement-extended   │
  │                                                                                                                     │
  └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
  Response status code does not indicate success: 404 (Not Found).
07:20:34Z config.sh exit code: 1
07:20:34Z result: not registered

This is the loud failure, and the easy one. A deploy pipeline that builds runner VMs from an old image fails at config.sh, and someone notices.

Registered, but past its end date, 2.335.1. This is the one to worry about. Registration works, the runner connects, and then:

runner 2.335.1

# the ASCII registration banner is trimmed from the output
$ REPO=The-DevOps-Daily/runner-floor-lab scripts/floor-test.sh 2.335.1
07:06:37Z runner 2.335.1 (arm64), label floor-2-335-1-1790751997
07:10:50Z config.sh --disableupdate
  # Authentication
  √ Connected to GitHub
  # Runner Registration
  √ Runner successfully added
  # Runner settings
  √ Settings Saved.
07:10:57Z config.sh exit code: 0
07:10:57Z run.sh
  runner status: offline, busy: false
07:11:34Z dispatched run 36682287765, polling for up to 180s
07:14:51Z run status after 197s: queued 
07:14:51Z runner process exited with code 0
07:14:51Z runner output (last lines):
  
  √ Connected to GitHub
  
  Current runner version: '2.335.1'
  2026-09-30 07:11:04Z: Listening for Jobs
  An error occurred: Runner version v2.335.1 is deprecated and cannot receive messages.
  Runner listener exit with terminated error, stop the service, no retry needed.
  Exiting runner...
07:15:09Z cancelled run 36682287765

Look at the order. "Runner successfully added" and "Connected to GitHub" both succeed. The runner even reports "Listening for Jobs". Then it logs one error and stops, and the process exits with code 0, the same code as a clean shutdown. The job is not rejected either. It was still queued when the script stopped polling after 197 seconds, and the script cancelled it. We did not measure how long GitHub would keep it queued.

The same runner, with the exit code turned on. Exit code 0 does not tell anything that checks exit status that the runner failed. The runner has a setting for this. With ACTIONS_RUNNER_RETURN_VERSION_DEPRECATED_EXIT_CODE=1 in its environment, run.sh exits with code 7 instead of 0 when the version is deprecated. By the runner's source and pull request #4285, the setting first shipped in 2.333.0. We tested it on 2.335.1:

runner 2.335.1, deprecated exit code on

# the ASCII registration banner is trimmed from the output
$ ACTIONS_RUNNER_RETURN_VERSION_DEPRECATED_EXIT_CODE=1 REPO=The-DevOps-Daily/runner-floor-lab scripts/floor-test.sh 2.335.1
07:25:07Z runner 2.335.1 (arm64), label floor-2-335-1-1790753107
07:29:19Z config.sh --disableupdate
  # Authentication
  √ Connected to GitHub
  # Runner Registration
  √ Runner successfully added
  # Runner settings
  √ Settings Saved.
07:29:29Z config.sh exit code: 0
07:29:29Z run.sh (ACTIONS_RUNNER_RETURN_VERSION_DEPRECATED_EXIT_CODE=1)
  runner status: offline, busy: false
07:30:04Z dispatched run 36684028728, polling for up to 180s
07:33:17Z run status after 185s: queued 
07:33:17Z runner process exited with code 7
07:33:17Z runner output (last lines):
  
  √ Connected to GitHub
  
  Current runner version: '2.335.1'
  2026-09-30 07:29:37Z: Listening for Jobs
  An error occurred: Runner version v2.335.1 is deprecated and cannot receive messages.
  Runner listener exit with deprecated version exit code: 7.
  Exiting runner...
07:33:18Z cancelled run 36684028728

Same error, and the new probe job stayed queued too, but now the process exits with a failure code. A supervisor that checks exit status can act on that. By default, systemd treats a non-zero exit as a failed service, and Kubernetes records the container's termination reason as Error instead of Completed. We did not run either of them. We set the variable in the shell that started run.sh, which is the only way we tested it.

Inside the window, 2.336.0 and 2.337.0. Both registered, picked up the job within seconds, and finished it:

runner 2.336.0

# the ASCII registration banner is trimmed from the output
$ REPO=The-DevOps-Daily/runner-floor-lab scripts/floor-test.sh 2.336.0
07:01:11Z runner 2.336.0 (arm64), label floor-2-336-0-1790751671
07:05:26Z config.sh --disableupdate
  # Authentication
  √ Connected to GitHub
  # Runner Registration
  √ Runner successfully added
  # Runner settings
  √ Settings Saved.
07:05:41Z config.sh exit code: 0
07:05:41Z run.sh
  runner status: online, busy: false
07:06:11Z dispatched run 36681802539, polling for up to 180s
07:06:29Z run status after 18s: completed success
07:06:29Z runner process: still running
07:06:29Z runner output (last lines):
  
  √ Connected to GitHub
  
  Current runner version: '2.336.0'
  2026-09-30 07:05:46Z: Listening for Jobs
  2026-09-30 07:06:14Z: Running job: probe
  2026-09-30 07:06:24Z: Job probe completed with result: Succeeded

By the API's schedule, 2.336.0 stops receiving jobs on November 5, 2026, unless it updates first.

Why this breaks quietly in practice

Runners with auto-update on should mostly follow the schedule by themselves: the runner updates when a new release is out. The risk is in setups where the runner stays on one version because it cannot or may not update:

• Container images. A Dockerfile with ARG RUNNER_VERSION=2.334.0 builds the same runner every time. If those runners do not update themselves, each new one starts on the old version and would behave like the 2.335.1 runner above once its date passes.
• Actions Runner Controller. ARC scale sets start runners from an image such as ghcr.io/actions/actions-runner:2.335.1, so the image tag tells you which version each new pod starts with.
• VM templates and golden images. An AMI or a Packer template built in the spring holds a spring runner. A VM from it starts on that version.
• --disableupdate in install scripts. Teams add it so runners do not change during a release window, then forget about them.

In our test the only clear message was in the runner's own output. The workflow run just showed "Queued".

Find every runner version you run

Start with the versions, then check them against the API. On a VM or bare-metal host, ask each installed runner directly. The runner binary prints its version:

# Run in each runner's install directory, for example /home/runner/actions-runner
./config.sh --version
# or: ./bin/Runner.Listener --version

On a fresh 2.337.0 download, both print 2.337.0. Loop over every install directory you have. A host with several runners has several versions.

On Kubernetes, list the runner images that pods run:

# Distinct runner image references, including tags such as :latest and digests
kubectl get pods -A -o jsonpath='{..image}' | tr ' ' '\n' | grep -i 'actions-runner' | sort -u

A version tag like :2.335.1 tells you the version each new pod starts with. A tag like :latest or a digest does not; resolve those before you check them. Treat both commands as a starting point, not a complete inventory.

Also check the places that build runners, not only the ones that run them: RUNNER_VERSION in Dockerfiles, Packer variables, Terraform user-data, and Helm values.

Then feed the versions to the check script. It calls the deprecations API for each version and prints the days left. It needs bash, python3 and gh, and it exits non-zero on bad input or an API error, so a broken check never reads as a pass:

check-versions

$ printf "2.337.0\nv2.336.0\n2.335.1\n2.329.0\n2.320.0\n" | REPO=your-org/your-repo scripts/check-versions.sh
VERSION    ENDS         STATUS
2.337.0    -            no runtime deprecation date announced
2.336.0    2026-11-05   36 days left
2.335.1    2026-09-24   ended 5 days ago: stops taking jobs
2.329.0    2026-01-23   ended 249 days ago: stops taking jobs
2.320.0    -            unknown to the API (it knows 2.321.0 and later): replace it

Anything that says "ended" is past its API date. In our test, the one such version we ran did not take jobs. Anything under three weeks needs a new image now.

The-DevOps-Daily/runner-version-window on GitHub

How to keep runners inside the window

Leave auto-update on for long-lived runners. It is the default. A runner that updates itself follows the schedule without anyone thinking about it. If you must control when runners change, schedule the update instead of disabling it.

Update the pinned version on every runner release. For images and templates, treat each actions/runner release as a change to make. Configure your dependency bot to bump the actual runner version or image reference, and check that it really opens those pull requests. Rebuilding an image that still pins the old version does not upgrade anything. Plan around GitHub's documented 30 days; the extra weeks we saw in the API are not a promise.

Alert on the date, not on the symptom. Run the check in CI on a schedule and fail when a version in use has less than 21 days left:

name: runner-version-check
on:
  schedule:
    - cron: '0 7 * * 1' # every Monday
  workflow_dispatch:

jobs:
  check:
    runs-on: ubuntu-latest # a GitHub-hosted runner, so the check never depends on the runners it checks
    steps:
      - uses: actions/checkout@v4
      - name: Fail if a runner version has less than 21 days left
        shell: bash
        env:
          # A token that administers REPO. The default GITHUB_TOKEN cannot call the deprecations API.
          GH_TOKEN: ${{ secrets.RUNNER_AUDIT_TOKEN }}
          REPO: ${{ github.repository }}
        run: |
          set -euo pipefail
          # Copy scripts/check-versions.sh from the companion repo into yours.
          # runner-versions.txt lists the versions your images and hosts use, one per line.
          scripts/check-versions.sh < runner-versions.txt | tee report.txt
          if grep -E 'ended|unknown|^[0-9.]+ +[0-9-]+ +([0-9]|1[0-9]|20) days left' report.txt; then
            exit 1
          elif [ $? -ne 1 ]; then
            exit 2 # grep itself failed
          fi

Make an expired runner fail loudly. Where a container or unit starts run.sh, set ACTIONS_RUNNER_RETURN_VERSION_DEPRECATED_EXIT_CODE=1 in its environment, as in the test above, and alert on exit code 7:

# Part of a pod spec for a runner container whose command is run.sh
env:
  - name: ACTIONS_RUNNER_RETURN_VERSION_DEPRECATED_EXIT_CODE
    value: '1'

One warning, from reading the runner source, not from a test: the svc.sh service does not use run.sh. Its wrapper, bin/RunnerService.js, has no case for code 7 in 2.335.1 or 2.337.0. It treats an unknown code as a failure and starts the listener again after 5 seconds. It only gives up if GITHUB_ACTIONS_SERVICE_EXIT_AFTER_N_FAILURES is set to a positive number and that many unknown exits happen in a row. On hosts that use svc.sh, rely on the version check instead.

Watch for queued jobs. A job that sits in "Queued" for more than a few minutes on a self-hosted label is worth an alert of its own. It catches this failure and every other reason runners stop, such as capacity, networking, or a broken image.

Plan for the documented rule, not the measurement. The API scheduled every version about 64 days after its successor. The documentation says 30. Build to the documented number, and treat any extra weeks as slack you did not count on. The schedule is GitHub's to change.

What we could not test

• Enterprise Cloud and data residency. We tested one free organization. The changelog names Enterprise Cloud, but we did not run a runner there.
• Supervisors. We started run.sh directly. We did not test how a systemd service, the runner's svc.sh wrapper, or ARC reacts to exit code 0 or 7. The svc.sh behavior above comes from the source code.
• Every version. We ran four versions. We did not test 2.329.0 itself or any other expired version besides 2.335.1.
• GitHub Enterprise Server. GitHub says it is not affected, and we did not test it.
• How the dates are set. The 62 to 70 day pattern comes from one snapshot of 18 versions. It is a measurement, not a published rule, and it can change.
• Time. All of this is one day of data: September 30, 2026, the day after enforcement started.

Summary

The registration minimum and the runtime rule answer different questions, and GitHub's June changelog says so. 2.329.0 only decides whether a runner can register, and that failure is loud. The rule to plan around is the moving one: each version stops receiving jobs some time after the next release, 30 days by the documentation and about 64 days in the API's current schedule. In our test, the runner past its date registered, connected, logged one error and exited with code 0, while its job waited in the queue. In our direct run.sh test, one environment variable changed that to exit code 7.

Find the versions your hosts, images, and scale sets run. Check them against the deprecations API. Rebuild pinned images on every runner release, and alert on the end date before the queue tells you.

The cheapest number in our comparison, though, is one you cannot rely on. Wasabi comes out at $15.60 for that 50 TB month, and that month is outside Wasabi's free-egress policy. Every provider in this category has fine print, and it is different for each one: a ratio to your stored data, a fair-use policy, a price per read, a minimum storage duration, a bundle that runs out. We put nine storage offerings from eight providers into a small calculator with their September 2026 prices and the main fine print applied, and ran four workloads through it.

The-DevOps-Daily/object-storage-egress-calc on GitHub

TLDR

• For egress-heavy workloads, the zero-egress providers win by one or two orders of magnitude. Our 50 TB media month: R2 $62, Hetzner $76, Tigris $90, against $4,379 on S3 at list price.
• Wasabi's free egress is a policy, not a price. Egress should stay at or below the data you store. Above that there is no overage rate; Wasabi reserves the right to limit or suspend the service.
• Backblaze B2 is free only up to 3x your stored data, then $0.01/GB, unless you go through a partner CDN.
• R2 egress is free, but reads are not. In the media workload, GET requests were about half of the R2 bill.
• Minimum storage durations beat egress for backups. With 30-day retention, Wasabi's 90-day minimum triples its storage bill and puts it next to S3 at the bottom of the table.
• AWS changed the math in November 2025 with flat-rate CloudFront plans. A direct per-GB comparison against S3 is no longer the whole story.

Prerequisites

• A rough idea of your workload per month: data stored, data served to the internet, number of GET and PUT requests, and how long objects live.
• Python 3, if you want to run the calculator. It has no dependencies.

The headline prices

This is what the pricing pages lead with, as of 28 September 2026. Every price comes from the provider's own pages. The repository's providers.json links the sources for every price the calculator uses, and other claims link their sources where they appear. Scenario totals are the calculator's estimates, not quotes.

For context, the other two big clouds are in the same place as S3: Google Cloud Storage lists $0.12/GiB for the first 10 TiB to most destinations, and Azure includes 100 GB a month, then charges $0.087/GB for the next 10 TB from North America or Europe over its Premium Global Network.

Several of these prices are new. Backblaze went from $6 to $6.95/TB on 1 May 2026 and made standard API calls free (event notifications still cost). Wasabi went from $6.99 to $7.99/TB on 1 July. Hetzner raised its base price from EUR 4.99 to EUR 6.49 on 1 April. Storj moved to its "Simplified" pricing on 1 July. Comparisons written in 2025 are out of date.

Four workloads

The calculator prices one month for each provider and prints a note wherever the workload breaks a rule. Here are the four workloads from scenarios.sh, with the results exactly as recorded in runs/scenarios.txt:

Media: 2 TB stored, 50 TB egress, 100M GETs per month

List prices on 28 September 2026, from the calculator in the linked repository. Wasabi's figure breaks its free-egress policy (egress above stored data), so it is not a price you can rely on. AWS S3 Standard for the same month is $4,379.20 at list price, off this scale.

Two things stand out. The order changes completely between workloads: Wasabi is cheapest for media and second most expensive for backups. And the reasons it changes are all in the fine print, not in the headline egress price. The rest of this post goes through that fine print, one rule at a time.

Wasabi: free egress is a policy, not a price

Wasabi's pricing FAQ is explicit: if your monthly egress is at or below your active storage, your use case "is a good fit" for free egress. If it is above, it "is not a good fit", and if that happens "on a regular basis, we reserve the right to limit or suspend your service." There is no overage price to pay instead.

So Wasabi is a poor fit for workloads that regularly serve more than they store, such as downloads, media and most public assets, however good the number looks. The calculator prints the price with a note rather than hiding the row, because the policy is judged over time and a single heavy month is not the same as a pattern.

Wasabi also has three minimums: a 1 TB monthly minimum (the side project pays $7.99 for 50 GB), a 4 KB minimum object size, and a 90-day minimum storage duration, which is the next rule.

Minimum storage duration: why Wasabi loses on backups

A backup bucket with 30-day retention deletes every object after 30 days. On a provider with a 90-day minimum, each of those objects is billed as if it stayed 90 days. In a steady state that is three times the storage you actually hold, and it is why Wasabi's backup month costs $234.00 instead of about $78.

Backups: 10 TB stored, 30-day retention, 200 GB restored

List prices on 28 September 2026, from the calculator. Wasabi bills each object for 90 days, so 30-day retention is charged at about three times the stored data. Storj and R2 Infrequent Access have 30-day minimums, which 30-day retention just meets.

The minimums across the providers here: Wasabi 90 days, and overwrites count as deletes. Storj 30 days. R2 Infrequent Access, Tigris Infrequent Access and DigitalOcean Cold Storage 30 days, Tigris Archive 90 days. None for S3 Standard, R2 Standard, B2 or Tigris Standard. Any bucket that churns, such as a build cache, CI artifacts or daily exports, pays the minimum on every object, whatever the egress price.

Backblaze B2: free up to 3x what you store

B2's egress is free "up to 3x their average monthly storage", measured in byte-hours over the month, then $0.01/GB. It is unlimited only when you download to or through partner CDNs and compute providers (the page names Fastly, Cloudflare, bunny.net, CacheFly, CoreWeave, Equinix Metal, Vultr and phoenixNAP), or on B2 Overdrive, which needs a multi-petabyte commitment.

That is why B2 does well on backups and less well on the downloads site: 500 GB stored gives 1.5 TB of free egress, and the other 3.5 TB costs $35. The overage rate is still a ninth of S3's, and a partner CDN in front removes it.

Cloudflare R2: egress is free, reads are not

R2 charges nothing for egress, with no ratio and no policy on the pricing page. It does charge for operations: every GET and HEAD is a Class B operation at $0.36 per million after the free 10 million a month. In the media workload, 100 million GETs cost $32.40 of the $62.25 total, more than the storage.

R2 Infrequent Access is a different product. It adds a $0.01/GB retrieval fee on every read or copy, a 30-day minimum, and no free tier. That is fine for backups, and after S3 it is the most expensive choice here for anything people download: $610.90 for the media month, because 50 TB of reads is $500 of retrieval.

Two more details. The r2.dev public URL is rate-limited and meant for development; public buckets in production should use a custom domain. And Cloudflare's service terms say that, unless you are an Enterprise customer, serving video and other large files through its CDN requires one of its paid services, such as the Developer Platform (which includes R2), Images or Stream. Check those terms before you put a large-file origin hosted elsewhere behind Cloudflare.

Bundles: a fixed amount of free egress, then per GB

Some providers sell a monthly bundle instead of a zero price:

• DigitalOcean Spaces: $5 a month includes 250 GiB of storage and 1,024 GiB of outbound transfer shared across all buckets, then $0.02/GiB stored and $0.01/GiB transferred. The built-in CDN is included, and its traffic counts against the same allowance. Transfer from Spaces to Droplets in the same datacenter group is free, so a Spaces bucket next to DigitalOcean compute only pays for what leaves for the internet.
• Hetzner: $7.99 (EUR 6.49) a month includes about 1 TB of storage and 1 TB of egress, accrued hour by hour, so a 30-day month holds 1.08 TB of egress. Extra egress is only $1.20/TB, which is why Hetzner is right behind R2 in the media workload. You pay the base price for any hour you have a bucket, even an empty one, and unused quota does not carry over.
• Storj: no bundle and no free egress. $7/TB for storage and $7/TB for egress from the first byte, with a $5 minimum invoice (accounts that pay in USDC are exempt).

Bundles are easy to predict and cheap for small projects. DigitalOcean's $5 covers the side project workload in full. What matters is where the included transfer runs out: DigitalOcean's overage is $0.01 per GiB, the same number as B2's $0.01 per GB above 3x, and it is the main line on its media bill.

Small objects pay more than their size

Several providers bill a minimum object size: Hetzner 64 KB, Storj 50 kB, Wasabi 4 KB, DigitalOcean 4 KiB (128 KiB for Cold Storage, on storage and on every read). A bucket of 10 KB thumbnails is billed at 6.4 times its real size on Hetzner. The calculator does not model this, so if your average object is small, check it against the list before you trust the result. R2, Tigris, S3 Standard and B2 state no minimum.

AWS: the free 100 GB is per account, and CloudFront changed the math

AWS's 100 GB of free internet egress is shared by the whole account, across all services and Regions, and the volume tiers also count all services together. A bucket in an account that also runs busy EC2 instances may never see the free 100 GB.

The bigger change is CloudFront. Transfer from S3 to CloudFront has long been free, and since 18 November 2025 CloudFront has flat-rate plans with "no overage charges": Pro is $15 a month for a 50 TB allowance and 10 million requests, Business $200 for 50 TB and 125 million requests. Above the allowance, AWS says the first spike up to 3x will not affect service that month, but if you keep exceeding it without upgrading, "your traffic delivery might be adjusted", for example served from fewer or more distant edge locations.

We did not model this, because the bill then depends on cache hit rates and on how AWS treats sustained overuse. But it means "S3 costs $4,379 for 50 TB" is only true if you serve straight from the bucket. With S3 behind a flat-rate plan, the AWS number for a cache-friendly workload moves toward storage plus the plan fee, and that belongs in the same comparison as R2.

Bytes you pay for but never deliver

AWS bills the bytes it sent when a client aborts a download, with its own example of 3 GB billed for a request cut off at 2 GB. DigitalOcean bills early disconnects "up to the full object size". For large downloads behind flaky networks, or players that seek around video files, billed egress can be noticeably higher than delivered bytes. Free-egress providers avoid this one by design.

Units and rounding

The providers here do not agree on what a gigabyte is. AWS, Tigris, DigitalOcean and Wasabi bill binary units (GiB, or 1 TB = 1,024 GB), Storj and Hetzner decimal ones; R2 and B2 do not say. A GiB is about 7.4% larger than a decimal GB, and a TiB about 10% larger than a decimal TB. R2 rounds each line up to the next whole billing unit, and Storj rounds usage up to the whole GB. The calculator ignores all of this, which is fine for choosing a provider and not fine for forecasting a bill to the cent.

Run your own numbers

object-storage-egress-calc

$ python3 calc.py --storage-gb 500 --egress-gb 5000 --gets 20000000 --puts 10000
500 GB stored, 5,000 GB egress, 20,000,000 GETs, 10,000 PUTs, objects live 365 days (prices as of 2026-09-28)
  Wasabi (pay-as-you-go)           $      7.99  billed for 1024 GB minimum; egress above stored data: outside Wasabi's free egress policy (no overage price; service may be limited)
  Cloudflare R2 Standard           $     10.95
  Hetzner Object Storage (USD)     $     12.69
  Tigris Standard                  $     19.85
  Backblaze B2                     $     38.41
  Storj Standard                   $     38.50
  DigitalOcean Spaces              $     49.76
  Cloudflare R2 Infrequent Access  $     73.09
  AWS S3 Standard (us-east-1)      $    460.55

--lifetime-days applies minimum storage durations, and every price in providers.json has its source URL, so you can update a number when a provider changes it.

How to pick

What we could not include

• CDN plans in front of storage, such as CloudFront's flat-rate plans or partner CDNs in front of B2. They can change the answer, but they depend on cache hit rates.
• Committed-use and enterprise pricing. Contracts and volume discounts are outside this comparison.
• Scaleway. Its page shows 75 GB of free egress and "EUR 0.01" after that, but does not print the unit clearly, so we left it out of the calculator rather than guess.
• Performance. This is a price comparison. We did not measure latency, throughput or availability.

Summary

Zero egress is real, and for workloads that serve more than they store it changes the bill by one or two orders of magnitude compared with S3 list prices. But every provider pays for it somewhere else: a policy instead of a price, a ratio to storage, a charge per read, a minimum duration, a bundle that runs out. The right answer depends on the shape of your workload, not on the headline egress price, so run your own numbers with the fine print switched on before you move a bucket.

The changelog makes it sound like a lot: a new config format, removed CRI APIs, registry mirrors on their way out, schema 1 images gone. We wanted to know which of those actually bite, so we ran the upgrade. Same old node config, same data directory, containerd 1.7.36, then 2.3.6, then 2.4.1, recording everything each version said. The config still loaded. The mirrors still worked. What broke was pulling schema 1 images, even on the upgraded node. The catch is that nothing pulls on an upgraded node until something forces it, so the failure tends to appear later, on the next fresh node.

The-DevOps-Daily/containerd-2-upgrade-check on GitHub

TLDR

• containerd 1.7 support ends in September 2026, and Kubernetes dropped containerd 1.x support in 1.36.0. Recent EKS AL2023 node images already ship containerd 2.2.7.
• An old version 2 config still loads on 2.x. containerd migrates it in memory at startup and logs a warning. containerd config migrate writes version = 4, and it copies the dead sections into the new file.
• Inline registry mirrors still work on 2.3 and 2.4. Our logging mirror received five requests per pull on all three versions. Their removal date has moved from v2.1 to v2.4 to v2.7 depending on which version you ask.
• Pulling schema 1 images is the real break. An image pulled and converted under 1.7 is still in the image store after the upgrade, but any schema 1 pull on 2.x fails: schema 1 image manifests are no longer supported. It shows up when a node has to pull: a fresh node, a garbage-collected image, or imagePullPolicy: Always.
• A read-only check script in the repository reports all of this for a real node.

Prerequisites

• Kubernetes nodes that run containerd. (We did not test Docker Engine hosts.)
• Root access to one node, to read its config and run ctr.
• kubectl if you want the cluster-wide view.

Why now

Three dates line up:

• containerd's own release table lists 1.7 as an LTS release with end of life in September 2026, extended to that date for Kubernetes 1.30 to 1.32 on GKE. See RELEASES.md.
• Kubernetes agreed that the last release to support containerd 1.x is 1.35, with support dropped in 1.36.0. The kubelet exposes a kubelet_cri_losing_support metric: when it appears with a version label of 1.36.0, that node's containerd is too old for the next Kubernetes version. See the Kubernetes v1.35 sneak peek.
• Managed node images have moved already. The EKS AL2023 AMI release v20260923 ships containerd 2.2.7. On managed Kubernetes the containerd version usually comes with the node image, so check your node image's release notes before you upgrade a pool.

To see what every node in a cluster runs today:

kubectl get nodes -o custom-columns=NAME:.metadata.name,RUNTIME:.status.nodeInfo.containerRuntimeVersion

The experiment

The demo runs three containerd versions one after the other, against the same --root and --state directories, the way an in-place node upgrade reuses the node's data. Each one starts with the same config, written the way many Kubernetes nodes were set up in the 1.x era:

version = 2

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.9"

  [plugins."io.containerd.grpc.v1.cri".containerd]
    snapshotter = "overlayfs"
    default_runtime_name = "runc"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
        SystemdCgroup = true

  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
    endpoint = ["http://127.0.0.1:5055"]

[plugins."io.containerd.runtime.v1.linux"]
  shim = "containerd-shim"
  runtime = "runc"

The docker.io mirror points at a tiny local server that logs every request and answers 404, so containerd falls back to Docker Hub. That turns "is this setting still honoured?" into a number we can count.

One data directory, three versions

1. containerd 1.7.36 old config, schema 1 pull
2. containerd 2.3.6 LTS, same root
3. containerd 2.4.1 latest, same root
4. runs/ every command and warning

For each version the script records ctr deprecations list, a CRI image pull through crictl, how many requests reached the mirror, the schema 1 steps, and the daemon's warnings. The terminal output below comes from those recordings; the commands are shown without the demo's socket flags, and trailing spaces are trimmed.

Finding 1: the old config still loads

Both 2.x versions started with the version 2 config. They converted it in memory and said so once, at startup:

containerd 2.3.6 daemon log

time="2026-09-28T23:41:57+03:00" level=warning msg="Configuration migrated from version 2, use `containerd config migrate` to avoid migration" t="31.333µs"

That is friendlier than the changelog suggests, and it is also a trap: a config that loads is not a config that is doing what it says. Running containerd config migrate with 2.3.6 or 2.4.1 printed a complete config with version = 4, and two details in it matter:

• The runtime v1 section came through unchanged as [plugins.'io.containerd.runtime.v1.linux']. Runtime v1 was removed in 2.0, so that block now configures nothing, silently.
• The inline mirrors came through too, moved to the new io.containerd.cri.v1.images section.

So config migrate gives you the new layout, not a clean config. Read the output and delete what no longer exists before you ship it.

Finding 2: the mirrors still work, and the deadline keeps moving

The inline registry.mirrors setting has been deprecated since containerd 1.5. We expected 2.x to ignore it. It did not. Each version pulled a different busybox tag through the CRI, and the logging mirror recorded five requests per pull. Here is 2.4.1:

containerd 2.4.1

$ crictl pull docker.io/library/busybox:1.38.0
time="2026-09-28T23:42:10+03:00" level=warning msg="Config \"/etc/crictl.yaml\" does not exist, trying next: \"/home/biliev/projects/containerd-2-upgrade-check/bin/crictl.yaml\""
Image is up to date for sha256:d2482869a6b838d3b4c9cad0c8085c06277909482771eb525d298fc3a7d07927
# the local mirror's request log for that pull
requests the docker.io mirror received: 5
HEAD /v2/library/busybox/manifests/1.38.0?ns=docker.io
GET /v2/library/busybox/manifests/sha256:fd7dc98638c8e305f4dc34e979f1c0fdfdcaeb0fbf8fcff77ae834b6da3d7e6e?ns=docker.io
GET /v2/library/busybox/manifests/sha256:365a051f12e05767b598e643676f14a450fb678a75ccf2beb0052c95d5c73b83?ns=docker.io

What changed is the removal date in the warning. Each version records it with ctr deprecations list:

A deadline that has moved twice is still a deadline. There is also a side effect today: with inline mirrors in the config, 2.x logs Found 'Registry.Mirrors' in CRI config which is incompatible with transfer service ... Falling back to local image pull mode. In other words, keeping the old setting also keeps the old pull path.

The replacement is config_path plus one hosts.toml per registry:

# /etc/containerd/config.toml (version 3 or later)
[plugins.'io.containerd.cri.v1.images'.registry]
  config_path = '/etc/containerd/certs.d'

# /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://registry-1.docker.io"

# Give "resolve" (tag lookups) only to mirrors you trust.
[host."https://mirror.gcr.io"]
  capabilities = ["pull"]

Finding 3: schema 1 fails, but not where you are looking

Docker image manifest schema 1 is the format images were pushed in before Docker 1.10 introduced schema 2 in 2016. Some are still served that way. scripts/find-schema1.sh asks for newer formats first, the way a current client does, and docker.io/library/busybox:1.24, gcr.io/google-containers/busybox:1.24, gcr.io/google-containers/pause:0.8.0 and quay.io/coreos/etcd:v2.2.5 still came back as schema 1. containerd 1.7 still pulls them, converting on the way and warning about it; our 1.7.36 run did that with busybox:1.24. containerd 2.0 disabled schema 1 pulls (2.0.x can turn them back on with CONTAINERD_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE=1), and 2.1 removed them, according to containerd's deprecation table.

On 1.7.36 the pull worked, and the deprecation was recorded:

containerd 1.7.36

# the demo keeps the last two lines of ctr's progress output
$ ctr -n demo images pull --platform linux/amd64 docker.io/library/busybox:1.24
unpacking linux/amd64 sha256:dc53f06b7ab95434d92cd20c00409fe5b5ded8d3b68f861e709f2e4b49067242...
done: 323.503075ms
$ ctr deprecations list
ID                                                LAST OCCURRENCE                   MESSAGE
io.containerd.deprecation/cri-registry-mirrors    2026-09-28T20:41:42.439195342Z    The `mirrors` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.1. Use `config_path` instead.
io.containerd.deprecation/pull-schema-1-image     2026-09-28T20:41:54.047880144Z    Schema 1 images are deprecated since containerd v1.7 and removed in containerd v2.0. Since containerd v1.7.8, schema 1 images are identified by the "io.containerd.image/converted-docker-schema1" label.

Then we upgraded the same data directory. On 2.3.6 and on 2.4.1 the converted image was still in the image store, with its label. (It is an amd64 image and the demo ran on an arm64 Pi, so we listed it but did not run it.) Pulling it again failed:

containerd 2.4.1, same data directory

$ ctr -n demo images ls
time="2026-09-28T23:42:17+03:00" level=warning msg="DEPRECATION: The `mirrors` property of `[plugins.\"io.containerd.grpc.v1.cri\".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.7. Use `config_path` instead."
REF                            TYPE                                       DIGEST                                                                  SIZE      PLATFORMS   LABELS
docker.io/library/busybox:1.24 application/vnd.oci.image.manifest.v1+json sha256:dc53f06b7ab95434d92cd20c00409fe5b5ded8d3b68f861e709f2e4b49067242 661.3 KiB linux/amd64 io.containerd.image/converted-docker-schema1=sha256:8ea3273d79b47a8b6d018be398c17590a4b5ec604515f416c5b797db9dde3ad8
# the demo keeps the last two lines of ctr's output
$ ctr -n demo images pull --platform linux/amd64 docker.io/library/busybox:1.24
time="2026-09-28T23:42:17+03:00" level=warning msg="DEPRECATION: The `mirrors` property of `[plugins.\"io.containerd.grpc.v1.cri\".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.7. Use `config_path` instead."
ctr: schema 1 image manifests are no longer supported: invalid argument

That is the whole trap. The pull fails everywhere, the upgraded node included. But an upgraded node that already has the image on disk has no reason to pull it, so nothing visible happens on upgrade day. The failure appears later, whenever a node has to pull:

• A new node from the autoscaler or a node pool upgrade has an empty image store and has to pull.
• Image garbage collection on a busy node deletes the cached copy, and the next pod start pulls again.
• imagePullPolicy: Always, or a rollback to an old tag that the new nodes have never pulled.

Each of these looks like a random ImagePullBackOff on one node, days after an upgrade that "went fine".

To find the images before they find you, look for the label containerd adds on conversion. The check script does this across every namespace, which on a Kubernetes node includes k8s.io. The fix is to stop depending on the old manifest: rebuild the image and push it with a current tool, which writes a schema 2 or OCI manifest, or move to a newer tag that already has one. Retagging the old image does not change its manifest.

Check a real node

scripts/check-node.sh is read-only. Run it as root on a node before you upgrade. This is its output against the demo's 1.7.36 daemon after the schema 1 pull, with the paths overridden for the demo:

check-node.sh

$ sudo CONTAINERD_CONFIG=configs/old-node-config.toml CONTAINERD_ADDRESS=/tmp/ctd-check/containerd.sock CTR=bin/containerd-1.7.36/ctr scripts/check-node.sh
== containerd version
  Version:  v1.7.36
  Revision: 2892c2042ee7fbd3be0e5bdc675e07b5acedb0bf

== config: configs/old-node-config.toml
config version: 2
WARN inline registry.mirrors: deprecated; move to config_path and hosts.toml files
WARN runtime v1 section: removed in 2.0; containerd 2.x ignores it

== deprecations containerd has already seen (1.7.x and later)
ID                                                LAST OCCURRENCE                   MESSAGE
io.containerd.deprecation/cri-registry-mirrors    2026-09-28T20:48:14.827082624Z    The `mirrors` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.1. Use `config_path` instead.
io.containerd.deprecation/pull-schema-1-image     2026-09-28T20:48:19.606197042Z    Schema 1 images are deprecated since containerd v1.7 and removed in containerd v2.0. Since containerd v1.7.8, schema 1 images are identified by the "io.containerd.image/converted-docker-schema1" label.

== images that were converted from schema 1 (they keep working; pulling them again on 2.x fails)
k8s.io  docker.io/library/busybox:1.24

ctr deprecations list is the most useful line in there. It lists the deprecations containerd has observed on that node: config problems it saw at startup, and deprecated features something actually used, such as a schema 1 pull. It is not a complete audit, but it reflects the node, not the changelog.

An upgrade checklist

1. List runtime versions across the cluster with the kubectl command above, and watch for kubelet_cri_losing_support.
2. Run the check on one node per node pool or image type.
3. Rebuild and push every schema 1 image it finds, so it gets a schema 2 or OCI manifest, and search your manifests for old tags the check cannot see because no node has pulled them yet.
4. Move mirrors to config_path and hosts.toml.
5. Run containerd config migrate, then delete the sections for things that no longer exist, such as runtime v1, before you ship the new config.
6. Upgrade one node pool, then drain a node onto a fresh one, so the new node has to pull every image from scratch. That is where schema 1 fails, so test it on purpose.

What we could not test

• The CRI v1alpha2 removal. containerd 2.0 removed the old CRI API. Any kubelet recent enough to still be supported uses CRI v1, but older tools that speak v1alpha2 will stop working. We had no such client to show it.
• A real cluster. The recordings come from standalone daemons on a Raspberry Pi 4 (arm64). The schema 1 images are amd64 only, so we pulled them with --platform linux/amd64 and did not run them.
• aufs, custom runtimes, NRI plugins and GPU setups. A node that relies on any of these needs its own test.
• Every managed platform. Node images decide the containerd version and its defaults. Check your provider's release notes, as we did for EKS.

Summary

The containerd 1.7 to 2.x upgrade is gentler than its changelog. An old config loads, deprecated mirrors keep working for now, and images already on the node stay in the image store. That gentleness is exactly what makes the one real break dangerous: schema 1 pulls fail on every node, but you only see it when a node has to pull, which usually means the fresh node next week, not the one you just upgraded. Run the check, rebuild the old images, clean up the config by hand after config migrate, and test the upgrade on an empty node before the node pool does it for you.

It produced three pull requests that we merged, each for about two cents of model tokens. It also produced one that passed the test suite and was still wrong, which is the most useful part of this post. Below is the whole setup, the recorded runs, the check we added after the wrong one, what the permission rules did and did not stop, and the gotchas that cost us time.

The-DevOps-Daily/do-agent-issue-to-pr on GitHub

TLDR

• One script turns an issue into a pull request. It creates a Harness Runtime session, clones the repository into the microVM, prompts the agent once with no human attached, reads the diff back out, runs the tests again, and opens the pull request with credentials the agent never sees.
• The model runs on DigitalOcean. HARNESS_INFERENCE_MODEL: deepseek-v4-pro plus a model access key; no Anthropic or OpenAI account involved.
• Three issues, three merged fixes, about $0.02 to $0.03 of model tokens each at DigitalOcean's list price.
• One pull request passed its tests and was wrong. The agent added tests where unittest never runs them. The script now refuses a change that touches tests/ without raising the number of tests that run.
• Tool rules are not the boundary. OpenCode's edit and write tools were refused by our policy, so the agent wrote files with bash instead. The sandbox, the missing credentials and the egress allowlist are what actually held.

Prerequisites

• A DigitalOcean account with Managed Agents (public preview since 21 September 2026) and an API token that can use it.
• A DigitalOcean model access key for serverless inference.
• doctl 1.175.0 or later and the GitHub CLI.
• A repository with tests the agent can run. Ours is a deliberately small Python module so the runs are easy to read.

The shape of it

The target is a small duration parser, parse_duration("1h30m"), with three open issues: combined durations return the wrong number, days are not supported, and unknown units are silently ignored. All three are real bugs in the code as first committed, and the existing tests pass on all of them.

The design rule was simple: whatever the agent can touch, it cannot publish. It edits files in a microVM. Everything that talks to GitHub with write access happens outside, in a script we control.

One issue, one session, one pull request

1. GitHub issue labelled agent
2. issue-to-pr.sh holds the GitHub token
3. Harness Runtime OpenCode in a microVM
4. DO inference DeepSeek V4 Pro
5. Pull request a person reviews

The agent spec

A session is described in YAML. This is the file every run uses:

name: issue-to-pr
agent: opencode
size: mars-2vcpu-4gb
idle_timeout: 10m
env:
  HARNESS_INFERENCE_MODEL: deepseek-v4-pro
secrets:
  HARNESS_INFERENCE_API_KEY: ${DO_INFERENCE_KEY}
# Naming a host turns egress into a deny-by-default allowlist. The platform
# adds GitHub (for the clone) and the model endpoint on its own.
egress:
  - api.github.com
permissions:
  default: deny
  filesystem:
    mode: workspace-write
  rules:
    - tool: file.read
      action: allow
    - tool: file.write
      match: { path: '/workspace/**' }
      action: allow
    - tool: bash
      action: allow
    # Last matching rule wins, so these override the allow above.
    - tool: bash
      match: { command: 'git push*' }
      action: deny
    - tool: bash
      match: { command: 'curl *' }
      action: deny

What each part is for:

• agent: opencode picks the OpenCode adapter. Managed Agents also has built-in adapters for Claude Code, Codex CLI, Hermes and LangGraph.
• HARNESS_INFERENCE_MODEL and HARNESS_INFERENCE_API_KEY route the model through DigitalOcean serverless inference. The key goes under secrets, which are stored separately and never returned by the API. ${DO_INFERENCE_KEY} is filled in from your shell when you create the session.
• egress is open by default. Naming a single host switches the session to an allowlist, and the platform adds the hosts the adapter needs.
• permissions start from deny. For native actions the last matching rule wins, which is why the two bash denies come after the broad bash allow.

You can check a policy before paying for a session. The validate endpoint returns a verdict per rule, and all five of ours came back exact (response):

curl -X POST https://api.digitalocean.com/v2/agents/sessions/policy/validate \
  -H "Authorization: Bearer $DIGITALOCEAN_ACCESS_TOKEN" \
  -H "Content-Type: application/x-yaml" \
  --data-binary @agent.yaml

It rejects a spec that still has a ${VAR} placeholder in it, so validate a copy with the secret filled in or replaced by a dummy value.

The script, step by step

scripts/issue-to-pr.sh is about 120 lines of bash. The parts that matter:

1. A fresh microVM per issue.

doctl harness-runtime create --spec "$root/agent.yaml" --name "$session" \
  --wait-timeout 300 >/dev/null

2. Clone from outside the agent loop. The session has no GitHub credentials, so it could not clone a private repository by itself. exec runs a command in the sandbox directly, as root, so the checkout is handed to the agent user the model runs as:

doctl harness-runtime exec "$session" -- sh -c \
  "git clone -q https://github.com/$repo $workdir && chown -R agent:agent $workdir"

3. One headless run. prompt sends a single prompt, waits for the run to finish, and exits 0, 1 or 124 on timeout. --on-hitl reject refuses anything the policy would otherwise stop to ask a person about, because nobody is there to answer:

doctl harness-runtime prompt "$session" --on-hitl reject --timeout 1200 \
  -o json - <<<"$prompt" >"$out/answer.json" 2>"$out/progress.log"

The prompt includes the issue title and body and four rules: change only what the issue needs, add tests under tests/, run the test suite, and do not commit or push. With -o json the answer comes back with the run status and token counts.

4. Check the work instead of trusting the summary. The script reads the diff out of the sandbox and runs the tests there itself. It also counts the tests before and after, for a reason the next sections explain:

doctl harness-runtime exec "$session" -- sh -c \
  "git -c safe.directory=$workdir -C $workdir add -A && git -c safe.directory=$workdir -C $workdir diff --cached" \
  >"$out/change.patch"

5. Open the pull request with our credentials. The patch is applied to a fresh clone on the machine running the script, pushed to a branch named after the session, and opened with gh pr create. The agent's closing summary becomes the pull request body.

6. Remove the session. An EXIT trap saves the session's event log and removes the session, so a failed run does not leave a sandbox behind. It ignores errors from both commands, so check doctl harness-runtime list now and then.

A real run

This is issue #3, "Reject durations with unknown units", exactly as the terminal printed it:

issue-to-pr.sh

$ scripts/issue-to-pr.sh 3
12:58:31 issue #3: Reject durations with unknown units
12:58:31 creating session issue-3-1790600274
12:58:51 tests before: 8
12:58:51 running the agent
12:59:36 tests pass in the sandbox: 8 before, 10 after
remote: 
remote: Create a pull request for 'agent/issue-3-1790600274' on GitHub by visiting:        
remote:      https://github.com/The-DevOps-Daily/do-agent-issue-to-pr/pull/new/agent/issue-3-1790600274        
remote: 
12:59:43 opened https://github.com/The-DevOps-Daily/do-agent-issue-to-pr/pull/7
https://github.com/The-DevOps-Daily/do-agent-issue-to-pr/issues/3#issuecomment-5870355576
12:59:49 removing session issue-3-1790600274

The change it produced, pull request #7, checks the whole string before parsing it:

 _UNITS = {"s": 1, "m": 60, "h": 3600, "d": 86400}
 _PART = re.compile(r"(\d+)([smhd])")
+_VALID = re.compile(r"(\d+[smhd])+")

 def parse_duration(text: str) -> int:
     text = text.strip().lower()
     if not text:
         raise ValueError("empty duration")
+    if not _VALID.fullmatch(text):
+        raise ValueError(f"not a duration: {text!r}")

It also added two tests, one for "1h30x" and one for "abc". A reviewer would point out that the old "no parts" check below it is now unreachable, which is the kind of note that belongs in a review, not a reason to reject the fix. We merged it. Issue #1 went the same way: a one-character fix (total = to total +=) and two tests, merged as #4.

The pull request that passed and was wrong

Issue #2 asked for days, "7d" and "1d12h". The first run changed the code correctly and reported that all tests passed, and our script, which at that point only checked that the suite passed, opened pull request #5.

The tests it added looked like this at the bottom of the file:

if __name__ == "__main__":
    unittest.main()

    def test_days(self):
        self.assertEqual(parse_duration("7d"), 604800)

They are inside the if __name__ block, not inside the test class, so unittest never collects them. The agent's own summary said "All 8 tests pass (6 pre-existing + 2 new ones)". The test run our script did afterwards in the same sandbox printed Ran 6 tests.

A person reading the diff caught it. The script could not, because "the tests pass" was the only thing it checked. So it now counts how many tests run before and after the agent's change, and refuses to open a pull request if the agent touched tests/ but the count did not go up. It is a coarse check, since it cannot tell which new test ran, but it catches this failure:

count_tests() {
  doctl harness-runtime exec "$session" -- sh -c \
    "cd $workdir && python3 -m unittest -q 2>&1" | sed -n 's/^Ran \([0-9]*\) tests\{0,1\}.*/\1/p'
}

The second run of issue #2 added four tests that ran (6 before, 10 after), but failed to push because pull request #5's branch still existed; each run now pushes to its own branch. The third run added two tests that ran and became #6, which we merged.

The lesson is not specific to this platform. An agent's report of its own work is a claim. Check the claim with something the agent did not write, and keep a person on the merge button.

What the policy stopped, and what it did not

The network allowlist held. With egress limited to api.github.com, we ran these inside a session with doctl harness-runtime exec (recorded here):

inside the session

$ python3 -c "import urllib.request as u; print(u.urlopen('https://example.com', timeout=8).status)"
urllib.error.URLError: <urlopen error Tunnel connection failed: 404 Not Found>
$ python3 -c "import urllib.request as u; print(u.urlopen('https://pypi.org/simple/', timeout=8).status)"
urllib.error.URLError: <urlopen error Tunnel connection failed: 404 Not Found>
$ python3 -c "import urllib.request as u; print(u.urlopen('https://github.com', timeout=8).status)"
200

GitHub and the model endpoint still work, because the platform adds those hosts itself. An agent talked into leaking something by a malicious issue cannot reach an arbitrary server, though it can still reach the hosts on the list.

The tool rules did not do what we expected. The session log for issue #3 shows OpenCode's own editing tools being refused:

doctl harness-runtime logs issue-3-1790600274

▸ edit
[2026-09-28T12:58:59Z] run.tool_call_completed
  ✗ The user has specified a rule which prevents you from using this specific tool … (22ms)

The file.write rule for /workspace/** did not cover OpenCode's edit and write tools in these runs, so default: deny refused them. The agent did not stop. It changed the files through bash instead, with sed -i and cat > heredocs, which we had allowed. Every merged fix in this post was written that way.

DigitalOcean's docs warn about exactly this: "Rules match literally ... A denied action doesn't block the goal behind it." If bash is allowed, the agent can do anything bash can do inside the sandbox, whatever the other rules say. Treat tool rules as a way to shape the agent's behaviour, and treat the microVM, the missing credentials and the egress allowlist as the security boundary.

What it cost

Token counts come from the prompt command's JSON output; the price is DigitalOcean's list price for DeepSeek V4 Pro, $1.74 per million input tokens and $3.48 per million output tokens.

Two smaller costs sit on top. Each new session starts with a short readiness run, which used between 235 and 4,587 input tokens in ours. And the sandbox is billed while it exists: a Medium session lists at about $0.060 an hour under the current billing rule (25% of the allocated vCPUs plus peak memory), and by the script's timestamps each of ours existed for under two minutes, so compute came to a fraction of a cent per issue. These figures are derived from token counts and list prices; the new usage had not reached the invoice when we wrote this.

Gotchas we hit

• doctl wants an Anthropic key for Claude Code, even on DigitalOcean inference. doctl harness-runtime create refused a claude-code spec that used HARNESS_INFERENCE_*, asking for ANTHROPIC_API_KEY. In doctl's source, prepareClaudeCodeStart resolves that key for every claude-code session and checks it against Anthropic. Posting the same YAML to POST /v2/agents/sessions with Content-Type: application/x-yaml created the session (both recorded). OpenCode has no such check.
• Model access depends on your inference tier. Our key got "403 this model is not available for your subscription tier" for the Anthropic and OpenAI models we tried, both in a direct chat completion and inside a Claude Code session. Open models such as DeepSeek V4 Pro worked. Test the model you want with a plain chat completion before building on it.
• --gh-repo did not clone anything for us. Without a GitHub connection set up through doctl harness-runtime auth, the workspace stayed empty. Cloning with exec is explicit and works for public repositories.
• exec runs as root. Files it creates belong to root, and the agent could not edit them until we added chown. Git run as root then refuses the agent-owned checkout as "dubious ownership" unless you pass -c safe.directory.
• A closed pull request leaves its branch. Name branches after the session, not the issue, or the next attempt cannot push.

Wiring it to GitHub Actions

The repository includes a workflow that runs the same script when an issue gets the agent label:

on:
  issues:
    types: [labeled]

jobs:
  agent:
    if: github.event.label.name == 'agent'
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: digitalocean/action-doctl@v2
        with:
          token: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}
      - run: scripts/issue-to-pr.sh "${{ github.event.issue.number }}"
        env:
          GH_TOKEN: ${{ github.token }}
          DIGITALOCEAN_ACCESS_TOKEN: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}
          DO_INFERENCE_KEY: ${{ secrets.DO_INFERENCE_KEY }}

Two things to know before you turn it on. First, the DigitalOcean token in that secret should be able to do as little as possible, ideally in a separate DigitalOcean team: a token that can manage your whole account is a large thing to hand to a workflow. Second, pull requests opened with the workflow's own GITHUB_TOKEN do not trigger other workflows, so your normal test workflow will not run on them unless you open them with a GitHub App or a separate token. The script's own test run inside the sandbox covers the gap, but it is not a substitute for CI.

The runs in this post were made with the script from a terminal, not through Actions.

What we could not conclude

• Whether this scales past a toy repository. The target is a few dozen lines with a fast test suite. A real codebase means longer runs, more tokens, and tests the agent may not be able to run inside the sandbox without more egress.
• Why the file.write rule did not cover OpenCode's edit tools. We saw the refusals, not the mapping behind them.
• How other models compare. Our inference tier only allowed open models, so we did not run the same issues through Claude or GPT.
• Anything about speed. This is a how-to, not a benchmark, and Managed Agents is a public preview.

Summary

The pattern is small and reusable: a disposable microVM per task, a model key as the only secret inside, an egress allowlist, a script outside the sandbox that checks the agent's work with something the agent did not write, and a person on the merge button. On DigitalOcean Managed Agents the whole loop is a create, a few exec calls, one prompt and a remove, and the model can run on DigitalOcean too.

The agent did good work on three small issues for a few cents each. It also produced a confident pull request whose new tests never ran, and it worked around refused editing tools through bash without being asked to. Both are reasons to build the checks first and the automation second.

In 2026 that "somewhere else" became a product category. DigitalOcean put Managed Agents into public preview on 21 September, AWS shipped AgentCore Runtime V2 three days earlier, Microsoft made Foundry hosted agents generally available in July, Google renamed Vertex AI Agent Engine to Agent Runtime, and Cloudflare took Containers and its Sandbox SDK to GA in April. This post compares eight of them on the things that decide whether an agent survives production, and works out one cost scenario on every platform from list prices.

TLDR

• Best overall for running coding and tool-using agents today: DigitalOcean Managed Agents. Claude Code, Codex CLI and OpenCode start with one command, each session gets its own Firecracker microVM, checkpoints capture live memory, sessions have no 8 or 24 hour limit, and 16,000+ tools sit behind one MCP endpoint. It also has the lowest vCPU list price in the group. It is a public preview in one US region, and its network is open by default until you set an allowlist.
• Best for many small stateful agents: Cloudflare. Durable Objects give every agent its own storage and schedule, a hibernating agent is not billed for duration, and the Sandbox SDK adds a VM-isolated Linux box when an agent needs a shell.
• Best inside an enterprise cloud: AWS AgentCore, Google Agent Runtime or Microsoft Foundry, depending on which cloud already holds your identity, network and audit trail.
• Best when you run the agent loop yourself: E2B, Vercel Sandbox or Modal. They sell the sandbox, not the agent.

Prerequisites

• An agent you want to host: a coding CLI such as Claude Code or Codex, a framework agent (LangGraph, ADK, Agents SDK), or your own loop.
• A rough idea of your workload: session length, how much of the time the agent waits on a model, and which tools it calls.
• Familiarity with the Model Context Protocol (MCP) helps for the tool-access sections.

What we compared

Every platform here promises "run your agent in the cloud". They differ on six things that matter once real work runs through them:

1. Isolation. Can agent-written code reach anything it should not? A microVM per session is the strongest common answer.
2. State. Can a session pause and resume where it was, including running processes? Can you checkpoint it and try two approaches in parallel?
3. Tool access. How does the agent reach GitHub, a CRM or a database, and where do the credentials live?
4. Time to first agent. How much packaging stands between you and a running session?
5. Limits. Largest sandbox, longest session, regions.
6. Price. List prices, and what they are billed on.

The platforms fall into four groups, and the ranking makes more sense once you see them:

Four kinds of agent runtime

• Hosts for agents and coding CLIs bring a harness, get a microVM
  • DigitalOcean Managed Agents
  • AWS AgentCore Runtime
• Framework runtimes deploy an agent built on their SDK
  • Google Agent Runtime
  • Microsoft Foundry hosted agents
• Edge stateful agents one durable object per agent
  • Cloudflare Agents SDK + Sandbox
• Sandboxes for your own loop you orchestrate, they isolate
  • E2B
  • Vercel Sandbox
  • Modal

At a glance

List prices on 28 September 2026. "Active" means CPU is billed only while it is in use.

Note on DigitalOcean: its price is quoted per vCPU-hour of actual use, but active-CPU billing is "coming soon". Until then it bills 25% of the vCPUs you allocate, whatever the agent does. Microsoft's public pricing page shows placeholders; its rates above come from Microsoft's Azure Retail Prices API.

1. DigitalOcean Managed Agents

Managed Agents is two services that work together: Harness Runtime, which runs each agent session in its own Firecracker microVM, and Action Gateway, a managed MCP endpoint in front of more than 16,000 tools. It went to public preview for all users on 21 September 2026 after a private preview in August.

Why it is first. It is the only platform here where the agents most teams already use are first-class citizens. Claude Code, Codex CLI, OpenCode, Hermes and LangGraph are built-in adapters, and a custom container image covers the rest. Starting one is a single command:

doctl harness-runtime launch --harness claude-code --name repo-helper

The rest of the package is what a long-running agent needs:

• Real pause and resume. Pausing "freezes the sandbox in place. Processes, memory, and the workspace filesystem are all preserved", and idle sessions pause themselves after 15 minutes by default. A paused session costs nothing for compute.
• Checkpoints with live memory. A checkpoint captures the workspace and memory. From it you can fork up to four copies to try different approaches (on the Claude Code, Codex CLI and OpenCode adapters), or roll back in place. Fly.io Sprites restores files only; Daytona forks only its VM sandboxes.
• Room to work. Sizes go from 1 vCPU / 1 GB up to 16 vCPU / 32 GB with a 500 GB disk, more than AWS (2 vCPU / 8 GB), Microsoft (2 / 4) or Cloudflare (4 / 12) give a single session. There is no 8 or 24 hour session limit, only a cap of 744 active hours per session per month.
• Tools without credentials in the sandbox. Action Gateway exposes its catalog through three meta tools (search, invoke, and a code runner) instead of flooding the model's context. Its OAuth connections are resolved at the gateway, so the key never enters the sandbox. It also works on its own, from a Claude Code or Codex on your laptop.
• One policy layer. The coding adapters share the same allow, ask and deny rules, with approvals from the terminal or with doctl harness-runtime approve. Support varies in the details: Codex CLI cannot use default: deny, and triggered runs must not use ask.
• One key for every model. Instead of an Anthropic or OpenAI key, a session can use DigitalOcean serverless inference through HARNESS_INFERENCE_MODEL and HARNESS_INFERENCE_API_KEY, billed on the same DigitalOcean account.

A session is described in YAML. This one follows the environment spec reference and locks down the two defaults you most likely want to change, the open network and the permission rules:

name: repo-helper
agent: claude-code
size: mars-2vcpu-4gb
idle_timeout: 10m
env:
  HARNESS_INFERENCE_MODEL: anthropic-claude-5-sonnet
secrets:
  HARNESS_INFERENCE_API_KEY: ${DO_MODEL_ACCESS_KEY}
# Naming one host turns egress into a deny-by-default allowlist
egress:
  - api.github.com
permissions:
  default: ask
  rules:
    - tool: file.read
      action: allow
    - tool: bash
      action: ask

One catch from our hands-on run: doctl 1.175.0 still asks for a real Anthropic API key when you create a claude-code session, even with the inference fields above, and which models your DigitalOcean inference key may use depends on your account's tier. Issue to Pull Request with DigitalOcean Managed Agents walks through the setup we got working, with OpenCode and DeepSeek V4 Pro.

Price. $0.044 per vCPU-hour and $0.0095 per GB-hour of peak memory, the lowest vCPU list price in this comparison. Action Gateway calls cost $0.10 per 1,000, and the Exa search tool $10.10 per 1,000.

What to watch.

• It is a public preview: no SLA, support on weekday business hours (Pacific time), no data durability guarantee, and the terms warn that the API and spec may change without notice.
• It runs in one region, Richmond (RIC1), and data is processed in the US.
• Egress is open by default until you name a host, and ordinary secrets are readable inside the sandbox. Scoped secrets, which keep the real key at the egress proxy, are "not yet enabled everywhere".
• Billing runs from a prepaid balance shared with your other DigitalOcean products, with no per-session spend cap, and paused sessions still count toward the limit of up to 100 sessions per team.

Pick it if you want Claude Code, Codex or OpenCode working in the cloud this week, need sessions up to 16 vCPU with no fixed time limit, or want a large tool catalog without hosting MCP servers yourself.

2. Cloudflare Agents SDK, Durable Objects and Sandbox

Cloudflare approaches the problem from the other end. The Agents SDK turns each agent into a Durable Object with "a durable identity, local SQL storage, real-time connections, scheduled work, and recoverable execution", running in V8 isolates on Cloudflare's global network. When an agent needs a real Linux shell, the Sandbox SDK starts a container where "each sandbox runs in a separate VM". Containers and Sandbox went GA on 13 April 2026.

Why it is second. For agents that are many, small and long-lived (a support agent per customer, an email agent per inbox, a webhook agent per repository), it is built for exactly this. A Durable Object that hibernates is not billed for duration, CPU on Containers is billed only while active, Workers and Durable Objects run on Cloudflare's global network, and Containers can be placed by region or jurisdiction, including eu and FedRAMP. The 2026 egress controls add host allow and deny lists plus credential injection, so a sandbox can call an API without holding the key.

Price. Workers Paid ($5 a month), then Containers at $0.072 per active vCPU-hour and $0.009 per GiB-hour of provisioned memory. Durable Objects cost $0.15 per million requests and $12.50 per million GB-seconds of duration.

What to watch.

• The largest container is 4 vCPU, 12 GiB and 20 GB, and container disk is ephemeral (there is a backup and restore API). Heavy coding agents outgrow it.
• An agent handles 30 seconds of compute per request or message; long work goes into scheduled tasks or a sandbox.
• There is no hosted tool catalog. You build and host MCP servers yourself, and McpAgent is deprecated in favour of createMcpHandler.
• The agents package is still pre-1.0 (0.24.0).

Pick it if you run lots of stateful, mostly idle agents close to users, or want to host remote MCP servers at the edge.

3. AWS Bedrock AgentCore

AgentCore is AWS's set of agent building blocks: Runtime, Gateway, Identity, Memory, Code Interpreter and Browser. Each Runtime session "receives its own dedicated microVM", and Runtime V2, launched on 18 September 2026, restores sessions from snapshots with a P75 cold start of 1.9 to 2.0 seconds in AWS's own numbers.

Strengths. Runtime, Gateway and Identity run in 22 regions including GovCloud (V2 is in five so far), with IAM and VPC throughout. On V1 pricing, CPU "scales to zero during I/O wait". The Gateway turns your OpenAPI specs, Lambdas and MCP servers into tools for $0.005 per 1,000 calls. An Instances option runs sessions on EC2 capacity in your account for up to 14 days, with GPUs.

What to watch. A microVM session tops out at 2 vCPU and 8 GB and eight hours. Coding CLIs work (AWS has a walkthrough for Claude Code, Codex and others), but you package and push each one as a container yourself. The Gateway wraps your own APIs rather than shipping a SaaS catalog.

Pick it if your identity, data and audit trail already live in AWS.

4. Google Agent Runtime (formerly Vertex AI Agent Engine)

Google renamed Vertex AI Agent Engine to Agent Runtime on Gemini Enterprise Agent Platform in April 2026. It is a managed runtime for containerized agents, with full integration for Google's ADK and templates for LangGraph, LangChain, AG2 and LlamaIndex, plus Sessions and Memory Bank for state.

Strengths. $0.085 per vCPU-hour and $0.009 per GiB-hour, and "idle time spent waiting for the next prompt between turns is not billed", which suits chat-style agents that sit waiting for users. Containers go up to 8 vCPU and 32 GiB, in 23 regions, with Agent Gateway for MCP and A2A governance.

What to watch. It is a framework runtime, not a host for third-party coding CLIs. The Code Execution sandbox has "no network access" and does not let you install your own libraries.

Pick it if you build on ADK or LangGraph and already run on Google Cloud.

5. Microsoft Foundry hosted agents

Foundry Agent Service is GA, and its hosted agents reached general availability on 9 July 2026. Hosted agents use "per-session VM-isolated sandboxes" with persistent home directories, and a Toolbox puts curated tools (code interpreter, web search, OpenAPI, MCP and A2A) "behind one managed MCP-compatible endpoint".

Strengths. Entra identity, VNet integration, publishing to Teams and Microsoft 365, and 31 regions.

What to watch. Sessions are small (0.5 vCPU / 1 GiB up to 2 vCPU / 4 GiB, with up to 20 GiB of disk), idle timeouts run from 2 to 60 minutes, and hosted agents are Python and C# only. Hosted compute lists at $0.0994 per vCPU-hour and $0.0118 per GiB-hour in Microsoft's retail price API.

Pick it if your agents serve people who live in Microsoft 365.

6. E2B

E2B sells the sandbox, not the agent: "every E2B sandbox runs in its own Firecracker microVM with its own kernel", driven from its open source (Apache-2.0) SDKs while your code runs the loop.

Strengths. Pause saves "both the sandbox's filesystem and memory state", resumes in about a second, and a paused sandbox "is kept indefinitely". An MCP gateway inside the sandbox offers 200+ tools from the Docker MCP Catalog. Compute is $0.0504 per vCPU-hour and $0.0162 per GiB-hour.

What to watch. Continuous runtime is capped at 1 hour on the free Hobby plan and 24 hours on Pro ($150 a month plus usage), and compute is billed per second on the sandbox size while it runs.

Pick it if you are building a product with a code interpreter or coding agent inside it and want to own the orchestration.

7. Vercel Sandbox

Vercel Sandbox went GA in January 2026, and "each sandbox runs in its own Firecracker microVM with a dedicated kernel". Sessions can now run for 24 hours on Pro, persistence is on by default, and it is available in Vercel's compute regions.

Strengths. CPU is billed only while active ($0.128 per hour), so time spent waiting on a model is not counted, and the firewall "injects credentials into egressing traffic. The secrets never enter the sandbox." The default image ships with Node, Python and coding agents.

What to watch. No first-party MCP gateway that we found, a 45-minute session limit on Hobby, and memory at $0.0212 per GB-hour, more than twice DigitalOcean's.

Pick it if your agents are TypeScript and AI SDK code that already deploys to Vercel.

8. Modal Sandboxes

Modal runs sandboxes on gVisor, with VM sandboxes in beta, and is at its best for Python workloads and GPUs.

Strengths. Very high fan-out (Modal claims 100,000+ concurrent sandboxes), GPU support, and egress controls down to a domain allowlist. Good for RL, evals and batch agents.

What to watch. Billing is "whichever is higher: your resource request or your actual usage", sandboxes cost about three times Modal Functions, memory snapshots are alpha, and there is no first-party MCP gateway that we found.

Pick it if you run many short Python agents or need GPUs next to the sandbox.

One hour on every platform

To put the price lists side by side, take the example from DigitalOcean's launch post: one hour on a 2 vCPU / 4 GB sandbox, with the agent averaging 25% CPU. Platforms that bill active CPU pay for half a vCPU-hour; the rest pay for the full allocation. Memory is billed on whatever basis each platform uses. Free tiers, plan fees, storage, tool calls and model tokens are left out.

One hour at 2 vCPU and about 4 GB, 25% average CPU

Derived from list prices on 28 September 2026, not quotes. Cloudflare uses its predefined standard-3 type (2 vCPU / 8 GiB). Google counts the whole hour as active; its runtime does not bill idle time between turns, so a chatty agent can cost much less. Microsoft is left out because its billing basis is not confirmed.

The arithmetic for the first two, so you can redo it with your own numbers:

DigitalOcean  2 vCPU x 25% x $0.044  +  4 GB x $0.0095   = $0.022 + $0.038 = $0.060
AWS V1        0.5 vCPU-h x $0.0895   +  4 GB x $0.00945  = $0.045 + $0.038 = $0.083

Microsoft's retail rates imply about $0.246 if the full allocation is billed for the hour, but we could not confirm how hosted agents are billed.

Two things matter more than the hourly rate. First, a session that never pauses runs for 744 hours in a 31-day month: a medium DigitalOcean session costs $44.64 for that before storage, which is why auto-pause matters. Second, model tokens are billed on top of every number here. Price them before you optimise the sandbox.

How to pick

What we could not confirm

• Cloudflare does not name the hypervisor behind its container VMs, and Google does not name the sandboxing technology behind Agent Runtime.
• Microsoft's public pricing page for hosted agents shows placeholders; the rates here come from its retail price API.
• DigitalOcean's docs disagree with themselves in places: the limits page says VPC connections are not supported while the spec documents a vpc_uuid field, and resume is quoted as 200 ms in marketing and 305 ms in the launch benchmark.
• Prices and preview terms change fast in this category. Check each vendor's page before you commit.

Summary

The category split into two ideas in 2026. One says "give us your agent and we will run it": DigitalOcean, AWS, Google and Microsoft. The other says "we give you a sandbox, you run the agent": E2B, Vercel, Modal, and Cloudflare with its own twist of one durable object per agent.

For most teams that want an agent like Claude Code or Codex working in the cloud now, DigitalOcean Managed Agents is the shortest path: one command to a microVM, pause and resume with memory, sandboxes up to 16 vCPU / 32 GB, a large tool catalog, and the lowest vCPU price on the list. Go in knowing it is a preview in one region, set an egress allowlist on day one, and keep an eye on the prepaid balance. If you need GA guarantees, multiple regions or a specific cloud's identity model, Cloudflare and the hyperscalers are close behind, and the decision table above tells you which.

📌 Handpicked by DevOps Daily - Your weekly dose of curated DevOps news and updates!

⚓ Kubernetes

📄 How a missing kernel flag broke FIPS-certified containers in managed Kubernetes

A customer building a FedRAMP-compliant deployment found that Ubuntu Pro 22.04 FIPS container images were silently failing on standard, mainline Linux kernels, the kind used by managed Kubernetes envi

📅 Sep 28, 2026 • 📰 Ubuntu Blog

🔗 Read more

📄 The rise of agentic AI on Kubernetes: unleashing the new infrastructure layer

AI is changing expectations around infrastructure and operations, including Kubernetes management. When models run close to the data they use, The post The rise of agentic AI on Kubernetes: unleashing

📅 Sep 27, 2026 • 📰 The New Stack

🔗 Read more

📄 One Amazon EKS, many edges: How to choose your edge container strategy on AWS

Choosing the right edge container strategy across many locations can fragment your fleet into dozens of special cases. This post shows how to avoid that by standardizing on Amazon EKS, then choosing a

📅 Sep 25, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Building a single-pane NOC dashboard for Amazon EKS with Amazon CloudWatch

An incident is the worst moment to discover you cannot trust your Amazon EKS dashboard. This post shows what a trustworthy single-pane NOC for Amazon EKS on Amazon CloudWatch looks like, why each desi

📅 Sep 23, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Breaking the AI productivity paradox: an intelligent migration factory to modernize infrastructure and applications

The rise of generative AI promised a silver bullet, but for most enterprises, especially banks, digital transformation remains a slow, complex, and costly endeavor. Early reports suggested massive dev

📅 Sep 23, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Spotlight on SIG Apps

As Kubernetes adoption has grown, the conversation has shifted beyond running containers to managing increasingly complex application lifecycles. Modern platforms support stateless web services, state

📅 Sep 22, 2026 • 📰 Kubernetes Blog

🔗 Read more

📄 Kubernetes v1.37: Tracking When a PersistentVolumeClaim Was Last Used (Beta)

Kubernetes v1.37 promotes the PersistentVolumeClaimUnusedSinceTime feature gate to Beta (enabled by default). With this feature, the PersistentVolumeClaim (PVC) protection controller adds an Unused co

📅 Sep 21, 2026 • 📰 Kubernetes Blog

🔗 Read more

☁️ Cloud Native

📄 The case for a cloud native agent harness

Coding agents became useful when they stopped being a chat box. Four things changed the shape of the problem: capable tools, a shared repository and filesystem, subagents, and skills that capture what

📅 Sep 28, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Clearing the Vulnerability Backlog Safely: Agentic SecOps on SUSE AI Factory with the NVIDIA Open Agent Safety Platform

Scan a mid-sized container estate and you get back somewhere north of four thousand findings. Maybe thirty matter. The rest are unreachable code paths, packages that aren’t installed at runtime, image

📅 Sep 28, 2026 • 📰 SUSE Blog

🔗 Read more

📄 AWS named a Leader in the 2026 Gartner Magic Quadrant for Container Management

Gartner has recognized AWS as a Leader in the 2026 Gartner Magic Quadrant for Container Management for the fourth consecutive year. See what our latest container innovations across Amazon ECS and Amaz

📅 Sep 25, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Security Slam 2026 – Fall edition

Security Slam 2026 – Fall Edition is a 30-day virtual event from October 5 through November 6, 2026. What Is the Security Slam? The Open Source Security Foundation (OpenSSF) is partnering with the Clo

📅 Sep 25, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Manufacturing Trust for AI Agents | Docker’s WeAreDevelopers Keynote

Docker's WeAreDevelopers keynote shows how Sandboxes, Kits, and Cloud Sandboxes give AI agents strong isolation and reproducible authority.

📅 Sep 24, 2026 • 📰 Docker Blog

🔗 Read more

📄 From Dockerfile to Kit: the Docker Sandboxes Kit Specification

Docker's Sandbox Kit Specification v3 packages an AI agent's network rules, credentials, and volumes as an ordinary, pinnable OCI image.

📅 Sep 24, 2026 • 📰 Docker Blog

🔗 Read more

📄 Docker and CNCF partner on an open spec for agent permissions

Docker is bringing the open source Sandbox Kit Spec to the CNCF, so AI agent permissions become a neutral, vendor independent standard built on OCI.

📅 Sep 24, 2026 • 📰 Docker Blog

🔗 Read more

📄 Introducing Cloud Sandboxes: Start on Your Laptop, Finish in the Cloud

Run agents on your laptop, in the cloud, and move between them with one command, all safely. Earlier this year we launched Docker Sandboxes: microVM environments where coding agents can work autonomou

📅 Sep 24, 2026 • 📰 Docker Blog

🔗 Read more

📄 Observability Day: Where the community comes together at KubeCon + CloudNativeCon North America 2026

Observability Day returns to KubeCon + CloudNativeCon North America on November 9, 2026, in Salt Lake City, Utah, bringing together maintainers, operators, and end users from across the CNCF observabi

📅 Sep 24, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Which hat am I wearing right now?

Neutrality is quietly the hardest part of open source. It gets tricky the moment someone pays your salary — and staying honest about it takes more effort than anyone admits. Here’s something we don’t

📅 Sep 23, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Cilium at KubeCon + CloudNativeCon and CiliumCon North America 2026

Cilium is heading back to Salt Lake City this November for KubeCon + CloudNativeCon and CiliumCon North America 2026. Since coming…

📅 Sep 23, 2026 • 📰 Cilium Blog

🔗 Read more

📄 The Hidden Costs of AI: 7 Cost Drivers Your Enterprise AI Spend May be Missing

Ask most enterprises what their AI costs, and they’ll point at tokens. It’s the number on the invoice. The number in the pricing calculator. The number that often finds its way into the business case.

📅 Sep 23, 2026 • 📰 Kubecost Blog

🔗 Read more

🔄 CI/CD

📄 Leaked GitLab Email Tokens Can Reach Code, Secrets and CI/CD Pipelines

Security researchers have uncovered a GitLab behavior that could let attackers use a leaked project email address to push code, trigger CI/CD jobs and reach other repositories accessible to the addres

📅 Sep 25, 2026 • 📰 DevOps.com

🔗 Read more

📄 GitHub Copilot app for Beginners: How to build custom workflows with canvases

Describe the interface you need in plain English, then let the agent build a live surface you can both use and update—so you spend less time adapting to tools and more time getting work done. The post

📅 Sep 25, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Improving site performance by shipping more CSS

How we fully migrated github.com away from CSS-in-JS. The post Improving site performance by shipping more CSS appeared first on The GitHub Blog.

📅 Sep 25, 2026 • 📰 GitHub Blog

🔗 Read more

📄 CI/CD Security Best Practices: A 2026 Pipeline Checklist

CI/CD security best practices for 2026: secrets management, scoped access, dependency scanning, signed artifacts, and audit logging in one checklist. | Blog

📅 Sep 25, 2026 • 📰 Harness Blog

🔗 Read more

📄 When chat is the wrong UI

What is a developer to do when they need something more tangible than a chat box? Enter canvases. The post When chat is the wrong UI appeared first on The GitHub Blog.

📅 Sep 24, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Gitea Runner 4.0.0 is released

We are happy to announce the release of Gitea Runner 4.0.0. This is a runner-side release. It remains wire-compatible with existing Gitea versions; the major version bump reflects two breaking cha

📅 Sep 24, 2026 • 📰 Gitea Blog

🔗 Read more

📄 GitLab Critical Patch Release: 19.4.1, 19.3.3, 19.2.7

📅 Sep 23, 2026 • 📰 GitLab Blog

🔗 Read more

📄 How to design GitLab for enterprise scale

At enterprise scale, even small architecture choices can have outsized consequences. A deployment that works for a handful of teams can become a constraint once thousands of developers, repositories,

📅 Sep 22, 2026 • 📰 GitLab Blog

🔗 Read more

📄 How GitLab reduced code-per-agentic-flow ratio by 45%

GitLab Duo Agent Platform orchestrates and automates complex tasks through agentic flows. A key part of the platform is the Flow Registry, a declarative configuration framework, built from reusable co

📅 Sep 22, 2026 • 📰 GitLab Blog

🔗 Read more

🏗️ IaC

📄 Route every Claude Code message to the right model with Jev

Not every message you send to Claude Code needs the most capable model. A quick question about a Git command runs on the same model as a refactor across three services, unless you remember to switch m

📅 Sep 28, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 AI agents need continuity, not just context

Pulumi Neo works on infrastructure the way an engineer does: it clones repositories, edits files, installs dependencies, runs previews, and produces intermediate work along the way. A task is not only

📅 Sep 25, 2026 • 📰 Pulumi Blog

🔗 Read more

📊 Observability

📄 Rider 2026.2.3 Is Released!

Rider 2026.2.3 brings AI performance analysis to the Monitoring tool window and fixes an issue with the AI Agent Setup widget on Windows. You can update directly from the IDE, through the Toolbox App,

📅 Sep 28, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Exploring the OpenTelemetry Instrumentation Ecosystem

OpenTelemetry has a lot of pieces: APIs, SDKs, a protocol, semantic conventions, instrumentation, and tools like the Collector. The APIs and protocol define how telemetry is created and exchanged, whi

📅 Sep 25, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

📄 Best AI Security Solutions in 2026: A Buyer's Guide

Compare the best AI security solutions in 2026: LLM security, model monitoring, and AI-powered SOC tools, plus buying criteria and vendor questions. | Blog

📅 Sep 25, 2026 • 📰 Harness Blog

🔗 Read more

📄 What if your agent's hallucinations had a budget? How to start using SLOs for agent behavior

At Grafana Labs, observability is what we do. So as we started building AI agents, we naturally reached for the same instincts we bring to every system: measure it, set targets, and make reliability s

📅 Sep 24, 2026 • 📰 Grafana Blog

🔗 Read more

📄 Measuring the back/forward cache with Application Metrics

A bfcache hit leaves no trace in ordinary metrics. Here's how Sentry's bfcacheMetricsIntegration makes back/forward cache health measurable.

📅 Sep 24, 2026 • 📰 Sentry Blog

🔗 Read more

📄 Prometheus and OpenTelemetry interoperability in 2026: Survey results

We ran a survey asking users of OpenTelemetry and Prometheus how they collect, process, and store metrics. The goal was to understand, with real usage data rather than assumptions, how far the ecosyst

📅 Sep 22, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

📄 How to Build an SRE Agent That Actually Works (Without Blowing the Token Budget)

Learn how to build an SRE agent that delivers accurate, low-latency incident triage with multi-tiered memory and RAG—without blowing your token budget.

📅 Sep 22, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Announcing the 2026 Observability Forecast

The Observability Forecast 2026 offers insights from 2,575 IT and engineering leaders and practitioners worldwide on the future of observability.

📅 Sep 22, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Introducing New Relic Compound Alerts

Discover how New Relic Compound Alerts intelligently correlates related alerts into actionable operational issues to improve incident response and operational efficiency.

📅 Sep 22, 2026 • 📰 New Relic Blog

🔗 Read more

📄 MLOps Solutions for Production Machine Learning

Learn how MLOps solutions support experiment tracking, model serving, monitoring, feature management, governance, and production rollouts.

📅 Sep 21, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Grafana Alerting: Scale alert routing without scaling complexity using multiple notification policies

Alert routing often starts simple. A team creates a few contact points, adds some label matchers, and builds a notification policy tree that sends each alert to the right destination. But alerting con

📅 Sep 21, 2026 • 📰 Grafana Blog

🔗 Read more

📄 Announcing the 2026 OpenTelemetry Governance Committee Election

The OpenTelemetry project is excited to announce the 2026 OpenTelemetry Governance Committee (GC) election. Nominations are due by 16 October 2026 23:59 AoE. The list of eligible candidates will be sh

📅 Sep 21, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

🔐 Security

📄 Threats Making WAVs - Incident Response to a Cryptomining Attack

Guardicore security researchers describe and uncover a full analysis of a cryptomining attack, which hid a cryptominer inside WAV files. The report includes the full attack vectors, from detection, in

📅 Sep 28, 2026 • 📰 Linode Blog

🔗 Read more

📄 We Gave Our Agents Autonomy. Here’s How We Kept Control.

Co-authored by Stacey Miller & Troy Mangum Key Takeaways The problem: Agents have demonstrated their ability to escape lab environments and access unauthorized systems. Prompt and model safeguards gui

📅 Sep 28, 2026 • 📰 SUSE Blog

🔗 Read more

📄 GitHub’s Security Autofix Agent Now Remembers What It Fixed

GitHub’s agentic autofix now uses Copilot Memory to reuse repository-specific security fix patterns, helping Copilot apply lessons from past vulnerabilities across future alerts, reviews and coding wo

📅 Sep 28, 2026 • 📰 DevOps.com

🔗 Read more

📄 JFrog Artifact Alternatives: What to Look for When Migrating

Considering JFrog Artifactory alternatives? Here's what matters before you migrate: package coverage, security scanning, pricing, and migration risk. | Blog

📅 Sep 28, 2026 • 📰 Harness Blog

🔗 Read more

📄 Securing AI agents requires securing the systems around them

Enterprise AI is changing from software that primarily generates information to software that can take action. AI agents can call APIs, invoke tools, access files and credentials, communicate over net

📅 Sep 28, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Blitzy Makes Sandbox for Reverse Engineering Code Available at No Cost

Blitzy has made available a sandbox where DevOps teams can reverse-engineer up to one million lines of code, generate up to 25,000 lines of tested end-to-end code, and identify security vulnerabilitie

📅 Sep 25, 2026 • 📰 DevOps.com

🔗 Read more

📄 The path to a sovereign autonomous enterprise: Why open source is your clean core advantage

Key Takeaways Building an autonomous SAP environment requires balancing rapid AI innovation with strict data control and compliance. Keeping your SAP core clean relies on hybrid integration, private A

📅 Sep 25, 2026 • 📰 SUSE Blog

🔗 Read more

📄 {unscripted} Chicago and Columbus recap

{unscripted} Chicago and Columbus focused on AI code that doesn't ship, compliance as a feature, and the human cost of machine speed. | Blog

📅 Sep 25, 2026 • 📰 Harness Blog

🔗 Read more

📄 Red Hat Enterprise Linux 10 STIG automation now matches DISA STIG V1R2

For the U.S. Department of Defense (DoD) and its contractors, security has to hold up against a published baseline—not a general claim that systems are “secure.” The Defense Information Systems Agency

📅 Sep 25, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 AI-powered fuzzing with the GitHub Security Lab Taskflow Agent

In this blog post, I explain how to use the new fuzzing taskflow based on the GitHub Security Lab Taskflow Agent AI framework. The post AI-powered fuzzing with the GitHub Security Lab Taskflow Agent a

📅 Sep 24, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Your Vulnerability Backlog Is No Longer Technical Debt, It’s an Attack Surface

A growing vulnerability backlog is more than technical debt: it is an attack surface. Learn why outdated risk assumptions, automated attackers, and chained findings demand a new approach.

📅 Sep 24, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Accelerating delivery of CVE fixes with a new Kernel release strategy

When it comes to fixing security vulnerabilities, speed is crucial. Canonical is officially outlining a transition from its current 4-week regular and 2-week security kernel Stable Release Update (SRU

📅 Sep 23, 2026 • 📰 Canonical Blog

🔗 Read more

💾 Databases

📄 PLEASE_READ_ME: The Opportunistic Ransomware Devastating MySQL Servers

Guardicore Labs uncovers a Ransomware detection campaign targeting MySQL servers. Attackers use Double Extortion and publish data to pressure victims.

📅 Sep 28, 2026 • 📰 Linode Blog

🔗 Read more

📄 Building a Faster (Rust-Based) Python Driver for ScyllaDB

Binding Rust to Python with PyO3, zero-copy deserialization with yoke

📅 Sep 28, 2026 • 📰 ScyllaDB Blog

🔗 Read more

📄 The FinTech Scalability Crisis: How Distributed SQL Unlocks Innovation with Zero Downtime Operations

When Plaid’s Amazon Aurora MySQL fleet faced a major-version upgrade, the estimate came back at six engineering months and tens of minutes of downtime. For a company whose APIs sit underneath thousand

📅 Sep 26, 2026 • 📰 TiDB Blog

🔗 Read more

📄 Unlock 3x QPS and microsecond latency with Memorystore for Valkey 9.1

At Google Cloud, we are committed to delivering the best managed experience backed by open source software. Today, we’re announcing the general availability of Memorystore for Valkey 9.1, which achiev

📅 Sep 25, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Change your components, keep your infrastructure

Components let you turn a group of resources into a reusable building block. You can define a network, a database, or an application service once and share it across projects and teams. People using t

📅 Sep 25, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Vercel and TiDB Cloud Starter: The Full-Stack Playbook for AI Apps

You have an AI app running on Vercel, or a prototype that v0.dev generated in a few minutes, and now it needs a real database. The choice is harder than it looks, because serverless functions and edge

📅 Sep 24, 2026 • 📰 TiDB Blog

🔗 Read more

📄 How to Solve the Agent Handoff Problem

If your agents hand work to each other, or to a teammate's agents, this is the difference between the next agent building on a decision and rebuilding a dead end. Discover how to solve the agent hando

📅 Sep 24, 2026 • 📰 Yugabyte Blog

🔗 Read more

📄 PostgreSQL 19 Beta 4 Released!

The PostgreSQL Global Development Group announces that the fourth beta release of PostgreSQL 19 is now available for download. This release contains PostgreSQL 19 feature previews ahead of general ava

📅 Sep 24, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 The official FastAPI Redis SDK is now available

FastAPI now has an official Redis integration. With the fastapi-redis-sdk, we worked closely with the FastAPI team on what a naturally integrated Redis experience should look like, following FastAPI’s

📅 Sep 24, 2026 • 📰 Redis Blog

🔗 Read more

📄 Lakebase, TiDB X, and the Database Architecture AI Demands

An AI coding agent can generate an application, change its schema, test several implementations, and discard most of them within a single session. Another agent might spend that session updating order

📅 Sep 23, 2026 • 📰 TiDB Blog

🔗 Read more

📄 Stay up when a region goes down: Highly available Redis for Python apps

Active-Active Redis & client-side geographic failover Active-Active Redis distributes data across multiple regions, allowing each regional database instance to serve both reads and writes. What kind o

📅 Sep 23, 2026 • 📰 Redis Blog

🔗 Read more

📄 How moving from Azure Cache for Redis to Azure Managed Redis can cut costs by 40%

Moving to Azure Managed Redis can cut your monthly Redis service cost by around 40%. In four East US 2 Premium-to-Balanced price comparisons, the reduction is 43–44%, with high availability on both si

📅 Sep 23, 2026 • 📰 Redis Blog

🔗 Read more

🌐 Platforms

📄 The Oracle of Delphi Will Steal Your Credentials

Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic

📅 Sep 28, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger

In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum

📅 Sep 28, 2026 • 📰 Linode Blog

🔗 Read more

📄 Four months of VoidZero at Cloudflare: making the open-source JavaScript toolchain faster for all humans and agents

Since joining Cloudflare, VoidZero has delivered more than 80 releases that drastically speed up JavaScript compilation, linting, and testing. From a 10x faster React compiler to Vite+ 1.0, here’s how

📅 Sep 28, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Introducing Forge: the open source pipeline for generating SDKs, CLIs, docs, and more

Forge is a pluggable, open-source pipeline that runs in CI to generate SDKs, CLIs, and documentation directly from API definitions. By shifting generation upstream into individual team repositories, F

📅 Sep 28, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 The road to the agentic browser: A Kitesurf update

We’ve updated Kitesurf, our Workers-based browser for AI agents, with WebMCP support, improved DOM performance, and terminal-based rendering. With over 730,000 Web Platform subtests passing, agents ca

📅 Sep 28, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Introducing The Cold Start: pitch your startup live at Cloudflare Connect

Cloudflare is launching The Cold Start, a startup competition giving five early-stage companies five minutes on stage at Cloudflare Connect. Grand Prize winner receives $500,000 in credits, a San Fran

📅 Sep 28, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Nvidia launches Open Agent Safety Platform to lock down rogue AI agents

OpenAI, Anthropic, Meta, and Google have all recently disclosed that their models broke out of their test environments and reached The post Nvidia launches Open Agent Safety Platform to lock down rogu

📅 Sep 28, 2026 • 📰 The New Stack

🔗 Read more

📄 How DigitalOcean Manages Credentials for Autonomous Agents

Built to align with NVIDIA’s Agent Safety Platform strategy for securing autonomous agents Consider an agent investigating a duplicate charge. It reads the customer’s email, checks their billing histo

📅 Sep 28, 2026 • 📰 DigitalOcean Blog

🔗 Read more

📄 Amazon Transcribe adds customer-managed KMS keys for custom resources

Amazon Transcribe now lets you encrypt your custom vocabularies, custom vocabulary filters, and custom language models at rest with a customer-managed AWS KMS key that you own and control. Previously,

📅 Sep 25, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Amazon EC2 M8i and M8i-flex instances are now available in additional regions

Starting today, Amazon EC2 M8i and M8i-flex instances are now available in the AWS European Sovereign Cloud (Germany) region. These instances are powered by custom Intel Xeon 6 processors, available o

📅 Sep 25, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Amazon EC2 R8i and R8i-flex instances are now available in additional regions

Starting today, Amazon Elastic Compute Cloud (Amazon EC2) R8i and R8i-flex instances are available in the AWS European Sovereign Cloud (Germany) region. These instances are powered by custom Intel Xeo

📅 Sep 25, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Amazon EC2 C8i and C8i-flex instances are now available in additional regions

Starting today, Amazon Elastic Compute Cloud (Amazon EC2) C8i and C8i-flex instances are available in the AWS European Sovereign Cloud (Germany) region. These instances are powered by custom Intel Xeo

📅 Sep 25, 2026 • 📰 CloudFormation Updates

🔗 Read more

📰 Misc

📄 Visual Studio Code 1.140 (Insiders)

Learn what's new in Visual Studio Code 1.140 (Insiders) Read the full article

📅 Sep 30, 2026 • 📰 VS Code Blog

🔗 Read more

📄 Anthropic bought Stainless and shuttered its SDK generator. Cloudflare open-sourced Forge instead.

Cloudflare has announced an open source tool that takes an API definition and automatically produces the SDKs, command-line tools, documentation, The post Anthropic bought Stainless and shuttered its

📅 Sep 28, 2026 • 📰 The New Stack

🔗 Read more

📄 Air Teams: Bring Your Best Agentic Workflows to the Whole Team – and Automate Repeatable Work

Today, we’re introducing JetBrains Air Teams – the team layer for agentic development. It gives humans and agents the shared context, environments, tools, and instructions they need to work effectivel

📅 Sep 28, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 A More Reliable Compilation Scheme for Kotlin Multiplatform Modules

The current compilation approach to Kotlin Multiplatform projects works, but sometimes can lead to unexpected or hard-to-predict behavior. For example: With Kotlin 2.5.0-Beta1, we introduced an option

📅 Sep 28, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Canonical announces the alpha release of Charmed OpenShell to help secure autonomous AI agent fleets

To streamline enterprise deployment and lifecycle management, Canonical is announcing the alpha release of Charmed OpenShell.

📅 Sep 28, 2026 • 📰 Canonical Blog

🔗 Read more

📄 Why Red Hat is building secure agent onboarding

Every enterprise AI conversation we’ve had this year ends in the same place. A team has an agent that works. It writes code, calls internal APIs, fixes its own mistakes. Then someone asks what happens

📅 Sep 28, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Performance engineering from kernel analysis to AI: Adrian Cockcroft’s take

Over its five-year history, P99 CONF has hosted quite a few speakers who’ve offered pointed takedowns of the namesake metric. The post Performance engineering from kernel analysis to AI: Adrian Cockcr

📅 Sep 27, 2026 • 📰 The New Stack

🔗 Read more

📄 Dependency Mocking Approach That Gets More Accurate as Your Services Deploy More Often

Traffic-based dependency mocking turns frequent upstream deployments into opportunities to refresh mocks from real behavior and reduce integration test drift.

📅 Sep 25, 2026 • 📰 DevOps.com

🔗 Read more

📄 Managing multiple EKS clusters: why cluster #3 is the wall

Do the arithmetic before you read the rest of this. Count the people on your team who need access to a cluster. Multiply by the number of clusters you run. That result is how many access decisions som

📅 Sep 25, 2026 • 📰 SUSE Blog

🔗 Read more

📄 How to manage aircraft leases with AI agents

AI quickstarts are a catalog of ready-to-run, industry-specific use cases for your Red Hat AI environment. Each aims to provide a simple and practical way to solve real-world problems using AI on ente

📅 Sep 25, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Continuing to Move PHP Open Source Forward

A year ago, we introduced the first wave of PhpStorm’s open-source sponsorships. We truly believe in the power of open source and the importance of it to the PHP community, and so we want to do our bi

📅 Sep 24, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Visual Studio Code 1.139

Learn what is new in Visual Studio Code 1.139 Read the full article

📅 Sep 23, 2026 • 📰 VS Code Blog

🔗 Read more