Web Race Conditions: Coupons, Balances, and Workflows

This is a TOCTOU lab, not a “parallelize 200 requests and drain the coupon pool” cookbook. Target is a 40-line Go program with an in-memory counter and a tiny HTTP handler. Goal: show the counter go past the limit when the mutex is missing, then show it stop at the limit when the mutex is present. I do not publish a client that wins a production flash sale.

1Figure 1. Two 200s, one coupon. The bug is the window, not the HTTP method.
2goroutine A: if n<1 { n++ }   \
3goroutine B: if n<1 { n++ }   /  no mutex  => n>1

Lab layout

1labs/race_lab/
2  counter.go       # goroutines, limit=1
3  coupon.go        # httptest, same bug at /redeem

 1// counter.go — toy, no network
 2package main
 3
 4import (
 5    "fmt"
 6    "sync"
 7    "sync/atomic"
 8)
 9
10func main() {
11    var n int
12    var wins int32
13    var wg sync.WaitGroup
14    for i := 0; i < 32; i++ {
15        wg.Add(1)
16        go func() {
17            defer wg.Done()
18            if n < 1 {          // check
19                n++             // use  — no mutex
20                atomic.AddInt32(&wins, 1)
21            }
22        }()
23    }
24    wg.Wait()
25    fmt.Println("n", n, "wins", wins)
26}

Artifact: lost update, measured

1$ go run counter.go
2n 12 wins 12
3$ go run counter.go
4n 9 wins 9
5$ go run counter.go
6n 4 wins 4

Limit was 1. n is not 1. That is the whole web-race story, without HTTP. 32 goroutines, no lock, check-then-increment. I keep three runs because a single run can accidentally look fine.

The race detector agrees:

 1$ go run -race counter.go
 2==================
 3WARNING: DATA RACE
 4Read at 0x[REDACTED] by goroutine 8:
 5  main.main.func1()
 6      /labs/race_lab/counter.go:18
 7Write at 0x[REDACTED] by goroutine 7:
 8  main.main.func1()
 9      /labs/race_lab/counter.go:19
10==================
11n 8 wins 8
12Found 1 data race(s)
13exit status 66

-race is the ASAN analog here. I treat DATA RACE as the crash.

Same bug behind HTTP

 1// coupon.go — lab listener on 127.0.0.1
 2package main
 3
 4import (
 5    "fmt"
 6    "net/http"
 7    "sync"
 8)
 9
10var (
11    mu      sync.Mutex
12    locked  = true // set false to demonstrate
13    remain  = 1
14)
15
16func redeem(w http.ResponseWriter, r *http.Request) {
17    if r.Method != http.MethodPost {
18        http.Error(w, "method", 405)
19        return
20    }
21    code := r.FormValue("code")
22    if code != "LAB" {
23        http.Error(w, "nope", 400)
24        return
25    }
26    if locked {
27        mu.Lock()
28        defer mu.Unlock()
29    }
30    if remain < 1 {
31        http.Error(w, "sold out", 409)
32        return
33    }
34    remain--
35    fmt.Fprintln(w, "ok", remain)
36}
37
38func main() {
39    http.HandleFunc("/redeem", redeem)
40    http.ListenAndServe("127.0.0.1:8080", nil)
41}

With locked = true:

 1seq 8 | xargs -P8 -I{} curl -s -o /tmp/r{} -w '%{http_code}\n' \
 2  -X POST http://127.0.0.1:8080/redeem -d code=LAB
 3# 200
 4# 409
 5# 409
 6# 409
 7# 409
 8# 409
 9# 409
10# 409
11$ cat /tmp/r*
12ok 0
13sold out

One 200. remain ended at 0. Good.

With locked = false (the bug):

 1# eight parallel POSTs, same command
 2200
 3200
 4200
 5409
 6...
 7$ cat /tmp/r*
 8ok 0
 9ok -1
10ok -2
11sold out

Three 200s, remain negative. App log:

12023-05-19T15:11:02+08:00  POST /redeem code=LAB src=127.0.0.1 remain=0
22023-05-19T15:11:02+08:00  POST /redeem code=LAB src=127.0.0.1 remain=-1
32023-05-19T15:11:02+08:00  POST /redeem code=LAB src=127.0.0.1 remain=-2

I do not need 10k QPS. Eight curls with xargs -P is enough on this handler because there is no DB latency. Production races often need more overlap; the code shape is the same: read, branch, write, no atomic compare-and-set.

Failed-auth is a different log line and I keep it so SOC does not merge the tickets:

12023-05-19T15:11:09+08:00  POST /redeem code=WRONG src=127.0.0.1
2  result: 400 nope
3# remain unchanged

Analysis steps I run on a real endpoint

Name the side effect: “decrement remaining uses of code LAB”.
Find the check: if remain < 1.
Find the write: remain-- or UPDATE coupons SET used=1.
Ask: is there a unique constraint / UPDATE ... WHERE used=0 / SELECT FOR UPDATE / mutex covering both?
If the write is used=1 without WHERE used=0, file a race even if I cannot win it today.

SQL shape I want, for the same ticket when a DB is involved (see also the financial-apps note):

1-- bad
2UPDATE coupons SET used = 1 WHERE code = 'LAB';
3
4-- good
5UPDATE coupons SET used = 1 WHERE code = 'LAB' AND used = 0;
6-- then inspect rows-affected == 1

I do not run the bad UPDATE against a shared database. The Go counter is the repro I keep.

Sanitized reproduction

1$ go run -race counter.go
2# DATA RACE + n != 1  → ticket evidence
3
4$ go test -race ./...
5# once coupon.go is a test:
6# --- FAIL: TestRedeemOnce (0.00s)
7#     remain= -2 wins= 3

A unit test is the right regression. A public “race exploit client” is not.

Crash analog if someone uses a map without a mutex (Go will actually panic):

1fatal error: concurrent map writes
2goroutine 19 [running]:
3internal/runtime/maps.fatal(...)
4# stack [REDACTED]
5exit status 2

That panic is a gift. File it. Do not “fix” it by adding go more.

Multi-node: the mutex is not enough

coupon.go is one process. Two binaries behind a load balancer each have remain=1. I demonstrate with two processes, not with a clever HTTP client:

1$ locked=true go run coupon.go          # :8080
2$ locked=true PORT=8081 go run coupon.go
3$ curl -s -X POST http://127.0.0.1:8080/redeem -d code=LAB
4ok 0
5$ curl -s -X POST http://127.0.0.1:8081/redeem -d code=LAB
6ok 0

Two 200s, one coupon, both mutexes were fine. The fix moves to the database:

1UPDATE coupons SET remain = remain - 1
2 WHERE code = 'LAB' AND remain > 0;
3-- RowsAffected must be 1

I keep this next to the Go race so nobody ships a mutex as the production patch for a horizontally scaled redeem endpoint.

Mitigation

Mutex / transaction around check+use. In process: sync.Mutex. In DB: UPDATE ... WHERE remaining > 0 and require RowsAffected()==1, or SELECT FOR UPDATE in a transaction, or a unique (user_id, coupon_code) insert.
Idempotency key on the POST (Idempotency-Key header), stored uniquely, so retries are not extra redemptions.
Do not “fix” with time.Sleep or a frontend disable-button.
Log remain after the atomic write, with request id. The negative remain in the lab log is how I noticed the bug without a debugger.
-race in CI for any handler that touches in-memory inventory.

1mu.Lock()
2defer mu.Unlock()
3if remain < 1 {
4    http.Error(w, "sold out", 409)
5    return
6}
7remain--

That is the entire patch for coupon.go. For Postgres, the patch is the WHERE used = 0 line, not a mutex in the app (two app nodes). HTTP/2 multiplexing on a single connection is enough overlap for this handler; I do not need a botnet. xargs -P8 already produced three 200s.

What I file after this lab

counter.go: 32 goroutines, limit 1, observed n in {4,9,12}, -race DATA RACE at lines 18–19
coupon.go unlocked: 3× HTTP 200, remain in {-2,-1,0}
coupon.go locked: 1× 200, 7× 409
Wrong code: 400, remain unchanged (authz/validation, not a race)
Fix: mutex (single node) or UPDATE ... WHERE used=0 (multi node)
Out of scope: a script that beats a production promo

Commands appendix

1go run counter.go
2go run -race counter.go
3go run coupon.go   # then xargs -P8 curl ...
4go test -race ./...

Lab layout#

Artifact: lost update, measured#

Same bug behind HTTP#

Analysis steps I run on a real endpoint#

Sanitized reproduction#

Multi-node: the mutex is not enough#

Mitigation#

What I file after this lab#

Commands appendix#