Single-Flight-Based Cache Miss Coalescing

Single-Flight is a concurrency pattern that merges duplicate calls for the same key into a single execution. If 100 requests arrive simultaneously for the same item while the cache is empty, the database query executes only once, and all remaining requests receive the same value or error from that single execution.

Patterns like Cache-Aside are straightforward under normal conditions, but they reveal a weakness during the brief window when a popular key expires.¹ When multiple requests see the same cache miss and each independently falls through to the database or an external API, the load that the cache was absorbing all floods the origin storage at once. This phenomenon is called a cache stampede, or thundering herd. Google Zanzibar addressed this problem by maintaining a per-server lock table, ensuring that only one request among those sharing the same cache key is allowed to begin processing.²

The golang.org/x/sync supplementary module for Go projects includes a singleflight package that implements exactly this behavior. It is not part of the standard library, and the package documentation describes it as "duplicate function call suppression." If a task with the same key is already in progress, subsequent calls wait until the initial execution finishes and receive the same value and error.³

The internal execution model is also worth comparing. In Group.Do, the first caller executes fn directly in its own goroutine. Duplicate goroutines arriving with the same key release the mutex that protects the map, then wait for the result at WaitGroup.Wait(). The global mutex is not held throughout the loader execution. DoChan, by contrast, dispatches the work to a separate goroutine via go g.doCall(...) and immediately returns a result channel. The claim that "Go singleflight never creates background goroutines" holds true for Do but is incorrect when DoChan is included. (Personally, this is still a point of confusion for me, since I'm not yet fluent with Go.)⁴

What this registry stores is not a completed value but an execution that has not yet finished. Once the task completes, the in-flight entry is removed. TTL, eviction, and stale data remain concerns of the cache policy; Single-Flight only reduces duplicate executions during the overlapping window.

Core formula:

Text

Single-Flight
= N concurrent jobs sharing the same key
→ only 1 actual loader execution
→ N waiters share the result

In-Flight Registry
= Key → the Promise / Task / Future currently in progress

Cache
= retains completed values for a set duration

Single-Flight
= only temporarily shares work that hasn't completed yet

Invariants:
For the same key, at most one loader runs at a time.
Loaders for different keys run independently of each other.
Once a loader finishes, its entry is removed regardless of success or failure.
If one waiter leaves, the loader still runs for the remaining waiters.

1. Problem scenario

Suppose a product detail service uses the following cache-aside flow.

Text

Product lookup request
→ cache lookup
→ if cache hit, return immediately
→ if cache miss, query DB
→ store in cache
→ return result

Under normal conditions this works without issue. However, if 100 requests arrive simultaneously at the moment the cache for the popular item product-42 expires, the system can behave as follows. (After all, 42 is the answer to life, the universe, and everything, so of course it would be a popular item.)

Text

Request 1 cache miss → DB query
Request 2 cache miss → DB query
Request 3 cache miss → DB query
...
Request 100 cache miss → DB query

The desired behavior is as follows.

Text

Request 1
→ acquires the loader execution lock for product-42
→ performs DB query and cache write

Requests 2–100
→ detects that the same product-42 loader is already in-flight
→ waits for the existing operation without issuing a separate DB query
→ shares the same result

Requests for other products must still be processed in parallel.

Text

product-42 → 1 loader
product-77 → 1 loader

Loaders for both keys can run concurrently

The scope of this document extends to merging concurrent cache misses for the same cache key, within a single process, into one loader execution. Coordination across multiple servers is covered separately later.

2. Core concepts

The common API is as follows.

Text

SingleFlight.run(key, loader, callerCancellation)

key
= the unit of deduplication

loader
= the shared work that actually queries the DB or API and populates the cache

callerCancellation
= cancels only the individual caller's wait

serviceShutdown
= a parent lifetime signal that can cancel the shared loader itself

Canceling a shared loader that 99 other callers are waiting on just because one caller disconnected is unacceptable. Therefore, caller cancellation and shared-task cancellation are kept separate.

If the return value is a reference type, all waiters may receive the same object. Treat shared results as immutable values, or copy them at the call boundary. Single-Flight does not protect against a scenario where one waiter mutates the result object and another waiter observes those changes.

C#

using System;
using System.Collections.Generic;
using System.Collections.Concurrent;
using System.Threading;
using System.Threading.Tasks;

public sealed class SingleFlight<TKey, TValue>
    where TKey : notnull
{
    private readonly ConcurrentDictionary<
        TKey,
        Lazy<Task<TValue>>> inFlight = new();

    private readonly CancellationToken shutdownToken;

    public SingleFlight(
        CancellationToken shutdownToken = default)
    {
        this.shutdownToken = shutdownToken;
    }

    public async Task<TValue> RunAsync(
        TKey key,
        Func<CancellationToken, Task<TValue>> loaderAsync,
        CancellationToken callerToken = default)
    {
        ArgumentNullException.ThrowIfNull(loaderAsync);

        Lazy<Task<TValue>>? candidate = null;

        candidate = new Lazy<Task<TValue>>(
            () => this.ExecuteAndReleaseAsync(
                key,
                candidate!,
                loaderAsync),
            LazyThreadSafetyMode.ExecutionAndPublication);

        Lazy<Task<TValue>> selected =
            this.inFlight.GetOrAdd(key, candidate);

        return await selected.Value
            .WaitAsync(callerToken)
            .ConfigureAwait(false);
    }

    public int GetInFlightCount()
    {
        return this.inFlight.Count;
    }

    private async Task<TValue> ExecuteAndReleaseAsync(
        TKey key,
        Lazy<Task<TValue>> entry,
        Func<CancellationToken, Task<TValue>> loaderAsync)
    {
        try
        {
            return await loaderAsync(this.shutdownToken)
                .ConfigureAwait(false);
        }
        finally
        {
            this.RemoveIfCurrent(key, entry);
        }
    }

    private void RemoveIfCurrent(
        TKey key,
        Lazy<Task<TValue>> entry)
    {
        // If you remove an entry by key alone, you may accidentally delete the new entry that replaced it.
        // Delete the current entry only when both the key and value match.
        this.inFlight.TryRemove(new KeyValuePair<
            TKey,
            Lazy<Task<TValue>>>(key, entry));
    }
}

The value factory of ConcurrentDictionary.GetOrAdd() can be invoked multiple times under contention. Only one of the pre-constructed values ends up in the dictionary; the factory itself is not guaranteed to run exactly once.⁵ For this reason, wrap the actual loader inside a Lazy<Task<TValue>>, and only start the Value of the Lazy that the dictionary selects.

Individual callers cancel only their own wait via WaitAsync(callerToken). Only the service-shutdown shutdownToken is passed to the shared loader.

Use TryRemove(KeyValuePair<TKey,TValue>) for cleanup. This overload removes an entry only when both the key and value match the current entry, preventing a stale task from deleting an entry that was already registered for the same key by a newer task.⁶

It is good practice to set a loader timeout separately from the service-shutdown token. Because all waiters are blocked on a single task, a loader that hangs indefinitely keeps both the entry and the waiters alive. Note, however, that a CancellationToken is not a hard-abort mechanism; the timeout takes effect only if the DB driver or HTTP client actually forwards the token to the underlying I/O.

TypeScript

export type AsyncLoader<TValue> = (
  shutdownSignal?: AbortSignal,
) => Promise<TValue>;

export class SingleFlight<TValue>
{
  readonly #inFlight =
    new Map<string, Promise<TValue>>();

  readonly #shutdownSignal: AbortSignal | undefined;

  public constructor(
    shutdownSignal?: AbortSignal,
  )
  {
    this.#shutdownSignal = shutdownSignal;
  }

  public async runAsync(
    key: string,
    loaderAsync: AsyncLoader<TValue>,
    callerSignal?: AbortSignal,
  ): Promise<TValue>
  {
    validateKey(key);
    callerSignal?.throwIfAborted();

    const sharedPromise = this.#getOrCreate(
      key,
      loaderAsync,
    );

    return await waitForSharedAsync(
      sharedPromise,
      callerSignal,
    );
  }

  public getInFlightCount(): number
  {
    return this.#inFlight.size;
  }

  #getOrCreate(
    key: string,
    loaderAsync: AsyncLoader<TValue>,
  ): Promise<TValue>
  {
    const existing = this.#inFlight.get(key);

    if (existing !== undefined) {
      return existing;
    }

    const created = Promise.resolve().then(() =>
    {
      return loaderAsync(this.#shutdownSignal);
    });

    this.#inFlight.set(key, created);
    this.#registerCleanup(key, created);

    return created;
  }

  #registerCleanup(
    key: string,
    promise: Promise<TValue>,
  ): void
  {
    const cleanup = (): void =>
    {
      if (this.#inFlight.get(key) === promise) {
        this.#inFlight.delete(key);
      }
    };

    void promise.then(cleanup, cleanup);
  }
}

function waitForSharedAsync<TValue>(
  sharedPromise: Promise<TValue>,
  callerSignal?: AbortSignal,
): Promise<TValue>
{
  if (callerSignal === undefined) {
    return sharedPromise;
  }

  return new Promise<TValue>((resolve, reject) =>
  {
    let settled = false;

    const cleanup = (): void =>
    {
      callerSignal.removeEventListener(
        "abort",
        onAbort,
      );
    };

    const onAbort = (): void =>
    {
      if (settled) {
        return;
      }

      settled = true;
      cleanup();
      reject(callerSignal.reason);
    };

    callerSignal.addEventListener(
      "abort",
      onAbort,
      { once: true },
    );

    // An abort can occur between the preliminary `throwIfAborted()` call and the listener registration.
    // After attaching the listener, check the state again to close that gap.
    if (callerSignal.aborted) {
      onAbort();
      return;
    }

    sharedPromise.then(
      (value) =>
      {
        if (settled) {
          return;
        }

        settled = true;
        cleanup();
        resolve(value);
      },
      (error: unknown) =>
      {
        if (settled) {
          return;
        }

        settled = true;
        cleanup();
        reject(error);
      },
    );
  });
}

function validateKey(key: string): void
{
  if (key.trim().length === 0) {
    throw new RangeError(
      "A Single-Flight key cannot be empty.",
    );
  }
}

In JavaScript, there is no await between Map.get() and Map.set(), so entry registration completes synchronously within the same event loop turn. Loader execution is deferred to after Promise.resolve().then(...) so that the entry is registered first.

A caller's AbortSignal cancels only the wait inside waitForSharedAsync(). The shared Promise already in flight continues executing for the benefit of other waiters. This aligns with the MDN guideline that an AbortSignal-based Promise API should reject an unfinished operation with the signal's reason.⁷

Checking throwIfAborted() at the start alone is not sufficient; if cancellation arrives immediately after that check but before the abort listener is registered, the event will be missed. The example addresses this by checking aborted once more after registering the listener, which is generally safe.

Python

from __future__ import annotations

import asyncio
from collections.abc import Awaitable, Callable
from typing import Generic, TypeVar


TValue = TypeVar("TValue")
AsyncLoader = Callable[[], Awaitable[TValue]]


class SingleFlight(Generic[TValue]):
    def __init__(self) -> None:
        self._in_flight: dict[
            str,
            asyncio.Task[TValue],
        ] = {}

        self._lock = asyncio.Lock()

    async def run(
        self,
        key: str,
        loader_async: AsyncLoader[TValue],
    ) -> TValue:
        _validate_key(key)

        task = await self._get_or_create(
            key,
            loader_async,
        )

        return await asyncio.shield(task)

    def get_in_flight_count(self) -> int:
        return len(self._in_flight)

    async def _get_or_create(
        self,
        key: str,
        loader_async: AsyncLoader[TValue],
    ) -> asyncio.Task[TValue]:
        async with self._lock:
            existing = self._in_flight.get(key)

            if existing is not None:
                return existing

            task = asyncio.create_task(
                self._execute_and_release(
                    key,
                    loader_async,
                ),
                name=f"single-flight:{key}",
            )

            self._in_flight[key] = task
            task.add_done_callback(
                _observe_task_completion
            )

            return task

    async def _execute_and_release(
        self,
        key: str,
        loader_async: AsyncLoader[TValue],
    ) -> TValue:
        try:
            return await loader_async()
        finally:
            current_task = asyncio.current_task()

            async with self._lock:
                current = self._in_flight.get(key)

                if current is current_task:
                    self._in_flight.pop(key, None)


def _observe_task_completion(
    task: asyncio.Task[object],
) -> None:
    if task.cancelled():
        return

    task.exception()


def _validate_key(key: str) -> None:
    if key.strip() == "":
        raise ValueError(
            "The Single-Flight key cannot be empty."
        )

asyncio.shield() prevents the shared task from being canceled when an individual waiter is canceled. This is also why the Python documentation warns you to keep a strong reference to the task passed to shield(): the event loop may hold only a weak reference to a task, so in this example the _in_flight dictionary keeps the shared task alive.⁸

Cleanup is performed in the loader's finally block. Because this makes the timing of entry removal clearer than scheduling a separate cleanup task in a completion callback, a done callback is retained so that exceptions are observable even when the loader fails after all waiters have left.

Rust

use std::collections::HashMap;
use std::future::Future;
use std::hash::Hash;
use std::sync::atomic::{
    AtomicU64,
    Ordering,
};
use std::sync::Arc;

use tokio::sync::{
    Mutex,
    Notify,
    OnceCell,
};

#[derive(Clone, Debug)]
pub enum SingleFlightError<TError> {
    Loader(TError),
    LoaderTaskFailed(String),
}

struct Entry<TValue, TError> {
    id: u64,
    result: OnceCell<
        Result<TValue, SingleFlightError<TError>>
    >,
    notify: Notify,
}

impl<TValue, TError> Entry<TValue, TError> {
    fn new(id: u64) -> Self {
        Self {
            id,
            result: OnceCell::new(),
            notify: Notify::new(),
        }
    }
}

pub struct SingleFlight<TKey, TValue, TError> {
    in_flight:
        Mutex<HashMap<TKey, Arc<Entry<TValue, TError>>>>,

    next_id: AtomicU64,
}

impl<TKey, TValue, TError>
    SingleFlight<TKey, TValue, TError>
where
    TKey:
        Eq
        + Hash
        + Clone
        + Send
        + 'static,

    TValue:
        Clone
        + Send
        + Sync
        + 'static,

    TError:
        Clone
        + Send
        + Sync
        + 'static,
{
    pub fn new() -> Arc<Self> {
        Arc::new(Self {
            in_flight: Mutex::new(HashMap::new()),
            next_id: AtomicU64::new(1),
        })
    }

    pub async fn run<TLoader, TFuture>(
        self: &Arc<Self>,
        key: TKey,
        loader: TLoader,
    ) -> Result<TValue, SingleFlightError<TError>>
    where
        TLoader:
            FnOnce() -> TFuture
            + Send
            + 'static,

        TFuture:
            Future<Output = Result<TValue, TError>>
            + Send
            + 'static,
    {
        let (entry, is_leader) =
            self.get_or_create(&key).await;

        if is_leader {
            self.spawn_loader(
                key,
                Arc::clone(&entry),
                loader,
            );
        }

        wait_for_result(entry).await
    }

    pub async fn get_in_flight_count(&self) -> usize {
        self.in_flight.lock().await.len()
    }

    async fn get_or_create(
        &self,
        key: &TKey,
    ) -> (Arc<Entry<TValue, TError>>, bool) {
        let mut in_flight = self.in_flight.lock().await;

        if let Some(entry) = in_flight.get(key) {
            return (Arc::clone(entry), false);
        }

        let id = self.next_id.fetch_add(
            1,
            Ordering::Relaxed,
        );

        let entry = Arc::new(Entry::new(id));

        in_flight.insert(
            key.clone(),
            Arc::clone(&entry),
        );

        (entry, true)
    }

    fn spawn_loader<TLoader, TFuture>(
        self: &Arc<Self>,
        key: TKey,
        entry: Arc<Entry<TValue, TError>>,
        loader: TLoader,
    )
    where
        TLoader:
            FnOnce() -> TFuture
            + Send
            + 'static,

        TFuture:
            Future<Output = Result<TValue, TError>>
            + Send
            + 'static,
    {
        let owner = Arc::clone(self);

        tokio::spawn(async move
        {
            // loader를 한 단계 안쪽 task로 실행한다.
            // loader 내부 panic!이나 task abort는 JoinError가 된다.
            // 이를 terminal error로 공유해야 waiter가 영구 대기하지 않는다.
            let loader_task = tokio::spawn(async move {
                loader().await
            });

            let result = match loader_task.await {
                Ok(Ok(value)) => Ok(value),
                Ok(Err(error)) => {
                    Err(SingleFlightError::Loader(error))
                }
                Err(join_error) => {
                    Err(SingleFlightError::LoaderTaskFailed(
                        join_error.to_string(),
                    ))
                }
            };

            let _ = entry.result.set(result);
            entry.notify.notify_waiters();

            owner.remove_if_current(
                &key,
                entry.id,
            ).await;
        });
    }

    async fn remove_if_current(
        &self,
        key: &TKey,
        entry_id: u64,
    ) {
        let mut in_flight = self.in_flight.lock().await;

        let should_remove = in_flight
            .get(key)
            .is_some_and(|entry| entry.id == entry_id);

        if should_remove {
            in_flight.remove(key);
        }
    }
}

async fn wait_for_result<TValue, TError>(
    entry: Arc<Entry<TValue, TError>>,
) -> Result<TValue, SingleFlightError<TError>>
where
    TValue: Clone,
    TError: Clone,
{
    loop {
        if let Some(result) = entry.result.get() {
            return result.clone();
        }

        let notified = entry.notify.notified();

        if let Some(result) = entry.result.get() {
            return result.clone();
        }

        notified.await;
    }
}

Required dependencies:

TOML

[dependencies]
tokio = {
    version = "1",
    features = [
        "macros",
        "rt-multi-thread",
        "sync"
    ]
}

The Rust implementation runs the loader as a separate Tokio task. Even if an individual waiter future is dropped, the loader continues executing and other waiters can still receive the same result.

If the loader panics, the original task may terminate before populating the OnceCell or removing the entry. In that case, waiters would never wake up and the entry would remain in the map. In the example, the loader runs inside an inner task and JoinError is mapped to LoaderTaskFailed, so that panics and aborts, not just successes and business errors, become terminal states visible to all waiters.

The ordering in wait_for_result(), where notified() is created before re-checking the result, is also intentional. Tokio's notify_waiters() guarantees that a notification will be delivered to a Notified future created before the notification occurs, even if that future has not yet been polled.⁹ If you replace this with notify_one() without relying on that guarantee, you must also consider additional enable() handling.

3. Call Site

Single-Flight is placed at the cache miss boundary of the cache-aside pattern.

TypeScript

type Product = Readonly<{
  productId: string;
  displayName: string;
  priceCents: number;
}>;

type ProductCache = Readonly<{
  getAsync: (
    cacheKey: string,
    signal?: AbortSignal,
  ) => Promise<Product | undefined>;

  trySetAsync: (
    cacheKey: string,
    product: Product,
    ttlSeconds: number,
    signal?: AbortSignal,
  ) => Promise<boolean>;
}>;

type ProductRepository = Readonly<{
  getByIdAsync: (
    productId: string,
    signal?: AbortSignal,
  ) => Promise<Product>;
}>;

export class ProductQueryService
{
  readonly #cache: ProductCache;
  readonly #repository: ProductRepository;
  readonly #singleFlight: SingleFlight<Product>;
  readonly #cacheTtlSeconds: number;

  public constructor(
    cache: ProductCache,
    repository: ProductRepository,
    singleFlight: SingleFlight<Product>,
    cacheTtlSeconds: number,
  )
  {
    validateCacheTtl(cacheTtlSeconds);

    this.#cache = cache;
    this.#repository = repository;
    this.#singleFlight = singleFlight;
    this.#cacheTtlSeconds = cacheTtlSeconds;
  }

  public async getByIdAsync(
    productId: string,
    callerSignal?: AbortSignal,
  ): Promise<Product>
  {
    validateProductId(productId);
    const cacheKey = productCacheKey(productId);

    const cached = await this.#cache.getAsync(
      cacheKey,
      callerSignal,
    );

    if (cached !== undefined) {
      return cached;
    }

    return await this.#singleFlight.runAsync(
      cacheKey,
      (sharedSignal) =>
      {
        return this.#loadAndCacheAsync(
          cacheKey,
          productId,
          sharedSignal,
        );
      },
      callerSignal,
    );
  }

  async #loadAndCacheAsync(
    cacheKey: string,
    productId: string,
    sharedSignal?: AbortSignal,
  ): Promise<Product>
  {
    const cached = await this.#cache.getAsync(
      cacheKey,
      sharedSignal,
    );

    if (cached !== undefined) {
      return cached;
    }

    const product = await this.#repository.getByIdAsync(
      productId,
      sharedSignal,
    );

    // If the origin lookup succeeded, a cache write failure should
    // be handled on a best-effort basis so it does not propagate into a full read failure.
    await this.#cache.trySetAsync(
      cacheKey,
      product,
      this.#cacheTtlSeconds,
      sharedSignal,
    );

    return product;
  }
}

function productCacheKey(productId: string): string
{
  return `product:detail:v1:${productId}`;
}

function validateProductId(productId: string): void
{
  if (!/^[A-Za-z0-9_-]{1,64}$/.test(productId)) {
    throw new RangeError(
      "The `productId` format is invalid.",
    );
  }
}

function validateCacheTtl(cacheTtlSeconds: number): void
{
  if (!Number.isSafeInteger(cacheTtlSeconds)
    || cacheTtlSeconds < 1
    || cacheTtlSeconds > 86_400) {
    throw new RangeError(
      "The cache TTL must be between 1 and 86,400 seconds (inclusive).",
    );
  }
}

Inside the loader, the cache is checked one more time, because another process may have populated the cache between the initial cache miss and the loader execution.

The Single-Flight registry and the cache use the same canonical key. If the key builders for these two diverge, items that are the same in the cache may appear as different operations in the registry, or conversely, different results may be coalesced into one.

Cache writes are treated as a best-effort boundary. If a successful DB lookup causes all waiters to receive a failure simply because a single Redis write failed, the cache ends up reducing availability.

The trySetAsync() implementation logs failures and records them as metrics but returns false, while errors from the origin store are propagated as loader errors.

Typical separation of concerns:

Text

ProductCache
= stores completed values for the TTL duration

ProductRepository
= the actual I/O that queries the DB or external API

SingleFlight
= shares an in-progress loader for the same key

ProductQueryService
= orchestrates the sequence of cache hit/miss and loader invocation

Caller Signal
= cancels the wait for an individual request

Shared Shutdown Signal
= cancels the shared loader on process shutdown

Cache TTL Policy
= determines data freshness and expiration timing

Single-Flight has no knowledge of Product's business semantics or TTL. The cache has no knowledge of how many waiters the current loader is shared among.

This separation is not strictly required, but it reflects what I arrived at after personally synthesizing most of the examples. In other words, this is the average-looking arrangement; what matters is understanding the rules.

An individual caller's cancellation must not cancel the shared loader.

Everything else is simply a matter of how things change depending on the scale of the project.

4. Key Sequence

Text

What is the merge key?
→ Do requests for the same operation truly use the same key?
→ Is the in-flight entry registered before the loader starts?
→ Is only one loader executed per identical key?
→ Can different keys execute in parallel?
→ Is the entry removed after completion, failure, or cancellation?
→ Is a loader failure propagated identically to all waiters?
→ Does canceling an individual waiter avoid canceling the shared loader?
→ Does the loader re-check the cache before proceeding?
→ Is it clear whether this registry is process-local or distributed?

The first thing to check is not the existence of a lock, but the key design that determines what counts as the same operation.

5. Boundaries and Misconceptions

It is important to understand that Single-Flight is not a cache. The two are similar but fundamentally different in nature. A cache stores a value, while Single-Flight stores a task.

Text

Cache:
Reuse completed results for the duration of the TTL

Single-Flight:
Share only identical operations that are currently in-flight
Remove the entry after completion

Therefore, if a new request arrives immediately after the loader finishes, the cache must be holding the result. If you use Single-Flight alone without storing results in a cache, each sequentially arriving request will re-execute the loader every time.

The coalescing key must include every condition that affects the result.

Text

Bad key:
productId

If the actual result varies depending on:
- tenantId
- locale
- currency
- permission scope
- API version

Recommended key:
tenantId + productId + locale + currency + policyVersion

If requests with different permissions are merged under the same key, one user's result may be delivered to another user. This is not merely a performance bug; it is a data exposure issue.

The implementation covered in this post operates only within a single process.

Text

Server instance A:
1 product-42 loader

Server instance B:
1 product-42 loader

Overall system:
Up to 2 loaders running concurrently

If you need to allow only one loader across multiple instances, you will need a distributed lock, a lease, an atomic primitive from your cache provider, or a separate coordinator. However, a distributed lock only determines who gets to execute; it does not propagate the result. You also need to decide whether the leader fills the cache and followers re-read from it, whether results are broadcast via pub/sub, and whether a fencing token is used to block a leader that finishes late after its lease has expired. Simply adding a Redis lock on top of a process-local Single-Flight does not give you the same guarantees.

Individual waiter cancellation policy also matters. If the first request starts the loader and then disconnects, canceling the shared loader would cause all other waiters to fail. In this implementation, caller cancellation stops only the wait, not the loader itself.

There is also a cost in the opposite direction.

Text

All waiters are canceled
→ the loader can continue executing

If the loader is very expensive, you could consider tracking a waiter reference count and canceling the loader when the last waiter leaves. However, this complicates the interaction between cancellation and re-entry races, so it is better not to add this without measuring first.

Failures are also shared to some extent.

Text

Same-key loader hits a DB timeout
→ all waiters receive the same timeout error
→ in-flight entry is removed
→ the next request can execute a new loader

You must not permanently store a failed result in the in-flight registry. If you do, the failure can keep replaying even after a transient fault has recovered. On the other hand, if repeated lookups for a non-existent item are the problem, that should be handled with a negative caching policy, not Single-Flight.

Even immediately after a failure, a small wave of requests can appear.

Text

loader fails once
→ 500 waiters receive the failure simultaneously
→ all upper layers immediately retry
→ 500 requests pile into the next single-flight loader again

Review retry limits, exponential backoff, jitter, short-duration failure damping, and negative caching together. Single-Flight reduces duplicate executions but does not replace a retry policy.

Loaders require an upper time bound. Because all waiters share the same completion object, a single loader that never terminates will hold both the registry entry and its waiters indefinitely. Timeouts must be applied to the shared loader's I/O boundary, not just to individual waiters.

Key cardinality should not be left unbounded. If external input flows directly into keys and an attacker sends a different value each time, each key becomes a new, unmerged operation. Apply whatever controls are appropriate: a maximum in_flight size, key validation, request rate limiting, a waiter cap, or admission control.

Cases that fail in production:

Text

A new loader is created on every cache miss
The loader starts before the in-flight entry is registered
Entries are not removed after completion, causing memory growth
Failed Promises or Tasks are retained indefinitely
Caller cancellation propagates to and cancels the shared loader
Tenant and permission information is excluded from the key
Using an object reference as the key prevents semantically identical requests from being merged
Process-local coalescing is mistaken for a distributed guarantee
The cache is not checked again inside the loader
Single-Flight is mistakenly assumed to also handle TTL and stale policies
Without a loader timeout, entries and waiters remain indefinitely
Attacker-controlled input causes in-flight key cardinality to grow without bound
Waiters mutate the shared mutable result, interfering with each other
After a loader failure, all waiters immediately retry at once
A distributed lock is assumed to also handle result sharing

6. Bad Example

TypeScript

async function getProductBadAsync(
  productId: string,
  cache: ProductCache,
  repository: ProductRepository,
  signal?: AbortSignal,
): Promise<Product>
{
  const cacheKey = productCacheKey(productId);

  const cached = await cache.getAsync(
    cacheKey,
    signal,
  );

  if (cached !== undefined) {
    return cached;
  }

  const product = await repository.getByIdAsync(
    productId,
    signal,
  );

  await cache.trySetAsync(
    cacheKey,
    product,
    300,
    signal,
  );

  return product;
}

Looking at a single request, this is correct cache-aside code. With concurrent requests, however, the following race occurs.

Text

Request A cache miss
Request B cache miss
Request C cache miss

A DB query
B DB query
C DB query

Why it's bad:

Text

There is no concurrency coordination between a cache miss and the DB query.
When a popular key expires, backend requests spike momentarily.
The same operation that populates the cache runs multiple times.
Every request consumes a DB connection and network slot individually.
The problem is difficult to detect in normal testing.

Placing a single global mutex around the entire function is not a good solution either.

Text

global lock:
Looking up product-42
→ product-77 lookup also waiting

Never forget that Single-Flight must serialize only the same key, not the entire service.

7. Production Scaling

Pin the core contract with concurrency tests.

TypeScript

import assert from "node:assert/strict";
import test from "node:test";

test("Concurrent requests for the same key execute the loader only once", async () =>
{
  const singleFlight = new SingleFlight<string>();
  let loaderCallCount = 0;

  const loaderAsync = async (): Promise<string> =>
  {
    loaderCallCount += 1;
    await delayAsync(20);

    return "product-42";
  };

  const results = await Promise.all(
    Array.from(
      { length: 50 },
      () => singleFlight.runAsync(
        "product-42",
        loaderAsync,
      ),
    ),
  );

  assert.equal(loaderCallCount, 1);

  assert.deepEqual(
    new Set(results),
    new Set(["product-42"]),
  );

  assert.equal(
    singleFlight.getInFlightCount(),
    0,
  );
});

test("Requests for different keys each execute their own loader", async () =>
{
  const singleFlight = new SingleFlight<string>();
  let loaderCallCount = 0;

  const loadAsync = async (
    value: string,
  ): Promise<string> =>
  {
    loaderCallCount += 1;
    await delayAsync(10);

    return value;
  };

  const [first, second] = await Promise.all([
    singleFlight.runAsync(
      "product-42",
      () => loadAsync("first"),
    ),
    singleFlight.runAsync(
      "product-77",
      () => loadAsync("second"),
    ),
  ]);

  assert.equal(loaderCallCount, 2);
  assert.equal(first, "first");
  assert.equal(second, "second");
});

test("Failed entries are removed so that the next request can retry", async () =>
{
  const singleFlight = new SingleFlight<string>();
  let attemptCount = 0;

  await assert.rejects(() =>
  {
    return singleFlight.runAsync(
      "product-42",
      async () =>
      {
        attemptCount += 1;
        throw new Error("temporary failure");
      },
    );
  });

  const result = await singleFlight.runAsync(
    "product-42",
    async () =>
    {
      attemptCount += 1;
      return "recovered";
    },
  );

  assert.equal(attemptCount, 2);
  assert.equal(result, "recovered");
});

test("Cancellation by one waiter does not cancel the shared loader", async () =>
{
  const singleFlight = new SingleFlight<string>();
  const controller = new AbortController();

  let loaderCallCount = 0;

  const loaderAsync = async (): Promise<string> =>
  {
    loaderCallCount += 1;
    await delayAsync(30);

    return "shared-result";
  };

  const cancelledWaiter = singleFlight.runAsync(
    "product-42",
    loaderAsync,
    controller.signal,
  );

  const survivingWaiter = singleFlight.runAsync(
    "product-42",
    loaderAsync,
  );

  controller.abort(
    new Error("caller disconnected"),
  );

  await assert.rejects(
    () => cancelledWaiter,
  );

  assert.equal(
    await survivingWaiter,
    "shared-result",
  );

  assert.equal(loaderCallCount, 1);
});

function delayAsync(
  milliseconds: number,
): Promise<void>
{
  return new Promise((resolve) =>
  {
    setTimeout(resolve, milliseconds);
  });
}

Operational metrics:

Text

single_flight.request.count
single_flight.loader.started.count
single_flight.waiter.shared.count
single_flight.loader.succeeded.count
single_flight.loader.failed.count
single_flight.in_flight
single_flight.waiter.duration_ms
single_flight.loader.duration_ms

Fix derived values by recording both the name and the formula together. Using only the name coalescing ratio can cause confusion, as different teams sometimes interpret it as the reciprocal.

Text

requests per loader
Total Single-Flight requests / Actual loader executions

e.g.,
1,000 requests
50 loader executions

requests per loader = 20

loader suppression ratio
= 1 - (Actual loader executions / Total Single-Flight requests)

loader suppression ratio = 95%

Do not include raw product IDs or full cache keys as metric attributes. Use only fields with limited cardinality, such as operationName, result, and cacheRegion. OpenTelemetry also warns that attributes with ever-growing unique combinations, such as user IDs or raw URLs, can drive up memory costs and cause cardinality overflow.¹⁰

Items to review together in production:

Text

- cache hit ratio
Number of concurrent waiters per key
- loader p95/p99 latency
DB connection pool usage
Loader failure rate
Requests per loader per instance
Number of in-flight keys and number of rejected key creations
Current number of waiters per key and number of waiter limit reached events

If loader call count drops after applying Single-Flight but waiter latency increases excessively, you may have a hot-key problem where too many requests are tied to a single slow loader.

8. Comparison Notes: C# / TypeScript / Python / Rust

Language	Idiomatic Expression	Key Caveats
C#	`ConcurrentDictionary<TKey, Lazy<Task<T>>>`	Must distinguish between duplicate factory invocations and the observation cost of `Count`
TypeScript	`Map<string, Promise<T>>`	The Promise must be registered in the map before the loader starts
Python	`asyncio.Lock`, shared `Task`, `asyncio.shield`	Requires the assumption that everything runs within a single event loop
Rust	`Arc<Entry>`, `Mutex<HashMap>`, `Notify`, background task	waiter drop, loader panic, and entry cleanup must all be handled together

Language: C#
Idiomatic Expression: ConcurrentDictionary<TKey, Lazy<Task<T>>>
Key Caveats: Must distinguish between duplicate factory invocations and the observation cost of Count

Language: TypeScript
Idiomatic Expression: Map<string, Promise<T>>
Key Caveats: The Promise must be registered in the map before the loader starts

Language: Python
Idiomatic Expression: asyncio.Lock, shared Task, asyncio.shield
Key Caveats: Requires the assumption that everything runs within a single event loop

Language: Rust
Idiomatic Expression: Arc<Entry>, Mutex<HashMap>, Notify, background task
Key Caveats: waiter drop, loader panic, and entry cleanup must all be handled together

In C#, you need to account for the possibility that the value factory of ConcurrentDictionary.GetOrAdd() may be called multiple times under race conditions. Therefore, wrap the actual work inside Lazy<Task<T>> and execute only the selected entry.

The GetInFlightCount() method in the example is intended for debugging and low-frequency observation. Because ConcurrentDictionary.Count uses an internal lock, calling it on every request can cause contention.¹¹

If you need high-frequency metrics, maintain a separate Interlocked-based counter or use an approximate value that does not interfere with lifetime management. The counter should be incremented when an entry is successfully registered and decremented when the current entry is successfully removed.

In TypeScript, a Promise itself is a shareable completion representation. However, if you call the loader first and insert it into the map afterward, an exception thrown in the synchronous segment or duplicate entry can occur, so you must fix the order: register the entry first, then start the loader.

In Python, task cancellation of one waiter can propagate to the shared task being awaited. Use asyncio.shield() to decouple the lifetime of the shared operation from the lifetime of individual waiters.

The get_in_flight_count() method in the Python example reads len() directly under the assumption that the object is confined to a single event loop and thread. Within the same loop, since there is no await inside this method, no other coroutine can modify the map in between. Usage that accesses _in_flight or asyncio.Lock from another thread is not supported. This safety should not be explained as relying on CPython's GIL, because Python supports a free-threaded build, and asyncio synchronization primitives are not tools for synchronization between OS threads.¹²

In Rust, if a waiter drops its future, the work inside that future may be dropped as well. Spawning the loader as a separate Tokio task and sharing the result entry via Arc decouples the lifetime of individual waiters from the shared operation. However, spawning alone is not enough: if you do not convert a loader panic into a terminal result, waiters may remain blocked forever.

Regarding failure representation in general: in C#, TypeScript, and Python, a loader exception is naturally propagated as the failed state of the shared Task, Promise, or asyncio.Task. A panic in a spawned Rust task is not a business error TError, so the example promotes the JoinError to SingleFlightError::LoaderTaskFailed. This distinction allows the caller to log the error returned by the loader separately from the abnormal termination of the task itself.

The common characteristics shared across all implementations can be summarized as follows.

Text

Represent any in-flight work for the same key as a single completion object,
and have all waiters wait on that same completion object.

9. Further Considerations

If results vary by tenant, locale, or permissions, which values must be included in the Single-Flight key?
Should cancellation by one waiter stop the shared loader, or should it stop only that waiter's own wait?
Is it beneficial to keep the loader running and populate the cache even after all waiters have left?
Is process-local coalescing sufficient, or is distributed coordination spanning multiple instances necessary?
Should loader failures be retried immediately, or is a short negative cache or backoff needed?
When thousands of waiters are hanging on a single slow loader for a hot key, how do you bound memory usage and tail latency?

Single-Flight is not always necessary. If the loader is cheap and call volume is low, the registry and cancellation policy add more overhead than they save. Forcing independent requests onto the same key only increases latency, and merging write operations can lose per-call side effects or audit records. For operations like payments, message publishing, or sequence-number generation, you need to ask first whether requests that look identical must each be executed separately.

10. Summary

Single-Flight merges concurrent loader executions for the same key into one.
The cache stores completed values, while Single-Flight shares only in-flight work.
An entry must be registered before the loader starts and must be removed after success or failure.
The cancellation lifetime of individual callers must be kept separate from that of the shared loader.
Keys should include the tenant, permissions, locale, and policy version that affect the result.
Set upper bounds on loader timeouts, key counts, waiter counts, and retry bursts.
Define boundaries so that a cache write failure does not turn a successful origin lookup into a failure.
Process-local Single-Flight does not prevent duplicate executions across multiple server instances.

Rule of thumb:

Cache

Values are populated after the operation completes
TTL, eviction, and invalidation are preserved
Subsequent requests reuse the cached value

Single-Flight

Exists only while the operation is executing
Entry is removed on completion, failure, or cancellation
Subsequent requests start a new operation

Text

If the same cache key is missing concurrently,
don't let each request fall through to the backend independently; let one fetch the data and have the rest wait for that result.

Personal note: The core logic I picked up this time is a pattern I discovered while studying code from Chinese sources.

Cache<K, V> maps K → a completed V

SingleFlight<K, Task<V>> maps K → the currently in-flight Task<V>. There is actually a somewhat tricky point here.

The fundamental instinct in programming is really the same as in management: deciding what the minimum unit of control and identity should be. This is the management approach when you treat the Task as that minimum unit.

The long sections above are essentially a manual summarized point by point, but the simple idea is this: instead of treating multiple calls as separate requests, you group them into a single in-flight operation that produces one shared result.

Most programming problems reduce to the following questions.

What do you treat as a single unit?

What do you manage independently?

How far do you define an atomic (minimum-unit) state transition?

What do you assign identity and lifetime to?

It can be simple, or it can be complex.

Footnotes

Microsoft Azure Architecture Center. Cache-Aside pattern ↩
Pang, R. et al. Zanzibar: Google's Consistent, Global Authorization System (PDF). 2019 USENIX Annual Technical Conference. The cache stampede section of the paper describes per-server lock tables and the coalescing of requests for the same cache key. ↩
Go Project. `golang.org/x/sync/singleflight`: duplicate function call suppression ↩
Go Project. `x/sync/singleflight` source. You can see Do's direct execution and WaitGroup waiting, as well as DoChan's goroutine execution. ↩
Microsoft. `ConcurrentDictionary<TKey,TValue>.GetOrAdd` Remarks. Explains that valueFactory runs outside the lock and may be called multiple times during contention. ↩
Microsoft. `ConcurrentDictionary<TKey,TValue>.TryRemove(KeyValuePair<TKey,TValue>)`. Removes only the entry where both the key and value match. ↩
MDN Web Docs. `AbortSignal`. Includes examples of abort handling for Promise APIs and passing signal.reason. ↩
Python Software Foundation. `asyncio.shield`. Explains the separation between caller cancellation and shared task cancellation, along with a note on keeping strong references to tasks. ↩
Tokio. `tokio::sync::Notify`. Explains the wake-up guarantees between notify_waiters() and the Notified futures that were created beforehand. ↩
OpenTelemetry. Metrics cardinality limits. Explains why unique attribute combinations increase metric state and memory usage. ↩
Microsoft. Slow `ConcurrentDictionary.Count` lookup. Explains the locking cost of Count and alternatives when it is called at high frequency. ↩
Python Software Foundation. asyncio and free-threaded Python. Explains that event loops and tasks must not be shared across threads, and that primitives such as asyncio.Lock are not intended for thread synchronization. ↩