Connection resilience

How Propulsion behaves when the database goes away or refuses to cooperate: what it detects, what it recovers from on its own, and what it deliberately leaves to you.

Everything here is off by default except the detection itself. The two recovery features spend something real — a round trip, or running your closure more than once — so neither is switched on without a deployment having decided to.

All of it matters most under a persistent worker, where a connection outlives the request that opened it and can be reaped between requests by an idle timeout, a load balancer, a failover or a restart. Under PHP-FPM the connection died with the request, so most of this has nothing to catch.

Dropped-connection detection

A connection whose server-side session has gone away is detected wherever a statement runs, and evicted from the pool, so the next Propulsion::getConnection() opens a fresh one instead of handing out the corpse.

On detection:

• the transaction depth counter is zeroed — whatever the server had open is gone;
• buffered shared-query-cache version bumps are discarded, as on a rollback;
• the prepared-statement cache is dropped;
• the connection is evicted from the pool;
• a PSR-3 warning is logged, if a logger is registered;
• the original exception is rethrown unchanged.

That last point is deliberate: this is a notification seam, not an error-handling one. Code already catching PDOException sees exactly what it saw before.

The statement is not retried, and cannot be. PDO has no reconnect — a dropped connection’s object stays dropped for its lifetime — and retrying one level up is not safe either, because losing the connection loses any open transaction with it, so re-running a single statement on a fresh connection would execute it outside the transaction the caller believes it is in. Recovery has to happen where the transaction boundary is known, which is what transaction retry below is for.

Why this needed PropulsionStatement

Detection used to live on exec(), query() and DebugPDOStatement::execute(), and essentially nothing goes through any of those: every generated Peer and ModelCriteria path prepares a statement and calls execute() on it, which outside debug mode was a plain PDOStatement that nothing wrapped. PropulsionStatement is now the statement class on every non-persistent connection, and DebugPDOStatement extends it, so debug mode inherits the detection rather than repeating it. useDebug(false) resets to PropulsionStatement, not to a plain PDOStatement — turning debugging off must not also turn off a correctness feature.

Persistent connections keep the gap. PDO refuses a custom statement class when PDO::ATTR_PERSISTENT is set, so they keep plain PDOStatement; Propulsion suppresses that failure rather than refusing to connect.

Pre-checkout liveness check

'connection' => [
    'liveness' => [
        'enabled'        => true,
        'idle_threshold' => 5.0,   // seconds; 0.0 pings every checkout
    ],
],

When enabled, Propulsion::getConnection() pings a pooled connection before handing it out, and replaces it if the ping fails. The ping is the cheapest statement that proves the server is still there — SELECT 1, or SELECT 1 FROM dual on Oracle.

Only genuinely idle connections are pinged. A connection that ran a statement within idle_threshold seconds is evidence of its own liveness, so under sustained traffic this collapses to roughly zero extra round trips while still covering the quiet period after which connections actually get reaped. Too low a threshold makes a busy process pay for checkouts where the connection demonstrably worked moments ago; too high and the window in which a reaped connection still looks fresh grows to match. The 5s default sits far below any idle timeout worth worrying about (MySQL’s wait_timeout is 8 hours, pgbouncer’s server_idle_timeout 10 minutes).

A connection is never pinged inside a transaction — the pool does not hand out connections mid-transaction, and a stray statement inside somebody’s transaction is worse than a stale connection.

This narrows the window between “we checked” and “you used it”; it cannot close it. A connection can still die in that gap, and then detection above takes over.

Transaction retry

'connection' => [
    'retry' => [
        'enabled'      => true,
        'max_attempts' => 3,      // total attempts, not retries
        'base_delay'   => 50,     // ms before the first retry
        'max_delay'    => 1000,   // ms ceiling
        'multiplier'   => 2.0,
        'jitter'       => 1.0,    // 1.0 = full jitter
    ],
],

With this on, Propulsion::transaction() retries the whole transaction when it fails transiently:

$book = Propulsion::transaction(function ($con) {
    $book = BookQuery::create()->filterByISBN($isbn)->findOne($con);
    $book->setStock($book->getStock() - 1);
    $book->save($con);

    return $book;
});

What gets retried

Only failures the adapter classifies as transient, plus connection drops before the commit:

Platform	Retryable
PostgreSQL	40001 serialization failure, 40P01 deadlock detected
MySQL / MariaDB	1213 deadlock (40001), 1205 lock-wait timeout (HY000)
SQLite	5 SQLITE_BUSY, 6 SQLITE_LOCKED
MSSQL	1205 deadlock victim
Oracle	ORA-00060 deadlock, ORA-08177 cannot serialize

These are the database working as designed — it aborts one transaction so another can proceed — and the loser is supposed to try again.

Backoff is exponential with full jitter: the delay before retry n is a uniform draw from [0, min(base × multiplier^(n-1), max)], not that bound itself. Un-jittered backoff is actively harmful here — a deadlock has at least two transactions in it, and if both back off by the same computed delay they collide again on the retry.

What does not get retried

• Anything the closure itself throws. A business failure is deterministic; re-running it burns transactions to reach the same answer.
• Non-transient database errors — constraint violations, syntax errors.
• A connection lost while the COMMIT was in flight. The important one: the transaction’s outcome is genuinely unknown, since the server may have committed and died before saying so, so re-running the closure could apply the work twice. It is rethrown for you to resolve, because you are the only one with the information to. A drop before the commit is issued is retried, because nothing was committed.
• Nested calls. A closure running inside an outer transaction gets a savepoint and is never retried: the failures being retried abort the entire transaction on most platforms, so the outer one is already dead. Retrying has to happen at the outermost boundary.

Your closure must be safe to run more than once

This is the price of retrying, and it cannot be paid on your behalf. Database work is undone by the rollback; anything the closure does outside the transaction is not — sending mail, charging a card, incrementing a Redis counter, mutating objects the caller kept a reference to. Keep side effects out of the closure, or opt one call out:

use Propulsion\Connection\RetryPolicy;

Propulsion::transaction($work, policy: RetryPolicy::none());

An explicitly passed RetryPolicy always overrides the configuration.

Doing it yourself

Everything above is automatic. If you catch a PDOException yourself — around raw SQL, or in a retry loop of your own — three methods give you the same primitives the automatic path uses.

Propulsion::isConnectionDropped(\Throwable $e): bool classifies an exception as a dropped connection rather than a query error. It checks the SQLSTATE class for PostgreSQL’s 08xxx connection-exception family, then falls back to matching the driver messages that carry no usable SQLSTATE — “server closed the connection unexpectedly”, “has gone away”, “connection reset by peer”, “broken pipe”, “connection timed out” and the rest:

try {
    $con->exec($sql);
} catch (\PDOException $e) {
    if (!Propulsion::isConnectionDropped($e)) {
        throw $e;                              // a real query error — not ours to retry
    }
    Propulsion::discardConnection($con);
    $con = Propulsion::getWriteConnection('bookstore');   // freshly built
    $con->exec($sql);
}

Propulsion::discardConnection(PDO|PropulsionPDO $con): bool evicts that exact object from the pool, whichever datasource and mode it was registered under, and returns whether it was pooled at all. Matching is by object identity because that is the only thing a caller reliably knows: a PropulsionPDO doesn’t carry the name it was registered under, and one object can occupy more than one slot — getSlaveConnection() stores the master under slave for a datasource with no replicas configured, so every matching slot is removed.

Propulsion::forceReconnect(?string $name = null) is the name-based version, and drops both the master and the slave entry for that datasource. Use it when you know the datasource but hold no connection object. It is the blunter of the two: discardConnection() won’t take a still-healthy sibling down with it.

Neither can revive a connection you’re holding

PDO has no reconnect. “Reconnecting” here only ever means the next getConnection() builds a fresh one — any caller still holding the old object holds a dead one, which is why the example above reassigns $con after discarding it.

What is still not handled

• Persistent connections get no statement-level detection, as above.
• The check-then-use window in the liveness check can be narrowed, not closed.
• DatabaseMaps survive a setConfiguration(). The connection map, the adapter map and the memoised default datasource name are all dropped and rebuilt; the schema maps are the one thing left alone, deliberately. See Known gaps.

Related

• Transactions — the transaction API this builds on, and the savepoint differences between platforms.
• Configuration file — the connection section’s keys.
• Observing queries — the other feature built on the same statement seam.
• Running under a persistent worker — where all of this matters.