Containment is measured on the calls that never needed you

Containment is the number every contact-centre programme is bought on: the share of sessions an automated agent finishes without handing the customer to a person. It is computed over sessions. The sessions an agent finishes are the ones that were never going to need a human, so the metric rises fastest when the agent is doing the least valuable work. It rises again when the caller gives up.

The other thing that separates this domain is that the agent can spend money. It prices a fare, holds a seat, moves a passenger from one aircraft to another. When it gets that wrong there is no rollback in the software sense, but there is a void if you catch it inside a window measured in hours. Miss the window and it is a refund, on a statutory clock.

This is what I learned building conversational booking and proactive rebooking for a national carrier: voice and chat, a global distribution system behind every useful action, purchases staged by the agent and approved by a person, and a holiday peak we surge-tested to and still failed.

What containment counts

Start with the denominator, because everything downstream inherits it. Over a representative week of 336,000 sessions, 74.1% ended without a transfer to a human agent. That is the number on the slide.

Per 1,000 sessions, 741 ended without a transfer and 259 were transferred. Of the 741, 86 were callers who hung up partway through. The formula scores those as containment because nothing was transferred. Another 41 produced a second contact on the same record locator within 72 hours. That leaves 614 resolved without a person, 61.4%, a 12.7-point gap between what the metric reports and what happened.

The larger correction is the denominator. Containment over all sessions measures the wrong population. Most of the traffic an agent picks up would have self-served or given up regardless. What a contact-centre director buys is human contacts avoided, which means knowing how many there would have been. We routed 4% of traffic past the agent to the previous flow for six weeks, 80,640 sessions, and 46.8% reached a human.

The agent arm has to be counted the same way, and the first pass didn’t do it. A transfer is not an arrival, and 18.5% of transferred calls hung up in the queue before anyone picked up. Counting only those that reached a person, plus the repeat contacts and call-backs that reach a desk, 26.0% did. That is 208 contacts per thousand removed, 44% of the counterfactual.

from dataclasses import dataclass

@dataclass
class Window:
    sessions: int
    no_transfer: int          # ended without a transfer to a human
    abandoned: int            # caller hung up before the task reached an outcome
    repeat_72h: int           # a later contact on the same PNR within 72 hours
    transferred: int
    holdout_sessions: int     # agent disabled, routed to the old flow
    holdout_reached_human: int

# a transfer is not an arrival: this share hangs up in the queue afterwards
QUEUE_ABANDON = 0.185

def containment(w: Window) -> dict:
    naive = w.no_transfer / w.sessions
    resolved = (w.no_transfer - w.abandoned - w.repeat_72h) / w.sessions
    # every route to a person, measured the same way the holdout arm was
    human = (w.transferred * (1 - QUEUE_ABANDON)
             + 0.62 * w.repeat_72h            # measured share of repeats reaching a desk
             + 0.39 * 0.71 * w.abandoned)     # call-back rate, then reach rate
    baseline = w.holdout_reached_human / w.holdout_sessions
    return {"naive": naive,
            "resolved": resolved,
            "human_contact_rate": human / w.sessions,
            "counterfactual": baseline,
            "deflection": (baseline - human / w.sessions) / baseline}

# one week, plus the six-week holdout it is measured against:
# Window(336_000, 248_976, 28_896, 13_776, 87_024, 80_640, 37_740) ->
#   naive 0.741   resolved 0.614   human_contact_rate 0.260
#   counterfactual 0.468   deflection 0.444

Having both numbers is what caught the experiment that would otherwise have shipped. The agent escalates when its confidence in completing a rebooking falls below a threshold. Lowering it from 0.72 to 0.55 makes the agent try harder before giving up. Over 42,000 sessions per arm, naive containment rose 3.3 points against a 0.58-point interval and resolution fell 3.6 points. Human contacts moved half a point the wrong way inside a 0.60-point interval, which I wouldn’t call either way. The mechanism is arithmetic. Transfers fell by 33 per thousand, but only 27 of those were arrivals at a desk. The extra abandons and repeat contacts sent 32 more people to a person by a slower route, so the net was five more human contacts per thousand sessions. The agent held onto harder cases, transferred fewer, and the same people rang back a day later.

Those queue abandons show up twice, which is how we found them. Amazon Connect ships outcome percentages per bot conversation and a handoff rate for AI agents, neither of which is containment over a whole contact. So both of ours are derived, and the two derivations disagree. Connect’s contact records count contacts and key everything on InitialContactId, so a transfer creates a second contact under the same initial id. Our store counts Lex sessions, and a transferred call is one session. Contact-record arithmetic reads 78.9% against our 74.1%, and the entire 4.8-point gap was that same population: 18.5% of 259 transfers per thousand is 48 sessions the contact records attribute to the queue and we attribute to the agent. Neither is wrong. They answer different questions, and only one belongs on a slide, with its definition printed underneath.

Underneath both sits a telemetry assumption I didn’t check for two months. Containment is derived from contact lifecycle events, and Amazon Connect’s contact events are documented as unordered and best effort. The stronger caveat turned out not to reach us at all. The note that a COMPLETED event might not be delivered is written about chat contacts, and it fires when an agent switches to offline without clearing the contact in the control panel. That means the contact had already reached an agent. Those are transfers, and a transfer is not contained under either definition, so the caveat lands on none of the population the metric is computed over.

What does bite is the plain best-effort delivery underneath it. A missing terminal event looks like a session that never ended, which our reducer aged out and scored as contained. It now reconciles nightly against contact records and marks the unmatched sessions unknown.

The gap the caller hears

What matters on a call is the gap between the caller finishing a sentence and hearing the first syllable back. On our traffic that gap has a median of 2,970 milliseconds on a turn that calls a tool, and the model accounts for 900 of it.

Medians and 95th percentiles over 2,940 instrumented voice turns across three weeks of production traffic, sampled at one in a thousand and reconstructed from span timestamps, not from any one service’s own latency field.

The two model passes account for 900 of the 2,970 ms, 30% of a turn that also carries a 740 ms third-party search. The single largest term is a timer. Amazon Lex decides the caller has stopped talking after a fixed silence, and the documented default for x-amz-lex:audio:end-timeout-ms is 600 milliseconds, spent on every turn before anything downstream is asked to do anything. On a turn with no tool call that is a third of the budget. Lowering it makes the agent feel sharper, but it also starts truncating callers who pause. On the slot where a caller reads a record locator we went the other way and raised it to 1,100, because people pause between characters. That added half a second to the turn that already had the highest error rate.

For scale: on the Fisher corpus the mean offset between one speaker finishing and the next starting is 300 milliseconds over almost eleven thousand conversations. The 228 ms standard deviation reported with it is the spread between speakers, not the spread of one speaker’s transitions, so an individual is steadier than that figure suggests. Our median no-tool turn is 1,810 ms, six times the mean. It sits on a transport separately budgeted against ITU-T G.114, whose 2003 edition recommends not exceeding 400 ms one way and puts transparent interactivity below 150 ms.

The guardrail line is a policy decision wearing a latency cost. Bedrock Guardrails in the streaming path runs synchronously by default, buffering response chunks until the scan completes. Asynchronous mode ships chunks immediately with no latency impact, but in AWS’s own words it does not support masking of sensitive information, and on a call where the customer reads a card number aloud to a desk agent on the same bridge, masking is not optional. So the 210 ms is what redaction costs, charged on every turn including the ones where nothing sensitive was said.

Now the bound check, because it is the mistake I have watched three teams make while building a latency budget in a spreadsheet. Measure the gap end to end, from the clock in the caller’s ear. Adding the component p95s gives 7,160 ms, although the measured end-to-end p95 is 4,180. Percentiles of independent components do not add, and the sum is a loose upper bound because the tail events rarely coincide. A budget built that way declares the design impossible and sends you optimising a component that is not on the critical path 95% of the time.

There is a null result here that I expected to be a finding. Bedrock’s Converse API takes performanceConfig={"latency": "optimized"}, and turning it on moved first-token p50 from 480 ms to 441. Across three weeks I couldn’t distinguish its effect on end-to-end turn p95 from week-to-week variation. Thirty-nine milliseconds is 1.3% of a turn whose two largest terms are a silence timer and a third-party search. We left it on, since it costs nothing, and moved the tuning effort to the endpointer and the filler policy.

The part of that API worth reading closely on a voice channel is the stop reason, because six of its nine values mean the caller is about to get silence:

import botocore.session

TERMINAL = {"end_turn", "stop_sequence"}          # say it and stop
RECOVERABLE = {"tool_use"}                        # another round trip, play a filler
DEGRADED = {"max_tokens", "guardrail_intervened", "content_filtered",
            "malformed_model_output", "malformed_tool_use",
            "model_context_window_exceeded"}

shape = botocore.session.get_session() \
    .get_service_model("bedrock-runtime").shape_for("StopReason")
assert set(shape.enum) == TERMINAL | RECOVERABLE | DEGRADED, sorted(shape.enum)
print(len(shape.enum), "stop reasons,", len(DEGRADED), "of them with nothing to say")

# 9 stop reasons, 6 of them with nothing to say

On a chat transport a degraded stop reason is a visible error state. On a call it is dead air, which the caller fills by talking, which the endpointer treats as a new turn against a session whose last turn never completed. Each of the six needs a spoken fallback and a transfer rule.

Interrupting an agent that is thinking

Callers interrupt the agent 0.7 times a session at steady state and 2.4 times during a disruption, when it is talking more and they are in a hurry. How fast the playback stops decides whether the thing feels like a conversation or like talking over a recording, and that is the one number in this article I never managed to measure.

Lex’s streaming conversation API emits a PlaybackInterruptionEvent carrying an eventReason of VOICE_START_DETECTED, DTMF_START_DETECTED or TEXT_DETECTED, plus the causedByEventId of the input that triggered it. Detection happens at Lex, against the audio the media leg is streaming up. The audio the caller hears is already committed to the media path, the carrier trunk and the player’s buffer.

Four terms sit between the caller’s first syllable and silence: the voiced frames the detector needs before it will call it speech, the trip up, the event coming back down, and the buffered audio draining out. Our working figures are 120, 90, 90 and 180 milliseconds, so about 480 in total. Of that, 37% is the buffer, and the buffer belongs to the telephony team. That is a model assembled from transport numbers we do measure elsewhere, not a measurement of the interrupt itself. The rest of this section is why.

A recent benchmark puts user-interruption response latency across four full-duplex systems at between 0.257 and 2.531 seconds, and I quoted that range for a while as though it made half a second unremarkable. It does not, and the paper’s own table is why. The slowest of the four also scores worst on content relevance, the second fastest scores second best on it, and the system the authors single out for handling interruptions well is credited with holding an acceptable latency while it does. There is no trade running along that axis to place 480 ms against. That costs us less than it sounds, because the point of this section is that the 480 was never measured.

Whether Python can be anywhere near that path is settled on the API page itself. The StartConversation reference carries an exhaustive allowlist of the SDKs that support the operation, and it is the AWS SDK for C++, the SDK for Java V2 and the SDK for Ruby V3. Boto3 is not on it. The generated See Also block further down the same page links nine SDKs and the CLI, Python among them, which is how a team talks itself into trying it anyway. The Nova Sonic documentation makes the allowlist’s point from the other side, listing the SDKs with bidirectional streaming support and sending Python users to a separate experimental package. The part worth an afternoon is why, because the reason is one line of botocore and it has nothing to do with audio:

import boto3, botocore.handlers, botocore.session

H2 = "Operation requires h2 which is currently unsupported in Python"
removed = [(event, fn.__name__)
           for event, fn, *_ in botocore.handlers.BUILTIN_HANDLERS
           if (fn.__doc__ or "").strip() == H2]

for event, fn in removed:
    print(event.split(".", 1)[1], "->", fn.removeprefix("remove_"))

model = botocore.session.get_session().get_service_model("lexv2-runtime")
client = boto3.client("lexv2-runtime", region_name="eu-west-1")
print("StartConversation in the service model:",
      "StartConversation" in model.operation_names,
      "| on the client:", hasattr(client, "start_conversation"))

# lex-runtime-v2 -> lex_v2_start_conversation
# qbusiness -> qbusiness_chat
# bedrock-runtime -> bedrock_runtime_invoke_model_with_bidirectional_stream
# connecthealth -> connecthealth_start_medical_scribe_listening_session
# polly -> polly_start_speech_synthesis_stream
# StartConversation in the service model: True | on the client: False

The docstring is the entire explanation: the operation needs HTTP/2 and botocore’s transport doesn’t speak it. Five operations are deleted from the class this way, at client construction, and one of them is qbusiness.Chat, which is text, so the criterion is the protocol rather than anything about live audio. StartConversation is fully described in the service model, input and output event-stream shapes and all, but the method is gone before you ever hold a client.

That decides an architecture. The media leg cannot live in the same runtime as the rest of the stack, so it runs inside Amazon Connect’s native Lex integration, with the Python orchestrator behind a Lambda hop and the chat channel talking to it directly. Interruption is then handled by two AWS services talking to each other with none of our code in the path.

Which is also where the measurement went. PlaybackInterruptionEvent is delivered on the StartConversation stream and nowhere else, so it terminates inside Connect. It has three fields, eventId, causedByEventId and eventReason, and none of them is a timestamp. Lex conversation logs record utterances rather than event-stream traffic. The contact record has no field for a caller talking over a prompt. We spent a month instrumenting the orchestrator before accepting that the interrupt happens where we have no instrument at all.

The worse instrument we settled for is the conversation log. It carries a bargeIn field per utterance, documented as a string rather than a boolean, next to the input timestamp and an audioProperties.duration that describes the caller’s own audio and not the prompt they talked over. So it yields counts and rates, how often callers interrupt and which prompts they interrupt, and nothing about how far into a prompt they got. That is enough to tune prompt length and filler policy, but it will never produce a latency. The 480 ms stayed an estimate on a whiteboard.

The coupling to tool calls is the part I didn’t anticipate. A turn that hits the low-fare search leaves the caller in silence for close to three seconds, read as a dropped call, so the agent plays a filler while the tool runs. The filler is playback, and playback is what you barge in through. So every tool the agent gains widens the window in which a caller can talk over it. The share of turns needing cover is 34% at steady state and 71% during a disruption, when the search is slow and every session is a rebooking.

One more detail on the control surface, since it produced our least reproducible bug. allowInterrupt exists in four places. Three are build-time: the prompt, the response specification, and a per-attempt map keyed by Initial and Retry1 through Retry5, so setting it once at prompt level still leaves a retry attempt carrying its own value. The fourth cost us a week. A runtime session attribute, x-amz-lex:allow-interrupt:<intent>:<slot>, defaults to true and is set per session at conversation start, so it varies call by call and does not appear in the bot definition at all. Two calls with different interruption behaviour export identically.

Six characters over a four kilohertz channel

The eval suite was 412 multi-turn scenarios across six intents and an everything-else bucket, three seeds each, 1,236 runs, passing 91.3% with a one-sided 95% Wilson lower bound of 88.7%. That bound is computed at n = 412, not n = 1,236, because three seeds of one scenario share the scenario; treating the runs as independent buys a point of confidence you did not measure. The suite was also entirely text, and the voice path does not fail the way text fails.

Telephony is narrowband, and I had the causality backwards for a while. Lex will take 16 kHz audio on RecognizeUtterance, and not on the streaming API this architecture is stuck with, where AudioInputEvent.contentType documents one accepted value and it is audio/lpcm; sample-rate=8000, 16-bit, mono, little-endian. The PSTN leg would have pinned us there anyway, G.711 A-law on the trunk with a usable passband of roughly 300 to 3,400 Hz. The two constraints agree. Everything above 3.4 kHz is gone before any model sees it, and that is where most of the energy separating one fricative from another lives.

The identifier at the centre of every rebooking conversation is a record locator read aloud one character at a time, which is exactly what the band removes. Errors concentrate in the E-set, the nine letters that rhyme with /iː/, where B against D against P against T turns on a burst the channel truncates. They concentrate again in the F-set, where S against F turns on frication that mostly sits above the cutoff. M against N is a separate confusion and not a bandwidth one, since the nasal murmur and the formant transitions that carry it are well below 3.4 kHz. Those errors we do see, and they track the noise condition instead of the codec.

We rebuilt the audio arm with 118 scenarios mixed against four backgrounds from ETSI ES 202 396-1, which publishes about thirty-five binaural recordings across fourteen categories along with the recording and calibration method behind them. We took Inside Car, Outside Traffic, Public Places and the single voice distractor. The database is 48 kHz, so the 8 kHz rendering is our downsampling and not the standard’s. That is worth saying out loud, because the filter you pick on the way down is a parameter of the experiment and it never made it into the eval config.

That arm passed 78.6%, lower bound 71.8% at n = 118 scenarios, and almost the entire gap was locators and surname spellings. Per-character error rises from 0.4% on the wideband set to 4.1% on a speakerphone in a departure hall, and six characters compound, so 97.6% first pass becomes 77.8%. Only 4.6 of those 19.8 points are the narrowband channel, however, and the other 15.2 are the room. Worth knowing before someone proposes a wideband codec as the fix that StartConversation would refuse anyway. Character errors are not independent either, so the tails are worse than the arithmetic implies. A phonetic alphabet recovers most of it, at the cost of a longer prompt the endpointer charges for.

The obvious fallback is the keypad, and it does not work: a record locator is alphanumeric and DTMF letter entry is ambiguous. The DTMF-safe identifier is the thirteen-digit ticket number, on a receipt nobody has open at a gate. The identifier the customer holds is the one the voice channel handles worst, and the channel that handles it best is chat, which is where the caller is least likely to be during the disruption that made them call.

Then there was the part that was not an ASR problem at all. Our slot validator was ^[A-Z0-9]{6}$, because every locator anyone had ever shown us was six characters. IATA’s PADIS PNRGOV implementation guide defines element 9956, the Reservation Control Number, as an..20, in both the Field Type column and the Common Usage column, so there is no narrower-usage escape from it. The guide’s own worked reservation-locator examples carry five, six and nine characters. Six is a convention most systems happen to follow.

I lost an evening to where the six came from and got nowhere. Everyone I asked at the carrier said it had always been six, nobody had a source, and the two explanations I was offered contradicted each other. It has no bearing on anything in this post and I am putting it here because it was the most enjoyable half-hour of that week.

import re
SIX = re.compile(r"^[A-Z0-9]{6}$")          # what we shipped
PADIS = re.compile(r"^[A-Z0-9]{1,20}$")     # what element 9956 permits

# locators taken from the implementation guide's own worked RCI examples
for s in ["12DEF", "123EF", "345ABC", "ABC456789"]:
    print(f"{s:12s} six-char {str(bool(SIX.match(s))):5s} padis {bool(PADIS.match(s))}")

# 12DEF        six-char False padis True
# 123EF        six-char False padis True
# 345ABC       six-char True  padis True
# ABC456789    six-char False padis True

Interline and codeshare itineraries carry the partner’s locator, and 12.7% of our rebooking traffic is interline. The validator rejected roughly a fifth of those, about 188 sessions a day. What made it expensive is how it presented. The caller read the locator correctly, the transcript was correct, the validator refused it, and the agent said it hadn’t caught that and asked again, three times, then transferred. Every log line said retry. Nothing had been misheard. It landed in the bucket we had labelled ASR failures and was 31% of it, and we spent six weeks on noise robustness before reading the transcripts of the failing sessions.

Capture now uses Lex’s AMAZON.AlphaNumeric slot with a PADIS-shaped regex. That is a SlotValueRegexFilter and it is validation only, though. It decides what the slot will accept once recognition has finished and changes nothing about what the recogniser hears. Biasing is a separate control, and the runtime hints are separate again.

Surging a system you are not allowed to surge

Every useful action in this system goes through a global distribution system, and load-testing a third party’s production booking platform is a contract violation before it is an engineering problem. So the capacity question for holiday peak gets answered against a model of somebody else’s service, built from the latency distribution and the error responses you can observe from outside it.

Two limits sit on that path and engineers usually know only one. The commercial limit is a look-to-book ratio written into the distribution agreement with an overage price attached, which makes shopping volume a monthly budget you burn. The technical limit is a per-second cap on the shopping endpoint, and that is the one that returns an error to a caller who is on the line. Burn the look-to-book and you get an invoice. Exceed the cap and a rebooking dies mid-sentence. Our surge harness models the second, and what it is really testing is our own queue and retry policy: recorded responses replayed at the observed latency distribution, behind a bucket admitting 12 transactions per second.

We surged to 4.1 times peak-hour session volume and passed at 11.17 search calls per second against the 12 per second cap. In December an irregular-operations event ran 2.6 times peak volume and broke the system. The reason is in the mixture and not the multiple, because a peak multiple is a scalar applied to a mix of intents that does not hold still.

Shares from 61,400 hand-labelled sessions over four weeks of steady traffic and one 14-hour disruption window; call counts are medians from the tool-invocation log over the same period.

Sessions at 2.6 times and calls per session at 1.96 times compound, and the December event drove 13.85 search calls per second, 24% more than the surge test that passed, on 37% fewer sessions. At that rate against a cap of 12, 13.4% of search calls are refused. A rebooking session makes 6.2 of them, so under independence 59% of rebooking sessions hit at least one throttle. Throttles cluster inside a session, though, so the real distribution is more concentrated. Either way it lands on the one intent where the caller has nowhere else to go.

More capacity was not the fix. What worked was making the peak multiple a property of the mix instead of the session count. The surge harness now replays a recorded disruption day’s intent sequence, which at the same 4.1 multiple asks for 21.8 search calls a second against a cap of 12.

Proactive outbound rebooking hit a cap of its own, and the way we hit it is the embarrassing part. StartOutboundVoiceContact does not appear in the exceptions table on Amazon Connect’s service quotas page, so it takes the blanket default of two requests per second with a burst of five, enforced per account per Region and shared with everything else in the account. The 31,600 disrupted passengers of that December event, 23% of them with no usable digital channel, are 7,268 calls. At two per second that is 61 minutes to dial each of them once, before anyone answers. A second attempt on the 59% who don’t pick up is another 36. The window we had been given was ninety minutes, so that arithmetic is what went into the review.

It was also the wrong arithmetic, and wrong in the flattering direction. What limits a dialling campaign is not how fast you can call the API but how many calls can be up at once. The same page sets concurrent active calls at 10 per instance, and an outbound attempt holds a call for something like 30 seconds whether or not anyone answers. So one instance sustains a third of a dial per second, and the first pass alone runs past six hours. Two dials a second needs about 60 calls in flight. The quota that decided the outcome was one we had already read, filed under inbound capacity, where the same 10 sits against a holiday peak that needs 510.

Both mistakes have the same shape. Every quota that hurt us was the default on a general-purpose call we reached for because it was the one in the tutorial. We were calling a per-contact API in a loop for a job with a purpose-built feature. Outbound campaigns take batches through PutOutboundRequestBatch at ten requests per second under their own concurrency quota, with the answering-machine detection and contact-list handling you would otherwise write badly. Raising a concurrency limit is a support form. Picking the right API is a rewrite, and we did both in the same week under an incident.

The write that might have landed

Everything above is about reading. The moment the agent writes, the failure modes change kind, because an itinerary change is not idempotent and the compensating action is not free. A ticketing call that times out leaves you in the one state that matters: you don’t know whether it landed. Retry and you get a passenger holding two reservations on the same flight and a charge for both. Don’t, and you get a passenger who thinks they are rebooked. The usual answer is an idempotency key. That works only if the key is derived from the intent rather than the attempt, but it also works only if the far side honours it, and for a GDS entitlement that belongs in the commercial negotiation, not the code review.

import hashlib, json
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from zoneinfo import ZoneInfo

@dataclass(frozen=True)
class TicketingIntent:
    contact_id: str
    record_locator: str
    passenger_key: str      # surname and date of birth hashed, never the raw values
    segments: tuple         # ((carrier, number, date, origin, destination), ...)
    fare_amount_minor: int
    pos_timezone: str       # zone name of the point of sale, never a fixed offset

    def idempotency_key(self) -> str:
        """Stable across attempts at the same booking at the same price. A
        re-quote after the fare moves is a different booking and gets a new key,
        which is correct and is also why a re-quote can never dedupe against the
        attempt it replaces."""
        body = json.dumps({"pnr": self.record_locator, "pax": self.passenger_key,
                           "seg": sorted(self.segments),   # order must not change the key
                           "amt": self.fare_amount_minor}, sort_keys=True)
        return hashlib.sha256(body.encode()).hexdigest()[:32]

    def void_deadline(self, issued_at: datetime) -> datetime:
        """End of the issuing day in the point-of-sale market. Two clocks get
        confused here. An integer UTC offset cannot express +05:30, and any
        fixed offset moves the deadline an hour on a DST transition, in the
        direction that loses the free void; and `issued_at` has to be when the
        write was sent, not when it failed, or a ticket issued at 23:59:55 that
        times out eight seconds later gets a deadline a full day late."""
        local = issued_at.astimezone(ZoneInfo(self.pos_timezone))
        return (local.replace(hour=0, minute=0, second=0, microsecond=0)
                + timedelta(days=1) - timedelta(seconds=1))

class AmbiguousOutcome(Exception):
    """The write may or may not have landed. Never retried, only reconciled."""

def issue_ticket(gds, events, intent: TicketingIntent, issued_at: datetime,
                 *, timeout_s: float = 8.0) -> dict:
    try:
        return gds.issue(idempotency_key=intent.idempotency_key(),
                         locator=intent.record_locator,
                         amount_minor=intent.fare_amount_minor,
                         timeout=timeout_s)
    except (gds.Timeout, gds.ConnectionReset) as exc:
        enqueue_reconciliation(events, intent, issued_at,
                               now=datetime.now(timezone.utc))
        raise AmbiguousOutcome(intent.idempotency_key()) from exc
    except gds.Rejected:
        return {"status": "rejected"}    # a clean no: safe to retry, safe to report

def enqueue_reconciliation(events, intent: TicketingIntent,
                           issued_at: datetime, now: datetime) -> None:
    """EventBridge is the transport, not the queue. It has no ordering and no
    delayed delivery, so the consumer writes to a table keyed on the deadline and
    the reconciler sweeps that index in cutoff order. The deadline comes off the
    issue time and the time remaining comes off the wall clock, which is at
    least one timeout later."""
    deadline = intent.void_deadline(issued_at)
    events.put_events(Entries=[{
        "Source": "booking.agent",
        "DetailType": "TicketingOutcomeAmbiguous",
        "EventBusName": "agent-bus",
        "Detail": json.dumps({
            "idempotency_key": intent.idempotency_key(),
            "contact_id": intent.contact_id,
            "record_locator": intent.record_locator,
            "passenger_key": intent.passenger_key,
            "segments": intent.segments,          # the reconciler searches on these
            "void_deadline": deadline.isoformat(),
            "seconds_to_cutoff": int((deadline - now).total_seconds()),
        }),
    }])

The shape that matters is the exception hierarchy. A rejection is a clean answer and can be retried or surfaced. A timeout or a reset is an absence of information, and the only correct response is to stop and hand the key to something that can go and look. The event carries the passenger key, the segments and the seconds to cutoff, and the reconciler searches on those inside the window.

That cutoff is the whole design. A ticket issued in error can be voided at no cost until the end of the issuing day in the point-of-sale market, under that market’s BSP local procedures, yet once that day closes the only way to undo it is a refund. The retry policy’s real parameter is therefore how many hours are left before a mistake stops being free: a wall clock, not an attempt count. A reconciler whose median is four minutes and whose tail runs to an hour is harmless mid-morning and a finance problem in the last hour before cutoff. So it works a deadline queue ordered by time-to-cutoff instead of arrival.

Before the reconciler existed, a plain three-attempt exponential retry produced about ten duplicate bookings a day out of roughly 9,400 writes, and the overnight batch caught 71% of them after the cutoff, so seven a day were refunded rather than voided.

The direct cost is smaller than it sounds and the size is the point. A refund returns the fare, so what the carrier eats is the merchant discount fee it doesn’t get back, about eight euros on a 340-euro ticket, plus fifteen minutes of desk time and the float: roughly twenty euros a case, 4,400 euros a month. That is too small to reach a roadmap, which is why a retry policy any reviewer would have approved ran for a quarter. What made it worth fixing was the tail, where a duplicate reaches the card statement before the batch does and becomes a chargeback. Reconciling left about four a week, voided the same day.

Three clocks govern this and they are routinely conflated. The void window is the carrier’s own, measured in hours, covering every ticket. On US-touching traffic, 14 CFR 259.5(b)(4) requires a reservation to be held at the quoted fare without payment, or cancelled without penalty, for at least twenty-four hours, and only when it was booked a week or more before departure, which covers 58% of ours. And 14 CFR Part 260 defines a prompt refund as within seven business days for a credit card purchase and twenty calendar days otherwise. An engineer who has internalised only the twenty-four hour rule will build a reconciler that misses the one deadline that was free.

The ranker owns a refund liability

The coupling that surprised me most runs from a search ranking function straight into the refund line, and it carries a scope condition that took a lawyer to get right. On the carrier’s US-touching traffic, 14 CFR 260.2 defines a significantly delayed or changed flight, and among the triggers, next to the familiar three-hour domestic and six-hour international thresholds, is an itinerary with more connection points than the original. Section 260.6(a)(2)(ii) is where a passenger offered alternative transportation gets the right to decline it and take the refund instead, and 260.10 is what puts that refund back in the original form of payment. So when the agent moves someone from a cancelled non-stop onto a one-stop that arrives sooner, it has created a refund entitlement, through a sort order chosen by whoever wrote the search tool on the reasonable grounds that people want to get there soonest.

We measured the take-rate. When the offered itinerary preserves the connection count, 4.1% of passengers take the refund instead. When it adds a connection, 18.4% do. Ranking by earliest arrival put an added connection in front of 31% of passengers and produced a refund rate of 8.5%. Ranking by connection parity first and arrival second dropped that to 9% and 5.4%. A 3.15-point difference with a 1.13-point confidence interval at 3,900 involuntary changes per arm. At 650 involuntary changes a day that is about twenty fewer refunded tickets daily, close to seven thousand euros of fares that stayed on the aircraft.

Intra-European traffic answers to a different instrument and the lever moves. EU Regulation 261/2004 gives reimbursement within seven days or re-routing under comparable transport conditions, and connection count is not a trigger there at all. The same ranker therefore has a different cost on the two halves of the network, and the parity rule is over-conservative on one of them. We shipped it network-wide anyway, since the expensive error is the other one. Revenue owns the exception list.

That decision is the one in this post I am least sure of. Preferring parity over arrival time means some intra-European passengers were offered a later flight than the ranker could have found them, for a refund entitlement that does not exist on their itinerary, and we never measured what that cost in arrival delay or in complaint volume. The case for shipping network-wide was that one exception list is cheaper to govern than two rankers. That is an operations argument wearing a customer-experience decision, and I would want the number before making it again.

The tool description said “alternate itineraries, cheapest first”, and that sentence was the entire specification of a seven-figure annual exposure. It passed review on both sides of the wall, because the ML team does not read the refund regulation and revenue management does not read tool descriptions. Wherever an agent picks between options for a customer, the choice function carries a liability, and the person who can price it does not know it exists.

Sizing the approval desk

The agent does not issue tickets above a fare threshold. It stages an action and a person approves it, which is what made the system approvable at all, and which puts a queue of human reviewers between the caller and the money. Purchases below the threshold complete on their own, which the register carries as an accepted risk and the reconciler is sized for.

We stage about 8,960 money actions a day, 18.7% of sessions: new tickets at a 34-second median review, voluntary changes at 71, involuntary changes carrying a waiver code at 132, refunds at 96. That is 164 review-hours a day, and with 9.4% of volume in the peak hour, 15.4 review-hours have to happen inside one hour. The first staffing model I handed over said sixteen people, which is what you get by rounding up. At the 85% occupancy the workforce team plans to and 30% shrinkage for breaks, training and absence, 15.4 hours of work needs 26 rostered.

While the approval was inside the call, its wait had a p50 of 9 seconds and a p95 of 41. Both are fiction, because the distribution is bimodal. About 8 seconds when a reviewer is free, around 200 when the queue is three deep, very little in between. The wait is either fine or it is over three minutes, and 41 is the gap between the two modes. Composing a system SLO from a p95 speech-to-speech budget and a p95 approval wait is the same error as adding component percentiles, arriving from a different direction.

So the approval came out of the call. The agent stages the action, tells the caller it will be confirmed shortly, and the desk works a queue. What that trades against is the fare, though. A held fare carries a ticketing time limit, so moving the approval out moves the risk onto that limit, and sometimes the price has gone. The compensating action for a lapsed hold is a re-quote and a disappointed customer. For a bad ticket it is a refund.

The surge arithmetic on the desk is the part that did not survive contact with December. We sized it by scaling volume, and the surge test at 4.1 times peak implied 63 review-hours in the peak hour, 107 rostered, which was already an uncomfortable conversation. The December event ran at 2.6 times and needed 105.6 review-hours, 178 rostered, because the stage rate went from 18.7% to 30.3% and the mix shifted far enough toward involuntary changes to pull the mean review from 66 seconds to 107. Volume rose 2.6 times and the requirement almost seven, because the surge test scaled arrivals and held service time fixed.

What gets signed

At the go-live steering committee a small number of risks get accepted in writing, by people whose names are on the acceptance. The engineering job by then is narrow. Make sure the list is complete and the numbers on it are the honest ones.

The first thing on the first slide is a metric definition rather than a metric. Containment is 74.1% and human contacts avoided are 44% of the counterfactual, and the second is the one tied to headcount. Presenting the first alone funds a programme on an outcome it will not deliver, but presenting it with the holdout attached keeps your right to report a drop later without that reading as failure.

Assisted conversion made the same point from the other side. Over the pilot, purchases within 24 hours of a shopping session ran 31.4% against 27.9% for the control, a 3.5-point lift with a 4.2-point interval, which is to say we couldn’t call it. Eleven weeks and 34,700 sessions per arm later it was 2.6 points, plus or minus 0.7.

The accepted risks are where the real conversation happens. Purchases below a fare threshold complete without desk approval, at a residual duplicate rate of about four a week, with a named owner for the reconciler and its deadline queue. The agent may quote a fare that lapses before the desk approves it. The 8 kHz recognition floor on locators is 77.8% first pass in the worst channel condition. The rebooking ranker prefers connection parity over earliest arrival, and revenue owns that ordering now. And the one that took longest, because procurement had to be pulled in to sign it: Bedrock cross-Region inference absorbs unplanned bursts, but it can route to Regions nobody enabled in the account, since manual enablement is not required for it to work. A geographic profile does not put enablement back. What it constrains is the geography, which is a data-residency control and the thing the lawyer was being asked about, so what the committee signed was the geography, the profile authorised inside it, and who approves a change to either.

Smaller items belong in the register because they will otherwise surface during an incident. Session state crosses from the flow into the agent through contact attributes, which Connect caps at 32,768 UTF-8 bytes in total and sends down the flow’s error branch when the set contact attributes block goes over. So an agent that accumulates state across a long rebooking hangs up on the caller while its own model trace shows a completed turn.

The orchestrator, a Strands agent on Bedrock AgentCore Runtime whose idle session timeout is 15 minutes, runs against a Lex idle TTL you set separately and a caller who can hold longer than either: three clocks, shortest wins, silently.

And CloudWatch keeps sub-minute data points for three hours before rolling them up to one-minute resolution for fifteen days. Three hours is shorter than the gap between an incident and its review, so the one-second barge-in rate we added to debug interruption behaviour was still there when we went looking and no longer carried the resolution the question needed. It is mirrored into logs at full resolution now.

None of what I would alarm on is model quality. The share of GDS calls refused by the cap, which predicts the rebooking experience two intents before latency notices it. Ambiguous write outcomes still unreconciled with under two hours to the void cutoff, the only alarm here denominated in money. The gap between the two containment numbers, because when it moves a definition has drifted. Second contacts within 72 hours on a locator the agent already touched. And the intent composition against the sizing assumption, watched as a distribution rather than a mean.

The agent itself was good enough eighteen months ago and took very little of this project. The time went into a booking flow with a general ledger attached: a system where undo has a price and a deadline, where the capacity you sized for is a shape as much as a volume, and where the metric everybody agreed on before you arrived is computed over a set of sessions that excludes most of the ones you were hired to fix.