A confidence score is not a reason

When a mortgage application is denied, the notice has to say why, and Regulation B §1002.9(b)(2) forecloses the two answers an automated pipeline would naturally give: statements that the adverse action was based on the creditor’s internal standards or policies, or that the applicant failed to achieve a qualifying score, are insufficient. The Official Interpretations tighten it further, requiring the reasons disclosed to relate to and accurately describe the factors actually considered.

An extraction pipeline produces a number between 0 and 1 per field. Nothing maps that number onto a factor actually considered, because it is not a fact about the applicant; it is a fact about a scan.

I spent a year on an underwriting automation pipeline at a US mortgage originator: intelligent document processing over forty-one document types, rule-and-model cross-document validation of income, assets and collateral, an exception queue with humans in it, an immutable audit trail, and a straight-through rate somebody reported to a board every month. Almost every hard problem in that year turned out to be a version of the same gap. The score the system is confident about and the sentence the regulation demands are different kinds of object, and the machinery between them is where the engineering lives. This post is that machinery: what the confidence numbers describe, what a threshold on them really is, how one exception queue ends up carrying two different regulatory events, and what it costs to get the ordering right.

Two scores, and the gate reads the wrong one

Azure AI Document Intelligence returns a confidence per extracted field, and the concept documentation describes it in terms that sound like a calibrated probability: a confidence value of 0.95 indicates the prediction is likely correct 19 out of 20 times, and confidence can be used to decide whether to accept a prediction or flag it for human review. That reads like an instruction, and it is the one everybody follows. I ran a gate on it for two months before reading further down the page.

Further down is the sentence that undoes the reading. Field-level confidence reflects the model’s confidence on the position of the value extracted. Word confidence, a separate number living on pages[].words[], represents the confidence of the transcription. The docs then tell you to look at both and generate a comprehensive confidence for the field, but give no formula for doing it. Two things about that passage are worth flagging before leaning on it. It sits under a heading scoped to custom models, and the note at the top of the page says field-level confidence includes word confidence scores with the 2024-11-30 GA version for custom models. Everything below is therefore an extrapolation from custom-model guidance onto prebuilt extractors, which is what a lender actually runs. The extrapolation held up on our documents, but it is an extrapolation and the vendor never blessed it.

So the extractor answers two questions. Did I find the right box, and did I read what was in it. A gate on DocumentField.confidence alone asks only the first. The join is available because the field’s spans and the words’ spans index into the same result.content string:

from azure.ai.documentintelligence.models import AnalyzeResult, DocumentField


def contributing_confidences(result: AnalyzeResult, field: DocumentField):
    """Field confidence scores the position of the value. Word and selection-mark
    confidence score the transcription. A field's spans index into result.content
    and so does every word and every mark, so the overlap is the join."""
    if not field.spans:
        return []
    windows = [(s.offset, s.offset + s.length) for s in field.spans]

    def under_field(span):
        start, end = span.offset, span.offset + span.length
        return any(start < w_end and end > w_start for w_start, w_end in windows)

    out = []
    for page in result.pages or []:
        out += [w.confidence for w in page.words or [] if under_field(w.span)]
        out += [m.confidence for m in page.selection_marks or [] if under_field(m.span)]
    return out


def composite_confidence(result: AnalyzeResult, field: DocumentField):
    """The product is a choice, not a derivation. Every t-norm returns at most
    its smallest input, min included; the product is the one that also penalises
    two mediocre inputs rather than ignoring the better of them."""
    if field.confidence is None:
        return None
    transcription = contributing_confidences(result, field)
    if not transcription:
        return None          # no evidence under the field is weak, not strong
    return field.confidence * min(transcription)

On a pay stub where the year-to-date gross sits in exactly the right box and the digits are a photograph of a fax, prebuilt-payStub.us returns field confidence 0.971 and a single contributing word at 0.712. The composite is 0.691. The field gate passes it at any threshold anyone would set; the composite does not, and the composite is the one describing whether the number is right.

The join is less clean than it looks. DocumentWord.confidence is required while DocumentField.confidence is optional, so the composite has to treat an absent field score as weak instead of skipping it, and the same rule has to run in the other direction. A checkbox has spans and no words under them: its value is a DocumentSelectionMark on pages[].selection_marks[], a separate object with its own required confidence. The same page that asks for a comprehensive confidence says which score belongs in it, and says it in one clause that is easy to read past: evaluate the OCR result for text extraction or the selection mark depending on the field type. So the mark score is the transcription term for occupancy and for every other field the 1003 records as a checkbox. A field with neither a word nor a mark under its spans has no transcription evidence at all, and the honest composite for it is not the raw field score. DocumentLine carries no confidence at all, so line-level reasoning is unavailable even though the line is the unit a human would look at. Table, row and cell confidences arrived with the 2024-11-30 GA version for custom models only, and in azure-ai-documentintelligence 1.0.2 the typed DocumentTable has no confidence attribute at all; the value is reachable only through the model’s mapping interface, as table["confidence"]. None of that leaves a prebuilt extractor without per-row scores, which is what I assumed for longer than I should have. A bank statement running on prebuilt-bankStatement.us returns transactions as a valueArray of valueObject fields, and every transaction and every subfield inside it is a DocumentField carrying its own confidence. What is missing is a table-shaped score over the rows collectively; per-row gating works and “is this table internally coherent” doesn’t. The composite has to be rebuilt per document type against whatever that type’s model actually returns, which is the first place the per-type build estimate stops being a constant.

Measured over one quarter: of gated fields that cleared 0.85 on field confidence, 19.4% had at least one contributing word below 0.85. On pages that arrived scanned, faxed or photographed, that was 41.2%.

The score before the scores

A mortgage file arrives from the broker portal as a single PDF, between two and four hundred pages, scanned end to end at whatever the sending office had. Our packages averaged 268 pages and 21.6 documents. Before any field confidence exists, something has to decide where each document starts and stops and what kind of document it is, and both decisions come from a model with a third confidence score on it.

Custom classification handles this, and the response is page ranges rather than a single label:

from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.ai.documentintelligence.models import (
    AnalyzeResult, ClassifyDocumentRequest, SplitMode)


def classify_package(client: DocumentIntelligenceClient, classifier_id: str,
                     pdf: bytes) -> AnalyzeResult:
    """splitMode defaults to "none" on 2024-11-30 GA, so a package submitted
    without it comes back as one document. The SDK spells the parameter split."""
    poller = client.begin_classify_document(
        classifier_id,
        ClassifyDocumentRequest(bytes_source=pdf),
        split=SplitMode.AUTO,
    )
    return poller.result()


def segments(result: AnalyzeResult, tau_class: float):
    """One entry per identified document: page range, type, and whether the type
    is trusted enough to choose an extractor from it. The classifier assigns
    every page to one of its trained classes, so a low score is the only signal
    that the package held something it was never shown. tau_class carries no
    default because it is the same kind of object as the composite gate, and it
    is read from the same versioned config rather than written down here."""
    for doc in result.documents or []:
        pages = sorted({r.page_number for r in doc.bounding_regions or []})
        yield {
            "doc_type": doc.doc_type,
            "page_range": (pages[0], pages[-1]) if pages else None,
            "confidence": doc.confidence,
            "route": doc.doc_type == "other" or doc.confidence < tau_class,
        }

The default in that first function is the whole story of one bad week. On the v3.1 API the classifier split multi-document files as a matter of course. The 2024-11-30 GA version changed the default for splitMode to none, and the docs say in terms that you have to set it to auto explicitly to preserve the previous behaviour. A version bump that reads like housekeeping turns a 268-page package into one document with one docType and one confidence, and nothing downstream throws; the pipeline happily runs a pay stub extractor over three hundred pages and returns a handful of fields with unremarkable scores. The documentation calls it splitMode while the Python SDK and the REST query parameter both spell it split, so grepping the codebase for the name in the documentation finds nothing.

Two other properties of the classifier decide how much of this you can govern. Training needs at least two classes and five samples per class, and it caps at 100 samples per class against a ceiling of 1,000 classes, which pushes the taxonomy in a direction nobody would choose on document grounds. There are more than a hundred distinguishable homeowners declarations page layouts in a national intake mix and you may keep a hundred, so the way to hold more of them is to split one document type across several classes and map them back to one type downstream, and incremental training on 2024-11-30 lets you append those classes to an existing classifier by passing its id as baseClassifierId rather than rebuilding from a dataset you may no longer keep. So the cap does not stop you; what it costs is that the class list stops being a list of document types and becomes a list of layouts, and the mapping between the two is now something you maintain. The classifier also assigns every page to one of its trained classes, so a document type it has never seen comes back as the nearest neighbour with a confident-looking score attached. The documentation is direct about the fix, which is an explicit other class plus a threshold on the score, and both of those are things you choose rather than inherit.

The reason this belongs in an article about field confidence is that a classification error is invisible in every score downstream of it. When the docType is wrong, the extractor selected from it is answering a question about the wrong schema, and it answers confidently. On 62% of our misclassified documents the wrong extractor still returned a populated field set whose mean composite cleared the gate. Field confidence scores the position of a value against the schema the extractor was handed, and it has no term at all for the schema being wrong for the page. What Microsoft says about low scores cuts against the story I first told myself: high accuracy with low confidence indicates the analyzed document appears different from the training dataset, and a low document type confidence is indicative of template or structural variations. Both describe a misclassified page scoring low rather than high, and a bit over a third of ours did exactly that. But the other 62% came back looking ordinary, and that is the number that decides how much work the downstream scores can be asked to do.

At 98.9% docType accuracy per document and 21.6 documents per package, 21.3% of files carry at least one misclassified document. Page boundaries are harder than labels, because the evidence for where a bank statement ends is usually printed on the page after it: at 97.6% per document, 40.8% of files carry at least one boundary in the wrong place. Those two numbers do most of the work in the largest bucket of the funnel. Of the 278 files per thousand that fail document-set completeness before anything is scored, 53 fail because a document was present and the splitter put it somewhere else.

The score is calibrated on someone else’s documents

The 19-out-of-20 sentence is a claim about frequency, so it is testable. We pulled 3,200 field instances over a quarter, stratified by confidence bin and by how the page was captured, and had two reviewers adjudicate each one against the source page with a third breaking ties. Aggregated, the scores looked defensible; split by capture class, they were two different instruments.

Read the right-hand end of that chart, not the left. At 0.85 to 0.90 the two populations sit 4.1x apart, and in the top bin, where nothing is ever reviewed, they sit 5.2x apart. The ratio widens as confidence rises. The score is least informative exactly where it is trusted most, and a single threshold across both populations sets two different error rates while looking like one policy.

None of this is a defect in the extractor. A confidence is calibrated, if it is calibrated at all, over the population it was fitted on, and a national vendor’s population is not a regional originator’s intake mix. Microsoft never claims the scores are calibrated; the 19-out-of-20 line implies it, but no page on Learn validates it or offers a numeric threshold, which is a reasonable place for the vendor to stop and a bad place for a lender to stop.

The obvious repair failed, and the way it failed was the useful part. I fitted isotonic regression per field type, mapping raw confidence to observed correctness on the adjudicated set, which is the textbook move. Brier score went from 0.061 to 0.048, which reads like a win until you ask what set it was computed on. The adjudicated sample is stratified by bin, so it deliberately oversamples low-confidence fields and carries a 16.3% error rate, where the population field error rate is 1.8%. A Brier score computed on the population would be a much smaller number saying much less; the two are not interchangeable. Inside the scanned segment, expected calibration error went from 0.152 to 0.141, which I couldn’t distinguish from resampling noise on that stratum. The recalibration had learned the aggregate; the aggregate is dominated by clean pages that were already fine.

Fitting it per capture class was the obvious next move, but the obstacle was not the one I expected. Capture class is not a field the API returns, but it is cheap to derive, and we already derive it: the presence of a text layer on the source PDF separates native from scanned almost perfectly, and among scanned pages the effective DPI and the JPEG quantization tables separate a flatbed scan from a fax from a phone photograph well enough to bucket on. What stopped the per-class fit was sample size. Isotonic regression is fitted per field type, and splitting each field type again by capture class turns 3,200 adjudicated fields into cells holding tens of observations. The fit that would work needs an adjudication set an order of magnitude larger, which is a budget question and not a modelling one.

What shipped instead is a reporting rule. Every accuracy figure the program publishes is broken out by capture class, and the pooled figure is not published at all, because intake mix moves with the channel split from month to month and a pooled number moves with it.

The unit of truth is a set of documents

Gate every field on the composite and the pipeline stops. At a 0.70 composite threshold, 8.7% of gated fields fall below the line, which is 4.7% on native PDF and 27.0% on scanned, faxed or photographed pages. A file carries about 78 gated fields, so the chance that a file has no field below the line is about eight in ten thousand. Per-field gating on the composite routes essentially every file that reaches extraction, roughly 2,740 a month.

It is worth being precise about which number makes per-field gating look merely bad instead of impossible. Gate on field confidence at 0.85 and only 3.34% of fields fall below, 7% of files come through clean, and 93% route. That is the arithmetic I did first, and it is the arithmetic you get from reading the score the previous section argued against. The honest version is worse, and it is worse in a way that settles the design question instead of complicating it. Nothing survives a per-field gate. The other 27.8% of intake never reaches the gate at all, because the document set is incomplete before anyone scores anything.

The escape is that the pipeline does not need field values. It needs about eleven assertions per file: qualifying monthly income, verified liquid assets, reserves after closing, subject property value, occupancy, employment at close, and so on. Each draws on roughly seven fields spread across several documents, and mortgage files are redundant by regulation and habit. Income appears on the pay stub, the W-2, the verification of employment and the 1003. Measured across the quarter, 94.1% of below-threshold fields have a corroborating source that itself clears the gate, and a low-confidence field that two other documents corroborate is not a reason to stop.

That last clause is doing real work, so the code has to mean it. A corroborator has to be a different document and has to clear the threshold itself, or two unreadable scans of the same garbled figure would confirm each other:

from dataclasses import dataclass
from typing import Literal

Verdict = Literal["agrees", "disagrees", "cannot_tell"]


@dataclass(frozen=True)
class Contribution:
    document_id: str
    field_path: str
    value: str | None        # None when the field was not extracted at all
    composite: float | None  # None when there was no score to compute one from


def dispose(contributions, required, verdict: Verdict, tau: float = 0.70):
    """One assertion, several documents, one reconciliation verdict over them.
    The cause is the output that matters: it decides which notice is owed."""
    present = [c for c in contributions if c.value is not None]
    if not present:
        return "route", "unreadable"
    if not required.issubset({c.document_id for c in contributions}):
        return "route", "missing"
    if verdict == "disagrees":       # semantic disagreement, not string inequality
        return "route", "disagreement"
    if verdict == "cannot_tell":     # the reconciler could not read it either
        return "route", "unreadable"
    # an unscored field is weak, not strong: DocumentField.confidence is optional
    strong = [c for c in present if c.composite is not None and c.composite >= tau]
    weak = [c for c in present if c not in strong]
    # a corroborator has to be readable itself, and has to be another document
    corroborators = {c.document_id for c in strong} - {c.document_id for c in weak}
    if weak and not corroborators:
        return "route", "low_confidence"
    return "accept", "corroborated" if len(corroborators) > 1 else "sole_document"

The verdict argument is not decoration. Comparing values by string equality is exactly what the next few paragraphs exist to avoid, so the semantic comparison runs before disposition and its answer arrives here as an input. It has three branches because the verdict has three values, and the third one is the branch I had to be shown twice. A reconciler that cannot tell has reported something about the copy in front of it, not about the borrower, and filing that as a disagreement puts a fact about a scan into the row the notice layer reads as a question about the applicant. The two live a section apart in law.

Routing assertions instead of fields took that queue from about 2,740 files a month to about 1,370. Here is what those routes look like counted by assertion, over 11,431 files in one quarter, with an assertion counted only on files where it applies:

That comes to 10.83 assertions and 0.665 routes per file. Since routes are close to independent across assertions, 50.1% of files carry at least one, and a file that carries one carries 1.33 on average.

One row in that table is not a quarter-long rate and the totals under it inherit the problem. Large-deposit sourcing did not exist in weeks one to four and its rate fell to 4.6% after an intake change in week eleven, so the 11.2% printed here is the middle regime, weeks five to ten, and it is the regime the per-file figures are computed on. The last section is about that quarter. Every other table in this post carries the same caveat more quietly, which is the usual trouble with reporting a quarter that had two releases in it.

The single-source column took longest to see. Four assertions have exactly one document behind them: hazard insurance comes off the declarations page, title and vesting off the title commitment, and both the appraised value and the condition rating off the 1004. Corroboration is unavailable to them at any threshold, so every below-threshold contributor routes, and together those four produce 36% of all routes from a third of the assertions. They still are not at the top of the table, but only because appraisals and title commitments arrive as generated PDFs, where the composite falls below the line on 4.7% of fields instead of 27.0%. Hazard insurance sits highest of the four, and it sits highest because insurance agents fax.

I want to sit on that for a paragraph, because it is the detail I tell people about this project and it is not really about confidence scores at all. The worst pages in a mortgage file are not produced by anybody in the mortgage industry. They come from an insurance agency with a multifunction device bought in about 2009, scanning at whatever the last person left it on, into a fax stack that re-encodes once more on the way out. I could predict the declarations-page composite from the sending agency to a degree that embarrassed me. There was a serious suggestion at one point that we call the twenty worst offenders and offer to walk them through their scanner settings, which is not an ML intervention, is almost certainly the highest-return one available, but did not happen because nobody could work out whose budget it came from. I still think it would have worked.

The comparison step is where a language model earns its place, because the failure it catches is semantic. A W-2 reporting wages of $44,033.90 against a pay stub showing year-to-date gross of $48,213.90 is not a discrepancy if the difference is a pre-tax deferral, but it is a serious one if it is not. Azure OpenAI with structured outputs gets a schema that makes the model cite both sides:

from typing import Literal

from openai import AzureOpenAI
from pydantic import BaseModel, ConfigDict

client = AzureOpenAI(azure_endpoint=ENDPOINT, api_key=KEY,
                     api_version="2024-10-21")


class Evidence(BaseModel):
    model_config = ConfigDict(extra="forbid")
    document_id: str
    page: int
    field_path: str
    verbatim: str            # the characters on the page, never a paraphrase


class Reconciliation(BaseModel):
    model_config = ConfigDict(extra="forbid")
    assertion: str
    verdict: Literal["agrees", "disagrees", "cannot_tell"]
    evidence: list[Evidence]
    difference: str
    rationale: str


completion = client.chat.completions.parse(
    model=DEPLOYMENT,
    messages=[{"role": "system", "content": RECONCILE},
              {"role": "user", "content": pair_markdown}],
    response_format=Reconciliation,
)

One note on that block. client.beta.chat.completions.parse is what Microsoft’s own sample still shows, but in the current SDK beta.chat is a property returning the same resource, so the non-beta path is the one to write. The strict schema is not free either: every field has to be required, every object needs additionalProperties: false, and the keyword you reach for to constrain verbatim is unsupported, so the instruction that it be copied off the page lives in the prompt and has to be checked afterward by string containment against the extracted content. We check it. About 3% of returned quotes are not in the source, and those reconciliations are dropped and the assertion routed to a person.

I spent three weeks trying to get a confidence out of the model itself and it went nowhere. Chat completions expose logprobs and top_logprobs, so the log-probability of the verdict token looks free. Over 812 adjudicated pairs, on which the reconciler’s verdict was correct 720 times and wrong 92, that log-probability separated the two with an AUC of 0.54. Run through Hanley-McNeil at that class balance the standard error is 0.031, so the 95% interval runs from 0.48 to 0.60 and straddles chance in both directions. The mechanism is specific to this design: a strict schema pins the verdict to a three-value enum, so the distribution being measured is over three tokens the decoder was already constrained to, and there is very little left in it. That same constraint disarms the trap you would otherwise walk into: top_logprobs caps at 20 and anything outside it comes back as -9999.0 rather than a log-probability, which turns into a silent zero under naive exponentiation, and it never bit here because a three-value enum keeps the verdict token near the top of the list. It bites the moment you widen the schema. The reasoning-model families do not support logprobs, which pins the experiment to the GPT-4 line whether or not that is the line you wanted for the reconciliation itself.

The finding this section produces has nowhere obvious to live, which took me a quarter to notice. “The W-2 and the pay stub disagree by $4,180” is a property of a pair, and the audit trail I inherited was keyed per document: one row per analyze call, holding the model id, the timestamp and the result. No row holds the conclusion, so a notice saying income could not be verified had nothing behind it in the store except two rows that individually looked fine. The record has to be keyed by the assertion instead, carrying the ordered set of contributors as document id, page, field path, span offsets and bounding polygon, the extractor version per document, the classifier version and the page range it assigned, the validator version, the threshold in force, and the disposition with its cause. A reviewer at the time and an examiner three years later both need to be shown the pixels the number came from, and the examiner will be asking about a pipeline that no longer exists in that form. A polygon is eight floats per contributor.

A constant in a YAML file decides who gets a human

The accept threshold decides which applications a human looks at. A human looking at a file changes its outcome. So the threshold is a rule about who gets human consideration of their loan application, which is a credit policy, and it was set by an engineer in a configuration file.

It also isn’t a number you can carry across from one score to another, which is the mistake I made when the composite went in. A field confidence of 0.85 and a composite of 0.85 are not the same policy, because the composite is a product of two numbers that each live below 1. Hold the threshold at 0.85 and move the gate to the composite and 29.6% of all gated fields fall below it. That figure decomposes into three groups: 3.34 points are fields whose own confidence was already below 0.85, 18.74 points are fields that cleared on field confidence and carry a word that did not, and 7.51 points are fields where both inputs cleared 0.85 and their product did not. The last group is the one that surprises people; a field at 0.90 whose worst word is 0.92 has two scores a human would call fine and a composite of 0.828. So the threshold had to be re-fitted for the new score, and the sweep below is what set it at 0.70.

I expected model risk management to be the forcing function on the governance side, and the ground moved underneath that expectation in April 2026. The interagency guidance the industry had run on since 2011 was superseded by SR 26-2, issued jointly by the Federal Reserve, the OCC and the FDIC, and the new definition is narrower in a way that lands directly on this pipeline. A model is now a complex quantitative method applying statistical, economic or financial theories, and the guidance expressly excludes deterministic rule-based processes and software with no such theory underpinning them.

Read that against the architecture. The extraction and classification models are models. The comparison prompt is generative, and footnote 3 puts generative and agentic AI outside the scope of the guidance entirely, so it is governed by whatever the organization decides rather than by this letter. The number that decides who gets a human is a constant in a YAML file, compared with <, and it is not a model under that definition. Neither is the rules engine that turns assertions into dispositions. Five things decide what this pipeline does and two of them are models. The other three sit outside the framework by two different routes, one written into the definition and one written into a footnote, and the one that decides who gets a human is among them.

Whether any of it binds this lender is a separate question with a blunter answer. An SR letter is supervisory guidance from the Federal Reserve to the banking organizations it supervises, and this originator is a nonbank independent mortgage bank: no depository charter, no holding company, supervised by the CFPB and by every state it is licensed in. SR 26-2 does not reach it at any asset size; the thirty-billion threshold in the letter scopes organizations that were already inside it. Two carve-backs are worth reading anyway. Footnote 1 preserves supervisory action for violations of law and for unsafe or unsound practices stemming from insufficient management of model risk, whatever the definition says, and the letter states that the principles may be relevant below the threshold wherever model-risk exposure is significant. Neither creates an obligation for a nonbank, yet both are the language a regulator uses when it means the guidance to be read as a floor by people it has no authority over, and state examiners do read it that way.

The practical answer is to version the threshold as if it were a model anyway, in the same registry as the extractors, with the same change record and the same approval, and to make the credit policy committee sign the number. That is a governance choice with no citation behind it, which is worth saying out loud.

Then measure what moving it buys, because the intuition is wrong. We replayed the quarter’s stored confidences at five composite thresholds.

Replay of 11,431 files over one quarter against stored field and word confidences, recombined offline. Error here means a field accepted above the threshold, uncorroborated, and wrong by enough to move an assertion past tolerance. It is measured at field level and propagated to file level assuming independence, which overstates it, since real extraction errors cluster on one bad document. The two segments are documentation types, not applicant groups: standard means a W-2 wage earner with one employer and is 71% of the quarter, nonstandard means self-employment, multiple income sources or rental income and is the other 29%.

Moving from 0.70 to 0.75 costs 6.4 points of straight-through rate and removes 24% of the surviving error, a fifth of it in the standard segment and a quarter in the nonstandard one. That is a considerably better exchange rate than the field-confidence gate ever offered, because the composite is correlated with whether the value is right and field confidence mostly is not. The error mass now sits where a threshold can reach it: at 0.70 on the nonstandard segment, 51.6% of surviving error sits in the ten points just above the line and only 11.7% above 0.90. So the threshold is a real control on correctness, but it remains a much stronger control on cost, and no setting of it closes the gap between the two populations. Across the whole sweep throughput falls 79% while the segment ratio falls 39%. The gap lives in the documents, not in the score, and nothing done to a number that reads the documents will remove it.

That asymmetry is what makes the reporting conflict real. Straight-through rate is the number the program is funded against; the way to raise it is to lower the threshold. Every point of it lands unevenly across the documentation mix, because the mix is what drives the error, and documentation type is correlated with who the applicant is in ways nobody chose. The regulatory footing here shifted too: the CFPB published a final rule on 22 April 2026 that does not merely delete the effects-test language from Regulation B §1002.6(a) but replaces it with an affirmative statement that the Act does not provide for an effects test at all, effective 21 July. But it does not touch the Fair Housing Act, where HUD’s discriminatory effects standard still stands, and a residential mortgage lender answers to both. Still standing is not the same as settled. HUD proposed on 14 January 2026 to remove and reserve the entire subpart that carries §100.500, and to strike the sentence in §100.5(b) that says an illustration of discrimination can be established by discriminatory effect without discriminatory intent. Comments closed on 13 February. The subpart is still in the CFR as I write, so the standard governs today and is under active reconsideration at the same time. Both statutes moved this year, in the same direction and at different speeds, and a pipeline built for a mortgage originator has to be defensible under whichever one survives the next five years.

One number people expect at this point does not exist. There is no four-fifths rule in lending. The four-fifths convention lives in 29 CFR 1607.4(D), the Uniform Guidelines on Employee Selection Procedures, it is scoped to employment practices, and it is hedged in both directions even there. ECOA, Regulation B, Regulation C and the Fair Housing Act regulations contain no disparity ratio and no significance threshold. We report the segment ratio at the shipped threshold every month, and the chart carries no reference line because there is no published one to draw.

One queue, two regulatory events

Every exception lands in one queue with one SLA and one reviewer pool, and the entries are not one kind of thing. Over the quarter, per thousand queue entries:

Counted at first fire over 11,431 files; an assertion contributes at most one entry. Four of those five rows are the causes dispose returns. The disqualifying row is not: an assertion that disqualifies was accepted, and the policy engine raises it downstream on a value the gate passed. It shares the queue and it shares nothing else. The bottom row is where a cannot_tell verdict lands alongside the assertions with nothing extractable at all, which is why it is not a tenth the size of the disagreement row.

The last row and the disqualifying row are the same shape in the database and different events in law. A file where the income was read correctly and does not support the loan is an adverse action; that notice needs the specific principal reasons. A file where the income documents are unreadable is not an adverse action, and the harder question is whether it makes the application incomplete. §1002.9(c) gives a specific instrument for an incomplete application: a written notice specifying the information needed, designating a reasonable period for the applicant to provide it, and stating that failure to respond ends consideration. We read an illegible document as falling outside that instrument, on the ground that the applicant supplied what was asked for, and I want to be plain that this is a reading rather than a rule. §1002.2(f) defines a completed application as one where the creditor has received all the information it regularly obtains and considers, and a document nobody can read is a fair argument that it has not. The commentary to 9(c) does not mention legibility in either direction.

What settled it for us is that the 9(c) instrument and a request for a legible copy are the same letter with a different clock attached, so we send the request, start no clock the regulation does not give us, and treat only an unanswered request as incompleteness. Counsel signed that, and a different lender could land somewhere else on it.

Two things about that position are worth saying, because I have watched people adopt it from a slide without either of them. The first is that it is only conservative in one direction. Not invoking 9(c) means we do not start a clock that could end consideration of the application, which protects the applicant, but what it costs is that a file can sit in the request state for longer than the 9(c) period would have allowed, and nothing in the regulation caps that. We put an internal cap on it, chosen by operations, which is a policy the regulation does not require and does not forbid, and which nobody outside the company would ever see. The second is that the reading is load-bearing on a fact about the pipeline rather than about the law: it assumes we can tell an illegible document from an absent one. The dispose function above returns unreadable and missing as separate causes precisely because the letter differs, and the separation is only as good as the classifier and the completeness check that produce it. A document that arrived, was misfiled by the splitter and never reached the assertion looks exactly like a document that never arrived. That is the boundary case where a legal position rests on a page-range decision made by a model with a confidence score on it, which is the whole argument of this post arriving somewhere I did not want it.

The same fork reappears in HMDA. Action taken under §1003.4(a)(8) distinguishes denied from closed for incompleteness as separate codes, and §1003.4(a)(16) requires the principal reason or reasons the application was denied, which the commentary caps at four and which is reported as not applicable when the file was closed for incompleteness. So a routing decision made by a threshold comparison inside an orchestrator determines a field on the loan application register that a regulator reads at the portfolio level. Nobody drew that line on any architecture diagram I inherited.

Which is why the disposition function returns a cause and not a boolean, and why the cause is what the notice layer reads:

# cause -> (the Regulation B instrument, the HMDA action taken if the file ends here)
CONSEQUENCE = {
    # the document is in the file and cannot be read, or the reconciler could not
    # tell from what was there. We read that as a request for a legible copy
    # rather than an incomplete application, which is a position and not a rule:
    # 1002.2(f) can be read the other way. Only an unanswered request becomes
    # 1002.9(c) and code 5.
    "unreadable":     ("request a legible copy of a document already received", None),
    # nothing arrived, so the application is incomplete on its face
    "missing":        ("notice of incompleteness", "file closed for incompleteness"),
    "disagreement":   ("review, then whichever path the resolved value implies", None),
    "low_confidence": ("review, then whichever path the resolved value implies", None),
    # never returned by dispose(): the policy engine raises it on an accepted value
    "disqualifying":  ("adverse action, with the specific reasons", "denied"),
}


def consequence(disposition, cause, qualifies=None):
    if disposition == "accept":
        return CONSEQUENCE["disqualifying"] if qualifies is False else ("none", None)
    return CONSEQUENCE[cause]

There is no branch from a confidence value to an adverse action reason, and that absence is deliberate rather than unfinished. Low confidence can produce a request for a better copy of a document. It cannot produce a sentence about the applicant, because it is not about the applicant, and a reason code derived from it would fail the requirement that reasons accurately describe the factors actually considered.

The corroboration case is the one that makes this concrete. Run the resolver on an income assertion where the pay stub is unreadable and the W-2 and the verification of employment agree: it accepts. The file never enters the queue, the applicant is never asked for a better pay stub, and the income figure on the decision is supported by two documents that were read cleanly. The naive per-field gate sends that file to a person and asks the borrower for a document the file did not need. Corroboration keeps 94.1% of below-threshold fields out of the queue, and it is the single largest source of avoided volume in the design.

The orchestrator inherits a clock it did not start

Exception routing runs on Durable Functions, which is a good fit for the shape: fan out one activity per document, fan in, resolve, then fan out again over the assertions that need a person and wait an unbounded amount of time on all of them at once. The waiting is where the regulatory clock enters, and where the first version was wrong.

from datetime import timedelta

import azure.functions as func
import azure.durable_functions as df

app = df.DFApp(http_auth_level=func.AuthLevel.FUNCTION)


@app.orchestration_trigger(context_name="context")
def underwrite_file(context: df.DurableOrchestrationContext):
    file = context.get_input()

    analyses = yield context.task_all(
        [context.call_activity("analyze_document", d) for d in file["documents"]]
    )
    assertions = yield context.call_activity("resolve_assertions", analyses)
    routed = [a for a in assertions if a["disposition"] == "route"]
    if not routed:
        return {"outcome": "straight_through", "assertions": assertions}

    # the budget is what is left of the notice clock, computed by an activity from
    # the application's completion timestamp; current_utc_datetime is the replay
    # clock, so it returns the same instant on every replay of this step
    budget = yield context.call_activity("notice_budget_seconds", file["file_id"])
    deadline = context.create_timer(
        context.current_utc_datetime + timedelta(seconds=budget)
    )

    # every routed assertion is queued before the orchestrator waits on any of
    # them, so reviewers work them in parallel and the clock runs once for all
    yield context.task_all(
        [context.call_activity("enqueue_for_review", a) for a in routed]
    )
    waiting = {context.wait_for_external_event(f"review:{a['assertion_id']}"):
               a["assertion_id"] for a in routed}

    resolved, escalated = [], False
    while waiting:
        context.set_custom_status({"waiting_on": sorted(waiting.values())})
        winner = yield context.task_any(list(waiting) + [deadline])
        if winner is deadline:
            escalated = True
            break
        resolved.append(winner.result)
        del waiting[winner]

    if not deadline.is_completed:
        deadline.cancel()          # a pending timer keeps the instance alive

    if escalated:
        yield context.call_activity("escalate_before_clock", file["file_id"])
        return {"outcome": "escalated", "resolved": resolved}
    return {"outcome": "reviewed", "resolved": resolved}

Three details in that function are the API teaching by use. Orchestrators are generators and must not be coroutines, so async def and await are both wrong here, although every other Python function in the codebase uses them. context.current_utc_datetime is the deterministic clock and datetime.now() is not, which matters more than usual because the timestamp on a notice is a regulated artifact and a replay reading the wall clock produces a different one each time. And TimerTask.cancel() raises if the task has already completed, so the is_completed guard is required, not defensive; without it, a review that arrives in the same instant as the timeout throws inside the orchestrator. The loop shape is worth a fourth line. An earlier version enqueued one assertion, waited for it, and only then enqueued the next, so a file with four routed assertions occupied four reviewer sessions end to end against a clock that was running once for the whole file. Queueing all of them before waiting and draining whichever comes back first is the same code and a different SLA.

The first version computed the deadline from the moment the assertion was routed, with a two-day budget that felt generous. That is the wrong clock. Regulation B gives 30 days from receipt of a completed application, and a file has usually been in intake, credit pull and initial review for days before an assertion routes. The budget the orchestrator needs is the remainder of a clock that started somewhere else, which is why it comes from an activity that reads the application record instead of from a constant. Thirty-four files out of 11,431 in the quarter reached their notice date with an open exception, which is 0.30% and was 0.30% too many, since every one of them was a person waiting.

There is a second clock with a different shape. §1003.4(f) requires the loan application register to be recorded within 30 calendar days after the end of the calendar quarter in which final action is taken. That is a quarterly boundary, not a rolling window, so a file finalized on the last day of March and one finalized on the first day of April have very different amounts of slack but the same code path. The pipeline that populates the register has to know which quarter a decision belongs to, and a decision that gets revised has to know whether its quarter already closed.

Which is the case I got wrong and would design differently from the start. Retraining a custom extraction model and reprocessing older files sounds like backfill, but it is not backfill at all. We reprocessed 1,140 files after a model update; a decision-relevant assertion value changed on 71 of them, 6.2%; and twelve of those had already had a notice sent. Reprocessing an application after the decision is a new decision on an old application, made under a model that did not exist when the clock started. Overwriting the old value is the failure there, because the file then says something the letter that already went out does not, and no query can tell you which of the two the decision used. The record has to carry both, keyed to the extractor version that produced each, and the register write has to read the value that was in force when the quarter closed.

Ranking the document types by the exceptions they remove

Effort per document type is roughly five days of integration plus a bit over a quarter of a day per gated field, plus three days when no prebuilt model exists and the extraction has to be trained, plus six when the type is handwritten or the layout varies by issuer. That estimate held up well across ten builds. Azure carries prebuilt models for more of this domain than people expect, including the mortgage family covering the 1003 application, the 1004 appraisal, the 1005 verification of employment, the 1008 transmittal summary and the closing disclosure. A prebuilt type costs three days less to build and, more to the point, carries no labelled corpus to collect, no retraining cycle and no model to register, so the catalog decides more of the year-two run cost than the day estimate shows.

Build effort estimated per type before the work began; keying time from a stopwatch study over 40 files; 45,600 files a year at a fully loaded $38 an hour. Multiply the middle three columns and these ten types account for 11.3 of the 21.6 documents in an average package and 68 of its 78 gated fields, which is the whole reason they were the ten. The other thirty-one types are just under half the document count and an eighth of the fields a validation rule reads.

Those ten standalone cases sum to $1.33M a year. The ten of them together delivered $0.69M, so the business case overstated the outcome by 1.93x, and the overstatement is double-counting. A file that still routes for another reason gets opened by a person anyway, and the fixed cost of opening, orienting and logging is paid once no matter how many of its documents were automated. Document types overlap on the fields they source, so the second document to carry an income figure saves the comparison and not the reading. And each new type adds gated fields, which adds routes. The standalone case for the tenth document type is the same arithmetic as for the third but is worth much less, because the third already took the file most of the way there. Summed across all forty-one types the standalone cases came to $1.88M, and comparing that against a delivered figure covering ten of them is how the gap first looked like 2.7x.

The correction is to rank by exceptions removed instead of by accuracy or by volume, and to know what an exception costs. Ours is not the 7.8 minutes a reviewer spends on it. Thirty-eight percent of routed assertions end in a call to the borrower, which means a loan officer, a wait and a re-verification, and that path is about $61 against $4.94 for one resolved from documents already in hand. Blended, an exception costs $26.24, and the 1,825 routed assertions a month are a $575,000 line a year.

Priced that way the ranking inverts in places. Taking a third off the asset-side route rates, which is prompt work and tolerance tuning on assertions that already exist, is worth about $47,600 a year. Automating the gift letter, a document type with a perfectly good business case on its own terms, is worth $12,130 at best but needs $26,712 to build. Accuracy on a document type nobody was going to review anyway buys nothing. We reordered the year-two backlog on that basis and dropped nine types out of it entirely.

What shipping a working check cost

Agency guidelines require a large deposit into an asset account to be sourced. Before bank statement extraction shipped, nobody checked that at scale, because checking it means reading three months of transactions on every file. In week five of the quarter the check went live, and the number the program is measured on went down.

Straight-through rate went from a 31.9% mean over the first four weeks to 28.5% over the next six. That is 3.4 points of straight-through rate given up by shipping a control that works. Every one of those exceptions was a real finding that had been passing silently, and the honest description of the before-state is that the pipeline was not performing the check.

The precision here deserves care, because a weekly series invites more of it than it can support. At about 880 files a week a single point carries a 95% interval of roughly ±3.0 points, which is wider than the whole effect, so no individual week means anything on its own and the bands on that chart are there to say so. The multi-week means, however, are a different matter. The four-week mean against the six-week mean is a 3.4-point drop at about 3.4 standard errors, which is a real move. The recovery is not: 1.8 points at 1.7 standard errors sits inside the noise, and I report it as something we watched rather than something we established.

The fix was not in the pipeline. Intake started asking for deposit sourcing at document collection, on the files where the deposit pattern was predictable from the loan characteristics, and the assertion’s route rate fell from 11.2% to 4.6%. Over weeks eleven to thirteen the rate averaged 30.4%, still below where it started, with a control in place that was not there before.

I don’t have a clean way to have avoided the dip. What I would do differently is put the check behind a shadow flag for a month first, so the exception volume and the finding rate are known before the number moves, and so the conversation about the drop happens with the count of real findings already on the table. The number drops either way. We do that now for anything that adds an assertion.

None of the machinery here makes an extraction confidence into a reason. It makes the system produce, for every file, an assertion with the documents that support it, the pages those documents were split from, the version of every model that read them, the threshold that was in force, and the cause of every exception. From that a notice can be written that says something true about the applicant, and a reviewer can be shown the page. The score stays what it always was, a number about a scan, and the design work is keeping it from being asked to be anything else.