Why your AI assistant is giving you an incomplete list of SOC 2 exceptions

If you work in third-party risk management at a bank or credit union, you’ve probably tried it: drop a vendor’s SOC 2 Type II report into ChatGPT, Copilot, or whatever AI tool your institution has approved, and ask it to summarize the exceptions.

It’s fast. It sounds authoritative. And if you’re relying on it as your primary review step, it’s probably missing things.

I’m not writing this to knock AI. I run engineering for a company built on the idea that AI can meaningfully improve third-party risk work. But “an LLM can help with this” and “a general-purpose chat client is safe to use unsupervised on a compliance document” are two very different claims, and the gap between them matters a lot more in our industry than in most.

Here is what’s actually going on when a general-purpose AI model processes a 100+ page SOC 2 report.

The document gets mangled before the model ever sees it

Most AI chat tools don’t “read” a PDF the way you do. They convert it to plain text (or run OCR if it’s a scanned image) before any analysis happens. SOC 2 reports are exactly the kind of document that conversion handles badly: multi-column layouts, control-and-exception tables that span pages, footnoted CUEC (complementary user entity control) sections. Any of these can get scrambled, merged, or silently dropped in that first conversion step, and the model never knows anything was lost.

Long documents get uneven attention

SOC 2 Type II reports routinely run 80 to 150+ pages. Independent of any technical failure, general-purpose LLMs have a well-documented weakness sometimes called “lost in the middle”: they’re demonstrably better at retrieving and reasoning about information near the start or end of a long input than information buried in the middle. If a testing exception shows up on page 64 of 110, it’s statistically more likely to get dropped from the summary than one on page 4.

“Exception” isn’t a keyword. It’s a judgment call.

Auditors rarely write a document that helpfully flags every deviation with the word “exception.” Instead you get “management noted,” qualifying language around a control’s operating effectiveness, remediation narratives, or a testing result phrased in a way that only an experienced reviewer would recognize as a deviation. A model doing a fast, keyword-influenced pass will reliably catch “the following exception was noted” and just as reliably miss the deviation described in a paragraph that never uses that phrase.

Findings depend on cross-references the model has to hold in its head at once

This is the one I’d flag hardest for TPRM specifically. A “no exceptions noted” statement for a control can be qualified elsewhere in the report, by a complementary user entity control the vendor is assuming you have in place, or by a subservice organization carve-out that shifts responsibility for part of the control environment. To correctly interpret that control, the model has to connect three things that might be 40 pages apart. That’s exactly the kind of multi-hop reasoning where general-purpose models are least reliable, and it’s exactly the kind of thing that determines whether a finding actually represents risk to your institution.

There’s no built-in verification step

Ask a general AI chat tool for a list of exceptions and it will give you one, confidently, cleanly formatted, ready to paste into your vendor file. It won’t tell you what it skipped. Unless you explicitly force a verification pass (quote-and-cite every finding, go section by section, cross-check against defined terms), you have no way to distinguish a complete answer from a plausible-sounding one.

Why this matters more for banks and credit unions than most industries

TPRM isn’t just an internal control for financial institutions. It’s an examined one. FFIEC guidance expects you to be able to demonstrate a defensible, repeatable process for vendor due diligence, and “the chatbot said there were no material exceptions” is not going to hold up to an examiner asking how you know your review was complete.

What to ask for instead

If your team is using AI to accelerate SOC 2 review (and you should be; the manual alternative doesn’t scale), push for tooling that:

• Processes the report section by section rather than in one pass, so nothing gets attention-starved by document length
• Returns verbatim citations and page references for every finding, so a human can verify against the source rather than trusting a paraphrase
• Explicitly cross-references CUECs, subservice organization carve-outs, and control descriptions rather than treating each section in isolation
• Is built and tuned specifically for audit and attestation document structure, not a general-purpose assistant repurposed for compliance work
• Or, if you’d rather skip the evaluation process altogether, talk to us. This is one of the problems we built the platform to solve.

AI absolutely has a role in modernizing TPRM. But the tools built for general conversation weren’t designed for the specific failure modes of long, cross-referenced, precisely-worded assurance reports, and in a regulated environment, “mostly complete” isn’t a standard you want to explain to an examiner.