Building a Meeting Summarization Pipeline: Transcription, and Extracting Signal from Noise

I implemented Zoom meeting summarization and video-to-text transcription for a meeting-intelligence feature at Revfx. The transcription part is close to commodity now — solid off-the-shelf speech-to-text gets you most of the way. The part that actually determines whether anyone uses the feature is everything downstream of the raw transcript.

Raw transcripts are close to useless on their own

A verbatim transcript of a 45-minute sales call is technically “information,” and almost nobody will read it. Filler words, cross-talk, a rep re-explaining the same point three different ways under objection — none of that is signal. The summarization layer’s job is compression with judgment: what actually happened, what was decided, what needs follow-up. Getting an LLM to do this well required more structure than “summarize this transcript,” because a generic summarization prompt tends to produce a generic summary — technically accurate, practically useless for a sales context specifically.

Structuring the summary around what a rep or manager actually needs

The useful version broke the summary into fixed sections instead of free-form prose: key points discussed, objections raised (and how they were handled), next steps with owners, and any pricing or timeline commitments mentioned. This structure did two things. First, it made the summary scannable — someone could check just the “next steps” section without reading the whole thing. Second, and less obviously, it made the LLM’s job easier — asking it to fill in named slots produces more reliable output than asking for open-ended prose, because there’s a stronger implicit contract about what belongs where.

Speaker attribution matters more than raw accuracy

Word-level transcription accuracy gets most of the attention in speech-to-text evaluation, but for meeting intelligence specifically, who said what often matters more than perfect word accuracy. A summary that correctly captures “the prospect raised a concern about implementation timeline” is useful even if a few words were mis-transcribed; a summary that attributes the prospect’s objection to the rep (or vice versa) because speaker diarization glitched is actively misleading. I spent more tuning effort on getting diarization boundaries right at cross-talk moments than on marginal transcription accuracy gains, because diarization errors corrupt the meaning of the summary, not just its wording.

The video-to-text pipeline shape

The actual pipeline: audio extraction from the Zoom recording, speech-to-text with speaker diarization, a cleanup pass (removing filler, merging fragmented utterances from the same speaker), then the structured summarization step over the cleaned transcript — not the raw one. Summarizing directly from raw diarized output produces noticeably worse results than summarizing from a cleaned intermediate representation, because the LLM has to do double duty (both interpreting garbled input and compressing it) instead of focusing purely on compression and structure.

Where this saves real time, and where it doesn’t

The clearest win was for reps documenting call notes into a CRM — a structured summary as a starting point they lightly edit is meaningfully faster than writing notes from memory or a raw transcript. Where it was less of a slam dunk: using summaries as a replacement for watching a call recording when something genuinely nuanced needed review (a borderline objection, a tone-sensitive moment). Summarization is compression, and compression loses information by design — treating a summary as a strict superset of the source call is a mistake worth avoiding explicitly in how you present the feature to users, so expectations stay calibrated.

What I’d tell someone building this today

Don’t evaluate a meeting-intelligence pipeline purely on transcription word-error-rate — evaluate the end product (does the summary help someone act faster and more accurately) and work backward from there. Structure the summarization prompt around what your specific users need to extract, not a generic “summarize this” instruction. And invest in diarization quality specifically, because attribution errors are the failure mode that actually erodes trust in the feature, more than imperfect wording ever does.