Lesson 4.2: Research, summarising and extraction prompts
Learn prompts for researching a topic, summarising faithfully and pulling structured facts from documents, with a sampling check for large batches.
By the end of this lesson you will be able to use an assistant to plan and carry out research without trusting it as a source, write summaries that stay faithful to the text and fit the reader, define a schema so the assistant pulls the same fields out of every document, and check a large batch of extractions by sampling instead of reading everything. You will see why a typical pilot reports different accuracy for different fields, and how to turn a small sample of checked outputs into an honest range for the whole batch. You should have read Lessons 2.2 and 2.4, since all three jobs depend on grounding answers in documents and on checking what comes back. No technical background is needed.
Much of professional work is reading: reports, contracts, regulations, emails, transcripts. An assistant can read faster than any person and is tireless, and that makes it tempting to hand over the reading and accept the result. The skill lies in using it for what it does well, which is compressing, sorting and structuring text, while keeping control of what is true. The three jobs in this lesson, research, summarising and extraction, differ in how much you can trust the output and how you check it.
Three jobs, three risk profiles
Job · What you ask for · Where facts come from · Main riskResearch · Understand a topic, find leads, compare views · The model's memory, or a search the tool runs · Invented or outdated facts, fake sources
Summarising · A shorter version of text you supply · Your document · Additions, omissions, shifted emphasis
Extraction · Specific fields pulled out of text you supply · Your document · Wrong values, missed items, inconsistent format
Research is the riskiest, because the facts come from the model, not from your document. Summarising and extraction are safer, since the material is in front of the model, but they have their own failure modes, and with extraction at scale a small error rate becomes many errors.
Research: use the model to think, not to know
An assistant is a poor encyclopaedia and a good research partner. It can help you break a question into parts, suggest where to look, explain unfamiliar terms and organise what you find. It should not be your source for facts you will rely on.
Predict first: you ask an assistant, with no search tool and no documents, "What were the main changes in digital lending rules in India over the last two years?" Which parts of the reply would you trust, and which would you check first? Decide before reading on.
You can reasonably trust the broad shape, such as the general direction of regulation. You should check every date, circular number, threshold and named rule, because these are specific, and the model may be out of date or may blend different rules. The reply is a map of what to look for, not a record of what is true.
A research workflow
1. Frame the question. "I need to understand X in order to decide Y." The decision shapes what counts as relevant.
2. Break it down. "List the five sub-questions I would need to answer, in the order I should tackle them."
3. Ask for a map. "Explain the key terms and the main positions on this topic. Label what is widely agreed and what is disputed."
4. Ask where to look. "Which kinds of sources would settle each sub-question: regulator publications, filings, industry reports, academic studies?"
5. Go to the primary sources yourself, or use a tool that fetches them, and open every link.
6. Bring the material back and ask the assistant to summarise, compare or extract from it, with quotations.
7. Log what you found, with the source and date of each claim.
A research log is simply a table with the columns Claim, Source, Date and Confidence. It stops a project from becoming a pile of half-remembered statements, and it is what you will cite later.
Prompts that make research better
Ask for both sides. "Give the strongest argument for and against, and say who makes each." This counters the model's tendency to follow the lean of your question.
Ask what is contested. "Which parts of this are disputed among experts, and which are settled?"
Ask what would change the answer. "What facts, if they turned out differently, would reverse your conclusion?"
Ask for a timeline, then verify each date. Sequences expose errors, because a wrong date often contradicts the one before it.
Ask the model to say when it does not know. "If you are not sure of a detail, say so and do not guess."
Set the time frame. The model's knowledge stops at some point. Tell it the period you care about, and treat anything recent as unverified unless a search tool fetched it.
When a tool can search the web, the quality of the result depends on the pages it picks, which you do not control. Ask for the links, open them, and judge each source: who published it, when and why. A regulator's own page outranks a news summary, and a news summary outranks an anonymous blog.
Summarising faithfully
A summary is a promise: everything in here is in the original, and nothing important has been dropped. Models are good at the first part when the text is in front of them, and less reliable on the second, because deciding what matters is a judgement. You can steer both with the prompt.
Controls that matter
Control · Example instructionReader · "For a branch manager who has not read the report"
Length and shape · "Five bullets of at most 25 words"
Focus · "Concentrate on changes since last quarter and on risks"
Exclusions · "Leave out methodology and background"
Fidelity · "Use only what is in the text. Do not add interpretation."
Traceability · "After each bullet, give the page or section it comes from"
The fidelity line is worth stressing. A summary that adds a plausible "so what" feels more useful and is a common source of error, since the added sentence was not in the text. If you want interpretation, ask for it separately, labelled as such: "Summary of the text, then a section titled Interpretation, marked as my assistant's view."
text
Summarise the report below for a branch manager who has not read it.
Five bullets, each at most 25 words, covering what changed since last quarter
and the main risks. Use only the report. Do not add interpretation. After each
bullet, give the section it comes from in brackets. Leave out methodology.
If something important is unclear in the report, list it under "Unclear" at
the end.
Report:
"""
[PASTE]
"""
Checking a summary
Two checks matter. First, additions: can you point to the sentence in the original that supports each bullet? The section references make this quick. Second, omissions: is there something important in the original that the summary leaves out? A follow-up prompt helps: "List the five most important points in the original that are not in your summary." Compare the answer with your own reading of the document.
Long documents and many documents
For text too long to fit, a common technique is summarise in parts: split the document into sections, summarise each, then summarise the summaries. It works, but information is lost at each stage, and a detail dropped in the first pass cannot return in the second. For important work, use the extract-then-answer approach from Lesson 2.2 instead: first pull out the passages that bear on your question, then work from those.
Summarising several documents is a different task from summarising one. The valuable output is the comparison. Ask: "For each of the three reports, list its main conclusion. Then say where the reports agree, where they disagree and what each leaves out. Cite the report for each statement." This turns three summaries into an analysis.
Compression saves reading time, and it is worth seeing how much. A 6,000-word report read at about 230 words a minute takes 26 minutes. A 300-word summary takes about 1.3 minutes, a 20-fold reduction, saving about 25 minutes per report. For five reports a week that is over two hours. But the saving is real only if the summary is faithful, which is why the checks matter.
Extraction: turning text into fields
Extraction means pulling specific facts out of text into a fixed structure: the parties, dates and amounts in a contract, the action items in meeting notes, the complaint type and amount in an email. Done well, it converts a pile of documents into a table you can sort, count and check.
Define the schema
The key step is to define exactly what you want, as a schema: the list of fields, what each means, its format and what to write when it is missing.
text
From the contract text below, extract these fields.
- partya: the first party's full legal name
- partyb: the second party's full legal name
- startdate: the effective date, as DD-MM-YYYY
- amount: the total contract value as a number in rupees, no commas
- terminationnoticedays: the notice period for termination, as a number
Rules: use only the text. If a field is not stated, write "not stated". Do not
guess or calculate. After each value, give the exact words from the text that
support it, in quotation marks.
Output as a table with the columns Field, Value and Supporting text.
Contract:
"""
[PASTE]
"""
Notice the features. Each field has a definition and a format, so "date" cannot come back as "the first of November" in one document and "1/11/26" in the next. There is a "not stated" exit. And each value comes with its supporting text, which makes verification a matter of searching for a phrase. An example of one completed extraction, as in Lesson 2.1, further reduces format slips.
Piloting before you scale
Do not run a new extraction on 500 documents and hope. Run it on a pilot of 30 to 50, check every field by hand, and report accuracy per field. Fields differ widely in difficulty. Names that appear once in a standard place are easy. Dates are harder, because documents use many formats. Amounts are harder still, with currency and units. Clauses that require judgement, such as whether termination is allowed for convenience, are the hardest.
The chart tells you where to spend effort. Do not worry about the names. Improve the amount field by specifying units and examples, and keep the termination clause under human review, since an error there matters most. Accuracy by field also decides which parts of the process you can automate and which need a person.
Checking a large batch by sampling
Once the pilot is good enough, you run the whole batch. You cannot read 500 outputs closely, so you check a random sample and use it to estimate the error rate. Suppose you check 50 of 500 and find 2 with an error. The observed rate is 2 ÷ 50 = 4 per cent. But 50 is a small sample, so the true rate could differ. The margin of error is 1.96 × √(0.04 × 0.96 ÷ 50) = 5.4 percentage points, so the plausible range runs from near 0 up to about 9.4 per cent. Across 500 documents, that means somewhere between none and about 47 outputs with an error, with 20 the best guess.
Two practical points follow. The sample must be random, drawn across the batch and not the first fifty, since early documents may differ from later ones. And what counts as an acceptable error rate depends on the use: a few per cent may be fine for a first-pass sort and unacceptable for a regulatory filing. Decide the threshold before you look at the result.
Judging a source before you rely on it
Whatever the assistant finds, or you bring, apply the same short test to each source before it goes in your research log. You can even have the assistant draft the answers from the page, then check them yourself.
1. Who published it, and what is their stake? A regulator, a listed company, a trade body, a consultancy selling a service and an anonymous blog have different reasons to say what they say.
2. When was it published, and is it still current? Rules and figures change, and a report from three years ago may describe a world that no longer exists.
3. Is it primary or secondary? A circular is primary. A news article about the circular is secondary, and an article about that article is further away still. Go as close to the original as you can.
4. How does it know? A figure with a method and a sample behind it carries more weight than one with none.
5. Does anyone independent say the same? One source is a lead. Two independent ones are a finding.
This is ordinary research practice, and it matters more with an assistant, because the tool will summarise a weak source as fluently as a strong one. Fluency hides quality. If the log records the answers to these five questions beside each claim, you will see at a glance which parts of your argument rest on firm ground.
Messy inputs: transcripts, chats and scans
Real documents are often untidy, and extraction suffers from it. A meeting transcript has no punctuation and several people talking over one another. A scanned form has been read badly by character recognition. A chat export has timestamps and reactions mixed into the text. A few habits help.
Clean what you can first. Strip headers, timestamps and repeated boilerplate before pasting, so the model reads content and not clutter.
Tell the model what the mess is. "This is an automatic transcript. Speakers are not labelled and some words are wrong. If a name or number seems garbled, mark it with [unclear] instead of guessing."
Ask for uncertainty to be marked. An instruction to flag low-confidence values, with a rule such as "use [?] after any value you are less than sure of", gives your reviewer a shortlist.
Check a scan's text before you extract from it. Ask for a transcription of one paragraph and compare it with the image. If the characters are wrong, the extraction will be too.
Split long inputs at natural boundaries, such as one meeting item or one contract section per chunk, and keep the heading with each chunk, so every extraction knows where it came from.
The common thread is that the model can only be as accurate as the text it is given. Time spent making the input readable is repaid in fewer errors downstream.
Worked examples
Example 1: a literature scan. A policy analyst needs to understand how three recent reports view digital lending risk. She downloads them, so the facts come from documents. She asks for each report's main conclusion with a quotation, then for agreement, disagreement and gaps, citing the report each time. She checks four quotations against the pages and finds one paraphrased. She marks it and moves on. She then spends her time on the disagreements, which is where the insight was.
Example 2: meeting transcript to actions. An operations lead pastes a transcript and asks for a table of actions with columns Owner, Task, Deadline and Supporting quote, with "not stated" for gaps. Seven actions come back, three with no owner. She sends the three to the chair for confirmation. The assistant did the sorting, and the missing owners were found because the schema forced an answer for every field.
Example 3: contract extraction at scale. A procurement team extracts five fields from 300 supplier contracts. A pilot of 40 shows names and dates above 94 per cent but the termination clause at 80. They add two examples and a clearer definition for that field, and send every termination clause to a lawyer anyway. After the run they check a random 60 documents, find three errors, and estimate the error rate at 5 per cent with a range up to about 10, which they accept for a first-pass register, with the high-value contracts reviewed in full.
Common mistakes
Treating the model as a source for research facts. Use it to plan and explain, and go to primary sources.
Not opening the links. A real-looking source may not exist or may not say what is claimed.
Summaries that add interpretation. Ask for it separately, and label it.
Never checking omissions. Ask what important points the summary left out.
Vague extraction fields. Define each field, its format and a "not stated" value.
Running a large batch without a pilot. Check 30 to 50 by hand first and measure by field.
Checking only the easy or the first documents. Sample at random.
Accepting an error rate without a threshold. Decide what is acceptable before you look.
Key takeaways
Research, summarising and extraction differ in where the facts come from and how to check them. Research is riskiest, because the model supplies the facts.
For research, use the model to frame questions, explain terms and suggest sources, then go to primary sources and keep a log of claim, source, date and confidence.
A faithful summary is controlled by reader, length, focus, exclusions and an instruction to add nothing. Check additions with section references and omissions by asking what was left out.
Extraction needs a schema: fields, definitions, formats, a "not stated" exit and supporting text for each value.
Pilot on 30 to 50 documents and measure accuracy per field. Fields requiring judgement need more help or human review.
Check a large batch with a random sample. Two errors in 50 gives about 4 per cent, with a plausible range up to about 9 per cent.
Check yourself
1. Why is research riskier than summarising?
In research the facts come from the model's memory, which can be wrong or outdated. In summarising the facts are in the text you supplied.
2. What three features make an extraction prompt easy to verify?
Defined fields with formats, a "not stated" exit and the supporting text for each value.
3. You check 40 of 400 documents and find 1 error. What is the observed error rate and the expected errors in the whole batch?
1 ÷ 40 = 2.5 per cent, so about 10 errors in 400 as a best guess, with a wide range.
4. How can you check that a summary left out something important?
Ask for the most important points in the original that are not in the summary, and compare with your own reading.
5. A 9,000-word document is summarised in 450 words. What is the compression, and how much reading time is saved at 225 words a minute?
20 times. (9,000 − 450) ÷ 225 = 38 minutes.
Try this
You need an assistant and about fifty minutes.
1. Choose a research question from your work and write the decision it supports. Ask for five sub-questions and a map of agreed and disputed points. Pick three claims and verify them against primary sources, opening each link.
2. Gather three short documents on one topic, such as reports or policies. Run the faithful summary prompt on each, with section references. Check three bullets against the originals.
3. Run the comparison prompt across the three: conclusions, agreements, disagreements and gaps.
4. Design a schema of four or five fields for a document type you handle often, such as invoices, contracts or complaint emails. Write the definitions, formats and "not stated" rule.
5. Run it on ten documents. Check every field by hand and record accuracy by field.
6. Use the sample-check simulator to see what your result would mean if you scaled to a batch of 200.
7. Fill in this table.
Field · Correct out of 10 · Error pattern · Fix to try
A good result is a verified list of research claims, a faithful summary with checked bullets, and a tested extraction schema with field-level accuracy and a sampling plan for a larger batch. In the next lesson we turn to numbers: how to use an assistant for analysis and data work without being misled by its arithmetic.
Tags: Prompt Engineering Research Summarising Data extraction Intermediate