Lesson 6.3: Privacy, bias and responsible use of AI at work
Learn what data not to paste, how to test a prompt for unequal treatment, and a practical rulebook for human oversight, disclosure and accountability.
By the end of this lesson you will be able to sort the information you handle into what may and may not go into an assistant, reduce what you share without losing the help, test a prompt for unequal treatment of different groups and read the result with its uncertainty, and apply a short set of habits for oversight, honesty and accountability that suit work with real people's lives and money. You will estimate how many personal records a small habit can expose over a year, and see why a comparison of 50 pairs of CVs cannot tell you whether a prompt treats two groups differently, and what sample size could. You should have read Lessons 2.2 and 5.2, since privacy begins with what you put into a prompt and with where a tool sends it, and Lesson 5.1, since the bias test uses the same logic as any comparison. This lesson gives practical guidance and is not legal advice: the rules that apply to you depend on your organisation, your sector and the law of the places where you work.
The other lessons in this module made prompts reliable and secure. This one asks whether it is right to use them in a particular way, and what you owe the people whose information passes through. For most professional work, the answers are straightforward and require habit more than expertise. For work that touches customers' money, health or livelihood, they matter a great deal, and a small slip can affect a person who never knew an assistant was involved.
Privacy: what goes into the prompt
Everything in a prompt travels to whatever system runs the model. Depending on the tool, that text may be stored, reviewed by staff, used to improve products or exposed through a breach. You cannot get it back. So the first privacy rule is also the simplest: decide what is allowed in before you paste it.
Sort your information
A practical sort has four classes.
Class · Examples · Default rulePublic · Published reports, your own public writing, general knowledge · Fine to use in any tool
Internal · Draft plans, internal procedures, non-public figures, meeting notes · Only in tools your organisation has approved
Confidential · Client contracts, unreleased results, source code, pricing, negotiation positions · Only in approved tools with proper terms, and often not at all
Personal or sensitive · Names with account details, ID numbers, contact details, health, financial or location data about identifiable people · Not in a general tool. Only in systems approved for it, with the minimum needed
Personal data deserves particular care because the harm falls on someone else. Identifiers such as government ID numbers, bank account and card numbers, phone numbers, addresses, biometric and health information, and anything about a person's finances or credit are the usual examples. Combinations matter too: a name plus an employer plus a small town can identify a person even when no ID number appears.
Predict first: an analyst pastes 30 customer complaint emails into a public chat tool to get a summary of the themes. The emails contain names, phone numbers and account numbers. She does not need any of those for the summary. What should she have done, and could the summary have been just as good? Decide before reading on.
She should have removed the identifying details first. The themes, such as late refunds, confusing fees and delays in account opening, do not depend on who wrote the email, and a summary of the same emails with the names and numbers replaced by placeholders would have been equally useful and far safer. The lesson is the principle of data minimisation: give the tool only what the task needs.
Reducing what you share
Four techniques cover most cases.
Remove. Delete names, numbers and addresses that the task does not use.
Replace with placeholders. Substitute "Customer A", "Account X" or "[PHONE]", and restore the real details afterwards if you need to. This is the habit from Lesson 2.2.
Aggregate. Work with counts, shares and summaries, not individual records. "12 per cent of complaints in March mentioned fees" says what you need without a single name.
Generalise. Replace exact values with ranges or categories: an age band for an age, a city for an address, "a salaried customer in their thirties" for a profile.
These steps lower the risk and do not remove it. Patterns in the text can still point to a person, particularly a small group or an unusual case, so for truly sensitive work the answer is a system approved for it, not cleverer redaction.
The tool matters as much as the text
The same text can be fine in one tool and a violation in another. Questions to settle, ideally before you start and with your IT or compliance team:
Is this tool approved for my organisation, and for this class of data?
Is my text stored, for how long, and who can see it?
Is it used to train or improve models, and can that be switched off?
Where is the data processed, and does the law or my regulator restrict that for this kind of data?
Is there a contract that covers confidentiality, security and breach notification?
Can I delete what I have shared?
Many organisations offer an approved workplace version of an assistant with stronger terms, and provide a list of what may be used with it. Use it. Where none exists, the safe default is to treat the tool as public and to put in nothing you would not publish. Laws on personal data, such as India's Digital Personal Data Protection Act, 2023, together with sector rules for financial and health information, set duties on organisations that collect and handle personal data, and using an unapproved tool can breach them. Ask your compliance team where the lines are, since you are the one pasting.
How small habits add up
A single paste feels harmless. Repeated across a team, it is not. Suppose 50 staff each paste text into a general tool five times a week for 46 weeks. If 30 per cent of those pastes contain personal data, with about 10 records in each, then the exposure is 50 × 5 × 46 × 0.30 × 10 = 34,500 personal records over a year. None of the individuals consented, and none of them knows.
Bias: unequal treatment
A model learns from vast amounts of human writing, and that writing carries human assumptions, gaps and stereotypes. The result can show up in several ways.
Form · What it looks like · ExampleStereotyped content · Text that assumes roles or traits for a group · A draft job advert that describes "a strong leader, he"
Unequal quality · Better performance for some languages, dialects or names · Weaker answers in a regional language or on names the model sees less often
Unequal outcomes · Different recommendations for people who differ only in a protected trait or a proxy · Scoring similar CVs differently by name
Representation gaps · Examples or images that leave out groups · An example set that only features urban customers
Hidden proxies · A neutral-looking field that stands in for a sensitive one · A postcode that reflects community or income
For everyday drafting, the response is attention and editing: read generated text for assumptions, ask for neutral language ("use gender-neutral wording and avoid assumptions about age, background or ability"), and review images and examples for who is missing. For decisions about people, such as who is shortlisted, approved, flagged or contacted, the stakes are higher and a test is needed.
A paired test
The simplest test for unequal treatment uses matched pairs. Take a set of inputs and make two versions of each that are identical except for one detail that should not matter: a name associated with a different community, a gender, an address in a different area. Run both versions through the prompt, and compare the results. If the prompt treats people fairly, the outcomes should be the same, apart from run-to-run noise.
Say you ask a model to recommend whether to interview a candidate, and you test 50 identical CVs under two names. With the first set of names, 34 are recommended, which is 68 per cent. With the second, 29 are, 58 per cent. The gap is ten points. Is it real, or chance?
The standard error of the difference between two proportions is √(0.68 × 0.32 ÷ 50 + 0.58 × 0.42 ÷ 50) = √(0.00435 + 0.00487) = 0.096, so the difference of 0.10 is only about one standard error from zero. The 95 per cent margin on the gap is about 19 points. The data cannot say whether the gap is real. To detect a ten-point gap with 80 per cent power you would need about 360 pairs per group: 7.84 × (0.68 × 0.32 + 0.58 × 0.42) ÷ 0.01 = 362. This is the same arithmetic as the sample sizes in Lessons 2.1 and 5.1, applied to a question of fairness.
Two cautions follow. A non-significant result is not a clean bill of health. A small test cannot see a modest bias, and a bias small per case can still wrong many people at scale. And fairness has many definitions that cannot all hold at once, such as equal approval rates, equal error rates and equal treatment of identical cases. Which one applies is a policy decision for people with the authority and the context, not something the prompt writer can settle alone.
What to do when you find a gap
Look at the cause. Is the prompt using a proxy, such as a name or a postcode? Remove the field, or instruct the model to ignore it, and test again.
Take the model out of the decision. For anything where unequal treatment is a serious risk, use the model to extract or summarise facts, and have a rule or a person decide.
Retest after every change, and keep the paired tests as part of your test set from Lesson 5.1.
Escalate. If the gap affects people in a regulated area such as credit or employment, tell the people responsible for compliance and fairness, and do not quietly patch it.
Responsible use: a short rulebook
Beyond privacy and bias, a handful of habits mark responsible use of these tools at work.
Keep a person accountable. Whatever the assistant drafts, you sign. The tool is not the author of record, and "the AI said so" is not a defence to a customer, a regulator or a colleague. A named person owns every output that leaves the building.
Match oversight to stakes. Lesson 2.4 gave a simple scale. Low-stakes drafting needs a read-through. Anything that affects a person's money, job, health or legal position needs a person who understands the subject to review it, and, where a decision is made about someone, a route for them to ask for a human review and to correct errors. Do not automate high-stakes decisions about people end to end.
Beware over-reliance. People tend to trust a fluent, confident system more than it deserves, especially when tired or busy. Build a deliberate check into the workflow, such as a second reviewer or a required look at the source, so that trust is earned each time.
Be honest about AI involvement. Do not present generated text as a person's own work where that would mislead. Follow the disclosure rules of your organisation, your clients and your regulators, and where there are none, lean towards saying so when it matters to the reader. Never use these tools to fabricate evidence, reviews, quotations or identities.
Respect other people's work. Do not paste material you have no right to share, such as a client's confidential document or a paid report, into a tool. Treat generated text as a draft that may echo existing writing, and check anything you publish.
Avoid harm in what you produce. Do not use assistants to create misleading or deceptive material, to harass or impersonate, or to produce content that targets individuals. Consider what a convincing but false document, image or message could do in the wrong hands, and apply the same standards you would to anything you wrote yourself.
Keep records. For anything important, keep the prompt, the version of the model and the output, with the date and who reviewed it. If a decision is later questioned, you can show how it was made.
Have a way to report. If something goes wrong, such as private data pasted by mistake, a harmful output or a tool behaving oddly, report it quickly to the right person. A prompt and honest report limits the damage and helps the organisation fix the cause, and blame for the report discourages it.
A checklist before you use a tool on a task
Question · If the answer is noIs this tool approved for this kind of information? · Use an approved tool, or remove the sensitive details
Have I removed what the task does not need? · Remove or replace it
Is anyone's rights, money or opportunities affected by the output? · If yes, add human review and a way to appeal
Have I tested for unequal treatment, if it concerns groups of people? · Run a paired test before relying on it
Can I explain how the output was produced and checked? · Record the prompt, the sources and the checks
Am I allowed to share this material and to use this output? · Check rights, licences and disclosure rules
Would I be comfortable if the person concerned saw what I did? · Rethink the approach
The last question is a reliable guide. If you would not want the customer, the candidate or the colleague to see exactly what went into the tool and how the result was used, that is a signal to stop and ask.
Worked examples
Example 1: a complaints summary, done safely. The analyst from the opening exercise now removes names, phone numbers and account numbers with placeholders, and works from a tool her organisation has approved. The themes she needs come out in the same way. A colleague checks that no identifying detail remains. The change costs ten minutes and removes the risk.
Example 2: a shortlisting prompt under test. A hiring team's prompt scores CVs. A paired test with 40 pairs shows a six-point gap with a z statistic of 0.6, which is inconclusive. Rather than accept it, they remove names and addresses from the CVs before the model sees them, so the prompt cannot use them, and they keep the model's role to summarising skills. A person decides who to interview. They plan a larger paired test each year, with 300 pairs per group.
Example 3: a policy that worked. A bank's team writes a one-page rule: personal data never goes into general tools, the approved workplace assistant is for internal and confidential material, every customer-facing draft is reviewed by its sender, and any use of AI in a decision about a customer is logged and reviewed by a manager. A quarterly check samples ten outputs. In the first quarter, two prompts are found with account numbers in them. The team fixes them and adds a placeholder step to the template. The policy is short enough that people follow it.
Common mistakes
Pasting first and asking later. Decide what may go in before you paste.
Assuming a tool is private. Check storage, training use and who can see your text.
Using real data when placeholders would do. Minimise what you share.
Believing a model is neutral. It reflects its training, so test the cases that matter.
Treating a small test as proof of fairness. A non-significant gap in 50 pairs means almost nothing.
Automating decisions about people end to end. Keep a person in the loop and a route for review.
Passing generated work off as your own where it matters. Follow disclosure rules, and be honest.
Hiding mistakes. Report problems quickly, and fix the cause.
Key takeaways
Everything in a prompt goes to the system that runs the model. Sort information into public, internal, confidential and personal, and apply a default rule to each.
Minimise what you share: remove, replace with placeholders, aggregate or generalise. An approved tool with proper terms is the answer for genuinely sensitive work.
Small habits scale: 50 people pasting five times a week, 30 per cent with personal data and 10 records each, expose about 34,500 records a year.
Models can show stereotypes, unequal quality and unequal outcomes. Test decisions about people with matched pairs, and read the result with its uncertainty: a ten-point gap on 50 pairs per group is inconclusive, and about 360 pairs are needed to detect it.
A non-significant result is not proof of fairness, and fairness has several definitions that need a policy decision.
Keep a named person accountable, match oversight to stakes, be honest about AI use, respect others' work, keep records and report problems fast.
Check yourself
1. What is data minimisation?
Giving a tool only the information the task needs, for example by removing or replacing identifiers.
2. Name three techniques for reducing the personal data you share.
Remove it, replace it with placeholders, aggregate to counts or shares, or generalise exact values into ranges.
3. In a paired test, 40 of 100 pairs favour group A and 30 of 100 favour group B. Roughly how many standard errors is the gap?
Gap 0.10. SE = √(0.4 × 0.6 ÷ 100 + 0.3 × 0.7 ÷ 100) = √(0.0024 + 0.0021) = 0.067. So z = 0.10 ÷ 0.067 = 1.5, below 2.
4. Why is a non-significant bias test not proof that a prompt is fair?
A small test may be too weak to detect a modest gap, and a small gap can still harm many people at scale.
5. Which decisions should not be fully automated?
Decisions that affect a person's money, job, health or legal position, without human review and a way to ask for correction.
Try this
You need an assistant and about fifty minutes. Use made-up or properly anonymised material only.
1. List ten kinds of information you handle at work and place each in public, internal, confidential or personal. Mark what you may put in your organisation's approved tool and what in no tool.
2. Take a real task that uses personal data. Rewrite the input with placeholders, aggregation or generalisation. Run both versions and compare the usefulness of the result.
3. Ask your IT or compliance team the six tool questions from this lesson for the assistant you use. Write down the answers, and any you could not get.
4. Use the exposure simulator with your own team's numbers. What does a year of habit amount to?
5. Choose a prompt that assesses or recommends something about people, such as a summary of a candidate. Create 20 matched pairs of inputs that differ only in a name or similar detail. Run both versions three times and compare the results. Use the gap simulator to say whether the difference is beyond noise.
6. If you find a gap, try one fix, such as removing the detail before the model sees it, and test again.
7. Write the one-page rule you would give a new colleague about using AI at work, covering data, review, disclosure and reporting.
Item · Your resultInformation classes and rules ·
Exposure per year at your numbers ·
Paired-test gap and z statistic ·
Fix tried and its effect ·
Your one-page rule ·
A good result is a classified list of information with rules, a minimised version of one real task, a paired-test result stated with its uncertainty, and a short rulebook you could hand to a colleague. In the final lesson you bring everything in this course together in one project: a tested, safe prompt library of your own.
Tags: Prompt Engineering Privacy Bias Responsible AI Advanced