What Pharma's AI Answer Problem Teaches Every Company About Its Own Documents

When a source document changes, every copy of the old version is out of date from that day, and an AI answer drawn from one of those copies can be wrong on whatever point changed. Pharma is a well-documented case of this problem, because a medicine's official reference document changes during its life and regulators and researchers publish what happens afterwards.

Companies outside pharma have the same structure, often with nobody outside the company checking. A price list, a travel policy, a product specification or a contract template changes, and the copies in SharePoint, the Microsoft 365 Copilot answers and the agents can keep using the old one.

What the pharma evidence shows

Every approved medicine has an official reference document, called the label in the United States and the summary of product characteristics in Europe. It states how the medicine should be used and what its risks are, and it is revised as evidence accumulates. FDA staff reviewed 339 new drugs approved between October 2002 and December 2014 and found that 88.5% had at least one safety change to the label. The mean was 4.2 labeling updates per product over a median follow-up of 13.1 years. The analysis was published on 7 August 2026.

The law ties company material to that document. In the European Union, Article 87 of Directive 2001/83/EC says advertising of a medicine "must comply with the particulars listed in the summary of product characteristics." A brochure that complied last month can therefore fall out of line this month without anyone having edited it.

Old versions stay in circulation anyway. In a case completed in March 2022 by the PMCPA, the authority that administers the UK pharmaceutical industry's code of practice, a revised reference document for one medicine reached the UK's public medicines compendium 146 days after the company's own internal notification of the change. In the same case, promotional material that pre-dated one update was withdrawn 77 days after that update's formal approval. The case included a public reprimand for failing to notify and implement changes to the reference documents.

AI answers can carry the problem forward. A study in BMJ Quality and Safety, published online on 1 October 2024, put patient questions about the 50 most prescribed US drugs to Bing Copilot and compared 484 answers with a reference source. It found 26.0% partly inconsistent and 3.3% fully inconsistent with that reference. The authors reported answers that used outdated thresholds for one common drug, and wrote that the chatbot "lacks the capability to verify their currentness". Those answers were generated in April 2023 by a consumer chatbot reading public websites, so the figures do not measure Microsoft 365 Copilot inside a company. They are one test in which an assistant's answers disagreed with the reference, sometimes because the advice was out of date.

Every company has reference documents that change

Worked example, for illustration: a company revises its standard payment terms. Finance owns the current version. Sales still has the previous terms as a PDF in a Teams channel, and a proposal template quotes them. Nothing in the company's Microsoft 365 environment marks one of these files as the reference.

Microsoft's documentation states the access rule Copilot follows. Microsoft 365 Copilot "only surfaces organizational data to which individual users have at least view permissions." That rule is about permission and says nothing about which version of a document is current, so an employee who can open the old PDF can receive an answer drawn from it. An agent built in Copilot Studio adds a second place to look, because the knowledge sources its maker configured supply the information for its answers.

Pharma has authorities that can reprimand a company for an outdated leaflet. A stale price list has no such check in many companies, and it can reach a customer before anyone notices the discrepancy. The six controls below apply the same discipline to the documents where a wrong answer has a real cost.

Six controls for documents that change

1. Name the reference version and its owner

List the documents where a wrong answer costs money or trust: price lists, HR and travel policies, product specifications, contract templates, safety procedures. For each one, record a single location that holds the reference version and a named person who approves changes. Keep the list short enough that every entry has an owner who will answer for it.

2. Make each change visible to people and to machines

An email announcing an updated policy does not change what an agent reads. Put the version number, the effective date, the owner and a short summary of what changed on the first page of the document and in the library columns beside it, as structured fields. SharePoint keeps earlier versions of a file in its version history, so the reference file can keep one address while its content changes. Keeping one address avoids broken links, but each connected system still has to pick up the revision, which is what control 5 checks.

3. Know where the copies live

Search for each reference document by title and by a distinctive phrase, and record every copy with its site and owner. Delete each copy or replace it with a link to the reference, unless the copy has to be kept as a record. If your reference documents carry their own sensitivity label, the data access governance reports in SharePoint Advanced Management include one that identifies sites containing files with a specific label. It gives you sites to investigate and does not trace individual copies. A file uploaded as agent knowledge is another copy to manage. Copilot Studio's documentation lists uploaded documents as a knowledge source held in Dataverse, the Power Platform data store, so plan to replace each uploaded file when its source changes.

4. Decide what Copilot and agents may read, and retire old versions from that set

Retiring a version means keeping it for the record while making it less likely that an assistant answers from it. Microsoft Purview retention labels apply at the item level and can retain a document for a set period and then delete it, with a disposition review if a person should confirm the deletion first. Until then, move superseded versions to an archive library and set "Allow items from this document library to appear in search results" to No. Microsoft documents that setting for search results and the page does not mention Copilot, so confirm the effect with the test questions in control 5.

For whole sites, Restricted Content Discovery limits discovery of a site's content in organization-wide search and Copilot responses without changing permissions. Microsoft describes it as a temporary governance control that requires a Copilot license and SharePoint Advanced Management. It does not affect content a user already has open, and for sites with more than 500,000 items an update "could take more than a week" to take effect. Avoid building on the older Restricted SharePoint Search, which Microsoft says is retiring, with new enablement blocked from July 31, 2026. A data loss prevention policy can also restrict labeled files. Purview documentation says such a policy can prevent Copilot and agents from summarizing them, although Copilot "can reference it with a link" so that the user can open the file directly.

5. Test a fixed set of questions after each change

For each reference document, write down the questions employees and customers ask about it, with the correct answer under the current version. After every change, run them through Copilot and each agent, and check both the answer and the file it cites. A citation to a superseded copy is a finding even when the answer happens to be right. For agents built in Copilot Studio, the evaluation feature runs a saved test set of up to 100 test cases against the agent, and several of its test methods compare each answer with an expected response. That page covers Copilot Studio agents, so plan for a person to run the same questions by hand in Microsoft 365 Copilot Chat.

6. Keep the record

A year from now someone may ask what the assistant was telling staff in March. The record that helps is the version in force on each date, the test results for that version, and a trace of what Copilot read. Purview audit records for Copilot include references to the files and sites Copilot accessed to answer a prompt. Microsoft notes that auditing does not capture the prompt or response text and points to eDiscovery for that. Copilot Studio keeps test results for 89 days, so export them after each run. None of this reconstructs every answer given, which is a reason to keep the tests and their dates.

Where this approach has limits

These controls cost owner time, so apply them to the short list from control 1 and leave ordinary working files alone. Hiding too much also has a cost: Microsoft warns that excessive use of Restricted Content Discovery can affect "the completeness and relevance of search results and AI-generated responses." The pharma evidence is also narrower than it may look. No study cited here measured Microsoft 365 Copilot against a company's internal documents after a change, and a test of that kind would be the best evidence for or against this argument. Use your own test results, together with what a wrong answer would cost, to decide which controls need more work.

What to do this month

1. List the documents where a wrong answer costs money or trust, and give each one a reference location and a named owner.
2. Add the version number, effective date, owner and change summary to the first page and the library columns of each one.
3. Search for copies of your most important reference documents, including files uploaded to agents, and decide what happens to each copy.
4. Write test questions with correct answers for each document, run them in Copilot and in each agent, and export the results.
5. Agree where superseded versions go, and which label or setting keeps them out of Copilot and agent answers.

Where QueryNow fits

QueryNow helps IT and knowledge teams put these controls in place in Microsoft 365 through its Copilot AI foundation service, and it makes Juncture, a product for pharma teams that flags where company content or an AI answer no longer matches a medicine's official information. Mahesh Tatiparthy, QueryNow's founder, will discuss this subject on a panel webinar later this year.

Update, 11 November 2026: the panel, Winning the AI Battle for HCP Attention, is confirmed for Thursday 17 December 2026, 16:00 to 17:00 CET. The registration link will be added here when it opens.