Inform a small documentation project with data

This is post 5 of 5 in the Data-informed documentation series. Start at the beginning: Practicing data-informed documentation.

When you take on a small documentation project, you can inform each step with data. It might seem like overkill. Small projects are often ones that you tend to consciously plan or build a process around — they’re mostly an execution task:

It’s easy to treat each small documentation request as an important task to deliver, especially with AI-driven workflows. Avoid bloating the content, and thus the context that your reader must absorb, by using data to inform even small documentation requests and determine whether to write something at all.

When you work on a project this small, the phases look somewhat different from those of a medium or large-sized project:

  1. Plan whether and where to make the update
  2. Write the update
  3. Evaluate the usefulness of the update

You also want to keep the time that you spend on each phase in scale with the size of the project itself. For a small documentation project, spend the most time planning the update, less time making the update, and even less time on evaluating the outcomes of your update.

Plan small documentation updates #

When planning a small documentation update, examine the request and determine whether a documentation update is the right approach. The request is the first, and most important, data source. As part of planning the project, you want to identify the actual problem leading people to request the documentation be updated. Based on that, you can then evaluate if a documentation update is the right approach.

Choose how much data, and which sources to use when planning your small documentation updates depending on your analytics capacity :

Capacity Key data sources
None or Basic Original request
Product exploration
Growing or Defined Page quality feedback
Product exploration, Documentation exploration
Mature A wide variety of customer submissions and research

Evaluate the request #

When evaluating the request, take a data-informed approach and use the following resources to identify the root cause and qualify the request:

Customer submissions
Page quality feedback
Support cases
Community forum questions
AI chatbot conversations
MCP server interactions
Social media posts
Search terms
Slack posts
GitHub issues and PRs
Research
Product exploration
Documentation exploration

No matter what data sources you use to contextualize the documentation request, consider the following:

  1. What is the request?
  2. How often is the request being made?
  3. Who is making the request, both how many users and of which type?

These are essentially triage questions, and the answers can reveal that something is inaccurate, needs clarification, or can lead to data loss or a security issue for customers.

For example, given the following documentation requests that you might receive:

With just the data in the request alone, it might seem like only one person is asking this infrequently. If you attempted to situate the requests on the following chart, they might all be an “A”:

Chart with four quadrants, with the x-axis labeled how often, ranging from never at 0 to all the time, and the y-axis labeled who is asking?, ranging from one person at 0 to every customer. In each quadrant is a comment box labeled with A (near one person and never), B (near one person but most of the time), C (near lots of customers but sometimes), or D (near every customer and all the time).

But what is actually true? For each of these requests, even the straightforward accuracy, clarification, and risky ones, you can provide better outcomes for your readers by adding data to inform your planning process.

You don’t need to use every data source available to you, especially if you have limited analytics capacity, but you want to look at a few different types to qualify the request and be able to answer the earlier questions with a reasonable degree of certainty.

For example, consider customer submissions :

Data source Questions
Page quality feedback Have customers submitted feedback about this topic before? Is there an existing ticket for this request?
Support cases/chats How many customers are contacting support about this issue? Who are they?
Community forum questions Are customers asking questions about this issue?
AI chatbot conversations Are customers asking the AI chatbot on the docs for help with this?
MCP server interactions Are customers asking their AI assistants for information about this?
Social media Are customers posting on Reddit, LinkedIn, or elsewhere about this?
Search terms Are there unsuccessful searches on the docs site for this information?
GitHub issues and PRs Are there existing issues open about this request? Has anyone opened a PR attempting to update the documentation?

From the volume of requests alone, you might recognize that lots of different customers are asking for this information pretty frequently — in which case, this is either an urgent small documentation task or a larger task that needs a medium-sized approach.

The data that you collect about a request might move it from an “A” to more of a “C” or a “D”, in which case delivering documentation becomes more relevant.

For example, take the request to add information about how frequently the data in a dashboard is updated. When you go to investigate how common of a question that is, you might see a wide variety of other questions about the same dashboard. For example:

You can then decide what to do next — expand the scope of your project or not. You can choose to identify, record, and address all three of these requests at the same time — effectively promoting the task to a medium-sized project. Or, you can note the larger (scope creep) tasks for later and address only the small-scoped item to add the data refresh details for now.

If you continue with the small task, add research to further qualify the request and plan your writing task :

Qualify the customer #

When analyzing customer submissions , you’re trying to identify roughly how many customers are affected or share the request, and what they might have in common. This qualifying step helps you determine if the request is a universal concern (and thus a good idea to document), or something specific to a customer’s configuration (and thus likely irrelevant to add to documentation).

For example, in response to page quality feedback requesting details about how often the data in a dashboard is updated, you might perform research to answer the following questions about the customer experience:

Explore the product and its code #

Further qualify the context of the request by answering the following questions about the product with product research .

For example, in response to a request for details about how often the data in a dashboard is updated, identify what the latency is and why it exists:

How easily you can answer these questions reflects the scope of the task and the analytics capacity of your organization. If the answers are easy to find, the task stays small and your analytics capacity might be growing or more mature than that. If the answers are undocumented internally, not available in code that you or AI tools can access, or otherwise difficult to find, the project can start to creep in scope due to unavailable data and context. Avoiding that scope creep is why it matters to identify your analytics capacity before deciding when and whether to add data to a project.

Refer to existing documentation #

You can also explore the documentation itself to help plan your approach and qualify the request .

Review the existing documentation. For the example of investigating dashboard latency, review the following:

Leave lessons for your future self and agent collaborators
If you use AI in your planning process (and your drafting process), the context gained from reviewing similar pages can be easily skipped because agents naively follow prompts. To prevent agents from updating one page without checking related pages, consider adding something to the file that instructs the agent behavior (AGENTS.MD, CLAUDE.MD) to review similar documentation pages for necessary changes when making documentation changes to one page.

With the research complete, finish your data-informed plan.

Plan what changes to make and why #

After performing research to evaluate and qualify the request, you can plan what changes to make to the documentation, and justify why:

What you found in the data Action to take
Latency of up to 4 hours is expected. Add a note to the dashboard documentation.
Misconfigured settings can cause a refresh delay. Add a note to the dashboard documentation.
A bug in the data pipeline causes latency for some customers. Add a known issue to the product release notes.
One customer misconfigured a setting and thought there was latency. Do nothing in the documentation.
Latency was documented but on a main page about all dashboards, not a specific dashboard page. Add the latency, or a link to the main page, to each dashboard page.

If you determine that writing a small documentation update is the right path, continue with that part of the process, otherwise skip to evaluating your decision.

Write small documentation updates #

In planning, you identified whether there is truly a documentation issue. Now, you execute and deliver on your data-informed plan:

If you use AI in your drafting process, it can be easier to add data into your workflow, but the size of a small documentation change can limit the utility of adding data in this phase. Some practical examples of using data with AI agents is to use your knowledge of the existing code base to prompt an agent to copy the structure of a note on another page, or to create a reusable snippet with some existing note text, and add that to a documentation page. You can rely on your documentation research to take AI-enhanced shortcuts.

Guard against context bloat #

A risk of using AI in your writing process , even in a data-informed practice, is that it is easier to respond to every request by automatically delivering on it. Request received in a Slack thread? Summon an agent immediately to make the requested update!

That sort of responsiveness can seem like ideal customer service, but it can also lead to bloated content that contains irrelevant information.

Knowing that AI agents are increasingly consuming your content might motivate you to add more content than you might otherwise, so that the agents have all the context available to them. However, if you continually make small updates to your documentation, adding new bits of information when requested, the key points of the content can get lost.

As Dachary Carey has discovered with her research, such as in Agent Web Fetch Spelunking, agents can miss content buried in large pages, and they have limited context windows.

If you have too much content, you risk context bloat and distracting your human and AI-powered content consumers. Instead, take the time to plan and use data to inform even small documentation changes. Contribute your expertise as a user advocate, curating the documentation experience.

Evaluate small documentation updates #

When evaluating a small documentation update, your main goal is to assess whether you sufficiently addressed the original request. The data you use to inform the project helps you reduce uncertainty about your direction, implementation, and validate the outcomes.

Small changes to the documentation often aren’t noticed by customers, so aggregate quantitative metrics don’t show any measurable difference. As a result, how you evaluate the change looks different from a medium-sized or large project.

Based on your uncertainty about the change, the scope of the change, and the amount of data in support of the change, choose different levels of effort when evaluating the change:

With this more elaborate approach, you might want to review the following data sources :

Data source What to look for
Page quality feedback Do you still get feedback comments about the confusion that led folks to write in?
Support cases Do you still get cases asking about this thing that you clarified?
AI chatbot conversations Does the chatbot answer the questions more clearly and correctly?
Search terms Do search terms about the issue find results, or have a higher % of clicks on the page that you updated? Search volume might be the same (or higher), but the journeys to get to the relevant information (the page you updated) are hopefully shorter.
Page views and click activity If you updated only the release notes, you can review the number of page views and click interactions with the note that you added.
Product telemetry If you updated only the product, look at usage of the page that you updated.

If you use quantitative sources, remember that the purely quantitative sources of data only reveal impressions, not understanding or direct interaction. They help you assess whether the change reached your users, but not whether they actually read it.

Evaluate a choice to not write documentation #

If you decide not to make a documentation update based on the data you consulted in the planning stage and instead decide to make an update to the product experience, you can evaluate that choice too!

For example, you added an on-page description and some tooltips for more confusing terms for a dashboard in the product, rather than adding a note to the documentation about the dashboard.

How do you know if those changes were effective on their own and don’t need to be partnered with additional documentation updates? Review additional data sources from customers :

In addition to changes in feedback patterns, the amount of feedback that you see from customers is also relevant. If you only see a few requests, or a few pieces of feedback about this issue over a quarter, the data isn’t large enough to treat as a representative sample.

An absence of data can be a good sign, but only in comparison to a relevant baseline. For example, if there was a strong pattern of consistent support cases for this dashboard, or search queries for the confusing term, and after the change you make there are no longer support cases or search queries for the term, that drop-off is a likely indication of success.

On the other hand, if customer submission data sources remain the same after the change, it’s possible that your change wasn’t effective. An even clearer signal is if requests to update the documentation remain consistent, or resume. In that case, return to your planning data sources, collecting and analyzing more to identify whether to actually update the documentation.

Practice data-informed documentation on a small project #

Taking the time to inform even small documentation projects with data strategically ensures that your content remains relevant and useful for your readers. Not all requests need to result in documentation updates, and not all seemingly small requests stay small.

By taking a data-informed approach, you can effectively deliver relevant, curated content updates without unexpected scope creep or bloated pages that confuse human readers and AI agents they might use.

This post is part of the Data-informed documentation series.