competitive analysis on This is important

Inform a medium-sized documentation project with data

Tue, 27 Oct 2026 10:33:00 +0000

If you are about to start a medium-sized documentation project, you can inform each step of the process with data. If you do, you can more easily assess the scope of the work, prioritize and plan the implementation, and write more relevant content.

A medium-sized project is one that might take you a week or two of work, like a new feature that needs one or two new documentation topics, or a few different pages that need to be updated with new content for some to-be-released functionality.

For example, you might be writing medium-sized documentation updates for one of the following features:

An interactive workflow to create and assign tickets to specific users
A new type of widget that can be added to a dashboard
A new SQL function

In this post, I’ll cover how to use data to inform each phase of a medium-sized documentation project like one of those examples:

Planning and writing are often intertwined

For a medium-sized project, planning and writing inform each other. Rather than creating a plan in full, then starting to write, you might go back and forth, acting on some plans as soon as you make them and expanding your plan as you learn new information about the project scope that wasn’t clear when you started. This is common when documenting new functionality — it’s still in development, so the scope often isn’t predetermined and can’t be planned for fully in advance!

Plan medium-sized documentation updates

Planning a medium-sized writing task involves identifying the following:

What is being built?
What do customers expect from it?
What already exists in the documentation?

Use these discovery steps to identify the scope of the project. For example, if your engineering team is adding a new type of dashboard widget that animates images in a repeating pattern, you can use the following data sources to help plan:

Expertise

Business understanding

Research

User research
Product exploration
Documentation exploration
Competitive analysis

System

Documentation metadata

Customer submissions

Feature requests
Customer success call transcripts
GitHub issues or PRs
Search terms
Product reviews
Community forum posts

The planning process relies mostly on qualitative data, so you can inform it with data at most analytics capacity levels. If your organization has a more basic analytics capacity, you can rely more heavily on the data collected through research. If you have a more mature analytics capacity, you can add the more tool and expertise-dependent quantitative analysis.

Plan for maintenance

When you plan to add new pages and content, consider your future self and plan for maintenance too. For example, if you add screenshots of the new dashboard widget, place them in a shared location and record and share the assets used to create them, such as an example dashboard in a shared staging environment. Consider recording the research artifacts that you use to inform your plan in a longer-lived dashboard or collection that is shared with others. This practice of storing your data artifacts can help accelerate the practice of using data to inform future documentation tasks too!

The results of your data collection informs the decisions that you make as part of your documentation plan, namely what types of content to write, how much to write, and where to place the content. Start by considering your expertise.

Rely on your expertise

To determine the release phase, priority, and goals of the project, rely on your expertise.

Expertise

Business understanding

Your business understanding can help you identify why the new functionality is being implemented in the product at all, and what the business is hoping might happen as a result of this new functionality being available. For example, this can guide you in deciding what example to use in a task, or whether a customer might be improving an existing manual workflow with one that is more automated now with the new feature launch.

Your knowledge of the release process and planned priorities can also inform what to write. For example:

In the case of a new dashboard widget that animates images in a repeating pattern, it is an early stage feature that is low priority, so you might only plan a basic task and reference content about how to add the widget to a dashboard and what settings can be configured.
For a new SQL function becoming generally available that unblocks a major customer use case, you might plan a basic reference and multiple detailed end-to-end examples describing the function for the specific use case.
For an interactive workflow to create and assign tickets to specific users, the workflow is the first of many steps to target a new customer base, so while it is an early stage feature it is also crucial for the business strategy. As a result, you plan to deliver a task, a concept, and end-to-end example about the feature. As a lower priority, but planned to align with a marketing launch, you can add tutorials and extra screenshots.

I recommend a similar strategy for building out minimum-viable documentation for a product, and it is still relevant after you have a more full set of documentation to consider.

Beyond considering your understanding of the business and its priorities, you can perform targeted research to inform your documentation plan with data.

Perform research

Inform your plan for the medium-sized documentation updates by performing targeted research to identify what and how much to write. Your main objective is to know as much as you can about the feature — why it’s being built, who it’s for, how it works — to write the best content that you can.

Research

User research
Product exploration
Competitive analysis

User research

To identify customer expectations to help identify what content to write, how much to write, and where to place it, perform user research.

Research

User research

Depending on your analytics capacity , user research might be formal interviews guided by a qualified user researcher, a short conversation with a specific customer that offered feedback about the product, a survey sent to existing customers, or notes taken by design or product leadership as part of the planning and development of the new feature.

Rely on the user research available to you, if any, to inform your documentation plan:

Identify if the target audience of an interactive workflow is likely to be familiar with the relevant tasks involved, or whether it is very new, and therefore they’d want a detailed walkthrough.
Identify preexisting mental models or customer expectations about dashboard widget capabilities that might be informed by the current dashboard widgets, or widgets available from popular competitors that your customers also use.
Identify what key use cases and problems the new feature might resolve for customers.
Define learning outcomes for your content informed by key use cases, such as “Can successfully add a spiral image to the dashboard widget”.

You can use the audience understanding that you build through user research to define details of a task or conceptual introduction to provide for a given feature.

For example, with a new dashboard widget that animates images in a repeating pattern, determine if there are specific customer use cases that led to the development of this widget. Perhaps a hypnosis service company has agreed to purchase your product if this widget is built, so it only has one key use case, but it is a critical one. Additional research and discussions with the product management and design teams reveal that they expect the widget to also be useful to customers nostalgic for Geocities website design and targeting the retro web market.

In addition to user research, you must explore the product itself (if possible).

Product research

Product research is a data gathering exercise , and it’s one that we’re intimately familiar with as technical writers. No matter what your analytics capacity is , you can explore the product and your documentation as a data collection exercise to plan the shape of your project work.

When you perform product research, you can experiment directly with the product, if you have access (and it’s complete enough to do so), you can explore the code directly, or use AI agents as a data gathering tool to either answer specific questions about the product or identify relevant sections of the code to read and review yourself.

Using AI tools for research can risk introducing inaccurate content and assumptions, so if you use AI-based tools for code-answerable questions, prompt the tool to cite specific files (and check its work), include reminders about feature flags or other gates that might affect feature visibility, and verify any claims about the user interface with independent in-product explorations if possible. Consider standardizing guidance in a shared skill.

When you perform product research, especially for a medium-sized project for net-new functionality, collect as much data as you can about the new feature, especially how it works:

What is changing?
What is being built?
How complex is the feature?
What is the entry point?
Is the feature being added to an existing part of the product, or a new location?
How easy is the feature to use?
Are there missing steps or confusing workflows?

The value of a strategic product advocate

Identifying issues like confusing product design or opportunities for alignment with preexisting product features in the planning stage is valuable so that you can give feedback about the design or implementation. As a customer advocate, you can make your strategic value as a technical writer clear.

Use this research to establish the scope of the feature that you plan to document. For a complex feature like a new authentication method for the product or anything that involves an integration with an Amazon product, writing documentation can take more effort. A simple feature, such as the tenth type of dashboard widget that has been added to the product, can be more straightforward to document.

Defining the scope of the project is an iterative and interrogative process. Don’t take the initial scope that you discover as the planned forever scope. As feature development progresses, the scope of the feature can change. What you initially discovered at the start of your research might be an incomplete scope.

These planning and research tasks share some similarities with those that you’d perform for a large project, but they’re different for the feature. In a medium-sized project, where you’re documenting something like a new dashboard widget, there might be more design planning and definition of the future state of the content. The future state planning is present in a large project, and more elaborate due to size, but often more constrained by existing content or project scale. With a medium-sized project, you’re limited mostly by time and priority of the project—namely, what else you’re doing at the same time.

Competitive analysis

For a medium-sized documentation project, it sometimes makes sense to perform a small amount of competitive analysis to gather data to inform your documentation plan.

Research

Competitive analysis

By performing a review of competitor documentation for similar features on the market, you can determine and validate use cases guiding feature development. This kind of review can also help you identify if there are interesting or relevant ways that competitors document the same kind of functionality. Maybe you want to borrow something from their content design, or identify a pattern that you want to avoid.

A competitive review can also help you understand what terminology customers might be familiar with, identifying keywords that you might want to include when writing your content.

Keep a competitive analysis short and contained for a medium-sized project, limiting it to validation and discovery.

It might be tempting to use AI tools for a quick competitive analysis , but I’d caution against relying fully on AI output for this research task. AI tools retrieve content according to what you prompt, thus making organic discovery of content display tactics less likely. It’s a best practice to review the cited sources manually to validate relevance and accuracy, so while you might use AI tools to identify relevant pages in competitor documentation, review the content yourself for relevant patterns and insights.

After performing different kinds of research yourself, you can review other qualitative data sources available to you for some additional planning context.

Examine customer data

If you don’t have much opportunity to perform user research, or if you have a limited understanding of the business, you might want to review different qualitative data sources available to you.

Customer submissions

Feature requests
Customer success call transcripts
GitHub issues or PRs
Search terms
Product reviews
Community forum posts

Customer submissions can be a revealing data source for use cases, terminology, conceptual mental models, and expectations, all of which are valuable for informing your documentation plan.

With data sources like feature requests, product reviews, or call transcripts with internal folks and customers, you can gather information about how customers talk about the new interactive workflow being developed, or the exciting new dashboard widget.

For example, based on customer feature requests in support cases and GitHub issues, as well as customer success call transcripts, you recognize that customers are concerned about the performance of dashboard widgets, which is unclear for most widgets. As a result, you can add an item to your documentation plan to test and provide details about the performance of the new dashboard widget that animates a repeating image.

You might not be able to prioritize that type of content for an early-stage, low priority feature, but by using data to inform your plan, you’re going beyond the basics of “document the feature” and instead being more strategic, planning to deliver more practical and relevant content to users.

A risk with these data analysis phases, compiling business understanding and user research with customer-submitted data, is that you weight internal assumptions and plans for product positioning more heavily than your findings from user research and customer data. Just because your organization wants something to be true and wants your customers to understand something about the new feature based on the planned marketing and product positioning, doesn’t mean that they will. You need to understand the customer’s mental model and preexisting expectations to write effective content — and that can require more effortful research and data analysis.

Use data to define what to write and where to place it

After you use your expertise, research, and review of customer-submitted data to define the scope and identify customer expectations, use that information to plan what to write, how much to write, and where to place it.

For example, identify the type of content to write based on the feature:

An interactive workflow to create and assign tickets to specific users might require you to write a task, a use case, and requirements.
The new dashboard widget might require you to write a task for how to add the widget, a reference for the available widget settings, and an example use case for the widget.
A new SQL function might require you to write a reference topic about the arguments and output, as well as several examples of the function in use.

For the content to be findable, you must identify where people (and AI agents) might look for information about the functionality. For example:

Content about an interactive workflow to create and assign tickets to specific users might be placed by existing triage and assign workflows, or organized with other content targeting the same audience.
Content about a new type of widget that can be added to a dashboard might be placed by other widget docs and added to a parent reference list of available widgets.
A new SQL function might be placed with similar types of functions, or even organized with the name of a competitor’s SQL function as an alias in the navigation to improve findability.

To take a data-informed approach, you can use the following data sources to plan how to structure the content and where to place the content:

Expertise

Business understanding

Research

Product exploration
Documentation exploration
Competitive analysis

Customer submissions

Search terms
AI chatbot conversations
Feature requests
Community forum

System

Documentation metadata

Most of these data sources are qualitative. Because you’re writing something new, you want to identify existing patterns and mental models that might exist in the industry, but otherwise you’re largely relying on your expertise and understanding of the customer and business, plus research about the existing documentation content and new product feature when deciding where to place the new content.

You don’t need to use all of these sources to help inform your key content decisions. Use the approach that makes sense, given what you can do easily, quickly, or effectively based on your available tooling and analytics capacity.

For example, to guide the following key content decisions as you define your plan of what to write, you can use the following data sources and analysis questions:

Key content decisions	Questions to ask of the data	Data source
What should I name the topics? What keywords should I mention in the opening of the topics?	What terms do customers use today when looking for information about this? Are there unsuccessful searches for this concept already?	Search terms
How should I frame the topic? What keywords should I mention throughout the topic? What terms should I make sure to use consistently? What use cases should I use for examples? What mental models should I make more explicit?	What questions do people ask of an AI chatbot? What terms do they use when looking for information about the subject that you’re documenting? Are there questions with made up responses in the AI chatbot conversation data?	AI chatbot conversations
What use cases should I use for examples?	Have customers asked for this functionality? What reasoning do they provide?	Feature requests
What other features should I mention to disambiguate the jobs-to-be-done for the feature?	What use cases are customers trying to accomplish already that this feature could address?	Community forum
What information should I include in the introduction? What use cases should I use for examples?	Why is this feature being built? Why now? What gap in the market does this feature fill?	Business understanding
How should I structure my content? What terms should I use in my content to improve findability	What content do competitors have about similar functionality? What information do they provide?	Competitive analysis

Your documentation explorations can help you understand how and whether similar, but different, functionality is documented for your customers. That research can guide how much content design work you need to do in deciding how to document the new dashboard widget, as well as identifying what patterns your customers might expect from your dashboard documentation content.

To help identify where to place the new page in the navigation and information architecture, answer the following questions:

How is the company positioning the new feature or functionality? Is there similar or existing functionality that is positioned in the same category?
What sections exist in your information architecture today? What architectures are possible with your tooling?
Where do competitors place documentation about similar functionality, if it exists?

To help you assess page structure, answer the following questions:

Where do people searching for information about this functionality end up today? What terms do they use?
How long are the existing pages about a feature? Consider using page length as one indicator to help you decide whether to write a new page or update existing ones, in addition to content type.

At this point in your medium-sized documentation project, you have a detailed plan outlining customer expectations, project scope, and a sense for what content to write, how to structure it, and where to place it.

Your next step is to write the content.

Write a medium-sized documentation project

The writing process is where you focus on delivering on your data-informed plan — both writing the content itself and placing it in the designed structure. Planning and writing are intertwined, so as you continue using the product to draft your task, you continue reevaluating the scope and updating your plan for what to write and how.

For example, start drafting a task outlining how to add the widget to a dashboard, featuring an example that combines with widget with another widget type, as well as a reference topic outlining the formatting options available for the widget. As you build an example dashboard to support your content about the new widget type, you might discover a missing setting that won’t be added in time for the release, so you write a note about it, updating your plan and your content in tandem.

Before AI-enabled writing workflows were common, writing was the stage where using data to inform your process was the least relevant, but that’s no longer true. If you draft content with AI agents in the loop, you can use the data from your planning stage as part of the context that you provide when creating a draft. For example, provide the specific use cases that the widget was designed to support, the terms used by customers and internal folks when describing the functionality, as well as references to existing similar pages to align structure and style of the content with existing dashboard widget task and reference pages.

If you don’t use AI in your writing workflow, you can still use the data sources and metrics that you reviewed during the planning process to inform your writing process, especially to validate assumptions about the right approach to take during the writing process:

As you write examples, consider the customer use cases to assess the relevance of the examples.
As you write headers, titles, and introductions for pages, review the customer search terms and keywords used internally to describe the feature to evaluate the discoverability of the content and the consistency of your language.
As you structure the content, evaluate if you’re matching the format of existing similar content types, or intentionally deviating.

Decide whether to add screenshots

When you plan new content, you can use data to help you decide: Should you add screenshots to documentation?

Screenshots are most helpful when they are relevant to the readers and useful for the context of the documentation. How do you evaluate that? With data!

For example, when adding documentation about a new dashboard widget that animates images in a repeating pattern:

Use on audience research:

Is this widget concept totally new to them?
Do they know how dashboard widgets generally work?
Are they familiar with a different configuration UI for widgets?

Use product and documentation research:

Is the widget setup design confusing?
Are you providing a reference, task, or example topic?
What is the goal and learning objectives of the topic?
Do similar topics have screenshots?

Use your business understanding:

What resources exist to help you maintain the screenshots?
Can you automate the screenshots with the tools available to you?

In this case, you might recognize that a new dashboard widget that animates images in a repeating pattern is a new concept for many readers, so adding a screenshot of how the widget looks when fully configured to the task documentation to illustrate an example can help support the business value of the feature.

If you have a higher analytics capacity, you can use additional data sources to validate the decision to add one screenshot.

Review page quality feedback for dashboard widgets:

Are people requesting more screenshots?
Do topics with screenshots get more or less page quality feedback?

Review website analytics for dashboard widget pages:

Compare the page views and referrer traffic for a high quality dashboard widget topic with screenshots to one without screenshots.
Compare the click activity on the similar pages, do users click or hover over the screenshots on the page with screenshots?

With a data-informed understanding of the product, your audience, your business capabilities, and current user activity, you can make a sustainable decision about screenshots.

You’ll likely redo some product research as part of the writing and revision process, because the product has likely changed since you started planning and writing. For example, check the product to validate the accuracy of your documentation draft, ensuring that you reference accurate UI strings and navigation paths, as well as relevant option and parameter names for API workflows.

After you write, revise, and publish your medium-sized documentation project, it’s time to evaluate it.

Evaluate medium-sized documentation changes

Were the docs useful to your customers? Are they accessing the docs? Are they asking questions answered in them and are the docs helping to answer? These are key questions to consider when evaluating medium-sized documentation changes.

When you add content about a new feature, there isn’t a baseline to compare against, but you can still evaluate what you produced.

First identify if you hit the most basic goal of delivering self-service support that is clear, useful, and relevant to your users.

One good signal can be the volume of internal questions. Oftentimes, the first readers of documentation about a new feature are the sales and support folks that are trying to understand it on behalf of customers that they’re helping, or prospects that they’re pitching. If you don’t see that many questions, or if the vast majority of questions can be easily answered with a link to the documentation, that’s a good initial signal that your documentation is useful and relevant.

However, the internal customers are not your primary audience, so you also want to evaluate more customer-specific metrics.

Short-term evaluation

In the short term after the release, monitor support cases and chat volume, product bugs and GitHub issue reports, and search queries to identify missed opportunities in the content or validate successes. Medium-sized documentation changes, or any changes to support a new feature, are perhaps the only changes for which you can do some preliminary evaluation immediately after a release.

What you can evaluate, how effectively, easily, and coherently, depends on your analytics capacity. The higher your capacity, the easier it is to evaluate quantitative metrics like feature adoption, documentation page views, and support case volume together on a graph and evaluate correlations against baseline usage. If you have a lower analytics capacity, you might not have a quantitative baseline, or one that you can make much sense of to help you evaluate this medium-sized change. In that case, you can use a more qualitative approach to evaluating your success in the short term.

Correlate quantitative metrics

After a release, you’re likely to see different post-release correlations across those three quantitative metrics:

Product telemetry
Support case volume
Website analytics

For example, for a new dashboard widget that animates images in a repeating pattern, if a lot of customers were waiting for it and closely followed the release notes, you might see both feature usage and documentation page views for the new pages increase after the release, such as in charts A, B, or C.

If the release was not hotly anticipated or perhaps is for a feature that was already broadly available and just reaching general availability, you might see a pattern like chart D in a short term evaluation. Compared to the product usage metrics generally, feature usage didn’t increase — not much has changed compared to before the release.

For a feature with immediate adoption, you can evaluate the short term success of your documentation by assessing the trend of the documentation usage and support case volume for pages and cases related to the new feature.

For chart A, assuming a feature going from private preview to public preview, people are using both the newly public feature and its documentation, but still confused enough to contact support with roughly the same relative frequency as when the feature was available only to select customers privately.

For chart B, people are using the feature and its documentation, but a brief dropoff in feature usage could indicate a lack of success with the feature. The continued documentation and support usage likely indicate confusion with the feature that probably isn’t fully resolved by the documentation. If the dropoff was due to a seasonal holiday, you’d expect to see usage of all three metrics go down at the same time, which is why it’s important to review the metrics together.

For chart C, the usage is a bit noisier in the data, but it’s generally a good signal for product usage and documentation usage to go up while support cases either stay flat or go down, indicating that folks are using the product generally successfully with help from the documentation as needed, without needing to contact support for much help. You can also perform a spot review of support cases before the feature was widely available and after, to determine whether the higher volume was customers asking about the availability of the feature or asking for help with it.

I recommend looking at all three metrics together. If you look only at documentation page views (as a way to proxy usage of the documentation pages that you added) you might draw misguided conclusions about the documentation usage.

For example, if your new dashboard widget documentation has very few page views, you might think it’s not useful. But if feature usage for the new widget is also very low, that indicates that the low documentation usage is likely unrelated to your content.

Instead, something else could be happening. If the feature wasn’t announced, or doesn’t address a gap in the product felt by a large segment of the product audience, product adoption might be slow. Discoverability of the feature itself influences interactions with your documentation, especially in the short term.

If the only data you have access to is documentation page view data, you can reliably use it to evaluate awareness of the feature, but not usage or success.

Even if you don’t have only sparse and statistically insignificant quantitative data, it can still be useful for decision-making and informing your documentation practice. Bob Watson elaborates in his post: If it’s not statistically significant, is it useful?.

Consider qualitative sources

Beyond correlating three quantitative metrics together, you can assess more qualitative data sources to attempt to extract more intent-rich insights. For example:

Customer submissions

Support cases
Product bugs
GitHub issues and PRs
AI chatbot conversations
Search terms

You don’t need to review every data source in this list, and you don’t need to review all content from each source. Reviewing a sample of just one qualitative source, especially if you have a basic or growing analytics capacity, can give you some useful information.

Data source	What to look for in the short term
Support cases	Do you get cases confused about the new feature, or with questions not addressed by the documentation? Is there a pattern to the questions? Are the questions about the feature more complex than before, indicating that perhaps more basic questions have been addressed by your documentation?
GitHub issues and pull requests	Do people open issues asking about functionality supported by the feature? Are there problems with its functionality for some customer deployments?
AI chatbot conversations	Are relevant questions being answered more accurately? How are the evaluation scores for the new content?
Product bugs	Are there any bugs with the feature discovered by customers or QA after the feature launched? Do any of them necessitate documentation updates? (Like a limitation that was missing from the docs)
Search terms	Do search terms about the issue find results? Are there fewer unsuccessful searches? Do you have a higher % of clicks on the page that you updated? Is search volume higher?

GitHub issues, and especially pull requests from contributors, can be an indication of product usage even if you don’t collect or have access to product telemetry. However, there is a risk of relying on those as a signal without correlation from other channels of data. People file issues and contribute pull requests for functionality that they’re using and want to improve — but a lack of issues doesn’t mean a lack of usage for a product feature — it might be frequently used but satisfactory in its current state.

Another assumption to avoid when analyzing product telemetry data (either directly or implicitly through product bugs and GitHub issues) is assuming that current and past usage patterns are indicative of future usage patterns by a customer. A customer testing and planning to roll out the new functionality to their user base looks different in the usage patterns than a fully scaled production workflow.

To mitigate these risks, correlate any decisions or conclusions drawn from product telemetry alone with user research, either by talking directly to customers using the capability, or at least the internal stakeholders that work directly with those customers, such as Customer Success Managers or Professional Services teams.

In some cases, you might want to perform a longer-term evaluation of the content you added for your medium-sized project.

Longer-term evaluation

If your product customer base is largely enterprise-sized organizations with slower adoption curves, or you’re attempting to assess documentation that supported a relatively low-key unannounced feature, a longer-term evaluation is more useful than a short-term data check.

In these cases, you can use the time immediately after the release as a pseudo-baseline, and correlate product telemetry for the feature with documentation usage to evaluate any patterns .

Data source	What to look for in the longer term
Website analytics	The page views and click interactions with the page, such as clicks on code samples or clicks to highlight text, increase. User sessions and referrer traffic indicate that the journeys to get to relevant information (the pages you created or updated) are shorter.
Product telemetry	Evaluate documentation usage with product adoption data to determine if there is a correlation.

For example, for readers using search or other pages to access information in your documentation, their paths might go from messy and convoluted session activity patterns:

To a more streamlined and simple approach where they land directly on a relevant page:

Interpreting session length depends on topic type

A shorter session length makes sense only in context of the documentation content.

A short web browsing session on the documentation site could mean any of the following:

Someone got frustrated and left without finding the answer they sought.
Someone found what they needed quickly and left to return to their task.
Someone was mid-exploration and they got interrupted by a surprise software update restarting their computer.

When attempting to interpret user session activity, compare a specific session against a baseline session length for similar topic types and audiences.

Someone with a long documentation session, visiting conceptual pages, reference pages, and task pages in the same session might be frustrated trying to find a specific piece of information, or they might be learning and trying to understand the feature before they start using it.

For example, if the topics are all about configuring authentication and provisioning new users, the session might be more likely to represent a thorough reader when compared to a similar session journey through dashboard documentation — but it might not be! Quantitative data cannot encode intent, only activity and volume.

Evaluate different pages differently

If, due to your analytics capacity , you rely primarily on page view analytics to evaluate your medium-sized documentation changes, you must evaluate different page types differently. You cannot derive one single metric to define the success of every documentation page that you publish. Instead, you can evaluate specific pages.

If you don’t know what the goal of a specific page is, you cannot evaluate its success.

Page	Goal	Measurement
Documentation homepage	Attract customers from search results Direct customers to other pages	Track sessions started from organic traffic Track on-page click volume and targets
Task (How to do X, Create and manage Y)	Help customers accomplish a task Provide instructions	Track support case volume Track search term volume landing on this page
Landing page	Provide a conceptual overview of content Direct customers to other relevant pages	Track page views with this page as a referrer Track page rank in search results for generic search terms about the topic
API or code reference	Help customers write code	Track clicks to copy code examples Track successful endpoint requests performed on the documentation page.

There are some universal data signals for documentation pages, but they require additional time-consuming analysis to be reliable, such as:

Tracking MCP server interactions that return the page, which requires validating the accuracy of the response to validate usefulness.
Tracking AI chatbot conversations that reference the page as a source, which also requires validating the accuracy of the response to validate usefulness.
Tracking page quality feedback on the page, which is a low-volume self-selecting signal that isn’t representative of all users.

The goal of evaluating your changes over the short term and long term is to evaluate the success of your content decisions. Keep track of what worked for a specific project, note what you might want to repeat, and share it with your colleagues.

Use data to inform a medium-sized documentation project

When you use data to inform a medium-sized documentation project, you can more effectively plan the scope of the work and design the content required. As you write and review the content, you can make decisions, iterate on the plan, and add context to AI-powered tools. When you attempt to evaluate the success of your project, the data you gather and analyze can validate the usefulness of your approach.

Inform a large documentation project with data

Tue, 27 Oct 2026 10:30:00 +0000

If you’re about to start a large documentation project, you can inform each step of the process with data. Why? So that you can be more confident in the decisions that you make, and to base those decisions in what customers are actually doing and asking about.

In the context of this post, a large documentation project is something like reorganizing a large set of existing documentation, or writing a large set of documentation from scratch to support a new feature.

Maybe a longstanding bit of functionality is being deprecated and replaced with another set of functionality
Maybe a new feature launch means that you need to reorganize and reframe an existing set of content.
Maybe a large change to the user interface is launching, requiring some new documentation and a lot of updates to existing content.

No matter the specifics, a large documentation project is one that will take you a few weeks or months to complete. A large part of that time is likely to be spent in planning — identifying the scope, direction, and desired output of the project.

In this post, I’ll cover how to use data to inform each phase of the project:

Plan your large documentation project

When you plan a large documentation project, you primarily want to identify the scope and the direction of the project itself, answering questions like:

What documentation pages are affected?
What changes need to be made?
How much work will the changes require?

Answering these questions can be easier when you add data.

Expertise

Business understanding
Documentation understanding

Research

Product exploration
Competitive analysis

System

Documentation contents
Documentation metadata

Customer submissions

Search terms
Website analytics
Community forum questions
MCP server interactions
AI chatbot conversations
GitHub issues and PRs

Even if your organization has a relatively basic or growing analytics capacity, planning a large documentation project can be a good place to start practicing data-informed documentation if you have the time and support. While your organization might have some gaps in existing data sources or expertise, you can start to build capacity with targeted data gathering through a content audit or documentation metadata collection.

Consider what you already know

A perhaps under-considered data source when planning a documentation project is your own knowledge It might be nontraditional to consider subjective information (your own personal expertise) as a type of data (it has been pre-processed and transformed into knowledge, so it isn’t data in its raw form!), but your personal expertise is essential context, and a crucial component of any documentation planning exercise. .

Each documentation project exists because something in the organization created a need for it.

Your personal context and understanding of the organization is essential to help plan and scope the documentation project itself. Depending on your tenure at an organization, you might apply this context subconsciously, but consider making it an active, intentional part of your planning process.

Especially if you use AI agents to plan your work, an agent does not have access to your own expertise unless you provide it as part of the context.

The data source of your understanding of the business is also possible to access for anyone, no matter their analytics capacity. Recognize your place in the organization and review what you know.

Take an example project where new functionality is being released, and you need to reorganize and reframe a large, existing set of content. Your knowledge about the organization can contribute to planning a large documentation project by answering the following questions:

How will the new functionality be positioned in the market?
Does the new functionality target a different buyer than the existing product functionality? If yes, who are they?
Is the new functionality a central part of product market positioning, or is something else the focus?
What company strategic initiatives does the new functionality support?

You might end up doing some research , or building some partnerships with folks in marketing, to more fully collect answers to these questions.

Why are these questions relevant to consider when planning a large documentation project? The answers can help you address scoping questions like:

Does the documentation architecture need to be updated to reflect changes to product positioning in the market?
What use cases might you want to address directly with examples?
Will specific kinds of decision support or conceptual overviews help support sales enablement or product marketing collateral?

In addition to compiling data from your knowledge about the company and product positioning, perform some focused research.

Perform research

Research is another form of data collection and analysis . Identifying what you need to know in order to effectively plan the shape of your content involves research into product capabilities and context about the way the product development process functions.

For a large project especially, it’s valuable to review what industry leaders and competitors document for similar large projects.

Do product research

To effectively plan documentation about a product, you need to know about the product. As part of the research you perform , gather data about the product development pace, patterns, and plans:

Is this functionality going to stay the same for several years, or will it change again in a few weeks or months?
How much development is going on in this feature area, now and in the future?
What future projects are planned in this feature area?
Is this a generally available release, or an alpha, beta, or preview release?

As part of your job you might have enough relevant product expertise to start planning the project already, but you might also need to do dedicated research to collect relevant data.

Do focused research to answer product-specific questions like the following:

Is this feature or functionality similar to existing functionality in the product? If so, how is it similar? How is it different?
How discoverable is this functionality in the UI? Does it have a top-level menu or is it buried several layers deep in a menu, behind an unlabeled icon, or something else?
How many changes are being made to the user interface?
Are there new permissions requirements being added as part of this project?
How confusing is the UI to use? Will it stay that way?

Overall, to help you plan you want to know: What is changing? How much is it changing?

As part of your research, use the product yourself! If possible, access a development or staging version of the product functionality and use it as a user might to perform specific tasks.

If, due to the type of product that you’re developing (or other reasons), you can’t directly access the product to explore it yourself, you can still use other research methods to answer these questions:

Interview the developers and designers that contributed to the product changes.
Review pull request descriptions, screenshots, and work-tracking tickets from a system like Jira or Linear.
Read the code, possibly with assistance from an AI agent prompted to gather data, to help answer relevant questions like required permissions, scope of the changes to the user interface, and more.

You can then use the data that you collect to scope the large documentation project. For example, given a large scale deprecation, a new feature launch, or a redesign of the user interface, consider the following:

Does I need to add new standalone pages for the new feature launch, or will I mostly be updating existing pages?
How many existing pages might I need to update?
Do I need to add a conceptual overview about the redesigned user interface?
If the new feature is similar to an existing feature, do I need to add decision support guiding users about when to use one or the other?
Especially if the user experience is confusing or the new functionality is hard to find, how much detailed content is relevant for me to add?
How many screenshots do I want to add to support the redesigned user interface?
Is a tutorial relevant and useful for users, or would an end-to-end example be more clear?

For example, if you are documenting new functionality that is incredibly complex and buried several layers deep in a menu when you test it out, you can give feedback to the product engineering team that they might want to update it, but you might also plan your documentation deliverables differently, perhaps planning to add more screenshots to your task steps or more detailed wayfinding guidance in introductory pages.

Perform competitive analysis

For some large documentation projects, it can be useful to perform a competitive analysis or review of competitor documentation and websites. Gather data from the broader industry and how your documentation exists in that context :

What are your competitors doing with their documentation?
What content structures are in use elsewhere?
What terms are commonly in use to describe different features or functionality?
What types of content exist for this type of feature in similar products?

For example, given a large scale deprecation, or a redesign of the user interface, you might review the content of industry leaders for best practices to use as models for your own content:

What content types support a large scale deprecation or redesign?
How do other organizations communicate deprecation timelines?
What do you like or not like about the choices others are making in communicating the changes?

For a large documentation project supporting a new feature launch or a new product line, you might review competitors with similar features or products:

What content types exist?
What sections exist and how are they organized?
What terms do those organizations use to name the feature or product capabilities?
What use cases and examples are referenced as supporting content?
What do you like or not like about the architectural decisions?
How do the architecture choices align with (or not) the current state of your organization’s documentation?

You can use the competitive review and analysis to plan the shape of the work that you want to produce for your own large-scale documentation project:

What does the ideal end state of this content look like?
Is aligning terminology going to be part of the project?
What content types need to be included for this type of documentation?

Documentation-specific data

Data about your organization’s existing documentation is crucial to inform scoping questions for the planning process.

Documentation-specific data is the broadest category of data, and also accessible in different ways to organizations with different analytics capacities . If you have a lower analytics capacity, you might only be able to perform basic tasks from this section, possibly a content audit or some small-scale tooling assessments. If you have a defined or mature analytics capacity, you can likely take advantage of most of the guidance in this section.

To help you plan your large-scale project, take advantage of the following documentation-specific data sources:

Assess the current state of your documentation team, processes, and tooling.
Collect data about the current state of your documentation with a content audit.
Review metrics and metadata to inform and prioritize the changes in your plan.

Assess your documentation understanding

To help plan what resources are available to you as part of the large project, collect answers to the following questions about your documentation team:

How many people are working on related projects?
Who might have additional bandwidth available?
Who has different helpful skills that might help make the project more successful, like a background in instructional design, visual design, or video production?
Is translation or localization of content a consideration?

To help plan what content shapes, structures, and tactics are available for your large documentation project, collect answers to the following questions about your tooling:

Is content reuse supported?
Are there platform limitations for different types of information hierarchies?
Can you programmatically update screenshots or text across multiple pages of content at once?
Can you use AI tooling with the content management system?
Do you use version control systems to track changes over time?

The answers to those questions can help you plan the project:

Identify the need for a release branch strategy in git to streamline collaboration of multiple writers on related content.
Determine the amount of work necessary to complete and whether additional writers or other collaborators are necessary to finish the project during the available time.
Recognize tasks that might take more or less time depending on the limitations of the content management system.
Identify limitations in tooling that you need to plan around in advance, such as navigation structure limitations.
Plan where to draft the long-lived content for the large-scale project.
Account for expected time to complete translations to make localized content available.

With an understanding of the current state of your documentation site, tooling, and content, you can plan the scope of the project more accurately, considering issues like:

What content structures and designs are possible? How can I structure the end result?
Will this project affect the translated docs?
If yes, do I need to finish my documentation sooner so that it can be translated in time for the release?
Where do I place the content in the table of contents for the site?
How many other writers or teams do I need to coordinate this project with?

Perform a content audit

A content audit is essentially a targeted data collection exercise, reviewing the current state of the documentation . Often, a content audit is performed as a quality evaluation of an entire set of content. However, for a content audit to support planning a large documentation project, you can scope it to just the relevant sections of the site.

For example, if you are planning documentation to support a product deprecation, you need to perform a content audit of all the pages that mention that product (which is hopefully not every page in your company’s documentation).

A content audit is important data collection. With a good understanding of the current state of the relevant parts of your documentation, you can more accurately plan the scope of the changes you need to make as part of your project.

As part of the audit, answer questions like the following for each page:

How accurate is the content?
How complex is the page structure?
Who is the audience?
What is the goal or learning objective of the content?
Is the content up to current style guidelines?

Depending on how much time you have available to perform an audit, you can provide in-depth answers or perform surface-level evaluations.

For the example of a deprecated feature, a content audit might help you recognize the following about the affected documentation pages:

Most pages are accurate, but some are inaccurate and will need additional updates as part of the deprecation work.
Some pages are structured with a lot of complex tasks, so those pages will need to be restructured and rewritten to make it easier to communicate the deprecation, for example by separating the tasks specific to the deprecated functionality to a dedicated section, instead of being integrated with the page sections.

The content audit produces data you can use to consider the scope and shape of the plan for your large documentation project.

Review documentation metadata

Documentation metadata can help you scope the project as well. You can gather and analyze documentation metadata like the following :

Metadata	Example value	Source	Level
Page count	`812`	Content management system	Site
Section title	`Administer`	CMS	Page
Last updated date	`2026-08-23T16:04:34Z`	Version control or CMS	Page
Header count	`8`	Computed	Page
Header depth	`3`	Computed	Page, per header
Link count	`4`	Computed	Page
Unique link count	`3`	Computed	Page
Image count	`1`	Computed	Page
Image has alt text	`True`	Computed	Page, per image
Word count	`1273`	Computed	Page
Code sample count	`4`	Computed	Page
Code language	`Python`	Computed	Page, per code sample
Code test results	`Pass`	Test suite	Page, per code sample

You can gather this data through a manual content audit, through an API, or compute it yourself based on the contents of your documentation pages with a script. Some metadata might be available in the files containing your content, such as in page frontmatter. Other metadata might be site-specific and configure the navigation or table of contents of your site. Other metadata might be available from a version control system, like git. Other metadata might need to be computed.

The approach you might take to collect documentation metadata depends on the tooling that you use and your analytics capacity. The more API-accessible your tooling and the greater your analytics capacity, the more likely you can perform a complex, scripted collection & analysis of this metadata. With a more basic analytics capacity, you’re likely counting screenshots by hand and inputting data into a spreadsheet Speaking from experience… .

Version-control-specific metadata

If you use version control, you might also have access to git commit metadata, for example the details of a pull request squash-merged to your release branch in GitHub.

Version control systems can also usually show you a line-by-line difference of when a page was last updated, making it easier to see precisely when an image was last updated, for example. Alternatively, if you host your own images, you can use the last updated date for the image file itself as a signal.

The data that you collect might be well-structured like the following, or it might be a chaotic spreadsheet. Either way, it can be useful metadata for planning a large-scale documentation project.

Example JSON-formatted documentation metadata

{  
  "section": {
    "title": "string",
    "pages": 
        {"pageId": {  
            "title": "string",  
            "url": "string",  
            "last_updated": "datetime",  
            "contents": {
                "body":"string",
                "word_count":"number",
                "headers": {
                    "total_count": "number",
                    "header": {
                        "depth": "number",
                        "text": "string"
                    }
                },
                "links": {
                    "total_count": "number",
                    "link": {
                        "path":"string",
                        "external":"boolean"
                    }
                },
                "code_samples": {
                    "total_count":"number",
                    "code": {
                        "sample":"string",
                        "language":"string",
                        "test_results": "boolean"
                    }
                },
                "images": {
                    "total_count": "number",
                    "image": {
                        "path":"string",
                        "alt":"string",
                        "last_updated_date":"datetime"
                    }
                }
            }
        },
        "pageId": {  
            "title": "string",  
            "url": "string",  
            "last_updated": "datetime",  
            "contents": {...}
        }
      }
   }
}

With whatever metadata you have available, you can answer at least some questions like the following:

Question	Metadata
How many pages are there?	`page_count`
How many pages in each section?	`section_title`
When was each page last updated?	`last_updated_date`
How many headers exist?	`header_count`
What depth are the headers?	`header_num`
How many links exist?	`link_count`
To which pages do the links point?	`link_count_distinct`
How many screenshots and images exist?	`image_count`
Do the screenshots and images have alt text?	`image_alt_text`
Are the screenshots and images up-to-date?	`last_updated_date`
Are there code examples on the page?	`page_code_fence_count`
Does the code run?	`code_test_results`

The answers to these questions, in combination with other, more targeted data collection like the content audit or competitive analysis results, help scope and outline the shape of your large project.

For example, given a large scale deprecation, a new feature launch, or a redesign of the user interface, consider the following to scope your project:

How many pages have images that might need to be updated or removed entirely?
How many pages include content about the deprecated functionality? How many include UI-based steps that will be changing?
How many links do you need to update?
When might you want to update existing new pages or add new pages because the current ones are too long or complex?
How many pages will need general accuracy updates or rewrites when they’re updated for this project?

The metadata by itself doesn’t offer enough context to help scope the project, but when combined with your documentation research and product research, you can plan the shape of the large-scale project.

Examine customer data

When planning a large scale documentation project, customer data can be a valuable source when making more tactical plans about what content to write and how to write it. To plan the specifics of what you will write, you can gather data sources like the following:

Search terms
Website traffic, referrer data, and other analytics
Community forum questions
MCP server interactions
AI chatbot conversations
GitHub issues and PRs

Depending on your analytics capacity, this data might be available to you in a boilerplate analytics tool like Google Analytics, a dashboard in your content management, a dedicated data analytics tool like Snowflake, Sigma, Tableau, or others. You might also not have access to any of this data.

You can use the data sources available to you to answer specific questions like:

Do the search terms used by customers match the terms used in the documentation?
Which keywords commonly lead people to the documentation that you’re updating?
Do people ask about this functionality when interacting on the community forum or with an AI chatbot?
What use cases and challenges do people talk about on the community forum or with an AI chatbot in context of this functionality?
Which websites and content types refer people to the documentation?
How many unique viewers and views does each page have?
Which agent-derived keywords through MCP server interactions generate traffic to the pages in your documentation that you plan to update?

The answers to those questions then inform your plan for the large-scale documentation project:

Inform what terms to include in the topics that you plan to write, such as guiding title or header terminology choices.
Guide which content to move or not move, based on referral traffic or organic search traffic that might make a URL slug change expensive.
Identify pages with strong, existing information scent (popular search destinations) that might need special handling to make sure you don’t affect existing content findability.
Prioritize specific use cases or examples to implement in the content.

Each data trail is a trace left by a user that you can use to derive vague usage patterns that you can model and shape as you redesign the content experience with your own updates and changes as part of the large documentation project.

Plan the shape of the work

As you collect and analyze relevant data sources, you can plan the shape of the large documentation project and the end state that you are working toward.

With the competitive analysis, you identified different content types to write and navigation structures to implement.
As part of the content audit and documentation metadata review, and assessing the documentation-specific data you identified the scope of the pages affected, constraints and opportunities from the team and tooling, and types of changes that need to be made.
You reviewed customer data and activity to guide content decisions.

When you know the shape of the work, you can identify what success looks like so that you can evaluate it later. For example, for a deprecation, success might be an easy reference page to send in an email about the deprecation; for a UI redesign, success might be fewer questions than expected about the UI changes in support chats. Take a moment to define a measure of success as part of your plan.

Write a large documentation project (informed by data)

Writing is the stage where using data to inform your process is perhaps the least relevant. You’ve already made a data-informed plan — this step is about delivery. Enter that fugue state and get to writing.

For example, given a large scale deprecation, a new feature launch, or a redesign of the user interface, this is when you start making the changes:

Surgical updates to pages that link to others to mention deprecated functionality
Sweeping changes to task steps to change the navigation instructions
Scaffolds of a new section of the navigation and accompanying new pages to match
Meticulous updates to screenshots

As you start drafting and reviewing, you might discover new data that requires additional planning — some previously unidentified scope of the project, or new user requirements that involve updating additional pages and more product research. Writing and planning can be an iterative process, especially for a large project that runs over many weeks.

But keep the data that you collected as part of the planning process close at hand. There are still some opportunities to use that data to inform your writing tasks in a large documentation project.

Guide AI-driven prompts

If you use AI tools as part of your drafting process, whether to draft content, make larger mechanical changes like link slug updates, or create new files that you then author, the data sources that you used as part of planning can be valuable context to guide AI-driven prompts.

Use search terms as context for header, introduction, and page title drafting.
Use existing, up-to-date pages as reference models for new file creation.
Use curated community forum questions, AI chatbot conversations, and GitHub issues as fodder to create a troubleshooting guide.
Include product-positioning context and audience details to output relevant code examples based on actual customer use cases.

If your data sources are available with an MCP server, it’s even easier to add relevant context to your drafting, for example with a prompt like:

Use the MCP server to look up support chats that mention an error or bug from the last 60 days and identify the related features. For issues related to the to-be-deprecated functionality, propose troubleshooting sections in order of frequency based on the findings.

Or for automating content updates related to the UI redesign:

Follow the pattern established on this reference page /path/to/human-authored-draft.md to update the other pages in the same section. For guidance about terminology, use the MCP server to look up the terminology list.

If you don’t use AI tools as part of drafting, but you do use AI tools as part of the content review process, you can follow a similar prompt for the deprecated functionality example:

The following pages are new and address the same use case as the to-be-deprecated functionality on these other 3 pages. Review the headers and introductory sentences for each new section for consistency with the search terms that lead users to click on the 3 preexisting pages.

Data is context for AI tools, and the output can be improved with relevant, curated context for your task.

Whenever you plan to use AI with documentation drafting, make sure that you have an established AI strategy for documentation to help ensure responsible, relevant, and accurate usage.

Inform tactical choices

You can also use data to inform tactical choices throughout the writing process. As you draft content, you might encounter judgment calls like how to phrase a heading or whether to put content on a new page or leave it where it is. In those cases, review the data you collected during the planning process to validate your assumptions about the right approach to take.

Decide what to name a feature

For a specific task like determining what to name a feature, it’s straightforward to use data to inform your decision.

Start by gathering information about existing terms. Which terms are in use, how are they used, and how common are they in the industry? Review sources like:

Competitor documentation, blogs, and websites
Google ngram and Google Trends
Documentation site search terms
Google Search keywords leading to the documentation and company websites
AI agent conversations

After reviewing the data, you can make a decision (or a recommendation, if you don’t have decision-making power).

If you get pushback, you can share the data that you used as backup for your decision. A discussion is also another opportunity to gather data — maybe you’re missing some context about product positioning, business direction, or lack access to a data source that offers a compelling counterargument.

Data is not an impenetrable backstop. Be curious, not defensive, and build connections to gather information and context that can help you do your job better — and help the user succeed with your product.

Other tactical choices that you can inform with data are the readiness to deploy your changes. As part of your plan, you likely prioritized high-value pages to update based on usage. As you continue to write, you can evaluate the progress against your plan and your knowledge of the product development and release process to identify when your work is “done enough” to release.

Advise review decisions

As you finish the drafting process, data also informs how and when you might send out your draft content for review.

Based on your understanding of the organization and product priorities , you might know that content related to this project is high priority and iterative review cycles with smaller drafts will be easier for folks to review (show that you are doing something), rather than one final dedicated review process that takes multiple weeks with fully drafted content.

Data about your output can also help here — consider how many pages you updated, which ones you’re more or less confident about based on your product research , and the different projects taking time from your technical reviewers — to prioritize what content to send for review when.

You can also use data to assist with the actual review process itself, using a style linter like Vale to collect data about the adherence of your content updates with the style guide rules of your documentation site and validate consistency in terminology.

Evaluate large documentation changes

After you publish your large-scale documentation changes, you’ll certainly let out a big sigh of relief. While documentation updates are never completely done, after you release a large set of changes you can evaluate the results.

As part of your plan, you might have defined what success looks like. With a complex and large project, it can be difficult to measure the effectiveness of the changes if you don’t know what the goals of your content, and the project, are. For example:

For the UI redesign, measure support chat volume.
For the deprecated functionality, measure referrer traffic.
For the new feature, measure user engagement.
For the reorganized content, measure discoverability improvements

For many changes to existing content, you want to wait several months and compare user activity against a baseline set from before the change, either from the same time of year or not, depending on how much your content and metrics have changed.

Part of the reason for this is that it takes user behavior several months to adjust to change. Especially for evaluating changes that are not tied to a specific date or customer communication, a longer, baseline-driven analysis is more valuable for evaluation than checking analytics a week after the release and using that as your only evaluation metric.

For guidance on how to determine a baseline, refer to the informative posts from Bob Watson, How you can make sense of your site analytics, and You’ve tamed your analytics! Now what?.

The less analytics capacity you have, the less likely you can rely on baseline data (like comparing before and after changes in data points), but you can still perform pointed data collection to evaluate the effectiveness of your content.

If you don’t see any difference in any of your success measures even after a few months, you could wait a little longer, turn to qualitative data sources like UX research to try to get more feedback, or consider whether your success measures are flawed and consider other metrics to review.

Evaluate support chat volume

For the UI redesign, maybe you hope that less than 5% of support cases are navigation questions related to confusion with the new UI.

Customer submissions:

Fewer support cases about the navigation questions, and those that do happen include links to the documentation as reference to help assist the customer.
More clicked results in the search terms leading to the page about the changed navigation and fewer unsuccessful searches.

One risk of this success metric is that the support team didn’t know about your documentation, and perhaps your documentation introducing folks to the redesigned UI (compared with the previous UI) was not included with the communications sent to customers. So even if support chats are down, your page views might be low and very few support chats actually link to your documentation.

However, you can mitigate this risk. This sort of pattern in the data can be a reminder that often, publishing the documentation isn’t enough. You also need to evangelize the existence of the content internally, and perhaps externally as well. Build partnerships with colleagues in support, customer communications, sales enablement, customer success, and marketing. Help ensure that blog posts about the changes, email communications about the changes, and internal messages about the changes all include reference to your content as the authoritative customer resource.

Another risk about this success metric is that it assumes that one canonical page is responsible for customer success with the new redesign (one explaining the before and after changes), and doesn’t consider the effect of correct or incorrect task steps in specific documentation pages.

Evaluate referrer traffic and user behavior

For the deprecated functionality, maybe your proxy metric for success is a lot of referrer traffic to the main migration guide for the change, indicating that it is shared as a primary resource.

Website analytics:

Track referrer data to see if there is organic and email referral traffic to the main migration page.
Track user session activity to identify if users are accessing the migration page from content about the deprecated functionality.

However, there are risks with relying on this sort of data as a success metric.

Quantitative data is limited

Quantitative data isn’t universally or consistently collected for all users across all browsers. For example, referrer data is only tracked if someone’s browser settings, and the website they left, allow that to happen.

As a result, you can’t assume that each user_id that you’re tracking in quantitative data like website analytics or search terms is a specific individual, or just a different browser on the same device One person might be using different devices or browser profiles to access information on your site as part of the same information-gathering session, but might look like separate individuals in your quantitative data.

Associating unique user IDs with specific users is often a fruitless endeavor (unless your documentation site requires authentication, in which case you trade findability for precise user attribution).

Quantitative data is useful, but limited. Treat quantitative data as a supplemental source, rather than a primary one, due to these limitations.

An alternative metric that might be useful if you have partnered with the folks sending customer communications is evaluating patterns in website traffic to the page and correlating those patterns with email send dates, open rates, and click-through rates (if that is data you have access to).

Evaluate user interaction with use cases

For the new feature, maybe you expect to get a lot of clicks on the example use cases and users copying the code samples included, indicating engagement with the content and likely relevance of the code samples.

Website analytics:

Click activity on code samples
Page views on example use case pages
Referrer traffic to example use case pages

Using website analytics as the sole data source to measure success is risky. You can’t derive user intent from user activity directly. Click activity on code samples means that the code has been copied, but not necessarily that it was useful or even ran without errors.

An alternative, albeit harder to measure, metric is organic shares of the content on social media, community forum, support chats, or internal Slack messages. If someone asks for help with the new functionality, do users (not on the documentation team) share your content? Using organic sharing of your content as a success metric again relies on successful awareness-raising of your content.

If you think you did a good job raising awareness of the content but organic shares and user engagement with the content are both sluggish, correlate the documentation usage with the product usage to contextualize it. Is usage of the new product or functionality similarly sluggish, or is product usage high while documentation visits are low, even for a complex feature that you might expect to require assistance from the documentation?

If none of the possible success metrics indicate success, you might need to go back to the planning phase and consider if the use cases and examples that you chose to represent were accurate with the product positioning and customer challenges.

Redo some data collection by reading blog posts announcing the new functionality and customer questions posted on the support chats, community forum, posed to AI chatbots, and shared on social media, then evaluate if you want to make updates to the documentation content to be more relevant and useful.

Evaluate customer journey improvements

For example, if your project was to reorganize the documentation to support specific customer journeys, such as configuring a data connection, you might look for signs that those specific customers are succeeding on those journeys more quickly than before the documentation change:

Website analytics:

Track user session click paths to and through pages related to configuring a data connection to identify if they are shorter and more focused (fewer clicks on unrelated pages in the same session).
Track page views to see if they go up for pages about configuring a data connection.
Track referrals to see if there are more from organic search traffic.
Track search terms to see if there are more clicked results leading users to this page and fewer unsuccessful searches for related terms.

Product telemetry:

Measure the time it takes before a new customer configures a data connection after creating a new account.

User research:

Test users in the target customer group and determine if they can more quickly configure a data connection

Some risks of evaluating these data points to measure success is that your documentation doesn’t exist in a vacuum. Other, related customer enablement or product design efforts can produce similar trends in the product analytics and user behavior data that your content might not have influenced.

That doesn’t mean that your content wasn’t useful, but limits the success that can be attributed to your content if there are other initiatives complementing your work, like dedicated training courses, blog posts, or UI tweaks to improve the process. Those content initiatives are not a bad thing — the goal of documentation is to support the user’s success with their goals (through your product), and ultimately any improvement is a good thing.

For guidance performing qualitative user research on your documentation changes, see Bob Watson’s posts Tips for conducting documentation research on the cheap and Documentation research requires more curiosity than money.

Quantitative data encodes activity not intent

You can’t use quantitative data alone to understand why someone did something, only to track what they did. You know that someone searched for “list widgets” but you have no context about why they looked for that term. You know that your page about dashboard layouts got 10,000 views last week when it usually only gets 4,000, but you can’t know from the site analytics definitively why that is the case.

Because user activity data can only tell you what, supplement it with qualitative data like user interviews, well-designed surveys, and text-rich interaction sources like AI chatbot conversations or community forum data. However, what qualitative data gains in clarity of user intent, it obscures that through difficulty of analysis.

AI tools can assist with qualitative analysis, but there are tradeoffs. Data analyst Randy Au captures them in his posts:

Use data to inform a large documentation project

When you use data to inform a large documentation project, you can more effectively plan the amount, complexity, and type of work associated. As you write and review the content, you can inform tactical choices, inform revisions, and add context to AI-powered tools. When you attempt to evaluate the success of your project, the data you gather and analyze can validate the usefulness of your approach.