thread on This is important

Data maintenance matters

Thu, 29 May 2025 22:25:39 +0000

Data requires maintenance to stay accurate. For machine-generated data, monitoring the structure to ensure you’re collecting valid records is probably enough maintenance. For other types of data, however, it’s a bit more involved. Geospatial data, for example.

Molly Taft, writing for Wired in Trump Cuts Are Killing a Tiny Office That Keeps Measurements of the World Accurate shines a light on the National Geodetic Survey (NGS), which is responsible for maintaining spatial data for the United States of America:

NGS is currently responsible for maintaining and updating what’s known as the National Spatial Reference System, a consistent system of physical coordinates used across federal and local governments, the private sector, and academia. This includes not only latitude and longitude, but also measurements of depth and height as well as calculations around Earth’s gravitational field—crucial mathematics that inform much of the basic infrastructure around us, from constructing bridges to mapping out water and electric lines.

The Earth isn’t static — it’s constantly moving (remember, our land masses are basically just floating on magma!) and our data need to adapt to that movement. Modern engineering demands more precise data, and thus more precise measurements.

If you want to build a high speed train network across miles of desert, you need to know exactly where to lay that train track and build the bridges — not just in each acre but across the entire length of the rail line.

Measurements don’t just appear, they have to be made, and then accurately modeled to form an accurate spatial representation that is usable across the country (and continent). Who makes these precise measurements and builds the models? Geodesists. Unfortunately, the United States hasn’t spent much on their geodesy prowess:

In 2022, a group of leading geodesic experts authored a paper on what they dubbed the US’s “geodesy crisis,” detailing how other world powers have invested in training geodesists over the past three decades while the US has wound down funding and training. China has invested particularly heavily in creating more geodesists: the country graduates between 9,000 and 12,500 geodesy students per year, many of whom are then employed by the government.

Without these experts, what you’re left with instead is high demand without expertise:

Many industries now “want that high accuracy positioning” that comes with advanced geospatial technology, [Dave] Doyle [a former chief geodetic surveyor at NGS] says, “yet they don’t understand the basics of the science. Now you’ve got all these people punching buttons and getting numbers, and only a tiny percentage of them really understand what the numbers mean, and how one set of numbers relates to another.”

Whenever you encounter something built with data, remember that someone was there in the background, crafting and curating that data to ensure its accuracy, relevance, and utility. When we ignore, devalue, and defund these data workers, the quality of our data is diminished, and so too is the quality of anything that we build with that data.

Via The Map Room blog, in their post NOAA Cuts Threaten Spatial Reference System Update.

Where search keyword data comes from

Tue, 27 May 2025 23:27:51 +0000

As someone who does a lot of armchair-yet-informed speculation about how various things get defined in music metadata, this deep dive into how keywords and entity relationships have been formed by Google over the years was fascinating to me.

Deborah Carver, based on her years of expertise in the content industry, explores where SEO keyword data comes from in CT No.112: Demystify your algorithms: The origin of keywords:

If you receive a list of target keywords or a keyword research spreadsheet, do you know how they were sourced? Where exactly does your keyword data come from? Your answer is likely “from my browser extension,” “from our audience editorial team,” “from my keyword research tool” or “from our digital marketing agency.”

But of course, those keywords had to come from somewhere else, first! She details the different tools that Google has offered over the years, and defines the different datasets that contribute to the keywords identified by Google—including those that SEO tools add on.

As she points out, it’s advantageous to Google because these investments in keywords help drive the success of its primary revenue driver:

Giving away raw data and free training on its software has proved wildly profitable for Google. It’s a juggernaut business strategy in the open source tradition. The more people who learn to use keyword data, the more people understand search engine marketing, the more ads Google can sell.

Perhaps the most fascinating aspect for me, as a fellow content professional, is her clarification about the different ways that people use search keywords compared with how they use natural language:

Keyword research also helps stakeholders understand that people don’t use marketing speak in their search terms. People also don’t search with the same words they use to talk or chat or tweet. Imagine if, when suggesting dinner, your partner said, “pizza near me now.” No one talks that way.

Many people search that way—17,800 times each month across the globe, on average. Search language is its own behavior, one that’s ever-evolving.

I don’t think that “search language” as a behavior will change anytime soon, even with the rise of LLMs. Google and other search engines are adding LLMs to the results and processing of search queries in a purported effort to make searching “easier” and “more natural”, but so far, it seems to me that adding LLM output to search results is diluting the quality of results as people realize their trained habits for looking for information no longer work the same way.

And too, directly using an LLM to produce information requires specific language techniques to get better results. The same way search language has evolved over time, prompt engineering is emerging as its own type of language. Whether prompt engineering is a skill that, like search language, everyday people will need to pick up remains to be seen.

The technology might be changing, but at the end of the day, we’re all just learning new ways to speak to machines.

On the objectivity of data-driven decisions

Mon, 26 May 2025 21:30:43 +0000

Back in 2021, writing about the seemingly-objective Supreme Court Justices making politicized decisions, Benn Stancil interrogates the idea that data-driven decisions are equivalent to objective decisions in Tilt and tilted:

Making arguments from data, like interpreting the law through legal deliberation, isn’t inherently problematic. Quite the opposite, in fact—to the extent that it’s possible, data should be foundational. But, also like the law, it’s a foundation built on less level ground than we often admit.

Though we think of data as irrefutable ground truth, it is, in fact, also almost “entirely self-referential and made up.” Often, data—and its computational cousin, the metric—isn’t an abstract representation of an innate natural quality we’re attempting to quantify; it’s an accounting identity.

Unfortunately, the common perspective of using data to make decisions (or do anything) is that data is the truest form of objectivity:

To use data is to be level-headed. The surest sign of fairness is to support your claims with numbers; the surest sign of prejudice is to fail to do so. Data is both a sword and a shield: It is a weapon for prosecuting your point, and a defense for protecting yourself as reasonable and impartial.

But bias isn’t something that can be easily removed:

For those of us who work with data, the solution isn’t to make our data or analysis more objective. We can’t. Our raw material is too tilted, as are we.

It’s very common to see data as objective, but it’s important to remember that it isn’t any more objective than we are.

AI and a duty of care

Fri, 12 Jul 2024 23:09:09 +0000

A recent post from the American Alpine Club has me considering the potential duty of care that websites like AllTrails and Mountain Project owe to their readers.

In The Prescription—July 2024, the editor Pete Takeda describes two incidents, one where two people were stranded due to their inexperience climbing snow, and another where one person in a group of nine slipped and fell to her death.

Recent accidents in Grand Teton National Park

The American Alpine Club Prescription highlights a route on the Teewinot mountain in Grand Teton National Park that has claimed many lives. Introducing the accidents, Takeda points out:

While these accidents differ in immediate cause and final outcome, they share a common origin: the use of hiking-specific applications for selection, preparation, and route-finding, versus the use of climbing-specific resources.

In the conclusion and analysis, he goes further, pointing out the use of hiking-focused apps as a key root cause:

Use of Hiking Apps. This team relied on information taken from a popular hiking application (as previously discussed in this Prescription). This climb in particular is listed on several platforms as a hike. Interviews with survivors revealed they were under the impression that the East Face of Teewinot was a traditional hike. It is a fifth-class climb.

Based on this information, he researched the AllTrails description of the route:

On the app, the updated route description reads like a ChatGPT-derived synthesis of crowd-sourced user comments. What it lacks in human nuance, it compensates with a tinny stridency.

To me, that immediately makes me wonder — is it actually generated text?

Is it generated?

When you open the AllTrails page for Teewinot, the route is described as a mountaineering route, but the description sounds largely innocuous:

Proceed cautiously on this 5.9-mile out-and-back trail near Moose, Wyoming. Generally considered a highly challenging route, it should only be attempted by experienced adventurers. This trail is great for rock climbing, and it’s unlikely you’ll encounter many other people while exploring. The best times to visit this trail are July through September.

As Takeda also highlights, referring to this route as a trail in the summary is extremely misleading, and the text is almost certainly AI generated. How can I be so confident?

A press release published by AllTrails on June 13, 2024, titled AllTrails Reveals Major Update advertises Redesigned Trail Pages:

Redesigned Trail Pages

AllTrails overhauled its more than 400,000 Trail Pages to help trailgoers plan with confidence, making essential information more accessible and visually engaging. Key details like trail length, format, and estimated duration are more prominent, and the pages now include the ability to visually tour the trail in a dynamic Photo Gallery, view the route in 3D, assess trail conditions, and read an AI-powered summary of community reviews.

However, there’s nothing on the page to indicate that the summary is AI-generated, nor any details about what prompt is being used to shape the community reviews into a summary of the route.

A local route near me that I discovered on AllTrails contains similar language:

Enjoy this 59.5-mile out-and-back trail near San Francisco, California. Generally considered a challenging route. This trail is great for road biking, and it’s unlikely you’ll encounter many other people while exploring.

As someone who spends a lot of time crafting consistent content, this type of structured description is unsurprising. What is surprising is the use of the word trail for routes that a person wouldn’t describe that way.

Using the term trail to refer to a mountaineering route or a cycling route indicates to me that this text is almost certainly machine-generated, and that the instructions include guidance to use that term.

Data cleanliness and prioritization

The use of the word trail to describe these non-hiking adventure routes isn’t a matter of data cleanliness. The fact that these routes aren’t for hiking and are instead for climbing or cycling is known, but not clearly communicated.

It’s an issue with how the data is prioritized and communicated on the site.

If you filter for hiking trails in Grand Teton National Park, this route does not appear. However, if you filter by the activity “Rock Climbing” this route does show up. The data is there.

However, on a trail description page, there is no “activity” filter or label — only a loose set of barely noticeable tags or labels that describe the route:

If this were an issue of unclear or missing data, filtering by activity wouldn’t be possible at all. Instead, the data is there but isn’t clearly highlighted or communicated on the page.

Don’t try to be the tool for everything

A platform can be immensely valuable in a specific niche. AllTrails is fantastic for offering granular route details, community-sourced conditions, and other relevant information for hiking trails.

But AllTrails was built for hiking trails, and the machine generation tooling used to create summaries was clearly also built for hiking trails—despite the site now including a variety of routes for other activities.

The expertise and audience that built AllTrails into what it is today for hiking doesn’t automatically translate to other activities like rock climbing and cycling, and as such, isn’t appropriate to offer support for rock climbers (and probably cyclists too) the way that dedicated sites can.

Mountain Project, the go-to source for most online climbing information, is part of a conglomerate that has expanded over time to include mountain biking, hiking, trail running, and skiing. And it’s telling that the sites are separate — because the audiences and the risks are distinct.

Expanding to a new customer base requires building intentional content, and in the outdoors, access to that content often requires local community experts. Other sites that serve climbers and mountaineers, like SummitPost, or the climbing app Kaya, offer support for community experts and are mindful stewards of information.

It might not seem like a big deal to generate summaries for the different routes listed in AllTrails. It might even seem like a net benefit for consistency and clarity, making the outdoors more accessible and available.

But if those summaries are crafted by a poorly-instructed machine, rather than a person that understands the context and the risk involved in providing such a summary, you end up with a mountaineering route called a trail, and a 60 miles cycling route called a trail.

A product with “AI-powered summaries” that are actively misleading is a product that is neglecting its duty of care to the outdoor enthusiasts it aims to support.

What about GIFs instead of screenshots?

Sat, 06 Jan 2024 18:16:57 +0000

After I published Should you add screenshots to documentation?, I got some comments from folks who prefer GIFs to screenshots because GIFs can more clearly show how to use a complicated user interface.

I agree that GIFs are cool and useful, but they’re also MUCH harder to keep up-to-date than screenshots and have extra accessibility considerations if you decide to use them.

WCAG level A standard requires that:

For any moving, blinking or scrolling information that (1) starts automatically, (2) lasts more than five seconds, and (3) is presented in parallel with other content, there is a mechanism for the user to pause, stop, or hide it

If you use a GIF, you need to consider those accessibility constraints, as well as the standard consideration when providing visual task steps: Can someone complete the task successfully if the screenshot, GIF, or video is unavailable?

So if you decide to use GIFs instead of screenshots, keep in mind these extra considerations:

Can someone pause, stop, or hide the GIF?
Does the GIF stop moving in less than 5 seconds?
Did I write effective alt text for the GIF?
Is the source file for the GIF stored somewhere accessible to the rest of my team?
Do we have clear and consistent style guidelines for creating GIFs?

Recommended blogs

Sat, 30 Dec 2023 12:03:59 +0000

Inspired by Matt Webb’s post 10 blogs for your newsreader, I decided to share my own favorite blogs to follow.

I also use RSS feeds to consume most of the content I encounter, and always get sad when a site doesn’t provide an RSS feed to let me follow new posts. I end up following many newsletters through RSS as well, because I’m better about keeping up with my feeds than I am my email inbox—or at least Feedly, my feed reader of choice since the demise of Google Reader—removes unread posts that are older than 30 days by default.

These are the blogs where I read, or at least intend to read, almost every post:

Elezea by Rian van der Merwe. Rian does a great job sharing his thoughts about product management and also sharing valuable and fascinating articles having to do with product, organizations, and business.
Simon Willison’s Weblog by Simon Willison. Simon has been thoughtfully and technically covering AI for the last year, also sharing valuable articles that he comes across elsewhere. When he’s not writing about AI, he’s writing about his data tool, Datasette, and other data and content related thoughts.
Things That Caught My Attention by Dan Hon. Writing about technology from a personal, social, and organizational perspective, if you ever think “I should read thinking in systems” Dan is someone who is thinking and writing about those systems.
The Beautiful Mess by John Cutler. Another great product management blog, John writes short and thoughtful product management and product organization perspectives regularly.
Ask a Manager by Alison Green. Most of what I’ve learned about what to expect from my workplace, how to navigate professional norms, and how to look for a job, I’ve gleaned from Alison’s excellent and entertaining advice column.
Labnotes by Assaf Arkin. Assaf has been writing this for years, aside from a brief hiatus when he was hit by a car, and I always enjoy the links and posts that he shares about technology and the world. I follow many newsletters like this and there’s always new and interesting content in what Assaf finds.
Interconnected by Matt Webb. Explorations and wide-ranging perspectives on “the future of technology, design, and society”.
Internal Exile by Rob Horning. Rob used to be an editor for The New Inquiry and Real Life (rip) and now writes this newsletter. I’ve been following his writing all the while.
Monkeynoodle by Jack Coates. I worked with Jack for a little while many years ago, and now I follow his great blog about product management and organizations.

Other favorites that I don’t read as consistently, but I wish I did:

Rest of World which I started following to gain a perspective about tech in countries like China and India. The focus of the site has since broadened, supported by a vision “to become an indispensable source of information that captures people’s experiences with technology outside the West”. Highly recommend, especially their reporting on the folks performing data labeling for AI tools.
404 Media founded this year by some folks from Motherboard and are doing incredible journalism already, especially on data security, privacy, and AI.
She’s a Beast by Casey Johnston. Casey has been writing “Ask a Swole Woman” for years, and wrote an ebook, Liftoff: Couch to Barbell. She’s been on a mission to empower women and others to get stronger, and along the way, trust their bodies and themselves.
★❤✰ Vicki Boykis ★❤✰ by Vicki Boykis. I started following Vicki when she was writing Normcore Tech about ordinary data and the bread and butter of data work. As life happens, she’s largely stopped writing the newsletter but still occasionally posts on her blog.
Counting Stuff by Randy Au. Randy covers the bread and butter of data work as well (He worked with Vicki to put on Normconf with many other talented folks) and it’s so straightforwardly useful.
Untangled by Charley Johnson. Charley writes about the intersections of society and technology and the role of power along the way.
The AI Ethics Brief by Abhishek Gupta of the Montreal AI Ethics Institute. A brief yet thorough overview of research and policy updates in AI ethics and other movement in the space.

The format of your online technical content matters

Wed, 29 Nov 2023 22:59:33 +0000

How you choose to make technical content available online sends a message about how you do business and what you think about your content and customers…

🌐 Available fully for free on the web: You want your technical content to be as easy to find as possible. You want it to be easily searched, shared, and copied. You don’t mind if competitors know how your product works, because you also want prospective and current customers to know. You want your product and its technical capabilities to be part of the online and offline conversations. You can easily update the content and make sure that people access the most recent version.

🔐 Available on the web, but only if you log in first: You fear your competitors and think that secrecy is your product’s competitive advantage. Existing customers can access your content, but only them. Prospective customers can read your marketing content and talk to sales. If they aren’t paying you, they can’t have a conversation about what your product really does.

📄 Available only as a PDF: You want to make your content available on the web, but just barely. You want to pretend your content is easy to find, but it isn’t easy to search for the content inside the PDF, share a specific part of the content, copy information from it, or link on the web. If you publish content as a PDF, you don’t want to be part of the conversation about the content—you don’t care how people get or share the content, or what questions they might have about it. It’s a bit onerous to publish a new version of the content, and difficult to ensure that people that accessed the earlier versions of the PDF get access to the latest updated one.

With all this in mind, I personally don’t understand why Meta chose to publish their Responsible Use Guide for Llama 2 as a PDF, instead of as part of the user guide for the model. Their own user documentation is reduced to pointing to the PDF and instructing readers to “refer to pages 14–17” of the PDF.¹

Maybe there were some legal restrictions preventing Meta from publishing the guide as part of their official documentation for Llama 2, but it’s certainly sending a clear message—Meta wants you to have information about responsibly using the model, but they don’t care if you find it, or what you do with it.

Fun fact, I couldn’t link to the header for the guidance about reducing hallucinations because the header is actually a <p> tag and not a true HTML header. If you’re going to make content available on the web, please use semantically correct HTML. ↩︎

Why web design sucks now

Thu, 26 Oct 2023 22:05:43 +0000

Heather Buchel’s post It’s 2023, here is why your web design sucks about the current state of web design (and web app design) resonated with me, especially this quote:

“Design decisions can only be pushed so far to the left before we realize the system is broken”

If you bisect design and development into different professions, you can end up with designers that aren’t technical enough to design what’s possible, or developers without enough design prowess to make design decisions independently.

I wonder too if tooling has something to do with it — in an era of Figma and Canva and Webflow, it’s easier than ever to design websites and web applications without having to interact with HTML, CSS, or JavaScript, let alone the DOM of a web browser and poking around in DevTools to get something to display properly.

A similar issue can happen with technical writers in a “throw it over the wall” documentation culture, where a writer relies on functional specs, design docs, and engineering requirements documents to write content rather than actually using the product.

Part of it could be a hiring prioritization motive, where engineers are the first and most-valued hires, while designers and technical writers are seen as support staff and hiring is calculated according to team:designer or team:writer ratios, rather than a partnership-based approach.

In my opinion, cross-functional partnerships with empowered, collaborative product development, are crucial for building a high quality product that customers can use.

The value in research gaps

Mon, 20 Mar 2023 10:57:21 +0000

There’s value in the holes. If you search for information about a topic and don’t find very much about it, it can be a clear signal that more research is necessary or desired to find the answers.

A research gap is exactly how Dr. Tina Lasisi, interviewed for the Melaninology episode of the Ologies podcast, describes how she got into this field of research [starting around 6:22]:

I have always been aware that different people have different skin color, but I never thought about how it was patterned around the world. What about other traits? How do those vary, and how do those evolve? And my immediate question as a Black woman, was okay what about my hair? Like okay, I understand why my skin is brown, but why is my hair curly? And the wild thing is there wasn’t a good answer! What should’ve been a really quick Wikipedia search that satisfied my curiosity, became this rabbit hole where I basically had this Postdoctoral Fellow who was at our college who took me under his wing and was like hey, let’s talk about BioAnth [Biological Anthropology], and I was like, oh, so I have all these questions and I can’t find anything about like, hair, and he basically was like well, sounds like that could be something for like your undergraduate thesis! And like as an undergraduate I decided ok let me get hair samples and measure them, and like yeah, long story short, basically this thing that should’ve been a short Wikipedia search ended up being a decade plus journey into understanding this trait and like why humans have it. (transcription mine)

I’d wager many other academics have a similar origin story for their own research projects.

Unfortunately, if you rely on a large language model to do research for you, or help give you ideas about what to research, the nature of the tool means that instead acknowledging that there are no results for your prompt, you instead can get output full of “best guess” citations based on words that are semantically similar to the words that you prompted.

Ben Davis writes about this phenomenon for Artnet in We Asked ChatGPT About Art Theory. It Led Us Down a Rabbit Hole So Perplexing We Had to Ask Hal Foster for a Reality Check. He and his colleagues found that if you attempt to perform research with ChatGPT, you are instead likely to receive a list of nonexistent citations:

Sometimes—if, for instance, I ask it to “give me a list of citations about the influence of Artificial Intelligence on European Medieval Art”—it accurately tells me that this query makes no sense—but then provides a list of made-up references anyway

Ben makes several attempts to get the system to provide references about a topic that likely has very little, if anything written about it, going so far as to specifically ask for citations that actually exist:

When I ask a follow-up, specifying that the references now be actually “real,” my chatbot helper is again very helpful, but again just makes stuff up

Ultimately, he concludes that this flaw is not a surprise, and is fact completely expected due to what ChatGPT is built on:

The glitch seems to be a linear consequence of the fact that so-called Large-Language Models are about predicting what sounds right, based on its huge data sets. As a commenter put it in an already-months-old post about the fake citations problem: “It’s a language model, and not a knowledge model.”

In other words, this is an application for sounding like an expert, not for being an expert—which is just so, so emblematic of our whole moment, right?

This flaw makes large language models a decent application for things like thought leadership, a common forum for sounding like an expert. But for circumstances where it truly matters to be an expert—like academia, or writing documentation—it’s a serious problem.

It’s one thing to confabulate thought leadership — it’s quite another to lie about product functionality in contractually offered software documentation.

There’s value in identifying the holes in knowledge and understanding. If you gloss over knowledge gaps with generated drivel, you miss out on the opportunity to dig deeper to learn about something—and so too does anyone that might discover what you learned in your research.

Considering types of meetings

Fri, 17 Mar 2023 08:22:55 +0000

The next time you’re in a meeting, wishing you weren’t, you might want to consider why the meeting feels so insufferable.

Often it’s because no one has bothered to consider the purpose of the meeting.

Cam Daigle devised a classification system that provides an excellent framework for improving meetings. They declare that There are three types of meetings:

I believe all meetings can either be defined as either Status meetings, Feedback meetings, or Decision meetings.

For each type of meeting, Cam identifies the goal, appropriate scale, power dynamic, and risks inherent in each type. For example, for status meetings:

The goal of a Status Meeting should be to disseminate information. That’s it. If the attendees of the meeting come away with current and relevant information about whatever the hell the meeting was intending to communicate, it’s done its job.

My favorite meeting type is the feedback meeting—mostly because meetings of this type are the most frequent devolution state of a status meeting or a decision meeting (in my experience):

Everyone is in a room talking about what they accomplished in the last 24 hours, and then a few folks start offering feedback on a specific approach to solving a problem that someone took. A status meeting just became a feedback meeting.
Or maybe you’re in a meeting room, trying to decide whether to build something based on a given design, and you get an hour of feedback on the design but no clear path forward. A decision meeting became a feedback meeting.

For a feedback meeting, Cam is careful to identify 2 crucial devolution states, of which the second is more critical:

Be extremely careful about decisionmaking.

The moment the presenter stops receiving feedback and starts reacting to it, the meeting dynamic is at risk.

That’s right, a feedback meeting can also devolve into a decision meeting.

I highly recommend reading Cam’s entire post. Since doing so, every meeting I go to and thinkpiece about meetings that I read has me sitting there wondering what type of meeting I’m in, or is being discussed.

Is my 1:1 with my manager a decision meeting or a status meeting?
Is this product review a feedback meeting or a decision meeting?

The next time you schedule a meeting, give it an agenda, but also consider which type of meeting it is—and whether it should be happening at all.

Chat apps are no substitute for documentation

Wed, 15 Mar 2023 09:25:14 +0000

As a technical writer, I have a mixed opinion of chat applications like Discord and Slack. On the one hand, they make it easy to quickly get ahold of someone who can answer your questions, which is a relief if you’re struggling to gather information you need to write a draft.

On the other hand, because it’s easy to quickly get ahold of someone who can answer your questions, that convenience can implicitly incentivize folks to neglect documentation. This is true on both sides:

Folks with questions might ignore documentation that exists because it seems easier or faster to just ask in the chat app.
Folks answering questions might prefer not to spend lots of time writing system design documentation, detailing the decisions they made and why, when they can just answer those questions as needed.

Jason Scott shares this mixed opinion. As he points out in Discord, or the Death of Lore:

“I have no disputes as the popularity of the places, the things that happen there, and the unquestioned vivaciousness of being the party that never seems to end and everyone wants to join.

I just happen to be the sort of person who notices there’s no decent fire exits and most of the structure is wood and there’s an… awful lot of pyrotechnics being set off.”

I feel the same way.

I wonder if the folks who have their focus time interrupted by detailed, somewhat archaic questions that require a backstory, ever wish they had documented the answer so that they could respond with a link and move on with their day.

Chat apps like Discord end up diluting the available knowledge because the content shared in them isn’t persistent, and the allure of an always-available answer breaks down when the person that could answer is no longer available. Jason refers to this as the “lore-to-knowledge transfer”:

The danger in this process, the potential lost ballast in the rise to the skies, is that the lore-to-knowledge transfer is lossy, messy, and arbitrary. Maybe those in the know want to keep the information to themselves, so it won’t be given to whoever the person or persons are who are laying down the written form. Maybe the chronicler of information has blind spots they don’t know about and not enough people to correct them. Or, more likely, you have to set the “noise filter” of the information to not go down the rabbit and rat holes of contingencies that maybe a dozen or two people will even want to know about, to the favor of that which everyone will need. The outcome is always the same: Lore loses in the long run.

The process of documenting information can break down silos (information is more available), expose blind spots in an approach (something is missing here), and enable asynchronous knowledge transfer (documentation is always online).

Etsy blogged about an interesting approach five years ago, Etsy’s experiment with immutable documentation, where they built a plugin that engineers could use within Slack to update or create documentation:

At Etsy we’ve developed a system for adding how-docs directly from Slack. It’s called “FYI”. The purpose of FYI is to make documenting tactical details – commands to run, syntax details, little helpful tidbits – as frictionless as possible.

I’d be curious to learn whether they’re still using this system, and how it has aged.

Measuring data (and documentation) quality is hard

Mon, 13 Mar 2023 10:24:27 +0000

Gwen Windflower asks Are you actually measuring data quality?:

Data is a woven net thrown over the world, capturing it in a grid of variable resolution. We want to capture as much detail as we possibly can, but it’s then crucial to make judicious decisions about how we translate what we capture into a useful map.

Documentation involves a similar discernment process as data analysis and data engineering—given a large set of inputs, identify how to make sense of the output and where to start first. Defining priorities and classifying information is vital.

When it comes to quality in particular, and how to measure quality, Gwen addresses the common practices of data analysts:

Typically though, when it comes to quality, we proceed to measure things like the internal consistency of multiple printings of our map, the resolution of the ink, and how quickly we print new maps in response to changes in the terrain. Two related maps stay internally consistent? Quality. Granular resolution of detail? High quality. Map updates in near-realtime? The highest quality. We often assume that making these kinds of measures higher in our output is an unalloyed good.

Again, this is a common practice in documentation. It’s tempting to align documentation quality measures with documentation coverage (parity with product functionality), documentation freshness (updated quickly), or with particular measures about the writing quality itself, such as sentence length, page length, or word complexity.

But as Gwen points out, measuring quality in these ways is flawed:

There’s a problem with these self-referential, almost tautological measures though: none of them tells us if a map is fulfilling its purpose.

Maps should be measured on how well they get you where you want to go. To do that maps need to be as detailed and accurate as necessary and no more.

If you want to measure the quality of your output, you also need to assess its usefulness.

To measure quality, we need to look beyond the data (or the documentation) itself, and look at the function. Gwen makes this clear, continuing with the map metaphor:

to know the proper level of detail, pace, and presence for our maps, we need to know where we want to go.

If you want to hike through a national park, you want a trail map, not a road map. Similarly, you need to understand why you’re creating, refining, or analyzing data so that you can produce something that is high quality for the purpose.

The payroll department might have an objective to improve payroll quality, measured by reducing paycheck errors across the company from 5% to 1%. If you’re managing their data pipeline, knowing what the data is used for and what the quality issues are can make a big difference in how you approach your work. You might start by improving the accuracy and reliability of the data before you worry about reducing latency of the data to be near real-time.

Similarly, documentation is a supporting team. We do our own work, but it’s relatively useless if it doesn’t help folks meet their objectives. To write high quality documentation, we must consider the goals of our audience, and what they want to accomplish with the product, to provide relevant and useful documentation.

As a supporting team, data teams face a challenge that Gwen identifies:

There is constant discussion of how hard it is to measure the impact of data teams, and that’s directly because of a lack of intentionality with our data.

That’s something that applies to tech writing teams. The impact of a data team, and the impact of a documentation team, can be measured by identifying the objectives that those teams are helping others meet. Who does our work support, and did we do our work with the goals of that audience in mind?

Technical documentation as a map

Mon, 19 Sep 2022 20:46:08 +0000

Matt Webb wrote a post about organizing data and mapping the experience of the web, and that made me consider how the decisions about documentation structure, especially in the early stages, require similar decisions.

Technical documentation functions as a map for your product. For specific users, your documentation provides wayfinding guidance and provides the information necessary to navigate the space relevant to your product.

Beyond a map for your product, technical documentation can also map out specific workflows within your product—specific routes, one might say.

For example, diagramming and describing how data flows through the 55+ different systems that comprise Facebook. Without that documentation, you might find yourself in the same situation as some Meta engineers did in March¹, admitting that:

“we have a somewhat strange engineering culture compared to most where we don’t generate a lot of artifacts during the engineering process. Effectively the code is its own design document often.” If the code serves as the documentation, that’s the same as having street signs or individual transit stations, but no sense of where you are within a larger system. You need a map to add a layer of intelligibility to the individual parts of an overall system.

Considering your documentation as a map can help you when you’re struggling to know where to start when writing minimum viable documentation.

What kind of map might you draw on a napkin about your product, or about a specific data interaction or user interaction?

What is the key wayfinding information? What are the intersections or landmarks to be aware of?
What would confuse someone if you left it out of the map? Are your steps complete enough to help someone get to their desired destination?
Do you understand the destination and the person you’re guiding well enough to draw such a map?

If you consider those questions the next time you need to write documentation about a product, a data flow, or a user flow, you and your customers will be much better situated.

As disclosed in a court filing reported by The Intercept in Facebook Engineers: We Have No Idea Where We Keep All Your Personal Data and further reported on by Vice in Facebook Engineers Admit They Don’t Know What They Do With Your Data. ↩︎

Self-Analytics

Fri, 28 Jun 2013 21:40:33 +0000

Using Wolfram Alpha’s Facebook Report tool, I can examine my own patterns of Facebook activity–and confirm suspicions of my own patterns and habits.

I work an 8-5 job, so you see spikes of activity over lunchtime and after I get home from work.

The majority of what I post to Facebook is links to share with my friends.

I post a lot of links.

Encounters with the Internet

Thu, 27 Jun 2013 20:29:03 +0000

Don’t read books, buy shoes. Urban Outfitters knows what you really wanted, and it wasn’t reading.

Sometimes, Facebook has the best ads.

This, for example, is a real play.

Disco pants let you be a Human Supernova. Minus the probable death that might be included in that.

If you have a lot of tabs open, and it takes you awhile to get back to them, sometimes they’ll completely glitch out before they recover.

In which privilege is revealed...

Wed, 05 Jun 2013 21:26:16 +0000

Gap gets presumptive about the class of its cardholders. “In what city is your vacation home?” Also doesn’t believe in maiden names with apostrophes, apparently.