From 0b01f6fc8aabed68982c73331cf067c1d9c6e9f3 Mon Sep 17 00:00:00 2001 From: Dave Welsch Date: Wed, 19 Feb 2025 14:57:54 -0800 Subject: [PATCH 1/6] Add Vitess analysis document. Signed-off-by: Dave Welsch --- analyses/0014-vitess/vitess-analysis.md | 1019 +++++++++++++++++++++++ 1 file changed, 1019 insertions(+) create mode 100644 analyses/0014-vitess/vitess-analysis.md diff --git a/analyses/0014-vitess/vitess-analysis.md b/analyses/0014-vitess/vitess-analysis.md new file mode 100644 index 0000000..be739ee --- /dev/null +++ b/analyses/0014-vitess/vitess-analysis.md @@ -0,0 +1,1019 @@ +--- +title: Vitess Documentation Analysis +tags: Vitess +created: 2025-02-19 +modified: 2025-02-19 +author: Dave Welsch (dwelsch-esi) +--- + + + +## Introduction + +This document analyzes the effectiveness and completeness of the +[Vitess][project-website] open source software (OSS) project's documentation +and website. It is funded by the CNCF Foundation as part of its overall effort +to incubate, grow, and graduate open source cloud native software projects. + +According to CNCF best practices guidelines, effective documentation is a +prerequisite for program graduation. The documentation analysis is the first +step of a CNCF process aimed at assisting projects with their documentation +efforts. + +### Purpose + +This document was written to analyze the current state of Vitess +documentation. It aims to provide project leaders with an informed +understanding of potential problems in current project documentation. +A second [implementation] document outlines an actionable plan for improvement. +A third document is a [list of issues][issues list] to be added to the project +documentation repository. Theseissues can be taken up by contributors to +improve the documentation. + +This document: + +- Analyzes the current Vitess technical documentation and website +- Compares existing documentation against the CNCF’s standards +- Recommends a program of key improvements with the largest + return on investment + +### Scope of analysis + +The documentation discussed here includes the entire contents of the website, +the technical documentation, and documentation for contributors and users on +the Vitess GitHub repository. + +The Vitess website and documentation are written in Markdown and are compiled +using the Hugo static site generator with the [Bulma] CSS framework and +apparently served from Netlify. The site does not appear to use a theme, or +uses a default theme (there is no theme given in the configuration file.) +The site's code is stored in its own repository in the Vitess GitHub project. + +#### In scope + +- Website: https://vitess.io/ +- Documentation: https://vitess.io/docs/ +- Website repo: https://github.com/vitessio/website +- Project repo (for reference): https://github.com/vitessio + +#### Out of scope + +- Other Vitess repos: https://github.com/vitessio/* (In general, + other Vitess code repos are out of scope) + + +### How this document is organized + +This document is divided into three sections that represent three major areas +of concern: + +- **Project documentation:** concerns documentation for users of the Vitess + software, aimed at people who intend to use the project software +- **Contributor documentation:** concerns documentation for new and existing + contributors to the Vitess OSS project +- **Website:** concerns the mechanics of publishing the documentation, and + includes branding, website structure, and maintainability + +Each section begins with summary ratings based on a rubric with appropriate +[criteria] for the section, then proceeds to: + +- **Comments**: observations about the existing documentation, with a focus on + how it does or does not help Vitess users achieve their goals. +- **Recommendations**: suggested changes that would improve the + effectiveness of the documentation. + +The accompanying [implementation] document breaks the recommendations down into +concrete actions that can be implemented by project contributors. Its focus is +on drilling down to specific, achievable work that can be completed in +constrained blocks of time. Ultimately, the implementation items are decomposed +into a series of [issues] and entered as GitHub issues in the webiste +[repository][project-doc-website]. + +### How to use this document + +Readers interested only in actionable improvements should skip this document +and read the **[implementation] plan** and **[issues list]**. + +Readers interested in the current state of the documentation and the reasoning +behind the recommendations should read the section of this document pertaining +to their area of concern: + +- [Project documentation](#project-documentation) +- [Contributor documentation](#contributor-documentation) +- [Website and documentation infrastructure](#website-and-infrastructure) + +Examples of CNCF documentation that demonstrate the analysis criteria are +linked from the [criteria] specification. + +#### Recommendations, requirements, and best practices + +This analysis measures documentation against CNCF project maturity standards, +and suggests possible improvements. In most cases there is more than one way to +do things. Few recommendations here are meant to be prescriptive. Rather, the +recommended implementations represent the reviewers' experience with how to +apply documentation best practices. In other words, borrowing terminology from +the lexicon of [RFCs][rfc-spec], the changes described here should be understood +as "recommended" or "should" at the strongest, and "optional" or "may" in many +cases. Any "must" or "required" actions are clearly denoted as such, and pertain +to legal requirements such as copyright and licensing issues. + +## Project documentation + +Vitess is a **graduated** project of CNCF. This means that the project should +have [_very high_][criteria] standards for documentation. + +| Criterion | [Rating (1-5)] | +| -------------------------- | -------------- | +| Information architecture | 2. Needs improvement | +| New user content | 2. Needs improvement | +| Content maintainability | 4. Meets or exceeds standards | +| Content creation processes | 1. Not present | +| Inclusive language | 2. Needs improvement | + +### Comments + +The following sections contain brief assessments of each element of the Project +Documentation rubric. + +#### Information architecture + +**Conceptual content**: Is there high level conceptual (“About”) content? + +Yes, the site contains conceptual material explaining what Vitess is and how it +works, in both the [Overview](https://vitess.io/docs/21.0/overview/) and +[Concepts](https://vitess.io/docs/21.0/concepts/) sections. + +The introduction to the Overview could be more helpful to new users. + +**Feature complete**: Is the documentation feature complete? (i.e., each product +feature is documented)? + +Yes, the documentation seems to cover all features of Vitesss (as far as I can +tell). However, see the following regarding task instructions. + +**Task instructions**: Are there step-by-step instructions (tasks, tutorials) +documented for features? + +Yes, there are task instructions covering all major classes of tasks required to +use Vitess: + +- [Installation](https://vitess.io/docs/21.0/get-started/) +- Setup and use +- Administration + +Except for Installation, all task documentation is in the +[User Guides](https://vitess.io/docs/21.0/user-guides/). + +Despite being labeled "Task-based guides," the task instructions in the User +Guide are written from a feature-oriented perspective, which lessens the +documentation's efficiency. + +**No features missing tasks**: Is the documentation free of any key features +which are documented but missing task documentation? + +It looks like there are features that don't have adequate task descriptions. +Features are represented, but often the task descriptions are not written as +procedures. For examples, see +[Information architecture](#information-architecture-1) in +[Recommendations](#recommendations). + +**Happy path**: Is the “happy path”/most common use case documented? + +Vitess is a complex system with many configuration options. That said, I +believe that setting up and running a production server in Kubernetes is the +main use case, and is covered. + +**Isolated, atomic tasks**: Does task and tutorial content demonstrate atomicity +and isolation of concerns? (Are tasks clearly named according to user goals?) + +No, tasks are not isolated. The User Guide has the following issues: + +- Tasks are intermingled with discussion of the features +- Multiple tasks are presented per page +- Task names are often not present in table of contents (TOC) headings +- Where present, task names are ambiguous or misleading + +Titles of pages and sections are often feature-based, making it hard to find +procedures to complete tasks. + +**Escalation path**: If the documentation does not suffice, is there a clear +escalation path for users needing more help? (FAQ, Troubleshooting) + +No clear escalation path exists. There are Troubleshooting guides in the +following sections: + +- Running in Production +- Advanced > Distributed Atomic Transactions +- Migration + +There is one special-purpose FAQ, for VReplication in the Reference. + +There is no glossary. + +There is a Vitess Community page on the product website. This is accessible from +the documentation as well. It has headings for Contributing, Monthly Meetings, +Governance, Code of Conduct, etc. It lists channels (Slack, Stack Overflow, +Issue tracker) but does not prominently feature a “Support” or “Getting Help” +heading. + +**Complete API reference**: If the product exposes an API, is there a complete +reference? + +Not applicable. Vitess does not have an exposed API. + +However, Vitess does have several command-line interface to servers and +utilities, and these are documented in the Reference. + +**Accurate and up-to-date**: Is content up to date and accurate? + +Yes, documentation is versioned and prominently includes the latest Stable and +Development versions. + +#### New user content + +**Getting started entry**: Is “getting started” clearly labeled? +(“Getting started”, “Installation”, “First steps”, etc.) + +Yes, the Get Started section contains instructions to install, set up, and run +Vitess in four different environments (OSes and container solutions). + +The Get Started guide has better written instructions than the User Guide, but +procedures could still be clearer. There is no meta-text explaining when you +would want to use each install option (the test installs are based on platform, +but a discussion of development, test, and production environments would still +help). + +**Installation**: Is installation documented step-by-step? + +No, instructions are not documented step-by-step, strictly speaking. +Instructions are generally sequential down the page but not numbered. As with +the User Guide procedures, steps could be rewritten so that they're more +explicit. + +**Multiple OS**: If needed, are multiple OSes documented? + +Yes, multiple OSes (and container platforms) are documented. Instructions are +given for a Kubernetes (production) install, and three local installs: Linux, +Mac, and Docker. The local (non-Kubernetes) installs are "for test purposes," +but no further details are provided. The default local install for pre-compiled +binaries does not explicitly list what OSes are supported. + +**Getting started next steps**: Do users know where to go after reading the +getting started guide? + +Yes, there is a Next Steps section at the end of each install except the Docker +local install. However, the Next Steps section is perfunctory, pointing in all +cases to the Move Tables page in the Migration guide. + +**New user signpost**: Is your new user content clearly signposted on your site’s +homepage or at the top of your information architecture? + +No, there is no clear entry path for new users. They'll probably make their way through +the install and then try to figure out how to implement their migration case (which +probably fits one of the documented scenarios, but there is no clear way for them +to find it). + +**Sample code**: Is there easily copy-pastable sample code or other example content? + +Yes, command-line examples are plentiful. There is no click-to-copy, however. Users +must highlight and copy manually. + +#### Content maintainability & site mechanics + +**Searchability**: Is your documentation searchable? + +Yes, there is a text Search feature that is limited to the currently displayed version. +It seems to work very well. + +Search is full-text, so common search terms can result in an unwieldy number of results. + +**Localization & i18n directories**: Are you planning for localization/internationalization +with regard to site directory structure? + +Yes, there are full versions of the documentation in both English and Chinese. + +**Localization framework**: Is a localization framework present? + +Yes, there seems to be some infrastructure for multiple languages. I'm not sure how +translation is done in Hugo. + +**Versioning**: Do you have a clearly documented method for versioning your content? + +Yes, releasing a new version is scripted. The repository contains instructions. The +latest three released versions are available on the website. + +#### Content creation processes + +**Content creation process**: Is there a clearly documented (ongoing) contribution +process for documentation? + +There are instructions for building and releasing the documentation, as well as detailed +instructions for contributing instructional videos. However, there are no instructions +for contributing technical documentation or fixing documentation issues. + +Presumably a contributor can submit a pull request on the website repo to amend or +add to the documentation, but there is no procedure documented. There is a notice +that no grammar or typo fixes are accepted from accounts less than a year old. + +**Release process**: Does your code release process account for documentation creation +& updates? + +No, neither the [Contribute](https://vitess.io/docs/contributing/) documentation nor +the [CONTRIBUTING.md](https://github.com/vitessio/vitess/blob/main/CONTRIBUTING.md) +file in the product repo describes how to contribute documentation. There is no Contribution +section in the website repo. + +**Review and approval**: Who reviews and approves documentation pull requests? + +Unknown. The Governance document gives detailed descriptions of the User, Contributor, +and Maintainer roles. It does not mention documentation explicitly except as one of +the contributor tasks. + +**Site owner and maintainer**: Does the website have a clear owner/maintainer? + +No, the main project repo lists maintainers, along with areas of expertise. None of +them lists documentation as an area of expertise. There is no owner information on +the website repo. + +#### Inclusive language + +**Non-inclusive phrases**: Are customer-facing utilities, endpoints, class names, +or feature names free of non-recommended words as documented by the +[Inclusive Naming Initiative](https://inclusivenaming.org) website? + +Not entirely, but there have clearly been attempts to correct non-inclusive language +on the site. For example, “primary” replaces “master” to describe the database of +record in the Vitess product. Some non-inclusive language remains. Examples: +“sanity check”; “master database” (these can be found using the website's Search function). + +Of course, some of this language exists in product command elements and outputs that +cannot be changed in the documentation without first eliminating them in code (command +line options, for example). + +**Ableist language**: "Is the project free of language like 'simple', 'easy', etc.?" + +No. A few examples exist. These should be evaluated and replaced on a case by case +basis. + +### Recommendations + +#### Information architecture + +**Get Started** + +Write an introduction on the [Get Started](https://vitess.io/docs/21.0/get-started/) +landing page. This introduction should outline a path for new users something like +the following: + +1. Install test environment +2. Configure test environment +3. Test Vitess + +After this familiarization process, there should be a a pointer to a User Guide processes +that walk the reader through the product's "happy path" in a more or less linear path +(a series of tasks). That might look something like this: + +1. Plan production installation +2. Install production version +3. Configure production environment +4. Populate databases +5. Run queries +6. Maintain and add cells/shards/databases. + +The "Next steps" sections on the test installation pages should point readers to new-user +configuration and testing procedures. + +The "Next steps" section on the production installation page should point users to +whatever is next in the User Guide documentation. This can be more than one path. +Make it conditional based on the task a reader might want to accomplish after installation: +Test the installation? Plan a cluster? Something else? + +**Conceptual Info** + + Clarify the beginning of the [Overview](https://vitess.io/docs/21.0/overview/whatisvitess/) +("What is ...") The introduction from the README in the [code repository][project-website] +is pretty good: + +> Vitess is a cloud-native horizontally-scalable distributed database system that +is built around MySQL. + +Or the introduction that this replaced: + +> Vitess is a database clustering system for horizontal scaling of MySQL through generalized +sharding. + +**Tasks** + +The User Guides page *says* "Task-based guides for common usage scenarios". That's +the right idea, but implementation must be improved if readers are to find and carry +out the documented tasks. + +The User Guides require three types of changes to maximize their effectiveness: +- Rename +- Rewrite +- Reorganize + +*Rename* + +Rename the individual pages in the User Guides to reflect the tasks that are described +there. Use verbs ending in "-ing" (along with a noun that describes the object of +the task, if applicable). This helps readers find the right content in at least two +ways: +- Makes the TOC more coherent +- Gives meaningful results in the sitewide Search + +Here are three examples from the +[same page] +(https://vitess.io/docs/21.0/user-guides/configuration-basic/global-topo/) +(the content of these sections is not considered here): +- *Choosing a TopoRoot*: A good title. Describes the task ("choosing") and the object + of the task ("TopoRoot"). +- *Moving to a different TopoServer*: A good title. Describes the task ("moving"). + Normally I'd recommend a more direct reference to the object -- + "Moving TopoServers" -- but in this case the phrase ".. to a different TopoServer" + removes ambiguity. +- *Backups*: Not a helpful title. It's not clear or what you're backing up or what + what aspect of the backups you're talking about. I'd change this to "Backing + up TopoServer data". + +Also on the Global TopoServer page, by the way: + +> The following command line options are required for every Vitess component: +> +> ```--topo_implementation=etcd2 --topo_global_server_address= + --topo_global_root=/vitess/global``` +> +> To avoid repetition we will use in our examples to signify the above + flags. + +Remove this. The "" placeholder does not seem to have been implemented. +There are no mentions of " elsewhere in the documentation, and in any +case each would have to refer back to this page. + +*Rewrite* + +A reader typically consults the User Guide to find out how to do something. The User +Guides should consist primarily of procedures. + +There are some Vitess features for which the User Guide gives a description but does +not provide adequate instructions for their use. + +For example, look at [Creating a Backup](https://vitess.io/docs/21.0/user-guides/operating-vitess/backup-and-restore/creating-a-backup/) + +This page appears to be well named ("Creating a Backup" is a descriptive task title) +and properly placed (it resides in a logical location in the User Guide). However, +the page itself doesn't contain an actual command or set of steps to back up a database: +There is no actual command line given between *Configuration* and +*Common Errors and Resolutions* The page's author might assume that it's implied, +but it should be presented explicitly as a step in the procedure. + +Further down the page, another backup option, *Using mysqlshell*, has the same shortcomings: +No actual command is presented. + +Instead, write these pages as step-by-step descriptions of how to perform a task. +Each step should be a concise instruction, with any required instruction or text illustrated +and explained (much of the time, this is a monospace code/CLI block displaying the +command-line instruction to use). This can be an example, but only if it is obvious +how the reader should use the command. Other times, it means showing the required +form of the command, with placeholders for parameters, and explaining those parameters +in the following text. + +The [Creating a cell](https://vitess.io/docs/21.0/user-guides/configuration-basic/create-cell/) +page shows a CLI command: + +> ``` +> vtctldclient AddCellInfo \ +> --root /vitess/cell1 \ +> --server-address \ +> cell1 +> ``` + +It's not clear if `/vitess/cell1` is a user-definable parameter or not. The server +address placeholder, ``, is not defined in the text. + +Here's how I'd rewrite it, defining placeholders for the parameters: + +> ``` +> vtctldclient AddCellInfo \ +> --root \ +> --server-address \ +> cell1 +> ``` +> +> where: +> - is the root directory of the server installation +> - is the IP address of the topo server + +(or whatever the actual descriptions of the parameters are.) + +Here's an outline for a one-page procedure writeup: + +- Title ("ABCing XYZ") + - Context (Describe where the procedure fits in the overall use case) + - Prerequisites (Hardware and software requirements, dependencies, procedures that + must be completed first) + - Procedure ("To ABC an XYZ, do the following.") + - Step 1 (*One* CLI command, or action. Don't combine actions.) + - Step 2 (etc.) + - Result (Optional - include only if there's something remarkable about the outcome.) + - Next Steps (What procedure or use case typically follows this. If it depends on + context, explain the options.) + +*Reorganize* + +Ensure that the various sections of the User Guide cover all usage scenarios. Reorganize +the User Guide: + +First, organize by user role, if there are distinct roles. As I understand it, there +are basically three user roles in Vitess: +- Vitess admin +- Database admin +- Application developer + + User roles traditionally are the basis for a "User Guide": +- Admin Guide +- DBA Guide +- Programmer's Guide +- etc. + +Within roles, organize the tasks around use cases. Don't be afraid to split up tasks +that use the same tool (CLI or other). In other words, pick and choose commands and +actions that server a use case rather than trying to document everything you can do +with the command in one place. (There is a place to exhaustively document a tool, +but that's in the Reference, not a User Guide.) + +The existing Vitess User Guides are already partially organized around user roles, +but they can be regrouped. In any case, make the user roles explicit: + +*Vitess admin*: +- Configuration +- Running in production +- Operational +- Migration + +*DBA*: +- VSchema and Query Serving +- SQL statement analysis +- Making schema changes + +*Application programmer*: +- Query serving +- Making schema changes (don't duplicate the section here. Instead provide links to + any tasks that are identical) +- Query optimization + +(Again, these are my understanding of the Vitess user roles. Adjust the details if +they're different. + +**Troubleshooting** + +Consolidate all troubleshooting information into a Troubleshooting Guide. Or, create +one troubleshooting guide per user guide/user role. In either case, create an escalation +path for any problems with a task (the escalation path might be: +*troubleshooting procedure > Slack Channel > project Issue*). Get rid of the +[VReplication FAQ](https://vitess.io/docs/21.0/reference/vreplication/faq/) +in the reference section and put the information in a troubleshooting section. + +**Glossary** + +Write a glossary. This is different from the "Concepts" page – the explanations of +terms is less in-depth. The glossary contains not just key terms but any word or phrase +that the reader might not know: abbreviations and acronyms, definitions of Vitess-specific +terms, and explanations of jargon used in Vitess ("topo", for example). + +**Other recommendations** + +Rename ["Reference > Programs"](https://vitess.io/docs/21.0/reference/programs/) to +"Command line reference" or "Tools reference". Consider splitting into two or more +sections by type of application: by function or user role. CLIs should be labeled +as such and separated from GUI tools. + +#### New user content + +Is there an alternative to Kubernetes for running in production? Explain. Give an +explicit list of supported OSes. This can be done by expanding the Get Started landing +page (see [Information Architecture > Get Started](#information-architecture)). + +Rewrite installation instructions as step by step procedures. + +Expand the production install +[Next Step](https://vitess.io/docs/21.0/get-started/operator/#next-steps) +section to accomodate different configurations (conversion, green field, etc.). Differentiate +between Next Steps for test/development installations and production installation. + +Put enough information on the Get Started landing page to orient new users as described +in [Information Architecture > Get Started](#information-architecture). + +Rename sections so that it's easier to find the right page in Search: +- Use "-ing" verbs for task pages as previously described +- Use the word "reference" in CLI references +- Use nouns for architectural components, features, and concepts + +#### Content maintainability & site mechanics + +No recommendations. Technical aspects of infrastructure and maintenance seem excellent. + +#### Content creation processes + +There is no documnetation for website and tech doc content creation. Such documentation +could include: +- Instructions for documenting tasks associated with new features +- Instructions for fixing documentation issues +- Templates for task writeups and command references +- Guidelines for amending conceptual information with new features +- A style manual +- A review and approval process, separate from the code process +- A link from the code contribution instructions to the doc instructions +- A maintainer designated to be responsible for website maintenance and for documentation +changes and updates +- Getting started instructions for new documentation contributors + +In practice, open source projects rarely contain all this information. At a minimum, +document the following processes for contributors: +- Documenting a new release + - Generating, testing, and deploying the new release docs, including how to publish +release notes + - Demoting, archiving, or deleting down-level releases +- Creating a doc issue +- Fixing and closing a doc issue +- How to contact the website and documentation maintainer with questions +- Getting started instructions for new documentation contributors + +These processes should address documentation-specific concerns, not just parrot the +code release procedures (although there may be many of the same steps). + +#### Inclusive language + +Search the document for non-inclusive terms, especially tier-1 and tier-2 terms per +the [Inclusive Naming Initiative](https://inclusivenaming.org/). Replace with recommended +language where possible. + +## Contributor documentation + +Vitess is a **graduated** project of CNCF. This means that the project should +have [_very high_][criteria] standards for documentation. + +| Criterion | [Rating (1-5)] | +| ----------------------------------------- | -------------- | +| Communication methods documented | 3. Meets standards | +| Beginner friendly issue backlog | 2. Needs improvement | +| “New contributor” getting started content | 2. Needs improvement | +| Project governance documentation | 4. Meets or exceeds standards | + +### Comments + +The following sections contain brief assessments of each element of the +Contributor Documentation rubric. + +#### Communication methods documented + +**Text communication channel**: Is there a Slack/Discord/Discourse/etc. community +and is it prominently linked from your website? + +Yes, the community page lists a Slack channel. + +**Repository link**: Is there a direct link to your GitHub organization/repository? + +Yes, the community page and the footer list the GitHub page for the product. + +**Project meetings**: Are weekly/monthly project meetings documented? Is it clear +how someone can join those meetings? + +Yes, the community page lists the monthly community meetings. + +**Mailing lists**: Are mailing lists documented? + +No, there does not appear to be an email list. + +#### Beginner friendly issue backlog + +**Issue triage**: Are docs issues well-triaged? + +No, there does not appear to be any mechanism for prioritizing issues in the GitHub +repo. + +**Good first issues**: Is there a clearly marked way for new contributors to make +code or documentation contributions (i.e. a “good first issue” label)? + +Yes, there is a “good first issue” label in the website repo issues list. + +**Issue documentation**: Are issues well-documented (i.e., more than just a title)? + +Yes, issues seem to have at least a paragraph description. However, there is no issues +template or guidelines for documenting issues, so the quality of the descriptions +is inconsistent. + +**Issue freshness**: Are issues maintained for staleness? + +No, the oldest open issue is over 6 years old. + +#### New contributor getting started content + +**Community featured on website**: Do you have a community repository or section on +your website? + +Yes, there is a Community link in the site’s menu bar, leading to the Community page. + +**New contributor document**: Is there a document specifically for new contributors/your +first contribution? + +No, I could not find a new contributor's document. + +**New user help**: Do new users know where to get help? + +There is no prominent document to guide new users. + +#### Project governance documentation + +**Governance documentation**: Is project governance clearly documented? + +Yes, there are Governance and Code of Conduct documents in the project repo. + +### Recommendations + +#### Communication methods documented + +No recommendations. + +#### Beginner friendly issue backlog + +Create an issue template to ensure that issue descriptions are consistent and complete. + +Go through the issues backlog and close or update old issues. + +#### New contributor getting started content + +Write a "new contributors" document. Post prominently. + +#### Project governance documentation + +No recommendations. + +## Website and infrastructure + +Vitess is a **graduated** project of CNCF. This means that the project should +have [_very high_][criteria] standards for documentation. + +| Criterion | [Rating (1-5)] | +| ------------------------------------------- | -------------- | +| Single-source for all files | 5. Exemplary | +| Meets min website req. (for maturity level) | 2. Needs improvement | +| Usability, accessibility, and design | 3. Meets standards | +| Branding and design | 4. Meets or exceeds standards | +| Case studies/social proof | 2. Needs improvement | +| SEO, Analytics, and site-local search | TBD | +| Maintenance planning | 2. Needs improvement | +| HTTPS access & HTTP redirect | 5. Exemplary | + +### Comments + +The following sections contain brief assessments of each element of the Website +and documentation infrastructure rubric. + +#### Single source + +**Single source for docs**: Does the project have a single source for documentation? +If not, is there a reason? + +Yes, the website and tech doc content are all in one GitHub repo. + +#### Minimal website requirements + +**Website guidelines**: Are all guidelines satisfied? + +Yes, the website mostly meets all criteria for a CNCF graduated website, including +hosting, copyright, CNCF branding, and trademark. + +**Docs analysis**: Are all follow-up actions addressed? + +Pending: There is no reason to believe the Vitess team won’t follow through on recommendations +in this analysis. + +**Project doc: stakeholders**: Are all stakeholder needs identified? + +No, roles are not explicitly defined in the documentation. There is some differentiation +among roles implicit in the docs (admin permissions, RBAC support, etc.), but it is +not used to organize the content. + +**Project doc: hosting**: Hosted directly. + +Yes, the site is hosted on Netlify. + +**Project doc: user docs**: Fully addresses needs of key stakeholders? + +Yes, the documentation probably addresses all stakeholder needs, but is not organized +so that users can find the docs efficiently. + +#### Usability, accessibility and devices + +**Mobile**: Is the website usable from mobile? + +Yes, the site seems to adapt well to small screen use. + +**Readability**: Are doc pages readable? + +Yes, documentation pages are readable on all tested platforms and screen sizes. + +**Mobile navigability**: Are all / most website features accessible from mobile -- +such as the top-nav, site search and in-page table of contents? + +Most website features are accessible from mobile. + +**Mobile-first design**: Might a [mobile-first] design make sense for your project? + +No, this project might occasionally be accessed on mobile but it doesn't seem likely +to be the main use case. + +**Color contrast**: Are color contrasts significant enough for color-impaired readers? + +Yes, text is black on white. Font is a very legible sans serif. + +**Keyboard-only**: Are most website features usable using a keyboard only? + +Yes, most features are usable. However, tab navigation is awkward and time-consuming. + +**Text-to-speech**: Does text-to-speech offer listeners a good experience? + +Text-to-speech was not tested. + +#### Branding and design + +**Recognizable brand**: Is there an easily recognizable brand for the project +(logo + color scheme) clearly identifiable? + +Yes, the site has a consistent design. The orange “layered sheets” logo is recognizable. + +**Consistent branding**: Is the brand used across the website consistently? + +Yes, branding is used cosistently. + +**Typography**: Is the website’s typography clean and well-suited for reading? + +Yes, the font is a little small in places, but overall legible. Font scales gracefully +when magnified in the browser. + +#### Case studies & social proof + +**Case studies**: Are there case studies available for the project and are they documented +on the website? + +No, I did not see any case studies. Some of them many videos might contain case studies, +though. + +**Testimonials**: Are there user testimonials available? + +No, there do not seem to be testimonials available on the site. The logo wall does +not have live links. + +**Active blog**: Is there an active project blog? + +Yes, there is a blog. The last entry was two months ago, but it seems to be updated +semi-regularly. + +**Community talks listed**: Are there community talks for the project and are they +present on the website? + +Yes, there are many videos on the “Learning Resources” page, containing a variety +of topics including community talks. + +**Logo wall**: Is there a logo wall of users/participating organizations? + +Yes, there is a substantial logo wall on the product landing page. + +#### SEO, Analytics and site-local search + +**Analytics on production site**: Analytics: Is analytics enabled for the production +server? +- TBD +- TBD + +**Analytics disabled for non-production sites**: Analytics: Is analytics disabled +for all other deploys? +- TBD +- TBD + +**Google analytics: GA4**: Analytics: If your project used Google Analytics, have +you migrated to GA4? +- TBD +- TBD + +**404 reports**: Analytics: Can Page-not-found (404) reports easily be generated from +you site analytics? Provide a sample of the site's current top-10 404s. +- TBD +- TBD + +**Site indexing on production server**: Is site indexing supported for the production +server? +- TBD +- TBD + +**Site indexing disabled on non-default content**: Is site indexing disabled for website +previews and - TBD +- TBD + +**Intra-site search**: Is local intra-site search available from the website? + +Yes, the site text search funciton works well. + +**Analytics custodians documented**: Are the current custodian(s) of the following +accounts clearly documented: analytics, Google Search Console, site-search (such as +Google CSE or Algolia) + +No documentation maintainers of any sort are credited. + +#### Maintenance planning + +**Well supported website tooling**: Is your website tooling well supported by the +community (i.e., Hugo with the Docsy theme) or commonly used by CNCF projects (our +recommended tech stack?) + +Yes, the site is implented using Hugo, CNCF's recommended site generator, and hosted +on Netlify. + +**Cultivating website maintainers**: Are you actively cultivating website maintainers +from within the community? + +Unknown. There is no documented evidence of recruitment on the website or in the repository. + +**Site build times**: Are site build times reasonable? + +I did not try building the site but have no reason to believe that the build time +is excessive. + +**Maintainer permissions**: Do site maintainers have adequate permissions? + +Unknown. + +#### Other + +**HTTPS**: Is your website accessible via HTTPS? + +Yes, all site pages use HTTPS. + +**HTTP redirect**: Does HTTP access, if any, redirect to HTTPS? + +Yes, pages requested using HTTP are redirected to existing HTTPS pages seamlessly. + +### Recommendations + +#### Single-source requirement + +No recommendations. + +#### Minimal website requirements + +Identify stakeholder and user roles. Orgainize task documentation around user roles +(see [Information architecture](#information-architecture)). + +#### Usability, accessibility and devices + +No recommendations. + +#### Branding and design + +No recommendations. + +#### Case studies/social proof + +Include case studies and testimonials on the product website. If these are already +among the video content, tag them and feature them on the landing page. + +Update blog content at least monthly. + +#### SEO, Analytics and site-local search + +TBD + +List documentation maintainers in the website repository. + +#### Maintenance planning + +TBD + +#### Other + +No recommendations. + +## References and notes + +### Rating values + +The numeric rating values used in this document are as follows: + +1. Not present +2. Needs improvement +3. Meets standards +4. Meets or exceeds standards +5. Exemplary + +### References + +[criteria]: ../criteria.md +[implementation]: ./implementation.md +[issues list]: ./issues-list.md +[project-doc-website]: ?https://vitess.io/docs/ +[project-website]: ?https://vitess.io/ +[Rating (1-5)]: #rating-values +[rfc-spec]: https://www.rfc-editor.org/rfc/rfc2119 +[website guidelines]: ../../website-guidelines-checklist.md +[Bulma]: https://bulma.io/ From 46edb5621ef489c81d25c9d91f0b6ab6c01a51c4 Mon Sep 17 00:00:00 2001 From: Dave Welsch Date: Fri, 21 Feb 2025 13:47:05 -0800 Subject: [PATCH 2/6] Update criteria to include common findings from the last two years of CNCF doc analyses. Signed-off-by: Dave Welsch --- docs/analysis/criteria.md | 162 ++++++++++++++++++++++---------------- 1 file changed, 94 insertions(+), 68 deletions(-) diff --git a/docs/analysis/criteria.md b/docs/analysis/criteria.md index e6719fb..e26a3f3 100644 --- a/docs/analysis/criteria.md +++ b/docs/analysis/criteria.md @@ -1,25 +1,44 @@ # Assessment criteria and examples +We recommend reading the +[CNCF sandbox project documentation primer](../sandbox-doc-primer.md) +before reviewing these criteria. The primer contains key definitions concepts +that are referred to in the criteria. + ## Project documentation ### Information architecture -The overall structure (pages/subpages/sections/subsections) of your project -documentation. We evaluate on the following: +How your technical information is organized, laid out, and presented. +This includes structure (pages/subpages/sections/subsections), information +types (conceptual/instructional/reference), and matching documentation +content +to user expectations. -- Is there high level conceptual/“About” content? Is the documentation feature - complete? (i.e., each product feature is documented) -- Are there step-by-step instructions (tasks, tutorials) documented for - features? -- Are there any key features which are documented but missing task - documentation? -- Is the “happy path”/most common use case documented? Does task and tutorial - content demonstrate atomicity and isolation of concerns? (Are tasks clearly - named according to user goals?) +We evaluate on the following: + +- Is there high level conceptual/“About” content? +- Is the documentation feature-complete? (i.e., every product feature is + documented) +- Does the documentation define all user roles (personas) for the product? +- Are there instructions (tasks, tutorials) documented for features? +- Are there instructions for all major use cases for each user role? +- Are tasks organized by user role and use case? +- Does instructional content demonstrate atomicity — are individual tasks + clearly named according to their goals? +- Are tasks written as numbered step-by-step instructions? +- Do task descriptions in headings and the TOC describe the task with a verb + phrase? +- Is the documentation free of any key features which are documented but missing + task documentation? +- Is the “happy path” — the most common use case — documented? - If the documentation does not suffice, is there a clear escalation path for - users needing more help? (FAQ, Troubleshooting) + users needing more help? - If the product exposes an API, is there a complete reference? +- If the product has CLIs, are there complete references? - Is content up to date and accurate? +- Does the documentation separate conceptual, instructional, and reference + information? Example: @@ -27,12 +46,17 @@ Example: ### New user content -New users are the most avid users of documentation, and need content -specifically for them. We evaluate on the following: +New users are the most avid users of documentation and they have unique +documentation needs. + +We evaluate on the following: - Is “getting started” clearly labeled? (“Getting started”, “Installation”, - “First steps”, etc.) + “First steps”, or the equivalent.) +- Is there a “getting started” path for all user roles? - Is installation documented step-by-step? +- Are different types of installation documented (development, test, production) + if necessary? - If needed, are multiple OSes documented? - Do users know where to go after reading the getting started guide? - Is your new user content clearly signposted on your site’s homepage or at the @@ -49,14 +73,22 @@ Example: ### Content maintainability & site mechanics As a project scales, concerns like localized (translated) content and versioning -become large maintenance burdens, particularly if you don’t plan for them. +become large maintenance burdens if you don’t plan for them. We evaluate on the following: - Is your documentation searchable? -- Are you planning for localization/internationalization with regards to site - directory structure? Is a localization framework present? +- Are you planning for localization/internationalization as regards site + directory structure? +- Is a localization framework present? - Do you have a clearly documented method for versioning your content? +- Is release-specific information documented in release notes? +- Is the documentation free of duplicate or nearly duplicated sections of + information? +- Do informational graphics add value by providing information in a way that + would be difficult in text? +- Will graphics require frequent modifications due to software changes, GUI + updates, or translation? Example: @@ -64,7 +96,7 @@ Example: ### Content creation processes -Documentation is only as useful as it is accurate and well-maintained, and +Documentation is only useful if it is accurate and well maintained. It requires the same kind of review and approval processes as code. We evaluate on the following: @@ -101,11 +133,11 @@ to reach you. We evaluate on the following: -- Is there a Slack/Discord/Discourse/etc. community and is it prominently linked +- Is there a Slack/Discord/Discourse or equivalent community prominently linked from your website? -- Is there a direct link to your GitHub organization/repository? -- Are weekly/monthly project meetings documented? Is it clear how someone can - join those meetings? +- Is there a direct link to your GitHub project or repository? +- Can users find and join periodic (weekly or monthly) project meetings, if + applicable? - Are mailing lists documented? Example: @@ -136,8 +168,9 @@ in easily? We evaluate on the following: - Do you have a community repository or section on your website? -- Is there a document specifically for new contributors/your first contribution? -- Do new users know where to get help? +- Is there a document specifically welcoming new contributors and documenting a + first contribution process? +- Can new users find where to get help? Example: @@ -174,48 +207,32 @@ Alternatively, files should be brought into the website repo via If a project chooses to keep source files in multiple repos, they need a clearly documented strategy for managing mirrored files and new contributions. +We evaluate on the following: + +- Does the project have a single source for documentation? If not, is there a + reason? + ### Minimal website requirements Listed here are the _minimal_ website requirements for projects based on their -[maturity level][]: sandbox, incubating, graduated and archived. +[maturity level]: sandbox, incubating, graduated and archived. Except for archived projects, requirements are cumulative through project -maturity levels so, for example, incubating projects must satisfy the -requirements for sandbox projects. - -- **Sandbox** - - [Website guidelines](../website-guidelines-checklist.md): a majority of the - guidelines are satisfied - - **Project documentation** may or may not be present -- it is acceptable at - this maturity level to link out to documentation that hasn't yet been - integrated into the website - - _Example_: website with a single homepage, without any documentation or, - as was mentioned above, linking out to an external (preexisting) source - for docs - - _However_: consider reading the recommended practices in this repository - and implementing as many of the best practices as you can. This groundwork - will pay big dividends later when you need to upgrade your practices and - update your documentation as an incubating project. Assistance is - available from CNCF TechDocs anytime, including answers to individual - questions or a documentation workshop. -- **Incubating** - - [Website guidelines][]: all guidelines are satisfied. - - [Docs assessment][]: request an (re-)assessment through the CNCF [service - desk][] - - **Project documentation**: - - Stakeholders / personas are identified and their core needs (in terms of - docs) documented - - Hosted directly on the website (also see [Single-source requirement]) - - Comprehensive, addressing most stakeholder needs -- **Graduated** - - [Docs assessment][]: follow-through actions from any assessment are complete - - **Project documentation** fully addresses the needs of key stakeholders -- **Archived** - - The website repo is in an [archived state][] - - The archived status of the project must be obvious to those visiting the - website, such as through the use of a prominent banner. - - If a successor project exists, link to its website and/or migration - documentation. +maturity levels. So, for example, graduated projects must satisfy the +requirements for incubating projects. + +We evaluate on the following: + +- Incubating: Are all [website guidelines][] satisfied? Graduated: Are all + guidelines satisfied? +- Incubating: Was a [docs assessment][] requested through CNCF service desk? + Graduated: All follow-up actions addressed? +- Incubating: User roles identified and doc needs documented? Graduated: All + stakeholder need identified? +- Incubating: Website hosted directly? Graduated: Hosted directly? +- Incubating: Is the [docs assessment][] comprehensive, addressing most + stakeholder needs? Graduated: Fully addresses needs of key stakeholders? + [archived state]: https://docs.github.com/en/repositories/archiving-a-github-repository/archiving-repositories @@ -232,16 +249,23 @@ Most CNCF websites are accessed from mobile and other non-desktop devices at least 10-20% of the time. Planning for this early in your website's design will be much less effort than retrofitting a desktop-first design. +We evaluate on the following: + - Is the website usable from mobile? - Are doc pages readable? -- Are all / most website features accessible from mobile -- such as the top-nav, - site search and in-page table of contents? +- Are all or most website features accessible from mobile -- such as the + top-nav, site search, and in-page table of contents? - Might a [mobile-first] design make sense for your project? +- Are color contrasts significant enough for color-impaired readers? +- Are most website features usable using a keyboard only? +- Does text-to-speech offer listeners a good experience? [mobile-first]: https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps/Responsive/Mobile_first -Plan for suitable [accessibility][] measures for your website. For example: +Plan for suitable [accessibility] measures for your website. For example: + +We evaluate on the following: - Are color contrasts significant enough for color-impaired readers? - Are most website features usable using a keyboard only? @@ -258,8 +282,8 @@ this is branding and marketing. We evaluate on the following: -- Is there an easily recognizable brand for the project (logo + color scheme) - clearly identifiable? +- Is there an easily recognizable brand for the project (logo, font, and color + scheme) clearly identifiable? - Is the brand used across the website consistently? - Is the website’s typography clean and legible? @@ -274,7 +298,8 @@ organizations using it. We evaluate on the following: -- Are there case studies available for the project and are they documented on +- Are there case studies available for the project and are they documented + on the website? - Are there user testimonials available? - Is there an active project blog? @@ -329,3 +354,4 @@ Example: - Is your website accessible via HTTPS? - Does HTTP access, if any, redirect to HTTPS? +- Are links to external websites or applications working and current? From 3784d9b9f00911f19557594c241ad7371d266744 Mon Sep 17 00:00:00 2001 From: Nate W Date: Fri, 21 Feb 2025 15:33:50 -0800 Subject: [PATCH 3/6] removing analyses/0014-vitess/vitess-analysis.md Signed-off-by: Nate W --- analyses/0014-vitess/vitess-analysis.md | 1019 ----------------------- 1 file changed, 1019 deletions(-) delete mode 100644 analyses/0014-vitess/vitess-analysis.md diff --git a/analyses/0014-vitess/vitess-analysis.md b/analyses/0014-vitess/vitess-analysis.md deleted file mode 100644 index be739ee..0000000 --- a/analyses/0014-vitess/vitess-analysis.md +++ /dev/null @@ -1,1019 +0,0 @@ ---- -title: Vitess Documentation Analysis -tags: Vitess -created: 2025-02-19 -modified: 2025-02-19 -author: Dave Welsch (dwelsch-esi) ---- - - - -## Introduction - -This document analyzes the effectiveness and completeness of the -[Vitess][project-website] open source software (OSS) project's documentation -and website. It is funded by the CNCF Foundation as part of its overall effort -to incubate, grow, and graduate open source cloud native software projects. - -According to CNCF best practices guidelines, effective documentation is a -prerequisite for program graduation. The documentation analysis is the first -step of a CNCF process aimed at assisting projects with their documentation -efforts. - -### Purpose - -This document was written to analyze the current state of Vitess -documentation. It aims to provide project leaders with an informed -understanding of potential problems in current project documentation. -A second [implementation] document outlines an actionable plan for improvement. -A third document is a [list of issues][issues list] to be added to the project -documentation repository. Theseissues can be taken up by contributors to -improve the documentation. - -This document: - -- Analyzes the current Vitess technical documentation and website -- Compares existing documentation against the CNCF’s standards -- Recommends a program of key improvements with the largest - return on investment - -### Scope of analysis - -The documentation discussed here includes the entire contents of the website, -the technical documentation, and documentation for contributors and users on -the Vitess GitHub repository. - -The Vitess website and documentation are written in Markdown and are compiled -using the Hugo static site generator with the [Bulma] CSS framework and -apparently served from Netlify. The site does not appear to use a theme, or -uses a default theme (there is no theme given in the configuration file.) -The site's code is stored in its own repository in the Vitess GitHub project. - -#### In scope - -- Website: https://vitess.io/ -- Documentation: https://vitess.io/docs/ -- Website repo: https://github.com/vitessio/website -- Project repo (for reference): https://github.com/vitessio - -#### Out of scope - -- Other Vitess repos: https://github.com/vitessio/* (In general, - other Vitess code repos are out of scope) - - -### How this document is organized - -This document is divided into three sections that represent three major areas -of concern: - -- **Project documentation:** concerns documentation for users of the Vitess - software, aimed at people who intend to use the project software -- **Contributor documentation:** concerns documentation for new and existing - contributors to the Vitess OSS project -- **Website:** concerns the mechanics of publishing the documentation, and - includes branding, website structure, and maintainability - -Each section begins with summary ratings based on a rubric with appropriate -[criteria] for the section, then proceeds to: - -- **Comments**: observations about the existing documentation, with a focus on - how it does or does not help Vitess users achieve their goals. -- **Recommendations**: suggested changes that would improve the - effectiveness of the documentation. - -The accompanying [implementation] document breaks the recommendations down into -concrete actions that can be implemented by project contributors. Its focus is -on drilling down to specific, achievable work that can be completed in -constrained blocks of time. Ultimately, the implementation items are decomposed -into a series of [issues] and entered as GitHub issues in the webiste -[repository][project-doc-website]. - -### How to use this document - -Readers interested only in actionable improvements should skip this document -and read the **[implementation] plan** and **[issues list]**. - -Readers interested in the current state of the documentation and the reasoning -behind the recommendations should read the section of this document pertaining -to their area of concern: - -- [Project documentation](#project-documentation) -- [Contributor documentation](#contributor-documentation) -- [Website and documentation infrastructure](#website-and-infrastructure) - -Examples of CNCF documentation that demonstrate the analysis criteria are -linked from the [criteria] specification. - -#### Recommendations, requirements, and best practices - -This analysis measures documentation against CNCF project maturity standards, -and suggests possible improvements. In most cases there is more than one way to -do things. Few recommendations here are meant to be prescriptive. Rather, the -recommended implementations represent the reviewers' experience with how to -apply documentation best practices. In other words, borrowing terminology from -the lexicon of [RFCs][rfc-spec], the changes described here should be understood -as "recommended" or "should" at the strongest, and "optional" or "may" in many -cases. Any "must" or "required" actions are clearly denoted as such, and pertain -to legal requirements such as copyright and licensing issues. - -## Project documentation - -Vitess is a **graduated** project of CNCF. This means that the project should -have [_very high_][criteria] standards for documentation. - -| Criterion | [Rating (1-5)] | -| -------------------------- | -------------- | -| Information architecture | 2. Needs improvement | -| New user content | 2. Needs improvement | -| Content maintainability | 4. Meets or exceeds standards | -| Content creation processes | 1. Not present | -| Inclusive language | 2. Needs improvement | - -### Comments - -The following sections contain brief assessments of each element of the Project -Documentation rubric. - -#### Information architecture - -**Conceptual content**: Is there high level conceptual (“About”) content? - -Yes, the site contains conceptual material explaining what Vitess is and how it -works, in both the [Overview](https://vitess.io/docs/21.0/overview/) and -[Concepts](https://vitess.io/docs/21.0/concepts/) sections. - -The introduction to the Overview could be more helpful to new users. - -**Feature complete**: Is the documentation feature complete? (i.e., each product -feature is documented)? - -Yes, the documentation seems to cover all features of Vitesss (as far as I can -tell). However, see the following regarding task instructions. - -**Task instructions**: Are there step-by-step instructions (tasks, tutorials) -documented for features? - -Yes, there are task instructions covering all major classes of tasks required to -use Vitess: - -- [Installation](https://vitess.io/docs/21.0/get-started/) -- Setup and use -- Administration - -Except for Installation, all task documentation is in the -[User Guides](https://vitess.io/docs/21.0/user-guides/). - -Despite being labeled "Task-based guides," the task instructions in the User -Guide are written from a feature-oriented perspective, which lessens the -documentation's efficiency. - -**No features missing tasks**: Is the documentation free of any key features -which are documented but missing task documentation? - -It looks like there are features that don't have adequate task descriptions. -Features are represented, but often the task descriptions are not written as -procedures. For examples, see -[Information architecture](#information-architecture-1) in -[Recommendations](#recommendations). - -**Happy path**: Is the “happy path”/most common use case documented? - -Vitess is a complex system with many configuration options. That said, I -believe that setting up and running a production server in Kubernetes is the -main use case, and is covered. - -**Isolated, atomic tasks**: Does task and tutorial content demonstrate atomicity -and isolation of concerns? (Are tasks clearly named according to user goals?) - -No, tasks are not isolated. The User Guide has the following issues: - -- Tasks are intermingled with discussion of the features -- Multiple tasks are presented per page -- Task names are often not present in table of contents (TOC) headings -- Where present, task names are ambiguous or misleading - -Titles of pages and sections are often feature-based, making it hard to find -procedures to complete tasks. - -**Escalation path**: If the documentation does not suffice, is there a clear -escalation path for users needing more help? (FAQ, Troubleshooting) - -No clear escalation path exists. There are Troubleshooting guides in the -following sections: - -- Running in Production -- Advanced > Distributed Atomic Transactions -- Migration - -There is one special-purpose FAQ, for VReplication in the Reference. - -There is no glossary. - -There is a Vitess Community page on the product website. This is accessible from -the documentation as well. It has headings for Contributing, Monthly Meetings, -Governance, Code of Conduct, etc. It lists channels (Slack, Stack Overflow, -Issue tracker) but does not prominently feature a “Support” or “Getting Help” -heading. - -**Complete API reference**: If the product exposes an API, is there a complete -reference? - -Not applicable. Vitess does not have an exposed API. - -However, Vitess does have several command-line interface to servers and -utilities, and these are documented in the Reference. - -**Accurate and up-to-date**: Is content up to date and accurate? - -Yes, documentation is versioned and prominently includes the latest Stable and -Development versions. - -#### New user content - -**Getting started entry**: Is “getting started” clearly labeled? -(“Getting started”, “Installation”, “First steps”, etc.) - -Yes, the Get Started section contains instructions to install, set up, and run -Vitess in four different environments (OSes and container solutions). - -The Get Started guide has better written instructions than the User Guide, but -procedures could still be clearer. There is no meta-text explaining when you -would want to use each install option (the test installs are based on platform, -but a discussion of development, test, and production environments would still -help). - -**Installation**: Is installation documented step-by-step? - -No, instructions are not documented step-by-step, strictly speaking. -Instructions are generally sequential down the page but not numbered. As with -the User Guide procedures, steps could be rewritten so that they're more -explicit. - -**Multiple OS**: If needed, are multiple OSes documented? - -Yes, multiple OSes (and container platforms) are documented. Instructions are -given for a Kubernetes (production) install, and three local installs: Linux, -Mac, and Docker. The local (non-Kubernetes) installs are "for test purposes," -but no further details are provided. The default local install for pre-compiled -binaries does not explicitly list what OSes are supported. - -**Getting started next steps**: Do users know where to go after reading the -getting started guide? - -Yes, there is a Next Steps section at the end of each install except the Docker -local install. However, the Next Steps section is perfunctory, pointing in all -cases to the Move Tables page in the Migration guide. - -**New user signpost**: Is your new user content clearly signposted on your site’s -homepage or at the top of your information architecture? - -No, there is no clear entry path for new users. They'll probably make their way through -the install and then try to figure out how to implement their migration case (which -probably fits one of the documented scenarios, but there is no clear way for them -to find it). - -**Sample code**: Is there easily copy-pastable sample code or other example content? - -Yes, command-line examples are plentiful. There is no click-to-copy, however. Users -must highlight and copy manually. - -#### Content maintainability & site mechanics - -**Searchability**: Is your documentation searchable? - -Yes, there is a text Search feature that is limited to the currently displayed version. -It seems to work very well. - -Search is full-text, so common search terms can result in an unwieldy number of results. - -**Localization & i18n directories**: Are you planning for localization/internationalization -with regard to site directory structure? - -Yes, there are full versions of the documentation in both English and Chinese. - -**Localization framework**: Is a localization framework present? - -Yes, there seems to be some infrastructure for multiple languages. I'm not sure how -translation is done in Hugo. - -**Versioning**: Do you have a clearly documented method for versioning your content? - -Yes, releasing a new version is scripted. The repository contains instructions. The -latest three released versions are available on the website. - -#### Content creation processes - -**Content creation process**: Is there a clearly documented (ongoing) contribution -process for documentation? - -There are instructions for building and releasing the documentation, as well as detailed -instructions for contributing instructional videos. However, there are no instructions -for contributing technical documentation or fixing documentation issues. - -Presumably a contributor can submit a pull request on the website repo to amend or -add to the documentation, but there is no procedure documented. There is a notice -that no grammar or typo fixes are accepted from accounts less than a year old. - -**Release process**: Does your code release process account for documentation creation -& updates? - -No, neither the [Contribute](https://vitess.io/docs/contributing/) documentation nor -the [CONTRIBUTING.md](https://github.com/vitessio/vitess/blob/main/CONTRIBUTING.md) -file in the product repo describes how to contribute documentation. There is no Contribution -section in the website repo. - -**Review and approval**: Who reviews and approves documentation pull requests? - -Unknown. The Governance document gives detailed descriptions of the User, Contributor, -and Maintainer roles. It does not mention documentation explicitly except as one of -the contributor tasks. - -**Site owner and maintainer**: Does the website have a clear owner/maintainer? - -No, the main project repo lists maintainers, along with areas of expertise. None of -them lists documentation as an area of expertise. There is no owner information on -the website repo. - -#### Inclusive language - -**Non-inclusive phrases**: Are customer-facing utilities, endpoints, class names, -or feature names free of non-recommended words as documented by the -[Inclusive Naming Initiative](https://inclusivenaming.org) website? - -Not entirely, but there have clearly been attempts to correct non-inclusive language -on the site. For example, “primary” replaces “master” to describe the database of -record in the Vitess product. Some non-inclusive language remains. Examples: -“sanity check”; “master database” (these can be found using the website's Search function). - -Of course, some of this language exists in product command elements and outputs that -cannot be changed in the documentation without first eliminating them in code (command -line options, for example). - -**Ableist language**: "Is the project free of language like 'simple', 'easy', etc.?" - -No. A few examples exist. These should be evaluated and replaced on a case by case -basis. - -### Recommendations - -#### Information architecture - -**Get Started** - -Write an introduction on the [Get Started](https://vitess.io/docs/21.0/get-started/) -landing page. This introduction should outline a path for new users something like -the following: - -1. Install test environment -2. Configure test environment -3. Test Vitess - -After this familiarization process, there should be a a pointer to a User Guide processes -that walk the reader through the product's "happy path" in a more or less linear path -(a series of tasks). That might look something like this: - -1. Plan production installation -2. Install production version -3. Configure production environment -4. Populate databases -5. Run queries -6. Maintain and add cells/shards/databases. - -The "Next steps" sections on the test installation pages should point readers to new-user -configuration and testing procedures. - -The "Next steps" section on the production installation page should point users to -whatever is next in the User Guide documentation. This can be more than one path. -Make it conditional based on the task a reader might want to accomplish after installation: -Test the installation? Plan a cluster? Something else? - -**Conceptual Info** - - Clarify the beginning of the [Overview](https://vitess.io/docs/21.0/overview/whatisvitess/) -("What is ...") The introduction from the README in the [code repository][project-website] -is pretty good: - -> Vitess is a cloud-native horizontally-scalable distributed database system that -is built around MySQL. - -Or the introduction that this replaced: - -> Vitess is a database clustering system for horizontal scaling of MySQL through generalized -sharding. - -**Tasks** - -The User Guides page *says* "Task-based guides for common usage scenarios". That's -the right idea, but implementation must be improved if readers are to find and carry -out the documented tasks. - -The User Guides require three types of changes to maximize their effectiveness: -- Rename -- Rewrite -- Reorganize - -*Rename* - -Rename the individual pages in the User Guides to reflect the tasks that are described -there. Use verbs ending in "-ing" (along with a noun that describes the object of -the task, if applicable). This helps readers find the right content in at least two -ways: -- Makes the TOC more coherent -- Gives meaningful results in the sitewide Search - -Here are three examples from the -[same page] -(https://vitess.io/docs/21.0/user-guides/configuration-basic/global-topo/) -(the content of these sections is not considered here): -- *Choosing a TopoRoot*: A good title. Describes the task ("choosing") and the object - of the task ("TopoRoot"). -- *Moving to a different TopoServer*: A good title. Describes the task ("moving"). - Normally I'd recommend a more direct reference to the object -- - "Moving TopoServers" -- but in this case the phrase ".. to a different TopoServer" - removes ambiguity. -- *Backups*: Not a helpful title. It's not clear or what you're backing up or what - what aspect of the backups you're talking about. I'd change this to "Backing - up TopoServer data". - -Also on the Global TopoServer page, by the way: - -> The following command line options are required for every Vitess component: -> -> ```--topo_implementation=etcd2 --topo_global_server_address= - --topo_global_root=/vitess/global``` -> -> To avoid repetition we will use in our examples to signify the above - flags. - -Remove this. The "" placeholder does not seem to have been implemented. -There are no mentions of " elsewhere in the documentation, and in any -case each would have to refer back to this page. - -*Rewrite* - -A reader typically consults the User Guide to find out how to do something. The User -Guides should consist primarily of procedures. - -There are some Vitess features for which the User Guide gives a description but does -not provide adequate instructions for their use. - -For example, look at [Creating a Backup](https://vitess.io/docs/21.0/user-guides/operating-vitess/backup-and-restore/creating-a-backup/) - -This page appears to be well named ("Creating a Backup" is a descriptive task title) -and properly placed (it resides in a logical location in the User Guide). However, -the page itself doesn't contain an actual command or set of steps to back up a database: -There is no actual command line given between *Configuration* and -*Common Errors and Resolutions* The page's author might assume that it's implied, -but it should be presented explicitly as a step in the procedure. - -Further down the page, another backup option, *Using mysqlshell*, has the same shortcomings: -No actual command is presented. - -Instead, write these pages as step-by-step descriptions of how to perform a task. -Each step should be a concise instruction, with any required instruction or text illustrated -and explained (much of the time, this is a monospace code/CLI block displaying the -command-line instruction to use). This can be an example, but only if it is obvious -how the reader should use the command. Other times, it means showing the required -form of the command, with placeholders for parameters, and explaining those parameters -in the following text. - -The [Creating a cell](https://vitess.io/docs/21.0/user-guides/configuration-basic/create-cell/) -page shows a CLI command: - -> ``` -> vtctldclient AddCellInfo \ -> --root /vitess/cell1 \ -> --server-address \ -> cell1 -> ``` - -It's not clear if `/vitess/cell1` is a user-definable parameter or not. The server -address placeholder, ``, is not defined in the text. - -Here's how I'd rewrite it, defining placeholders for the parameters: - -> ``` -> vtctldclient AddCellInfo \ -> --root \ -> --server-address \ -> cell1 -> ``` -> -> where: -> - is the root directory of the server installation -> - is the IP address of the topo server - -(or whatever the actual descriptions of the parameters are.) - -Here's an outline for a one-page procedure writeup: - -- Title ("ABCing XYZ") - - Context (Describe where the procedure fits in the overall use case) - - Prerequisites (Hardware and software requirements, dependencies, procedures that - must be completed first) - - Procedure ("To ABC an XYZ, do the following.") - - Step 1 (*One* CLI command, or action. Don't combine actions.) - - Step 2 (etc.) - - Result (Optional - include only if there's something remarkable about the outcome.) - - Next Steps (What procedure or use case typically follows this. If it depends on - context, explain the options.) - -*Reorganize* - -Ensure that the various sections of the User Guide cover all usage scenarios. Reorganize -the User Guide: - -First, organize by user role, if there are distinct roles. As I understand it, there -are basically three user roles in Vitess: -- Vitess admin -- Database admin -- Application developer - - User roles traditionally are the basis for a "User Guide": -- Admin Guide -- DBA Guide -- Programmer's Guide -- etc. - -Within roles, organize the tasks around use cases. Don't be afraid to split up tasks -that use the same tool (CLI or other). In other words, pick and choose commands and -actions that server a use case rather than trying to document everything you can do -with the command in one place. (There is a place to exhaustively document a tool, -but that's in the Reference, not a User Guide.) - -The existing Vitess User Guides are already partially organized around user roles, -but they can be regrouped. In any case, make the user roles explicit: - -*Vitess admin*: -- Configuration -- Running in production -- Operational -- Migration - -*DBA*: -- VSchema and Query Serving -- SQL statement analysis -- Making schema changes - -*Application programmer*: -- Query serving -- Making schema changes (don't duplicate the section here. Instead provide links to - any tasks that are identical) -- Query optimization - -(Again, these are my understanding of the Vitess user roles. Adjust the details if -they're different. - -**Troubleshooting** - -Consolidate all troubleshooting information into a Troubleshooting Guide. Or, create -one troubleshooting guide per user guide/user role. In either case, create an escalation -path for any problems with a task (the escalation path might be: -*troubleshooting procedure > Slack Channel > project Issue*). Get rid of the -[VReplication FAQ](https://vitess.io/docs/21.0/reference/vreplication/faq/) -in the reference section and put the information in a troubleshooting section. - -**Glossary** - -Write a glossary. This is different from the "Concepts" page – the explanations of -terms is less in-depth. The glossary contains not just key terms but any word or phrase -that the reader might not know: abbreviations and acronyms, definitions of Vitess-specific -terms, and explanations of jargon used in Vitess ("topo", for example). - -**Other recommendations** - -Rename ["Reference > Programs"](https://vitess.io/docs/21.0/reference/programs/) to -"Command line reference" or "Tools reference". Consider splitting into two or more -sections by type of application: by function or user role. CLIs should be labeled -as such and separated from GUI tools. - -#### New user content - -Is there an alternative to Kubernetes for running in production? Explain. Give an -explicit list of supported OSes. This can be done by expanding the Get Started landing -page (see [Information Architecture > Get Started](#information-architecture)). - -Rewrite installation instructions as step by step procedures. - -Expand the production install -[Next Step](https://vitess.io/docs/21.0/get-started/operator/#next-steps) -section to accomodate different configurations (conversion, green field, etc.). Differentiate -between Next Steps for test/development installations and production installation. - -Put enough information on the Get Started landing page to orient new users as described -in [Information Architecture > Get Started](#information-architecture). - -Rename sections so that it's easier to find the right page in Search: -- Use "-ing" verbs for task pages as previously described -- Use the word "reference" in CLI references -- Use nouns for architectural components, features, and concepts - -#### Content maintainability & site mechanics - -No recommendations. Technical aspects of infrastructure and maintenance seem excellent. - -#### Content creation processes - -There is no documnetation for website and tech doc content creation. Such documentation -could include: -- Instructions for documenting tasks associated with new features -- Instructions for fixing documentation issues -- Templates for task writeups and command references -- Guidelines for amending conceptual information with new features -- A style manual -- A review and approval process, separate from the code process -- A link from the code contribution instructions to the doc instructions -- A maintainer designated to be responsible for website maintenance and for documentation -changes and updates -- Getting started instructions for new documentation contributors - -In practice, open source projects rarely contain all this information. At a minimum, -document the following processes for contributors: -- Documenting a new release - - Generating, testing, and deploying the new release docs, including how to publish -release notes - - Demoting, archiving, or deleting down-level releases -- Creating a doc issue -- Fixing and closing a doc issue -- How to contact the website and documentation maintainer with questions -- Getting started instructions for new documentation contributors - -These processes should address documentation-specific concerns, not just parrot the -code release procedures (although there may be many of the same steps). - -#### Inclusive language - -Search the document for non-inclusive terms, especially tier-1 and tier-2 terms per -the [Inclusive Naming Initiative](https://inclusivenaming.org/). Replace with recommended -language where possible. - -## Contributor documentation - -Vitess is a **graduated** project of CNCF. This means that the project should -have [_very high_][criteria] standards for documentation. - -| Criterion | [Rating (1-5)] | -| ----------------------------------------- | -------------- | -| Communication methods documented | 3. Meets standards | -| Beginner friendly issue backlog | 2. Needs improvement | -| “New contributor” getting started content | 2. Needs improvement | -| Project governance documentation | 4. Meets or exceeds standards | - -### Comments - -The following sections contain brief assessments of each element of the -Contributor Documentation rubric. - -#### Communication methods documented - -**Text communication channel**: Is there a Slack/Discord/Discourse/etc. community -and is it prominently linked from your website? - -Yes, the community page lists a Slack channel. - -**Repository link**: Is there a direct link to your GitHub organization/repository? - -Yes, the community page and the footer list the GitHub page for the product. - -**Project meetings**: Are weekly/monthly project meetings documented? Is it clear -how someone can join those meetings? - -Yes, the community page lists the monthly community meetings. - -**Mailing lists**: Are mailing lists documented? - -No, there does not appear to be an email list. - -#### Beginner friendly issue backlog - -**Issue triage**: Are docs issues well-triaged? - -No, there does not appear to be any mechanism for prioritizing issues in the GitHub -repo. - -**Good first issues**: Is there a clearly marked way for new contributors to make -code or documentation contributions (i.e. a “good first issue” label)? - -Yes, there is a “good first issue” label in the website repo issues list. - -**Issue documentation**: Are issues well-documented (i.e., more than just a title)? - -Yes, issues seem to have at least a paragraph description. However, there is no issues -template or guidelines for documenting issues, so the quality of the descriptions -is inconsistent. - -**Issue freshness**: Are issues maintained for staleness? - -No, the oldest open issue is over 6 years old. - -#### New contributor getting started content - -**Community featured on website**: Do you have a community repository or section on -your website? - -Yes, there is a Community link in the site’s menu bar, leading to the Community page. - -**New contributor document**: Is there a document specifically for new contributors/your -first contribution? - -No, I could not find a new contributor's document. - -**New user help**: Do new users know where to get help? - -There is no prominent document to guide new users. - -#### Project governance documentation - -**Governance documentation**: Is project governance clearly documented? - -Yes, there are Governance and Code of Conduct documents in the project repo. - -### Recommendations - -#### Communication methods documented - -No recommendations. - -#### Beginner friendly issue backlog - -Create an issue template to ensure that issue descriptions are consistent and complete. - -Go through the issues backlog and close or update old issues. - -#### New contributor getting started content - -Write a "new contributors" document. Post prominently. - -#### Project governance documentation - -No recommendations. - -## Website and infrastructure - -Vitess is a **graduated** project of CNCF. This means that the project should -have [_very high_][criteria] standards for documentation. - -| Criterion | [Rating (1-5)] | -| ------------------------------------------- | -------------- | -| Single-source for all files | 5. Exemplary | -| Meets min website req. (for maturity level) | 2. Needs improvement | -| Usability, accessibility, and design | 3. Meets standards | -| Branding and design | 4. Meets or exceeds standards | -| Case studies/social proof | 2. Needs improvement | -| SEO, Analytics, and site-local search | TBD | -| Maintenance planning | 2. Needs improvement | -| HTTPS access & HTTP redirect | 5. Exemplary | - -### Comments - -The following sections contain brief assessments of each element of the Website -and documentation infrastructure rubric. - -#### Single source - -**Single source for docs**: Does the project have a single source for documentation? -If not, is there a reason? - -Yes, the website and tech doc content are all in one GitHub repo. - -#### Minimal website requirements - -**Website guidelines**: Are all guidelines satisfied? - -Yes, the website mostly meets all criteria for a CNCF graduated website, including -hosting, copyright, CNCF branding, and trademark. - -**Docs analysis**: Are all follow-up actions addressed? - -Pending: There is no reason to believe the Vitess team won’t follow through on recommendations -in this analysis. - -**Project doc: stakeholders**: Are all stakeholder needs identified? - -No, roles are not explicitly defined in the documentation. There is some differentiation -among roles implicit in the docs (admin permissions, RBAC support, etc.), but it is -not used to organize the content. - -**Project doc: hosting**: Hosted directly. - -Yes, the site is hosted on Netlify. - -**Project doc: user docs**: Fully addresses needs of key stakeholders? - -Yes, the documentation probably addresses all stakeholder needs, but is not organized -so that users can find the docs efficiently. - -#### Usability, accessibility and devices - -**Mobile**: Is the website usable from mobile? - -Yes, the site seems to adapt well to small screen use. - -**Readability**: Are doc pages readable? - -Yes, documentation pages are readable on all tested platforms and screen sizes. - -**Mobile navigability**: Are all / most website features accessible from mobile -- -such as the top-nav, site search and in-page table of contents? - -Most website features are accessible from mobile. - -**Mobile-first design**: Might a [mobile-first] design make sense for your project? - -No, this project might occasionally be accessed on mobile but it doesn't seem likely -to be the main use case. - -**Color contrast**: Are color contrasts significant enough for color-impaired readers? - -Yes, text is black on white. Font is a very legible sans serif. - -**Keyboard-only**: Are most website features usable using a keyboard only? - -Yes, most features are usable. However, tab navigation is awkward and time-consuming. - -**Text-to-speech**: Does text-to-speech offer listeners a good experience? - -Text-to-speech was not tested. - -#### Branding and design - -**Recognizable brand**: Is there an easily recognizable brand for the project -(logo + color scheme) clearly identifiable? - -Yes, the site has a consistent design. The orange “layered sheets” logo is recognizable. - -**Consistent branding**: Is the brand used across the website consistently? - -Yes, branding is used cosistently. - -**Typography**: Is the website’s typography clean and well-suited for reading? - -Yes, the font is a little small in places, but overall legible. Font scales gracefully -when magnified in the browser. - -#### Case studies & social proof - -**Case studies**: Are there case studies available for the project and are they documented -on the website? - -No, I did not see any case studies. Some of them many videos might contain case studies, -though. - -**Testimonials**: Are there user testimonials available? - -No, there do not seem to be testimonials available on the site. The logo wall does -not have live links. - -**Active blog**: Is there an active project blog? - -Yes, there is a blog. The last entry was two months ago, but it seems to be updated -semi-regularly. - -**Community talks listed**: Are there community talks for the project and are they -present on the website? - -Yes, there are many videos on the “Learning Resources” page, containing a variety -of topics including community talks. - -**Logo wall**: Is there a logo wall of users/participating organizations? - -Yes, there is a substantial logo wall on the product landing page. - -#### SEO, Analytics and site-local search - -**Analytics on production site**: Analytics: Is analytics enabled for the production -server? -- TBD -- TBD - -**Analytics disabled for non-production sites**: Analytics: Is analytics disabled -for all other deploys? -- TBD -- TBD - -**Google analytics: GA4**: Analytics: If your project used Google Analytics, have -you migrated to GA4? -- TBD -- TBD - -**404 reports**: Analytics: Can Page-not-found (404) reports easily be generated from -you site analytics? Provide a sample of the site's current top-10 404s. -- TBD -- TBD - -**Site indexing on production server**: Is site indexing supported for the production -server? -- TBD -- TBD - -**Site indexing disabled on non-default content**: Is site indexing disabled for website -previews and - TBD -- TBD - -**Intra-site search**: Is local intra-site search available from the website? - -Yes, the site text search funciton works well. - -**Analytics custodians documented**: Are the current custodian(s) of the following -accounts clearly documented: analytics, Google Search Console, site-search (such as -Google CSE or Algolia) - -No documentation maintainers of any sort are credited. - -#### Maintenance planning - -**Well supported website tooling**: Is your website tooling well supported by the -community (i.e., Hugo with the Docsy theme) or commonly used by CNCF projects (our -recommended tech stack?) - -Yes, the site is implented using Hugo, CNCF's recommended site generator, and hosted -on Netlify. - -**Cultivating website maintainers**: Are you actively cultivating website maintainers -from within the community? - -Unknown. There is no documented evidence of recruitment on the website or in the repository. - -**Site build times**: Are site build times reasonable? - -I did not try building the site but have no reason to believe that the build time -is excessive. - -**Maintainer permissions**: Do site maintainers have adequate permissions? - -Unknown. - -#### Other - -**HTTPS**: Is your website accessible via HTTPS? - -Yes, all site pages use HTTPS. - -**HTTP redirect**: Does HTTP access, if any, redirect to HTTPS? - -Yes, pages requested using HTTP are redirected to existing HTTPS pages seamlessly. - -### Recommendations - -#### Single-source requirement - -No recommendations. - -#### Minimal website requirements - -Identify stakeholder and user roles. Orgainize task documentation around user roles -(see [Information architecture](#information-architecture)). - -#### Usability, accessibility and devices - -No recommendations. - -#### Branding and design - -No recommendations. - -#### Case studies/social proof - -Include case studies and testimonials on the product website. If these are already -among the video content, tag them and feature them on the landing page. - -Update blog content at least monthly. - -#### SEO, Analytics and site-local search - -TBD - -List documentation maintainers in the website repository. - -#### Maintenance planning - -TBD - -#### Other - -No recommendations. - -## References and notes - -### Rating values - -The numeric rating values used in this document are as follows: - -1. Not present -2. Needs improvement -3. Meets standards -4. Meets or exceeds standards -5. Exemplary - -### References - -[criteria]: ../criteria.md -[implementation]: ./implementation.md -[issues list]: ./issues-list.md -[project-doc-website]: ?https://vitess.io/docs/ -[project-website]: ?https://vitess.io/ -[Rating (1-5)]: #rating-values -[rfc-spec]: https://www.rfc-editor.org/rfc/rfc2119 -[website guidelines]: ../../website-guidelines-checklist.md -[Bulma]: https://bulma.io/ From 4a5282cd1c34ce9996479b660a7d6794ed10adff Mon Sep 17 00:00:00 2001 From: Patrice Chalin Date: Tue, 25 Feb 2025 14:13:48 -0500 Subject: [PATCH 4/6] Fix formatting Signed-off-by: Patrice Chalin --- docs/analysis/criteria.md | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/docs/analysis/criteria.md b/docs/analysis/criteria.md index e26a3f3..52576ad 100644 --- a/docs/analysis/criteria.md +++ b/docs/analysis/criteria.md @@ -1,19 +1,18 @@ # Assessment criteria and examples We recommend reading the -[CNCF sandbox project documentation primer](../sandbox-doc-primer.md) -before reviewing these criteria. The primer contains key definitions concepts -that are referred to in the criteria. +[CNCF sandbox project documentation primer](../sandbox-doc-primer.md) before +reviewing these criteria. The primer contains key definitions concepts that are +referred to in the criteria. ## Project documentation ### Information architecture -How your technical information is organized, laid out, and presented. -This includes structure (pages/subpages/sections/subsections), information -types (conceptual/instructional/reference), and matching documentation -content -to user expectations. +How your technical information is organized, laid out, and presented. This +includes structure (pages/subpages/sections/subsections), information types +(conceptual/instructional/reference), and matching documentation content to user +expectations. We evaluate on the following: @@ -85,7 +84,7 @@ We evaluate on the following: - Is release-specific information documented in release notes? - Is the documentation free of duplicate or nearly duplicated sections of information? -- Do informational graphics add value by providing information in a way that +- Do informational graphics add value by providing information in a way that would be difficult in text? - Will graphics require frequent modifications due to software changes, GUI updates, or translation? @@ -96,8 +95,8 @@ Example: ### Content creation processes -Documentation is only useful if it is accurate and well maintained. It -requires the same kind of review and approval processes as code. +Documentation is only useful if it is accurate and well maintained. It requires +the same kind of review and approval processes as code. We evaluate on the following: @@ -233,7 +232,6 @@ We evaluate on the following: - Incubating: Is the [docs assessment][] comprehensive, addressing most stakeholder needs? Graduated: Fully addresses needs of key stakeholders? - [archived state]: https://docs.github.com/en/repositories/archiving-a-github-repository/archiving-repositories [docs assessment]: ./howto.md @@ -298,8 +296,7 @@ organizations using it. We evaluate on the following: -- Are there case studies available for the project and are they documented - on +- Are there case studies available for the project and are they documented on the website? - Are there user testimonials available? - Is there an active project blog? From e1644182c5b5e294ba1b632470daee127ac22e37 Mon Sep 17 00:00:00 2001 From: Nate W Date: Wed, 26 Feb 2025 14:09:14 -0800 Subject: [PATCH 5/6] Updates based on PR feedback. Co-authored-by: Dave Welsch Co-authored-by: Nate W Signed-off-by: Nate W --- docs/analysis/criteria.md | 105 +++++++++++++++++++++++++++----------- 1 file changed, 76 insertions(+), 29 deletions(-) diff --git a/docs/analysis/criteria.md b/docs/analysis/criteria.md index 3ffed0f..a8831cd 100644 --- a/docs/analysis/criteria.md +++ b/docs/analysis/criteria.md @@ -1,24 +1,23 @@ # Assessment criteria and examples We recommend reading the -[CNCF sandbox project documentation primer](../sandbox-doc-primer.md) before -reviewing these criteria. The primer contains key definitions concepts that are -referred to in the criteria. +[CNCF sandbox project documentation primer](../sandbox-doc-primer.md) +before reviewing these criteria. The primer contains key definitions concepts +that are referred to in the criteria. ## Project documentation ### Information architecture -How your technical information is organized, laid out, and presented. This -includes structure (pages/subpages/sections/subsections), information types -(conceptual/instructional/reference), and matching documentation content to user -expectations. +How your technical information is organized, laid out, and presented. +This includes structure (pages/subpages/sections/subsections), information +types (conceptual/instructional/reference), and matching documentation +content to user expectations. We evaluate on the following: -- Is there high level conceptual/“About” content? -- Is the documentation feature-complete? (i.e., every product feature is - documented) +- Is there high level conceptual content? +- Is every product feature documented? - Does the documentation define all user roles (personas) for the product? - Are there instructions (tasks, tutorials) documented for features? - Are there instructions for all major use cases for each user role? @@ -60,7 +59,10 @@ We evaluate on the following: - Do users know where to go after reading the getting started guide? - Is your new user content clearly signposted on your site’s homepage or at the top of your information architecture? -- Is there sample code or other example content that can easily be copy-pasted? +- Is there easily copy-pastable sample code or other example content? + + + Example: @@ -211,23 +213,63 @@ We evaluate on the following: ### Minimal website requirements Listed here are the _minimal_ website requirements for projects based on their -[maturity level]: sandbox, incubating, graduated and archived. - -Except for archived projects, requirements are cumulative through project -maturity levels. So, for example, graduated projects must satisfy the -requirements for incubating projects. +[maturity level][]: sandbox, incubating, graduated and archived. We evaluate on the following: -- Incubating: Are all [website guidelines][] satisfied? Graduated: Are all - guidelines satisfied? -- Incubating: Was a [docs assessment][] requested through CNCF service desk? - Graduated: All follow-up actions addressed? -- Incubating: User roles identified and doc needs documented? Graduated: All - stakeholder need identified? -- Incubating: Website hosted directly? Graduated: Hosted directly? -- Incubating: Is the [docs assessment][] comprehensive, addressing most - stakeholder needs? Graduated: Fully addresses needs of key stakeholders? +*Note: Except for archived projects, requirements are cumulative through project +maturity levels so, for example, incubating projects must satisfy the +requirements for sandbox projects.* + + +#### Sandbox + +- Are a majority of the [Website guidelines](../website-guidelines-checklist.md) + satisfied? + - "Guidelines" is misleading; these are mandatory standards concerning: + - Maintaining open source status + - IP licensing + - Corporate sponsorship and support + - Copyright and trademark + - Community standards + - **Plan ahead**: projects must meet all guidelines before being promoted to Incubating status. +- Is there rudimentary **project documentation**, or a placeholder or substitute? + - It is acceptable at this maturity level to link out to documentation that + hasn't yet been integrated into the website. + - _Example_: website with a single homepage, without any documentation or, + as was mentioned above, linking out to an external (preexisting) source + for docs. + - _However_: consider reading the recommended practices in this repository + and implementing as many of the best practices as you can. This groundwork + will pay big dividends later when you need to upgrade your practices and + update your documentation as an incubating project. Assistance is + available from CNCF TechDocs anytime, including answers to individual + questions or a documentation workshop. + +#### Incubating + +- Are all [Website guidelines][] satisfied? +- Has a [Docs assessment][] or reassessment been requested through the CNCF + [service desk][]? +- Does **project documentation** meet these standards?: + - Stakeholders / personas are identified and their core needs (in terms of + docs) documented. + - Hosted directly on the website. + - The [single-source requirement][] met. + - Doc is comprehensive, addressing most stakeholder needs. + +#### Graduated + +- Are follow-through actions from the [Docs assessment][] complete? +- Does **project documentation** fully addresses the needs of key stakeholders? + +#### Archived + +- Is the website repo in an [archived state][]? +- Is the archived status of the project obvious to those visiting the + website, such as through the use of a prominent banner? +- If a successor project exists, are there links to its website and/or migration + documentation. [archived state]: https://docs.github.com/en/repositories/archiving-a-github-repository/archiving-repositories @@ -250,7 +292,7 @@ We evaluate on the following: - Are doc pages readable? - Are all or most website features accessible from mobile -- such as the top-nav, site search, and in-page table of contents? -- Might a [mobile-first] design make sense for your project? +- Might a [mobile-first][] design make sense for your project? - Are color contrasts significant enough for color-impaired readers? - Are most website features usable using a keyboard only? - Does text-to-speech offer listeners a good experience? @@ -258,7 +300,7 @@ We evaluate on the following: [mobile-first]: https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps/Responsive/Mobile_first -Plan for suitable [accessibility] measures for your website. For example: +Plan for suitable [accessibility][] measures for your website. We evaluate on the following: @@ -293,7 +335,8 @@ organizations using it. We evaluate on the following: -- Are there case studies available for the project and are they documented on +- Are there case studies available for the project and are they documented + on the website? - Are there user testimonials available? - Is there an active project blog? @@ -348,4 +391,8 @@ Example: - Is your website accessible via HTTPS? - Does HTTP access, if any, redirect to HTTPS? -- Are links to external websites or applications working and current? +- Are links working and current? + - Internal documentation + - External documentation + - External websites + - Web applications From 24b1311e6755d8f99a9a21301c9048002f658551 Mon Sep 17 00:00:00 2001 From: Nate W Date: Wed, 26 Feb 2025 14:16:24 -0800 Subject: [PATCH 6/6] fixing failing tests. Signed-off-by: Nate W --- docs/analysis/criteria.md | 50 +++++++++++++++++++-------------------- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/docs/analysis/criteria.md b/docs/analysis/criteria.md index a8831cd..51c7870 100644 --- a/docs/analysis/criteria.md +++ b/docs/analysis/criteria.md @@ -1,18 +1,18 @@ # Assessment criteria and examples We recommend reading the -[CNCF sandbox project documentation primer](../sandbox-doc-primer.md) -before reviewing these criteria. The primer contains key definitions concepts -that are referred to in the criteria. +[CNCF sandbox project documentation primer](../sandbox-doc-primer.md) before +reviewing these criteria. The primer contains key definitions concepts that are +referred to in the criteria. ## Project documentation ### Information architecture -How your technical information is organized, laid out, and presented. -This includes structure (pages/subpages/sections/subsections), information -types (conceptual/instructional/reference), and matching documentation -content to user expectations. +How your technical information is organized, laid out, and presented. This +includes structure (pages/subpages/sections/subsections), information types +(conceptual/instructional/reference), and matching documentation content to user +expectations. We evaluate on the following: @@ -217,10 +217,9 @@ Listed here are the _minimal_ website requirements for projects based on their We evaluate on the following: -*Note: Except for archived projects, requirements are cumulative through project +_Note: Except for archived projects, requirements are cumulative through project maturity levels so, for example, incubating projects must satisfy the -requirements for sandbox projects.* - +requirements for sandbox projects._ #### Sandbox @@ -232,19 +231,21 @@ requirements for sandbox projects.* - Corporate sponsorship and support - Copyright and trademark - Community standards - - **Plan ahead**: projects must meet all guidelines before being promoted to Incubating status. -- Is there rudimentary **project documentation**, or a placeholder or substitute? + - **Plan ahead**: projects must meet all guidelines before being promoted to + Incubating status. +- Is there rudimentary **project documentation**, or a placeholder or + substitute? - It is acceptable at this maturity level to link out to documentation that hasn't yet been integrated into the website. - - _Example_: website with a single homepage, without any documentation or, - as was mentioned above, linking out to an external (preexisting) source - for docs. - - _However_: consider reading the recommended practices in this repository - and implementing as many of the best practices as you can. This groundwork - will pay big dividends later when you need to upgrade your practices and - update your documentation as an incubating project. Assistance is - available from CNCF TechDocs anytime, including answers to individual - questions or a documentation workshop. + - _Example_: website with a single homepage, without any documentation or, as + was mentioned above, linking out to an external (preexisting) source for + docs. + - _However_: consider reading the recommended practices in this repository and + implementing as many of the best practices as you can. This groundwork will + pay big dividends later when you need to upgrade your practices and update + your documentation as an incubating project. Assistance is available from + CNCF TechDocs anytime, including answers to individual questions or a + documentation workshop. #### Incubating @@ -266,8 +267,8 @@ requirements for sandbox projects.* #### Archived - Is the website repo in an [archived state][]? -- Is the archived status of the project obvious to those visiting the - website, such as through the use of a prominent banner? +- Is the archived status of the project obvious to those visiting the website, + such as through the use of a prominent banner? - If a successor project exists, are there links to its website and/or migration documentation. @@ -335,8 +336,7 @@ organizations using it. We evaluate on the following: -- Are there case studies available for the project and are they documented - on +- Are there case studies available for the project and are they documented on the website? - Are there user testimonials available? - Is there an active project blog?