Introducing: an Eleventy starter project for WCAG reports

Published 24 May 2021 category: code

When I audit websites for accessibility, I write Markdown files that Eleventy turns into reports. Today, I’ve shared the code I use for that. In this post I’ll provide some context.

At eleventy-wcag-reporter, I have published my Eleventy starter for reports about conformance to the Web Content Accessibility Guidelines (WCAG). It is issue oriented, promotes rich text, comes with a PDF export script and supports two languages (English and Dutch).

Example report shows an issue with code example, a listing of wcag criterion, severity and difficulty and what the sample is There is an example report available.

Note: this is just one specific way to create accessibility conformance reports. This works for me and my clients. Other approaches and requirements exists, and I can’t guarantee this works for you.

Issue-oriented reporting

WCAG conformance reports document whether a website meets the criteria in the WCAG standard. It will tell the reader which Success Criteria met the standard, which did not.

Many WCAG conformance reports list WCAG in its entirety and then specify the result for each. Say, a website is tested for WCAG 2.1, Level AA and 10 problems are found. The report will show all 50 Success Criteria (SCs): 10 say “Failed”, 40 “Passed” (or something else, like “Cannot Tell”). The accessibility problems are metadata of the success criteria, so to say.

Screen3 In this Eleventy project, you’ll create one file for each issue

In my personal experience, teams that commission a report are mostly interested in the actual accessibility problems. They want to know “what needs fixing”. They may also want to know: how severe are the issues in comparison, which expertise can solve it (design? development?) and which assistive technology is affected. And, where relevant, what the issue or solution looks like in code. This is why I provide clients with an issue-oriented report: a list of issues, rather than a list of SCs. The Success Criterion is added as metadata of the issue, rather than the other way around.

A benefit of the issue-oriented approach, is that it can integrate better with a team’s workflow. Issue trackers are more common in workflows than SC-trackers. It also, as mentioned, makes it easier to include other meta data like responsibility and severity, and to include code examples and screenshots.

WCAG-EM, the standard methodology for accessibility conformance reporting, does not specify either of the two approaches. What is important for a conformance report is that all Success Criteria were evaluated, and that this fact is made clear. You want to be sure that “no issue filed for an SC” really means the sample did not contain any issues for that SC.

In other words: the report is only done when you’ve checked all sample pages for all WCAG Success Criteria in your scope (i.e. all 50 criteria for WCAG 2.1, Level A + AA). Always test them all. If your report follows the SC-based approach this is explicit, in the issue-based approach it is implicit.

The power of Markdown and YAML

One of the benefits of using Eleventy for my reports, is that it makes it easy to write rich text. That may sound like a no-brainer, but in many reporting tools I’ve seen, issues are described in plain text. The main reason I want to use rich text to describe issues, is that they can make tips more clear. Lists, links, images, code examples and even text level stuff like bold and italics can make a report more readable. This is important, because it increases the chance the team can fix the issues found.

Markdown is my favourite syntactic sugar for rich text, so that’s what I use, but you can use whatever else. HTML should work, too.

For each issue that is described, there is also a set of meta data, like which Success Criterion it relates to. It is described in Yaml front matter, so at the top of an issue.md, you would add something like:

---
sc: 2.1.1
severity: High
difficulty: High
title: Main menu does not work with keyboard
sample: homepage
---

Spelling out the meta data in front matter makes it powerful. We can do things like add up the issues filed for a specific page, or count how many were filed for a given Success Criterion.

Some essential metadata in WCAG conformance reports

Pointing out accessibility problems and solutions is the bread and butter of my conformance reports, as my goal is usually to help the team fix all issues. But accessibility reports are used in various contexts. There is some important metadata that helps someone looking at the report what’s going on, including monitoring bodies like governments. They may require publication of accessibility conformance reports to help them perform their regulatory duties, good metadata aids this monitoring.

Logius, the Dutch government department that keeps track of WCAG conformance reports that Dutch governmental organisations provide, uses a checklist to verify if conformance reports meet their requirements:

The above is my translation of the checklist on the Controle door Logius page; many of these are also recommended in WCAG-EM; the requirements likely vary where you or your client are based.

The eleventy-wcag-reporter starter project has variables for each of these checkpoints. A lot of the meta data is added to the report’s index.njk as YAML front matter, so that it can be displayed.

FAQ

After reading this, you may have questions before trying to use the code. The questions below weren’t actually asked frequently, but I hope they have useful answers.

Will this find accessibility issues?
No, you find issues, create a Markdown file for each and then add some frontmatter. The code will then include it in a pretty report. Or really, a report that you can make pretty.
Why not integrate with automated tooling?
There are tools that can find issues, for ~20% of WCAG criteria (other estimates exist). You can use those tools to find issues and add them manually. It is usually easier for a human to group issues in a way that is most useful to readers of the report. For instance, if a brand’s primary colour contrasts badly with a white background, you may not need to file that more than once.
Can more languages be supported?
Yes, feel free to contribute. We need all of WCAG in JSON format (I have code to scrape) and translations for strings.
How many issues should I add to a report?
There is no minimum or maximum, but you should check all pages in your sample for all Success Criteria in your scope, and have issues for all situations where a SC is not met, in order to create a full report.
What about PDFs?
I like HTML as a report format. Some organisations prefer to receive a PDF document. Like web pages, PDFs also need to be accessible, and simply pressing “Save as PDF“ in a browser will be unlikely to include all semantics. So I use PrinceXML, which supports a lot of print CSS. There are licenses for a machine or server, or you can use the SaaS version DocRaptor. The starter project has a Shell script to generate a PDF with PrinceXML, you will need a license or end up with a watermark in your resulting PDF.

Wrapping up

So, find my starter project at eleventy-wcag-reporter, please don’t laugh at my code. Feedback is very much welcomed by email, DMs or in GitHub. I may not be able to get back very quickly, as the open version of this is a side project. Happy accessbility conformance report writing!

Thanks to Roel, Peter, Rian, Jeroen and many others for discussions, ideas and suggestions related to accessibility conformance reports. (Thanks do not imply endorsements)

Comments, likes & shares (186)

Patrick H. Lauke 03 Jun 2021 00:53:08

Not sure if this handles this sort of thing already, but: sometimes, you have complex “compound” issues (or rather, situations where it makes more sense to coalesce various atomic failures into a single “this component is borked in quite a few ways…just go and use the correct ARIA pattern” or similar). Does this system allow for issues to be associated with more than one SC?

Myles Lewando replied: Ooooo nice!
Peter van Grieken replied: I’ve been using this project for a while and it has made writing WCAG reports so much easier for me.
Peter van Grieken replied: I’ve been using this project for a while and it has made writing WCAG reports so much easier for me.
Sophie replied: wow!
Wassim Chegham 🌱 replied: Hey @eleven_ty folks, check this out!
Wassim Chegham 🌱 replied: That's great! Thank you for sharing.
Robbert replied: I love that every issue can be linked to with the # fragment identifier! Would like to link to relevant issues on the component backlog for NL Design System. No more PDFs!
Jules Ernst replied: Just missing the installation for very dummies. It's me I know, but tried hard to get it installed with no success.
Hidde replied: heb de ”Set up” zojuist vast ietsje uitgebreid, maar ik stuur je ff een DM voor support! :-)
Sara Soueidan replied:
An @eleven_ty starter project for WCAG reports, kindly open-sourced by @hdv Code: github.com/hidde/eleventy… Intro & about: hiddedevries.nl/en/blog/2021-0… #a11y
Colin replied: nice work!
Hidde replied: thank you!
Peter replied: Good work pal. I’ll look forward to kicking the tyres on this 👍🏻
Allen vaccinated Hoffman! replied: Nicely done. Take a look at ACRT Accessibility Conformance Reporting Tool from the U.S. FEeral government. It uses JSON to store the test checkpoints and also the report data for sharing.
Danisaurio replied: Hi! Loved your project and I'll add spanish support :) thanks for doing this!
Sun lover replied: This is an excellent #ally resource! Thank you so much for sharing.
bertrandkeller replied: hiddedevries.nl/en/blog/2021-0… - belle initiative, mais un peu compliqué à suivre dans le temps. RGAA C’est 106 critères, 106 fichiers MarkDown a éditer, c’est un peu lourd.
Shawn Thompson replied: Thanks @mgifford for sharing this with me!
Priya replied: Introducing: an Eleventy starter project for WCAG reports hiddedevries.nl/en/blog/2021-0…
Pablo Lara H replied: Introducing: an Eleventy starter project for WCAG reports by Hidde de Vries @hdv #Eleventy #Eleventystarterproject #WCAG #webdev #WebContentAccessibilityGuidelines hiddedevries.nl/en/blog/2021-0…
Max Antonucci replied: So @hdv shared their @eleven_ty template for WCAG reports, and it is pretty damn awesome - hiddedevries.nl/en/blog/2021-0… #a11y #accessibility
Melanie Sumner replied: This is very cool.
Ben Myers 🦖 replied: Ohhh, I love this.
Bruno Pulis replied: Awesome initiative @hdv soon as possible I'll send a translation to Brazilian portuguese! :)
Web Axe replied: An Eleventy starter project for WCAG reports hiddedevries.nl/en/blog/2021-0… #wcag #a11y #dev #reports
Hidde replied: Haha this sounds very close to home! Makes a lot of sense.
Carie Fisher replied: 😻😻😻
Joe Lanman replied: awesome, thanks!
The A11Y Project replied: "When I audit websites for accessibility, I write Markdown files that Eleventy turns into reports. Today, I’ve shared the code I use for that." @hdv hiddedevries.nl/en/blog/2021-0…
Leave a comment
Posted a response to this?

This website uses Webmentions. You can manually notify me if you have posted a response, by entering the URL below.