Introducing: an Eleventy starter project for WCAG reports

category: code

When I audit websites for accessibility, I write Markdown files that Eleventy turns into reports. Today, I’ve shared the code I use for that. In this post I’ll provide some context.

At eleventy-wcag-reporter, I have published my Eleventy starter for reports about conformance to the Web Content Accessibility Guidelines (WCAG). It is issue oriented, promotes rich text, comes with a PDF export script and supports two languages (English and Dutch).

Example report shows an issue with code example, a listing of wcag criterion, severity and difficulty and what the sample is There is an example report available.

Note: this is just one specific way to create accessibility conformance reports. This works for me and my clients. Other approaches and requirements exists, and I can’t guarantee this works for you.

Issue-oriented reporting

WCAG conformance reports document whether a website meets the criteria in the WCAG standard. It will tell the reader which Success Criteria met the standard, which did not.

Many WCAG conformance reports list WCAG in its entirety and then specify the result for each. Say, a website is tested for WCAG 2.1, Level AA and 10 problems are found. The report will show all 50 Success Criteria (SCs): 10 say “Failed”, 40 “Passed” (or something else, like “Cannot Tell”). The accessibility problems are metadata of the success criteria, so to say.

Screen3 In this Eleventy project, you’ll create one file for each issue

In my personal experience, teams that commission a report are mostly interested in the actual accessibility problems. They want to know “what needs fixing”. They may also want to know: how severe are the issues in comparison, which expertise can solve it (design? development?) and which assistive technology is affected. And, where relevant, what the issue or solution looks like in code. This is why I provide clients with an issue-oriented report: a list of issues, rather than a list of SCs. The Success Criterion is added as metadata of the issue, rather than the other way around.

A benefit of the issue-oriented approach, is that it can integrate better with a team’s workflow. Issue trackers are more common in workflows than SC-trackers. It also, as mentioned, makes it easier to include other meta data like responsibility and severity, and to include code examples and screenshots.

WCAG-EM, the standard methodology for accessibility conformance reporting, does not specify either of the two approaches. What is important for a conformance report is that all Success Criteria were evaluated, and that this fact is made clear. You want to be sure that “no issue filed for an SC” really means the sample did not contain any issues for that SC.

In other words: the report is only done when you’ve checked all sample pages for all WCAG Success Criteria in your scope (i.e. all 50 criteria for WCAG 2.1, Level A + AA). Always test them all. If your report follows the SC-based approach this is explicit, in the issue-based approach it is implicit.

The power of Markdown and YAML

One of the benefits of using Eleventy for my reports, is that it makes it easy to write rich text. That may sound like a no-brainer, but in many reporting tools I’ve seen, issues are described in plain text. The main reason I want to use rich text to describe issues, is that they can make tips more clear. Lists, links, images, code examples and even text level stuff like bold and italics can make a report more readable. This is important, because it increases the chance the team can fix the issues found.

Markdown is my favourite syntactic sugar for rich text, so that’s what I use, but you can use whatever else. HTML should work, too.

For each issue that is described, there is also a set of meta data, like which Success Criterion it relates to. It is described in Yaml front matter, so at the top of an issue.md, you would add something like:

---
sc: 2.1.1
severity: High
difficulty: High
title: Main menu does not work with keyboard
sample: homepage
---

Spelling out the meta data in front matter makes it powerful. We can do things like add up the issues filed for a specific page, or count how many were filed for a given Success Criterion.

Some essential metadata in WCAG conformance reports

Pointing out accessibility problems and solutions is the bread and butter of my conformance reports, as my goal is usually to help the team fix all issues. But accessibility reports are used in various contexts. There is some important metadata that helps someone looking at the report what’s going on, including monitoring bodies like governments. They may require publication of accessibility conformance reports to help them perform their regulatory duties, good metadata aids this monitoring.

Logius, the Dutch government department that keeps track of WCAG conformance reports that Dutch governmental organisations provide, uses a checklist to verify if conformance reports meet their requirements:

The above is my translation of the checklist on the Controle door Logius page; many of these are also recommended in WCAG-EM; the requirements likely vary where you or your client are based.

The eleventy-wcag-reporter starter project has variables for each of these checkpoints. A lot of the meta data is added to the report’s index.njk as YAML front matter, so that it can be displayed.

FAQ

After reading this, you may have questions before trying to use the code. The questions below weren’t actually asked frequently, but I hope they have useful answers.

Will this find accessibility issues?
No, you find issues, create a Markdown file for each and then add some frontmatter. The code will then include it in a pretty report. Or really, a report that you can make pretty.
Why not integrate with automated tooling?
There are tools that can find issues, for ~20% of WCAG criteria (other estimates exist). You can use those tools to find issues and add them manually. It is usually easier for a human to group issues in a way that is most useful to readers of the report. For instance, if a brand’s primary colour contrasts badly with a white background, you may not need to file that more than once.
Can more languages be supported?
Yes, feel free to contribute. We need all of WCAG in JSON format (I have code to scrape) and translations for strings.
How many issues should I add to a report?
There is no minimum or maximum, but you should check all pages in your sample for all Success Criteria in your scope, and have issues for all situations where a SC is not met, in order to create a full report.
What about PDFs?
I like HTML as a report format. Some organisations prefer to receive a PDF document. Like web pages, PDFs also need to be accessible, and simply pressing “Save as PDF“ in a browser will be unlikely to include all semantics. So I use PrinceXML, which supports a lot of print CSS. There are licenses for a machine or server, or you can use the SaaS version DocRaptor. The starter project has a Shell script to generate a PDF with PrinceXML, you will need a license or end up with a watermark in your resulting PDF.

Wrapping up

So, find my starter project at eleventy-wcag-reporter, please don’t laugh at my code. Feedback is very much welcomed by email, DMs or in GitHub. I may not be able to get back very quickly, as the open version of this is a side project. Happy accessbility conformance report writing!

Thanks to Roel, Peter, Rian, Jeroen and many others for discussions, ideas and suggestions related to accessibility conformance reports. (Thanks do not imply endorsements)

Comments & mentions (185)

Patrick H. Lauke 03 Jun 2021 00:53:08

Not sure if this handles this sort of thing already, but: sometimes, you have complex “compound” issues (or rather, situations where it makes more sense to coalesce various atomic failures into a single “this component is borked in quite a few ways…just go and use the correct ARIA pattern” or similar). Does this system allow for issues to be associated with more than one SC?

Ross Mullen likes this
Terry O'Brien likes this
Sara Soueidan likes this
Joe Lanman likes this
Matthias Ott likes this
Myles Lewando likes this
Michael Hastrich likes this
Starfall Projects likes this
Anneke Sinnema likes this
Ronald Aveling likes this
Gaël Poupard likes this
Sarah Federman likes this
Sami Keijonen likes this
Wassim Chegham 🌱 likes this
Peter van Grieken likes this
Matthew Holloway | Black Lives Matter likes this
Guillaume Deblock likes this
Princiya 👧‍ likes this
Martin Laker Ockovsky likes this
Robert Spier likes this
Tommie Woef likes this
Alicia Jarvis (She/Her/Hers), CPACC, CSM 🇨🇦 likes this
Ruben Bos likes this
Taeke likes this
Arturo Castillo Delgado likes this
Grigor 🇵🇸 likes this
Søren Birkemeyer 🦊 likes this
Darek Kay likes this
Stephanie Eckles likes this
Tomas Pustelnik likes this
Katie Langerman likes this
Joan Joly likes this
Adam Juran likes this
Francesco Giudice likes this
Karmugil likes this
Jens Geiling likes this
PK 🐢 👩🏻‍💻 likes this
Fotis Papadogeorgopoulos likes this
Brett Jankord likes this
emily 🪱🌔 likes this
Bernard Nijenhuis likes this
Dan likes this
Towhid likes this
Vincent likes this
Gaëtan likes this
Pat Ramsey “it STILL goes over the nose” likes this
Matsuko likes this
Mario Vasquez likes this
Rafael Calvo 🔰 likes this
Jess Peck 🐍👾✨ likes this
👻 Zac Vineyard likes this
Ruben Nic likes this
Erwin Verbruggen likes this
Florian Geierstanger likes this
Dana Byerly likes this
Eleventy likes this
Phil Wolstenholme likes this
Phillip Faniola likes this
mason conkright likes this
corey benson likes this
lisica likes this
ned likes this
Juha Karttunen likes this
Niek likes this
Jens Grochtdreis likes this
Patrick Grey likes this
Dirk Sidney Jansen likes this
Károly Szántai likes this
Jeremy Van Staalduine likes this
Sven Wolfermann likes this
Saptak Sengupta likes this
Scott McKee likes this
Bunch likes this
audrey maniez likes this
Danisaurio likes this
Colin likes this
Matt Hobbs likes this
Dennis Lembrée likes this
Peter Antonius likes this
ASIC Rocky 🪙⛏ likes this
Dave Cunningham likes this
Honkel Steffi likes this
Jasper Moelker 🔋🌳🇪🇺 likes this
Juan Javier likes this
Scifibrarian, MS, MFA likes this
Peter likes this
Who me? likes this
Alex Edwards likes this
Christian Gloddy likes this
Ollie Boermans likes this
Natalie Chin likes this
Stephanie Hagadorn likes this
TRST_Blog likes this
Martin Underhill likes this
Bogdan Cerovac likes this
Brian Crumley likes this
Jamie and Lion likes this
a 🧈 likes this
Marcos de Miguel likes this
Laura Whitehead 💙 likes this
Gode vinkjøp likes this
negi4a likes this
João Figueiredo likes this
Karan Ganesan likes this
T Carter Baxter likes this
Marco useCauseAndEffect() likes this
Zach Leatherman likes this
AJ Klein likes this
Patrick Haug likes this
Yannis Güdel likes this
Heather likes this
Hélio Correia likes this
Iain Bean likes this
Riccardo Erra likes this
Corey Megown likes this
Clare Hyam (she/her) likes this
Bruno Pulis likes this
Melanie Sumner likes this
Craig Abbott likes this
Stephanie Eckles likes this
Gaël Poupard likes this
Matthias Weston likes this
Somascope likes this
Carie Fisher likes this
Pauline Vos likes this
Tom Hermans🤘🌈 likes this
Chris likes this
Conor likes this
Joe Lanman likes this
Michael Hastrich reposted this
Starfall Projects reposted this
Jeanine Kay Lineback reposted this
Comandeer reposted this
Alicia Jarvis (She/Her/Hers), CPACC, CSM 🇨🇦 reposted this
Sarah Federman reposted this
Sami Keijonen reposted this
Yakim reposted this
Darice de Cuba reposted this
mallory, alice & bob reposted this
Chris Power reposted this
Rian Rietveld reposted this
Bram Smulders reposted this
Eleventy reposted this
Jess Peck 🐍👾✨ reposted this
Patrick Grey reposted this
Juha Karttunen reposted this
Saravana reposted this
Eric Bailey reposted this
Scott McKee reposted this
Sven Wolfermann reposted this
Who me? reposted this
Hadley reposted this
Marcos de Miguel reposted this
Zach Leatherman reposted this
Web Axe reposted this
Clare Hyam (she/her) reposted this
Myles Lewando replied: Ooooo nice!
Peter van Grieken replied: I’ve been using this project for a while and it has made writing WCAG reports so much easier for me.
Peter van Grieken replied: I’ve been using this project for a while and it has made writing WCAG reports so much easier for me.
Sophie replied: wow!
Wassim Chegham 🌱 replied: Hey @eleven_ty folks, check this out!
Wassim Chegham 🌱 replied: That's great! Thank you for sharing.
Robbert replied: I love that every issue can be linked to with the # fragment identifier! Would like to link to relevant issues on the component backlog for NL Design System. No more PDFs!
Jules Ernst replied: Just missing the installation for very dummies. It's me I know, but tried hard to get it installed with no success.
Hidde replied: heb de ”Set up” zojuist vast ietsje uitgebreid, maar ik stuur je ff een DM voor support! :-)
Sara Soueidan replied:
An @eleven_ty starter project for WCAG reports, kindly open-sourced by @hdv Code: github.com/hidde/eleventy… Intro & about: hiddedevries.nl/en/blog/2021-0… #a11y
Colin replied: nice work!
Hidde replied: thank you!
Peter replied: Good work pal. I’ll look forward to kicking the tyres on this 👍🏻
Allen vaccinated Hoffman! replied: Nicely done. Take a look at ACRT Accessibility Conformance Reporting Tool from the U.S. FEeral government. It uses JSON to store the test checkpoints and also the report data for sharing.
Danisaurio replied: Hi! Loved your project and I'll add spanish support :) thanks for doing this!
Sun lover replied: This is an excellent #ally resource! Thank you so much for sharing.
bertrandkeller replied: hiddedevries.nl/en/blog/2021-0… - belle initiative, mais un peu compliqué à suivre dans le temps. RGAA C’est 106 critères, 106 fichiers MarkDown a éditer, c’est un peu lourd.
Shawn Thompson replied: Thanks @mgifford for sharing this with me!
Priya replied: Introducing: an Eleventy starter project for WCAG reports hiddedevries.nl/en/blog/2021-0…
Pablo Lara H replied: Introducing: an Eleventy starter project for WCAG reports by Hidde de Vries @hdv #Eleventy #Eleventystarterproject #WCAG #webdev #WebContentAccessibilityGuidelines hiddedevries.nl/en/blog/2021-0…
Max Antonucci replied: So @hdv shared their @eleven_ty template for WCAG reports, and it is pretty damn awesome - hiddedevries.nl/en/blog/2021-0… #a11y #accessibility
Melanie Sumner replied: This is very cool.
Ben Myers 🦖 replied: Ohhh, I love this.
Bruno Pulis replied: Awesome initiative @hdv soon as possible I'll send a translation to Brazilian portuguese! :)
Web Axe replied: An Eleventy starter project for WCAG reports hiddedevries.nl/en/blog/2021-0… #wcag #a11y #dev #reports
Hidde replied: Haha this sounds very close to home! Makes a lot of sense.
Carie Fisher replied: 😻😻😻
Joe Lanman replied: awesome, thanks!
Leave a comment
Posted a response to this?

This website uses Webmentions. You can manually notify me if you have posted a response, by entering the URL below.