Towards Automated eGovernment Monitoring

Paper G- Monitoring accessibility of governmental web sites in Europe.

Author: Morten Goodwin


This chapter was originally published as scientific conference or journal paper. Please see the original source for the complete paper.1

Original paper authors: Christian Bühler, Helmut Heck, Annika Nietzio, Morten Goodwin Olsen, and Mikael Snaprud

Christian Bühler, Helmut Heck and Annika Nietzio are with Forschungsinstitut Technologie und Behinderung (FTB) der Evangelischen Stiftung Volmarstein, Grundschötteler Str. 40 58300 Wetter (Ruhr), Germany.

Morten Goodwin Olsen and Mikael Snaprud are with Tingtun AS, Kirkekleiva 1, 4790 Lillesand, Norway, email:


Web accessibility is an important goal of the European i2010 strategy. Several one-off surveys of eAccessibility have been conducted in the past few years. In this paper, we describe an approach to supplement the results of such surveys with automated assessments, that can easily be repeated at regular intervals. The software basis is provided by the European Internet Accessibility Observatory (EIAO). We analyse how the data collected by EIAO can be compared to other surveys.


In the Information Society a persistently increasing amount of information is made available on the web. It is essential to make this content accessible to all people including people with disabilities. The declaration of the UN Convention on the Rights of Persons with Disabilities by the United Nations has drawn attention to the topic of inclusion worldwide. One of the key areas covered by the Convention is the right to access information and services on-line.

The European Commission recognised the importance of creating wider access for all in 2000 and defined eInclusion as part of the Lisbon strategy: "Ensure that every citizen should have the appropriate skills needed to live and work in a new Information Society for all." In the following years the eEurope action plans [1],[2] and the i2010 strategy [3] were adopted, placing great importance on eInclusion, accessibility requirements for public procurement of ICT, accessibility certification, and use of legislation.

To support the future development of EU policy in the eAccessibility field, measurements of the current situation are needed. A number of studies and surveys have been undertaken to provide further insights. Amongst others the take-up of web accessibility as defined in the W3C Web Content Accessibility Guidelines (WCAG 1.0) [4] has been analysed repeatedly.

Given their complexity and often also costly nature, such studies are usually conducted as a one-off exercise. The usefulness of the results could be greatly enhanced, if supplemented by regular periodically repeated evaluations. The initial survey usually is performed manually (or in a combination of automated and manual assessment), because this method produces the most reliable results. Subsequent monitoring of status and changes can be carried out cost-efficiently in an automated process. Although it cannot perform all necessary tests, the automated assessment of web accessibility is a valuable supplement. Its advantages are that it can measure certain features that can be utilised as indicators for accessibility and that it allows the monitoring of a large number of web sites.

The EIAO project2 has established the technical basis for an European Internet Accessibility Observatory (EIAO) consisting of:

Figure 1 shows how the main components interact in the automatic monitoring process of the Observatory.

Figure 1: The EIAO software architecture

The remainder of this paper is organised as follows. Section Studies on eAccessibility reviews recent international eAccessibility studies. Section The EIAO approach presents more details on the EIAO monitoring approach. In section Findings and discussion we discuss and compare the findings from the evaluations and conclude with prospects for future research directions.

Studies on eAccessibility


In the past few years a number of national and international studies on eAccessibility have been conducted. Some are covering mainly web accessibility [6], while others address a wider range of ICT products and services. Also the scope with regard to the sectors included is varying. There are targeted surveys of governmental, higher eduction, or library web sites.

In the following, we describe three widely recognised international studies, all including indicators of the accessibility of public sector and governmental web sites from the European Union. We concentrate on the latest available results, to capture the most recent status of the rapidly evolving web.

UK Cabinet Office, 2005


The study "eAccessibility of Public Sector Services in the European Union" [7] was commissioned by the UK Cabinet Office during the UK presidency of the EU in 2005. It includes a policy survey and a detailed analysis of public web sites. The analysis is based on a combination of automated and manual testing. In addition to WCAG 1.0 conformance, the manual evaluation reports also on a number of good practice examples.

Overall 569 web sites from governments of all 25 member states and from the European Commission were evaluated with automated testing. A web crawler retrieved a fixed number of pages, starting from the home page of the site, following links through the site until a maximum number of steps from the home page was reached. Each page was scanned for automatically detectable violations against WCAG 1.0. Only 436 of the sites yielded usable data, the remaining 23.4% produced no usable results due to problems during retrieval or testing.

The automated results are supported by an in-depth manual evaluation of 31 selected web sites, which was partly extrapolated to the full set of sites. The data is summarised into four conformance classes:

United Nations, 2006


The "United Nations Global Audit of Web Accessibility" [8] was commissioned by the UN in 2006. With the Convention on the Rights of Persons with Disabilities the UN have put the topic of inclusion, meaning also eInclusion, on the agenda of governments worldwide.

Based on the evaluation of 100 web sites from twenty countries around the world, 3 the audit seeks to present an indication of the existing web accessibility status. One of the five sites from each country is from the public sector. In most cases the main governmental web site was selected.

The methodology was to test the home page of each web site using a combination of automated and manual techniques. The targeted conformance level was WCAG 1.0 AAA. The report provides a per checkpoint summary including good and bad examples, as well as a detailed analysis per country. Only 3% of the sites were found to meet WCAG 1.0 level A. All three are from the European public sector. None of the sites conformed to WCAG 1.0 level AA or AAA.

Measuring progress of eAccessibility in Europe (MeAC), 2007


The "Assessment of the Status of eAccessibility in Europe" [9] was commissioned by the EC in 2007 to follow up on previous studies and support the future development of European policy in the field of eAccessibility. The study presents a policy survey, a status measurement based on a set of key indicators, and the results from questionnaires sent to stakeholder groups (ICT industries, user organisations, and public procurement officials).

The MeAC assessment covers a wide variety of ICT products ranging from TV and telephony to computer hardware and software. The accessibility of public and private sector web sites is also part of the survey.

Overall 336 public and private sector web sites were evaluated using a combination of automated and manual testing of WCAG 1.0 level A. In an automated retrieval process 25 pages were collected from each site. Only 314 of the sites could be evaluated successfully, the remaining 6.5% produced no results due to problems during retrieval or testing. The outcome is presented in the same conformance classes as used in the UK Cabinet Office report.

The EIAO approach


The European Internet Accessibility Observatory (EIAO) is an implementation of the automated monitoring application scenario of UWEM. The UWEM methodology has been developed by a group of experts and institutions from several European countries. It addresses the methodological needs of various web accessibility evaluation approaches ranging from in-depth evaluation of a single web site to large scale monitoring of several thousand sites.

UWEM describes methods for the collection of evaluation samples, test procedures according to WCAG 1.0 level AA, and several different reporting options. EIAO uses the most aggregated reporting format, the UWEM accessibility score card.

Collection of evaluation samples

The identification of resources belonging to a web site is a prerequisite for the actual sampling. This exploration is carried out by a crawler automatically following each identified site-internal link. 4 The crawler used in the Observatory is implemented with a breadth-first deterministic crawling strategy. The crawl starts outwith one or more seed resources, typically the home page of the site.

From a statistical point of view, an exhaustive evaluation of complete web sites is advantageous, because it yields the most representative results. However, if a web site is very large or if a large number of web sites are included in a monitoring evaluation, it may not be feasible to identify a complete list of resources belonging to each site. In this case the evaluator may choose to stop the crawling process when a sufficiently large number of resources has been identified. EIAO uses a stop criterion of 6000 web pages. Web sites with less than 6000 available pages are crawled exhaustively. See section Data Collection for details on number of identified web pages.

The EIAO sampling strategy uses a uniform random sample without replacement from the list of known URLs of the web site. On the one hand large samples yield the most reliable results. On the other hand the sample size is constrained by the evaluation capacity of EIAO. Depending on the evaluation capacity of the EIAO hardware a suitable set of statistical parameters was selected, taking into account the trade-off between performance and precision of the results. EIAO uses a stop criterion of evaluating 600 web pages. 5

Accessibility tests and reporting

The Web Accessibility Metric (WAM) component of EIAO is an implementation of 26 tests for WCAG 1.0 level AA that can be carried out automatically according to UWEM. Both HTML and CSS accessibility are addressed. The output of the WAM uses RDF/EARL reports [10] producing results that are stored in the data warehouse. The findings about one site are summarised in a single number called the UWEM accessibility score. This number is calculated as the percentage of failed tests per site. Finally, the score values are mapped into five different score card categories: A — E, where A is the best and E is the worst score.

Strengths and weaknesses

A fully automated accessibility evaluation process has some clear advantages over purely manual evaluations. The Observatory can process and evaluate a large number of web sites compared to an expert. Furthermore, the evaluations are repeatable which enables a comparison of evaluations from one month to the next. Additionally, in contrast to an expert, the results from the Observatory are not influenced by human factors such as the expert's prior experience.

In contrast, EIAO has some disadvantages compared to manual assessments. Most significantly, only a subset of the UWEM tests are fully automatable. Tests that typically require human judgement have not been implemented. Because of this, EIAO cannot provide the same level of detail as an expert evaluation. Furthermore, the crawler cannot retrieve hidden web pages such as pages that are password protected. Also the checking of other web content requiring user interaction, such as Flash or JavaScript, is not supported.

Data Collection


In March 2008, EIAO carried out a survey of 3232 public European web sites. The system was able to download and evaluate 2317 (71.9%) of the sites. 40 of the remaining 915 web sites could not be evaluated due to problems with the site, such as unavailability of the site or restrictions by robots.txt. Furthermore, 875 sites (27.1%) could not be evaluated due to problems with the EIAO software, such as wrong handling of redirection, character encoding, or other problems. Of the 2317 web sites, where no problems were encountered, the crawler retrieved on average 2406 pages. 1998 sites (86.2%) were crawled exhaustively.

The distribution of evaluation results is presented in figure 2. Note that the score B+ is not part of UWEM. In our study, this category includes web sites that passed all the automatic tests. 6

Figure 2: Score distribution from EIAO evaluation of 2317 European web sites carried out in March 2008. UWEM scores are presented on the x-axis. The y-axis shows the percentage of sites falling into each category.  

Findings and discussion


The comparison to related research plays an important role in the analysis and interpretation of the outcome of the large scale accessibility assessments.

To enable a comparison based on the conformance classes introduced in section UK Cabinet Office, 2005, we assume the following mapping. Eiao B+ corresponds to Limited pass, no automated tests fail. Eiao B is mapped to Marginal fail. The three remaining score card categories Eiao C, D, and E are summarised into Fail. EIAO does not involve manual testing, therefore there are no sites in the Pass category. Figure 3 shows the conformance categories reported by the UK Cabinet Office report, the MeAC study, and EIAO. 7

Figure 3: Accessibility of public web sites in Europe as reported by different surveys. 

Consistently, all three assessments show that the overall level of accessibility of European governmental web sites is rather poor. The positive trend that can be observed in a comparison of the Cabinet Office and the MeAC study seems be continued in the EIAO results. However, subsequent EIAO assessments are necessary to confirm this trend.

There are some methodological differences between the studies. The Cabinet Office and MeAC studies tested for WCAG 1.0 level A conformance, whereas EIAO contains automated tests for WCAG 1.0 level AA. This explains the lower number of automated pass results reported by EIAO.

The number of pages sampled from each site is 25 in the MeAC study. The Cabinet Office report does not disclose the number. EIAO operates with a much larger sample size of 600.


The encouraging results of the comparison to other studies shown that EIAO can provide valuable supplementary information on eAccessibility surveys undertaken by human experts. Also expert surveys rely increasingly on automated tools for web accessibility evaluation. Therefore the alignment of results becomes quite straightforward. For instance, the results in the category Limited pass are directly comparable.

The EIAO software provides all necessary components in one architecture. Furthermore, it is compliant to the European methodology UWEM. EIAO assessments are planned to continue on a monthly basis. Once several results are available, it will be possible to make further statements about the evolvement of eAccessibility in Europe. In the future, it should also be considered to run an EIAO assessment on the same set of URLs as used in the other studies to increase the comparability of the results. Also a closer collaboration on the methodology definition, such as sample size and conformance classes, would be desirable to achieve a seamless integration of extensive yearly surveys and automated monthly monitoring.


1. [The paper has been published in the Proceedings of the International Conference on Computers Helping People with Special Needs 2008: 410-417. Springer-Verlag Berlin, Heidelberg © 2008]

2. [The EIAO project is co-funded by the European Commission, under the IST contract 2003-004526-STREP.]

3. [Four European countries were part of the audit: France,Germany, Spain, and United Kingdom.]

4. [The crawler observes the Robot Exclusion Standard and does not retrieve URLs that are requested to be ignored in the robots.txt file.]

5. [Note that the sample size (600 pages) is intentionally smaller than the number of pages identified by the crawler (6000 pages) to allow a statistical sound uniform random selection of samples.]

6. [UWEM requires manual testing for a web site to obtain the highest score A.]

7. [The UN audit is not part of the comparison because it contains only four public sector web sites from Europe, which is too few for a meaningful comparison.]


[1] eEurope,European Commission,Retroeved 2002 from

[2] eEurope,European Commission,Retrieved 2008 from

[3] i2010 — i2010 - A European Information Society for growth and employment,European Commission,Retrieved 2008 from

[4] Web Content Accessibility Guidelines 1.0. W3C Recommendation 5 May 1999,World Wide Web Consortium,Retrieved November 4th, 2009, from

[5] Web Accessibility Benchmarking Cluster,2007,Retrieved November 4th, 2009, from

[6] Large Scale Web Accessibility Evaluation - A European Perspective,Snaprud, Mikael and Sawicka, Agata,150-159,2007,4556,Universal Access in Human-Computer Interaction. Applications and Services,Faculty of Engineering, Agder University College, NO-4876 Norway,Lecture Notes in Computer Science,Stephanidis, Constantine

[7] Cabinet Office,Retrieved 2006 from

[8] Nomensa,Retrieved 2007 from

[9] Kevin Cullen and Lutz Kubitschke and Ingo Meyer,Retrieved March 19th, 2010, from

[10] World Wide Web Consortium,Retrieved November 4th, 2009, from

The author of this document is:
Morten Goodwin
E-mail address is:
morten.goodwin __at__
Phone is:
+47 95 24 86 79