This document introduces a common framework for enhancing the quality
practices of the W3C
Working Groups in the areas of specification editing, production of test
materials, and coordination efforts with internal and external groups. It
presents introductory, roadmap, and orientation information for the Framework
document family of the Quality Assurance
(QA) Activity. This is the first of the
family of QA Framework documents, which includes the other existing or
in-progress specifications: Process & Operational Guidelines;
Specification Guidelines; and, Test Materials Guidelines.
This section describes the status of this document at the time of its
publication. Other documents may supersede this document. The latest status
of this document series is maintained at the W3C.
This document is a W3C Working Draft (WD), made available by the W3C Quality
Assurance (QA) Activity for discussion by W3C members and other interested
parties. For more information about the QA Activity, please see the QA Activity statement.
This version is the first public Working Draft, and incorporates the
discussions at the first QA face-to-face, plus several subsequent QA Working Group [QAWG] teleconferences,
and supersedes all previous drafts. It is expected that updated WD versions of this document will be produced
regularly, along with other members of the Framework documents family. Future
progression of this document beyond Working Draft is possible, but has not
yet been determined.
Please send comments to www-qa@w3.org,
the publicly
archived list of the QA Interest
Group [QAIG]. Please note
that any mail sent to this list will be publicly archived
and available, do not send information you wouldn't want to see distributed,
such as private data.
Publication of this document does not imply endorsement by the W3C, its
membership or its staff. This is a draft document and may be updated,
replaced, or obsoleted by other documents at any time. It is inappropriate to
use W3C Working Drafts as reference material or to cite them as other than
"work in progress".
A list of current W3C Recommendations and other technical documents can be
found at http://www.w3.org/TR.
1. About this QA framework
1.1 Introduction
1.2 Why do we need a QA framework?
1.3 The Framework is for the Working Groups
1.4 The Framework goals
2. QA resources
2.1 Expertise & consultancy
2.2 Technical assets
3. Structure
& content of Framework documents
3.1 Applicable domain
3.2 Document orientation & structure
3.3 Normative guidelines
3.4 Terminology
3.5 Documents roadmap
4. Guide to using Framework documents
4.1 Target audience
4.2 Framework primer and guide
5. Conformance
6. Acknowledgments
7. References
1. About this QA framework
1.1 Introduction
This document introduces a common framework for promoting and facilitating
the quality practices of the W3C Working Groups. It presents
introductory, roadmap, and orientation information for the Framework document
family of the Quality Assurance (QA)
Activity.
The documents in this family collectively aim to provide the W3C Working
Groups with resources and tools for all phases and aspects of their quality
and conformance activities,
- planning and process setup,
- writing better, more testable specifications,
- coordination with internal and external groups,
- and ultimately to building or acquiring conformance test materials.
The structure and content of the Framework documents family are detailed
in the later chapter, "Structure &
content of Framework documents", but the main (root) documents are:
- "QA Framework: Introduction" (this document) -- a roadmap and primer to
the Framework document family;
- "QA Framework: Process & Operational Guidelines" -- process,
organizational, and operational guidelines for groups starting
development or acquisition of conformance test materials;
- "QA Framework: Specification Guidelines" -- guidelines for writing
better, more testable technical reports;
- "QA Framework: Test Materials Guidelines" -- detailed guidelines for
test materials, i.e., for building test suites (TS) and test tools themselves
Some of these documents have an additional associated "Examples &
Techniques" document, which contain detailed, technology-specific methods and
examples for implementing the guidelines.
1.2 Why do we need a QA framework?
The ultimate mission of W3C is "Leading the Web to its full potential...",
and W3C's many active working groups contribute towards this goal, in large
part, by building a family of high-quality Web standards. These standards are
the building blocks of a architecture to maximize the Web's potential.
Most would agree that the writing of the standard is not, in itself, the
end purpose of the Working Groups (WGs). It is a means to achieve the real end:
the success of the standardized technology in practice and practical
application. One measure of the success that can be applied to most Web
standards is a set of complete, correct, high-quality, interoperable
implementations that actually realize the functionality of the standard.
In the past few years, several WGs have discovered that the early
production of test materials (TM)
has been a major contributor to the success of their standards. The benefits
of test materials work are several-fold, but a couple of the big benefits
are: early detection and correction of unintentionally vague, contradictory,
or unimplementable language in the technical
reports (TRs) [W3C-TR]; and, a solid basis for the
development of the interoperable implementations demonstration required by
the W3C Process Document.
- Better specifications are inherently more implementable;
- and, better implementations are our ultimate goal.
The individual conformance test materials efforts of the Working Groups
have in several cases been inspired and on target, in others have been
rudimentary or incomplete. Even amongst the best efforts, however, there is
little uniformity in the quality-control principles, methodologies, or even
such basics as terminology. Moreover, these efforts are distributed
throughout W3C, making it difficult or at least time consuming for WGs
pursuing their QA goals to find and take advantage of what has already been
done. Each WG has started from
scratch, researching the numerous existing TS activities and defining their own processes,
operational framework and deliverables.
This defines the mission of our QA Activity. QA aims assist the WGs by
assembling and organizing the best of a body of good practice, defining a
framework, and providing a roadmap for the working groups. These
best-practice guides should dramatically ease the job of planning and
implementing QA projects within the Working Groups.
1.3 The Framework is for the Working Groups
The last underscores a key reality of improved quality practices
associated with W3C technical reports. QA is not a comprehensive service
proffered to the Working Groups, nor is QA an aspect of WG operation that can be exported to an
external entity. Similarly, quality practices are not mandates imposed on the
Working Groups from outside the WG, despite a trend towards higher
expectations for Working Groups' QA deliverables and for the quality of their
products in general.
QA is properly an intergral part of the efforts of each Working Group.
Each WG will provide the majority of the resources needed to meet its QA
commitments. The QA Activity provides this Framework, as well as other tools
and assets, to aid and guide the WG efforts.
Therefore the QA Framework is for everyone who is involved with the work
of W3C's Working Groups. Some part of these Framework documents should be
useful and helpful to anyone who is an active participate in generating such
deliverables -- specifications and conformance test materials alike -- as
well as anyone who is involved in assessing or evaluating such
deliverables.
1.4 The Framework goals
Foremost amongst the purposes of defining a common QA Framework is the
principal goal of the QA Activity itself:
- to improve the quality of W3C standards' implementations in the
field.
As an integral part of the working modes of the WGs, QA is ideally a
development context which is applicable to a spec's development from its
earliest stages through completion. This Framework provides a collection of
best-practice principles and guidelines, which span the life of WG activities
from the identification of QA deliverables in the WG charters, through
post-REC and possibly even post-WG maintenance.
While some might perceive QA projects as a regrettable drain on WG
resources, there is ample experience, both within W3C as well as other
standards venues, that shows significant improvement to the products of the
WGs. This equates to a sound business case for the early investment of
resources -- the cost in resources is more than offset by the benefits of
more implementable and better implemented Recommendations. In fact, in some
cases there could be a net resource gain, by dodging problems that would be
costly to fix if detected late in a specification's development process.
This Framework document family should provide to those undertaking quality
assurance projects:
- a step-by-step guide to QA process and operational setup, as well as to
development;
- and, access to resources that the QA Activity makes available to assist
the WGs in all aspects of their quality practices.
More specific goals include:
- to encourage the employment of quality practices in the development of
the specifications from their inception through their deployment;
- to encourage development and use of a common, shared set of tools and
methods for building test materials;
- to foster a common look-and-feel for test suites, validation tools,
harnesses, and results reporting.
2. QA resources
A variety of resources are provided by the QA Activity, including:
- "The Matrix" [MATRIX], which is a definitive
survey of existing QA practice;
- this QA Framework document family;
- common QA Glossary [QA-GLOSSARY];
- QA Taxonomy description [TAXONOMY];
- Notes (white papers) on QA-related topics;
- the Issues List of the QAWG [ISSUES-LIST];
- expertise and consultancy;
- assets.
A comprehensive survey of the resources of QA Activity is beyond the scope
of this document, and may be located on the QA Working Group [QAWG] and QA Interest Group [QAIG] Web
pages. However the last two items above are pertinent to other parts of these
Framework documents.
2.1 Expertise & consultancy
The QA Working Group expects to have access to considerable expertise on a
spectrum of QA topics, methods, and techniques. One key aim of the QA
Activity is to make this expertise available to the Working Groups, at least
on an as-needed (request) basis, as they plan and implement their conformance
and quality projects. The degree of participation of QA experts in WG
projects will likely be limited by QAWG staff resource constraints, but a
consultancy role should almost always be possible.
The Framework document, "QA Framework: Process & Operational
Guidelines", explores this topic in some detail, including:
- the nature of QAWG
relationship to other Working Groups;
- level and degree of participation of QAWG experts in other WG
projects;
- and, means and mechanisms for managing WG requests for QA
assistance.
2.2 Technical assets
While the initial resources that the QA Working Group offers to other WGs
focuses heavily on the Framework documents and available consulting
expertise, it is intended that ongoing development within QA Activity will
result in a collection of commonly useful assets. Some anticipated offerings
include:
-
XML-based test
case description languages (TCDL) that can be
applied or adapted to diverse test suite projects.
- Scripts and transformation stylesheets to generate commonly needed
TS materials, harnesses, and
documents from TCDLs.
- Grammars and methods for granular construction of specifications, as
well as tools and methods for management of testable assertions and even
(in some cases) automatic generation of tests.
- Templates for setting up project TS process documents, ../Test/ Web sites,
and other conformance project logistics.
- Standardized and routine procedures (including forms and scripts) for
WGs to manage the process of (external) test case submission, review, and
integration.
- Results reporting languages (e.g., EARL), plus
methodologies and specific tools for working with them.
Most of these will be discussed in "Test Materials Examples and
Techniques" [TEST-EXTECH], to illustrate how to
meet the "Test Materials Guidelines" [QAF-TEST] with
common tools.
3.1 Applicable domain
Conformance test materials are being developed in an organizational
landscape within W3C (and without) which is characterized by:
- Working Groups at all stages of maturity, ranging from newly created
(i.e., just chartered), to mature, to extended (i.e., re-chartered);
- Specifications at all stages of document progression (i.e., Process
REC-Track), ranging from First Working Draft through REC, post-REC errata
processing and subsequent-edition publication;
- Conformance test materials of all types (i.e., described in [TAXONOMY]), ranging from content validators to test
suites and tools for products, interfaces, and APIs;
- QA Experience in developing test materials, ranging from minimal to
significant;
- Resources for developing test materials, ranging from insufficient
staffing levels to staffing levels commensurate for the QA
delverables;
- Venue for Development of test materials, ranging from intra-WG
development to combination of WG and external organization to external
organization development of complete test material.
These framework documents should be applicable and useful throughout this
diverse context.
3.2 Document orientation & structure
The Framework documents will utilize the style of structuring guidelines
and verifiable checkpoints, similar to WAI standards such as WCAG [WCAG10]. The degree to which a simple cookbook guide is
achievable is complicated by the factors enumerated previously -- diversity
of potential test materials, as well as organizational diversity. Complexity
and diversity will be managed in the Framework document family by:
- stating the guidelines and checkpoints as much as possible with
general, cross-taxonomy and all-WG applicability;
- and, for each of the Framework's guideline documents, having an
accompanying "Examples & Techniques" document.
The guideline documents are closely linked to their respective "Examples
& Techniques" documents. The linked material explores, defines, and
presents examples of taxonomy-dependent implementations of checkpoints. This
model from the WAI standards has served well to manage diversity and
complexity.
These checkpoint technique documents are not intended to be a
comprehensive survey of existing W3C practice. That is the purpose of the QA
matrix [MATRIX]. That notwithstanding, several of the
conformance test efforts will be drawn upon for examples.
3.3 Normative guidelines
The Framework guidelines -- or more precisely, the checkpoints associated
with the guidelines -- comprise the individual conformance requirements of
these Framework documents. Each guidelines document contains a conformance
chapter which unambiguously defines conformance to the guidelines and
checkpoints.
The relationship of the Framework documents to the W3C Process is
undefined at the time of publication. It is not precluded that all or parts
of these guidelines may ultimately be incorporated (by reference) into the
W3C Process Document [PROCESS].
3.4 Terminology
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as
defined in RFC 2119 [RFC2119].
Unusual terms in these framework documents are defined when first used,
and most generally useful QA-specific terms will eventually be in the QA
Glossary [QA-GLOSSARY].
The Framework document family includes:
- "QA Framework: Introduction" (this document)
- "QA Framework: Process & Operational Guidelines" [QAF-OPS]
- "Process & Operational Examples and Techniques" [OPS-EXTECH]
- "QA Framework: Specification Guidelines" [QAF-SPEC]
- "QA Framework: Test Materials Guidelines" [QAF-TEST]
These documents are explained in the following subsections.
3.5.2 Introduction
- Overview of the W3C QA Activity
- Identification of a those QA resources which are integral to the
Framework document family
- Description of the structure and content of the Framework document
family
- Guide and Primer for how to use the Framework document family
This document is for everyone interested in QA within the W3C. Especially,
it serves as a first read for people with the following roles:
- WG chairs that need to start a QA-related project for their particular
specification
- WG members that need to understand what the commitment to QA is
about
- QA TS moderators, contacts,
coordinators and editors
The introduction document serves as a starter for most QA-related
activities, especially as it contains pointers to more specific information
on particular steps of the QA process, both by examples and by linking to
that information explicitly.
3.5.3 Process & Operational Guidelines
This document is primarily targeted to people explicitly involved in QA
activities. This includes both those within the W3C Working Groups (as
mentioned in the previous section) as well as others from organizations
external to the W3C involved in developing test materials. The goal of this
document is to present procedural and operational guidelines for groups
undertaking conformance materials development. The document contains
information about:
- Process considerations and tasks for incorporating QA related
activities within the Working Group;
- Logistical setup and operational activities for building conformance
test suites and tools;
- Resource considerations for staffing QA effort;
- Interaction between WGs and QA Activity and between WGs and external
organizations developing conformance materials;
- Intellectual Property Rights (IPR) issues.
Accompanying the Process and Operational Guidelines is a companion
document, Examples and Techniques. The Process and Operational Examples and
Techniques document provides examples and pointers to existing QA work,
illustrating the principles and guidelines set forth in the Process and
Operational Guidelines.
3.5.4 Specification Guidelines
This document is primarily targeted to people who are involved with either
writing or reviewing the specifications of a WG, as well as those explicitly
involved in QA activities. The goal of this document is to present guidelines
for the writing of clearer, more implementatable, and better testable
technical reports. The document presents traditional best-practice guidelines
for:
- Aspects of W3C specifications relating to testability and construction
of conformance test materials, including
- conformance clauses,
- enumeration of behaviors,
- identification and addressability of specification-specific
testable assertions
- profiles, levels, conformance claims
- optional features, extensions, etc
It also addresses advanced and emerging methods for easier generation of
TS schemata, tests, and assertions:
- Schemata used for writing specifications
- Schemata used for test extraction
- Schemata used for non-complete automatically generated frameworks
serving as the primer for a TS
Accompanying the Specification Guidelines is a companion document,
Examples and Techniques. The Specification Examples and Techniques document
provides examples and pointers to existing QA work, illustrating the
principles and guidelines set forth in the Specification Guidelines.
3.5.5 Test Materials Guidelines
This document is primarily targeted to people who are actively involved
with either building or assessing and acquiring conformance test materials.
Accompanying the Test Materials Guidelines is a companion document, Examples
and Techniques. Together, the two documents cover:
- Principles and requirements for general test suite design, test case
content, reporting etc.
- Principles and requirements for issue tracking systems in TS production, matrices of existing,
expected and wanted tests and so forth
- Tools used to produce conformance test suites
- Technologies used in doing so (RDF, XSLT, DTD/Schema etc)
- Building or using harnesses for running the TS and generating relevant results
4. Guide to using Framework documents
This chapter provides an introduction to the use of the Framework document
family. It addresses:
- what parts should be helpful, according to a participant's role in WG
activities,
- a primer on getting started on the QA activities within the WGs, by
reference to the guidelines and techniques of the other parts.
4.1 Target audience
4.1.1 For the overall Framework
QA is properly an intergral part of the efforts of each Working Group. The
QA WG charter succinctly summarizes the point that has been made from the
earliest W3C conformance papers:
"[...] QA is to be considered a 'natural overhead' of any WG [...]. QA
will succeed only if every person inside W3C participates in it."
Some parts of these Framework documents should prove useful to everyone
involved with the work of W3C's Working Groups -- not only those WG members
specializing in conformance test materials, but all of a WG's participants,
and even external parties reviewing the products of the WGs.
4.1.2 For the Introduction
This first part, "Introduction", should be read by everyone involved with
the work of the WGs. In addition to reviewing the scope and goals of the QA
Activity, and QA activities within the WGs, it also provides a detailed
roadmap and guide to the Framework documents family.
As presently scoped, there will be seven documents in the Framework (see
"Documents Roadmap"). After the
"Introduction", those parts that will be useful to any individual WG
participant depends on the participant's role in the WG's work.
4.1.3 For the guideline parts
From the perspective of conformance and quality practices, several roles
are significant in a WG's activities. In the following it is assumed that,
associated with each WG, there will be a "Test" group (WG-TS), that is focused on
conformance test suites and tools. WG-TS consists of a subset of
the WG members, and possibly other W3C members from outside of the WG. (The
WG-TS requirement is
detailed in the process and operational guidelines.)
- all WG members
- For any (potential) WG member, the charter and
QA-commitment parts of "Process & Operational Guidelines" ([QAF-OPS], Guideline 1) should be helpful in
understanding what the WG has committed to deliver. Familiarity with
the "Specification Guidelines"
[QAF-SPEC] will be helpful to any member who
participates in the advancement of the WG's specifications to
Recommendation.
- WG spec editors & authors
- As for all WG members, the operational guidelines [QAF-OPS] are useful. A good working understanding
of specification guidelines [QAF-SPEC] will be
needed in order to satisfy the specification guidelines and
checkpoints, and "Specification Examples and
Techniques"
[SPEC-EXTECH] should be a
valuable resource in choosing document structure, formats, and
techniques that will facilitate satisfying the requirements.
- WG chair
- As the person ultimately responsible for both the advancement of the
WG's specifications and the WG's QA projects, a familiarity with the
guidelines for process and operations [QAF-OPS],
for specifications [QAF-SPEC], and for test
materials [QAF-TEST] will be useful.
-
WG-TS
participant
- Those who are active in building the conformance test materials of
the WG will need a good working understanding of both the guidelines
for test materials [QAF-TEST] and "Test Materials Examples and Techniques"
[TEST-EXTECH]. Because of the close dependency
of test materials on the functional specifications, a good familiarity
with the specification guidelines [QAF-SPEC] as
well as examples and techniques [SPEC-EXTECH] will also be helpful.
-
WG-TS
moderator
- The person who manages the WG's QA projects should have good working
understandings of guidelines and techniques for specifications ([QAF-SPEC] and [SPEC-EXTECH]), as well as the guidelines and
techniques for test materials ([QAF-TEST] and
[TEST-EXTECH]). In addition, knowledge of
all of the process and logistical setup guidelines [QAF-OPS] and techniques [OPS-EXTECH] are needed.
- non-WG spec reviewers
- Whether from other WGs, or the public at large, the specification
guidelines [QAF-SPEC]will be helpful to those
who review a WG's specifications.
- non-WG TM reviewers
- Whether from other WGs, or the public at large, reviewers of the test
materials of a WG should be familiar with guidelines for test materials
[QAF-TEST], and familiarity with the techniques
[TEST-EXTECH] as well would facilitate a
critical review.
- Reviewers of activity proposals & charters
- For those W3C members who will be reviewing Activity proposals and
proposed WG charters, and helping to form their AC representatives
positions, the charter- and
deliverable-related requirements defined in the process and
operational guidelines [QAF-OPS] are
helpful.
- QA Activity members
- Members of the QA WG are an expert resource for the W3C Working
Groups, and accordingly should be expert on all parts of the Framework;
members of the QA IG need thorough familiarity with all parts as well,
to effectively render some of the IG's chartered deliverables.
4.2 Framework primer and guide
This section examines use of the Framework documents from the perspective
of significant milestones in a WGs activities, from writing the charter
through publishing Recommendations, and building or acquiring test suites and
tools.
4.2.1 First step -- charter commitment
Because QA is considered to be an integral part of the activities of each
WG, each WG has to consider and commit to a set of QA deliverables. A
spectrum of possible commitments, and associated guidelines, is found in the
charter-commitment
checkpoint(s) of the process and operational guidelines ([QAF-OPS], Checkpoint 1.1).
If a WG is being newly formed, then the WG charter needs to document its
QA deliverables, just as it does all other WG deliverables. Again, see the charter-commitment
checkpoint(s) of the process and operational guidelines ([QAF-OPS], Checkpoint 1.1). If a WG is being
re-chartered, this case should be considered the same as a newly formed WG --
the charter should document the WG's committed set of conformance and test
related deliverables.
For an already-chartered Working Group undertaking new QA projects, if
these deliverables are not documented in the charter already, then the W3C
Process Document [PROCESS] suggests that the charter
should be amended to document the new deliverables. Again, see the charter-commitment
checkpoint(s) of the process and operational guidelines ([QAF-OPS], Checkpoint 1.1).
4.2.2 Set up processes & logistics
Once WG commitments to QA deliverables are made and documented in the
charter, the next step is to set up the processes and logistics that the WG
will use for managing its QA activities. These include, among other
things:
- test materials (QA) moderator;
- "/Test/" home page in the Working Groups Web space;
- -ts email discussion list;
- WG process document for QA;
- decisions about (IPR) and test
materials.
See the sections about staffing WG QA efforts ([QAF-OPS], Guideline 4) and producing the WG's QA process
document ([QAF-OPS], Guideline 6) in the process
and operational guidelines, about how to do this.
4.2.3 Planning and writing the specification
There is a very tight relationship between how the specification (Rec) is
written on the one hand, and on the other hand its testability and its
suitability as the foundation for interoperable implementations.
New specification. "Specification Guidelines" [QAF-SPEC] should be applied from very beginning. Among
the key topics that authors and editors should consult are:
- conformance clause
- normative language throughout
- granular grammar and testable assertions
- optional features, discretion, extensions
- profiles, levels, conformance claims
Consider the guidelines and checkpoints of "Specification Guidelines" [QAF-SPEC] even at the stage of planning the structure
and presentation style of the spec. Along with W3C "pubrules" (the
member-only rules document for W3C editors who are publishing technical
reports) and "Style Manual" [STYLE-MAN], document
authors and editors should refer to the spec guidelines throughout, about
testable language, clarity, conciseness.
New "Edition" of specification. A new "Edition" of the same functional
level of a specification is typically used for incorporation of errata (e.g.,
XML 1.0 Second Edition). Normally, this should not be considered as a good
time to bring a specification for "Specification Guidelines" conformance, as
the latter could significantly disrupt and restructure the specification.
New "Version" of specification. A new "Version" of the specification
refers to a significant functional change and enhancement. This presents a
good opportunity to improve the testability and implementability of the
specification, as just described for new specifications.
4.2.4 Reviewing and progressing the specification
This stage in the specifications life has two significant aspects:
- the advancement of the specification to and beyond the first public
working draft (FPWD);
- and, the need to synchronize production of test suites and tools more
closely with the spec progression.
When the specification is published in TR
space, then non-WG W3C members and the general public begin to review and
comment. Such reviewers should consult and understand the material in
"Specification Guidelines" [QAF-SPEC], in order to
have an informed set of evaluation criteria for the conformance, testability,
and interoperability aspects of the specification.
WG members and especially WG-TS members should refer to
the sections (checkpoints) of the process and operational guidelines",
regarding specification exit
criteria ([QAF-OPS], checkpoint 1.2) and synchronization of the test
materials with specification progression ([QAF-OPS], Guideline 5). The project enters "The Matrix"
[MATRIX] at Last Call WD (if not sooner). Additionally, a de-facto
process convention is emerging, that there should be significant conformance
test materials at the stage of CR-exit/PR-commencement. This is the same
timing as the explicit process requirement of two interoperable
implementations.
4.2.5 Designing and building test materials
There are several scenarios for how the WG "builds" its conformance test
materials:
- design and build test cases and test tools in the WG, from scratch;
- import completed test materials from an outside group or
organization;
- assemble the test materials from contributed test cases and
materials.
In-WG build. Before starting the
development, the WG-TS
members should be thoroughly familiar with the material in "Test Materials
Guidelines" [QAF-TEST]. There is useful information
for both high-level planning -- e.g., breadth-first Basic Effectivity versus
fully detailed suite? -- as well as specific detail for the building of
individual test cases. Another aspect of building test materials is an
acceptance procedure for the individual bits, as they are built. This is
addressed in the review-procedures
requirements ([QAF-OPS], checkpoint 6.5) of the
process and operational guidelines.
Import completed test materials. Several high-quality test suites have
been developed outside of the relevant W3C WG, and then transferred to the
WG. WG's which are considering such a transfer should refer to the test materials transfer
guideline ([QAF-OPS], Guideline 7) of the process
and operational guidelines. Clearly, the quality of the candidate test
materials should be carefully assessed, and for this the "Test Materials
Guidelines" [QAF-TEST] can provide useful assessment
criteria.
Assemble
contributions. Some test suites have been built by implementing processes to
assemble significant chunks of material from outside (or internal member)
contributions. The "Process and Operational Guidelines" [QAF-OPS] addresses the steps needed to complete such a
transfer -- the are the same as the preceding paragraph about transferring
completed test materials. In addition, there should be careful quality
assessment of contributions, for which "Test Materials Guidelines"
[QAF-TEST] can be useful. Finally, there must be procedures for submission, review,
and integration of contributions ([QAF-OPS],
checkpoints 6.2, 6.4, 6.5), which are described in several related
checkpoints of the process and operational guidelines.
4.2.6 Publication of test materials
Typically, a WG-TS
group will want to publish releases of test materials, particularly as the
specification advances through its final stages (e.g., PR) towards
Recommendation. Test material publication is addressed in
TM publication checkpoint ([QAF-OPS], checkpoint 6.6) of the process and operational
guidelines.
As soon as the test materials become public, then there is definite need
for a procedure to process challenges to correctness, make determinations,
and appeal decisions. This is addressed in a test appeal checkpoint
([QAF-OPS], checkpoint 8.3) of the process and
operational guidelines.
Publication of test materials often comprises an implicit (or explicit)
invitation for contributions. The considerations described in "Assemble Contributions" are equally
applicable here.
4.2.7 Specification publication and beyond
When the specification reaches Recommendation, there is typically a
concurrent publication of the test materials. This might be considered a
"final" publication, or ongoing development may still be planned according to
one of the mechanisms discussed above. In any case, a maintenance procedure
must be in place for the test materials. Firstly, there are tie-ins between
approved specification errata and validity or applicability of particular
tests -- mechanisms for this are discussed in "Test Materials Guidelines" [QAF-TEST], and the general process overview is
discussed in the errata
update checkpoint ([QAF-OPS], checkpoint 8.2) of
the process and operational guidelines. Secondly, there is the
above-discussed need for both challenge/review/appeal processes. Finally,
even if WG-TS active
development of test materials ceases, it may be desired to continue to
consider submissions, review and integrate them per the requirements of the
test-contribution
checkpoint ([QAF-OPS], checkpoint 8.1) of the
process and operational guidelines.
4.2.8 Life after WG
It is possible that the WG and
WG-TS may disband
after its charter expires. This situation, and what to do about test
materials, is discussed in a "secure repository"
checkpoint ([QAF-OPS], checkpoint 2.2) of the
process and operational guidelines.
5. Conformance
This introductory document to the Framework documents family is entirely
informative. It contains no conformance requirements. Therefore the concept
of conformance to this document is not applicable.
The guidelines documents of this Framework document family do contain
conformance requirements, and each such document contains a conformance
chapter with an unambiguous definition of conformance to the documents
guidelines and checkpoints.
The following QA Working Group and Interest Group participants have
contributed significantly to the content of this document:
- Daniel Dardailler (W3C),
- Dimitris Dimitriadis (Ontologicon)
- Karl DuBost (W3C),
- Peter Fawcett (RealNetworks),
- Kirill Gavrylyuk (Microsoft),
- Dominique Hazael-Massieux (W3C)
- Lofton Henderson (CGM Open),
- Ian Jacobs (W3C)
- David Marston (Lotus Development Corp),
- Olivier Thereaux (W3C),
- Lynne Rosenthal (NIST).
- Mark Skall (NIST)
- Andrew Thackrah (Open Group)
- ISSUES-LIST
- QA Activity Issues List,
maintained by the QA Working Group, available at
http://www.w3.org/QA/WG/#issues.
- MATRIX
- W3C-wide conformance activity survey covering all the WGs, "The Matrix", available
at http://www.w3.org/QA/TheMatrix.
- OPS-EXTECH
- "QA Framework: Process & Operational Examples and Techniques",
not yet published.
- PROCESS
- W3C Process
Document, 19 July 2001, available at
http://www.w3.org/Consortium/Process-20010719/.
- QA-GLOSSARY
- Comprehensive glossary of QA terminology. (Under construction.)
- QAF-OPS
- "QA
Framework: Process & Operational Guidelines", Working Draft
companion version to this document, available at
http://www.w3.org/TR/2002/WD-qaframe-ops-20020201/.
- QAF-SPEC
- "QA Framework: Specification Guidelines", not yet published.
- QAF-TEST
- "QA Framework: Test Materials Guidelines", not yet published.
- QAIG
- Quality Assurance Interest
Group of the W3C QA Activity, which may be found at
http://www.w3.org/QA/IG/.
- QAWG
- Quality Assurance Working
Group of the W3C QA Activity, which may be found at
http://www.w3.org/QA/WG/.
- RFC2119
- Key words for use in
RFCs to Indicate Requirement Levels, March 1997, available at
http://www.ietf.org/rfc/rfc2119.txt.
- SPEC-EXTECH
- "QA Framework: Specification Examples and Techniques", not yet
published.
- STYLE-MAN
- "W3C Manual of
Style", summarizing the style and publication rules for W3C
technical reports, available at http://www.w3.org/2001/06/manual/.
- TAXONOMY
- QA Activity test
taxonomy, a classification scheme for conformance test materials,
available at http://www.w3.org/QA/Taxonomy.
- TEST-EXTECH
- "QA Framework: Test Materials Examples and Techniques", not yet
published.
- W3C-TR
- Location of all published W3C
technical reports, see http://www.w3.org/TR/.
- WCAG10
- Web Content Accessibility
Guidelines, version 1.0, W3C Recommendation, 5 May 1999, available
at http://www.w3.org/TR/WCAG10/.