The PDF version 
Display Related Directives to this directive.
NOT MEASUREMENT
SENSITIVE
DOE G 414.1-1A
5-31-01
MANAGEMENT ASSESSMENT
AND
INDEPENDENT ASSESSMENT GUIDE
FOR USE WITH
10 CFR, Part 830, Subpart A, and DOE O 414.1A, Quality Assurance; DOE P 450.4, Safety
Management System Policy; and DOE P 450.5, Line ES&H Oversight Policy
[This Guide describes suggested, nonmandatory approaches for meeting requirements. Guides are not
requirements documents and should not be construed as requirements in any audit or appraisal for
compliance with the parent Policy, Order, Notice, or Manual.]
U.S. Department of Energy
Washington, D.C. 20585
DISTRIBUTION:INITIATED BY:
All Departmental ElementsOffice of Environment, Safety and Health
CONTENTS
FOREWORD . . . . . . . . . . . . . . . . . . . . . . . . . . . .v
BACKGROUND . . . . . . . . . . . . . . . . . . . . . . . . . .vii
ACKNOWLEDGMENTS. . . . . . . . . . . . . . . . . . . . . . . . ix
1. INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . .1
2. APPLICATION. . . . . . . . . . . . . . . . . . . . . . . . .2
3. BACKGROUND . . . . . . . . . . . . . . . . . . . . . . . . .3
3.1 Assessment Program Expectations. . . . . . . . . . . .3
3.2 Assessment Benefits. . . . . . . . . . . . . . . . . .3
3.3 Graded Approach. . . . . . . . . . . . . . . . . . . .4
4. GENERAL INFORMATION. . . . . . . . . . . . . . . . . . . . .5
4.1 Purpose of Assessment. . . . . . . . . . . . . . . . .5
4.2 Types of Assessment. . . . . . . . . . . . . . . . . .5
4.3 Organizational Activity Levels . . . . . . . . . . . .6
4.3.1 Process Level Assessments . . . . . . . . . . .7
4.3.2 System Level Assessments. . . . . . . . . . . .7
4.3.3 Program Level Assessments . . . . . . . . . . .7
4.4 Assessing for Compliance, Effectiveness, and Performance8
4.4.1 Compliance Assessment . . . . . . . . . . . . .9
4.4.2 Effectiveness Assessment. . . . . . . . . . . .9
4.4.3 Performance-Based Assessment . . . . . . . . .9
5. GUIDELINES . . . . . . . . . . . . . . . . . . . . . . . . 10
5.1 Assessment Personnel . . . . . . . . . . . . . . . . 10
5.2 Assessment Program Planning. . . . . . . . . . . . . 11
5.2.1 Assessment Programs . . . . . . . . . . . . . 11
5.2.2 Management Assessment Planning. . . . . . . . 12
5.2.3 Independent Assessment Planning . . . . . . . 12
5.2.4 Planning Updates. . . . . . . . . . . . . . . 13
5.3 Assessment Integration . . . . . . . . . . . . . . . 13
5.4 Assessment Agendas . . . . . . . . . . . . . . . . . 13
5.5 Performance Criteria . . . . . . . . . . . . . . . . 14
5.6 Assessment Planning Tools. . . . . . . . . . . . . . 15
5.7 Independent Assessment Process . . . . . . . . . . . 16
5.7.1 Preassessment Meetings. . . . . . . . . . . . 16
5.7.2 The Entrance Meeting. . . . . . . . . . . . . 16
CONTENTS (continued)
5.7.3 Performing Independent Assessments . . . . . 17
5.7.4 Independent Assessment Techniques . . . . . . 17
5.7.5 The Exit Meeting. . . . . . . . . . . . . . . 19
5.7.6 Assessment Reporting. . . . . . . . . . . . . 19
5.7.7 Releasing and Responding to Assessment Reports 21
5.7.8 Corrective Action . . . . . . . . . . . . . . 21
5.7.9 Follow-up . . . . . . . . . . . . . . . . . . 22
5.8 Management Assessment Process. . . . . . . . . . . . 22
5.8.1 Defining the System . . . . . . . . . . . . . 23
5.8.2 Assessment Scheduling . . . . . . . . . . . . 23
5.8.3 Performing Management Assessments . . . . . . 24
5.8.4 Assessment Reporting. . . . . . . . . . . . . 24
5.8.5 Follow-up . . . . . . . . . . . . . . . . . . 24
5.8.6 Feedback. . . . . . . . . . . . . . . . . . . 25
APPENDIX A CONSENSUS STANDARDS AND REFERENCES. . . . . . . . .A-1
APPENDIX B ASSESSMENT FUNCTIONAL AREAS . . . . . . . . . . . .B-1
APPENDIX C TOOLS FOR ASSESSMENT PLANNING AND CONDUCT . . . . .C-1
APPENDIX D INDEPENDENT ASSESSMENT PLANNING . . . . . . . . . .D-1
FOREWORD
This Guide is approved for use by the Department of Energy (DOE) Office of Environment, Safety
and Health. It is intended and available for use by all DOE/National Nuclear Security Administration
components and contractors.
Suggestions for improving this Guide are welcome and should be sent to
Gustave E. Danielson, Jr.
EH-53/270CC
U.S. Department of Energy
19901 Germantown Road
Germantown, MD 20874-1290
Phone: (301) 903-2954
Fax: (301) 903-6172
e:mail: bud.danielson@eh.doe.gov
DOE Guides provide supplemental information for fulfilling requirements contained in rules, Orders,
Notices, and regulatory standards. Guides are also used to identify Government and non-
Government standards and methods DOE finds acceptable for implementing the Department's
requirements. Guides are not substitutes for requirements nor do they replace technical standards,
which are used to describe established practices and procedures for implementing requirements.
BACKGROUND
Since the1994 revision of this Guide, assessment practices have evolved. This revision reflects
current assessment practices, international standards, and changes in Department of Energy (DOE)
expectations, including the following:
feedback and improvement requirements of DOE P 450.4, Safety Management System Policy;
new requirements in DOE O 414.1A, Quality Assurance, for corrective action plans in response
to independent oversight assessments;
assessment requirements of 10 CFR 835;
requirements in the DOE implementation plan for Defense Nuclear Safety Board
Recommendation 98-1;
new and revised international management system standards [e.g., International Organization for
Standardization (ISO) 9001 and ISO 14001] and the integrated environmental and quality audit
standard (ISO CD.3 19011); and
DOE P 450.5, Line Environment, Safety and Health Oversight.
The Assistant Secretary for Environment, Safety and Health (ES&H) established a team to prepare
the initial version of this Guide (published August 1996). The team identified all assessment
requirements in the DOE ES&H Orders that were effective as of November 1994 and determined
that 10 CFR 830, Subpart A, and DOE Order 5700.6C (superseded by DOE O 414.1) contained
adequate assessment requirements. The Assistant Secretary for ES&H approved a significant
reduction in prescriptive assessment requirements. The team recommended a guide for assessment
be issued to convey helpful information from canceled Orders and current trends in assessment
methodology. The team's work yielded the following positive outcomes.
More than 400 requirements, rooted in more than 40 Orders, were replaced by the 2 found in
10 CFR 830, Subpart A, and DOE Order 5700.6C (now DOE O 414.1).
Factors that contribute to "stovepipe" assessment programs were deleted.
Flexibility to plan and implement assessment programs that provide value to an organization was
increased.
Ability to respond to customer needs was greatly enhanced.
Redundant and excessive assessments were no longer mandated or implied.
Ability to focus assessment resources on activities that pose the greatest risks and stand to
benefit the most from improvement opportunities was enhanced.
ACKNOWLEDGMENTS
The Department wishes to acknowledge and thank the following team members and additional
contributors. Without their expertise and dedication, the effort could not have succeeded.
Process Improvement Team Members:
Gustave E. Danielson, Jr.
DOE, Office of Nuclear and Facility Safety
Policy
Keith Rademacher, Team Leader,
Foster Wheeler Environmental Corp. Hanford
Margy Beckmeyer
Westinghouse Savannah River Company
David Brown
DOE, Richland Operations Office
Ken Gidlow
Dyn McDermott Petroleum Operations Company
Strategic Petroleum Reserve
John Gran
Westinghouse TRU Solutions LLC
Waste Isolation Pilot Project
Judith Malsbury
Princeton Plasma Physics Laboratory
Chuck Moseley
BWXT Y-12, L.L.C.
Y-12 National Security Complex
Denise Viator
Oak Ridge Institute for Science and Education
Additional Contributors:
Paul Chimah
DOE, Albuquerque Operations Office
Amy Ecclesine
University of California
Los Alamos National Laboratory
Gene Langston
Science Applications International Corporation
John Palmer
University of California
Lawrence Livermore National Laboratory
Donald White
Waste Management Federal Services
1. INTRODUCTION
The Department of Energy (DOE) and its contractors are required to perform management and
independent assessments [ Title 10, Code of Federal Regulations (CFR), Part 830, Subpart A,
"Quality Assurance Requirements," and DOE O 414.1A, Quality Assurance]. DOE P 450.4, Safety
Management System Policy, and the Federal Acquisition Regulation, 48 CFR 970.5223-1, also
require assessments as an element of the feedback and improvement safety function. Also, DOE
P 450.5, Line Environment, Safety and Health Oversight, relies upon contractor (self-) assessment
programs to reduce DOE oversight. This Guide gives information on establishing processes and
performing effective assessments in support of these Policies, regulations, and Orders.
(Appendix A contains a list of consensus standards and other references that support and
supplement the guidance in this document.)
Assessments integrated with the management system add value to products and services by
providing feedback and linking the management and conduct of work to meaningful improvement
actions. To enable management to take such actions, an assessment program should embody the
following principles.
Managers are involved in the assessment process to ensure results contribute to improved
performance of the programs, systems, and work processes.
Managers receive timely, objective feedback from assessments on the effectiveness of policies,
requirements, standards, processes, and procedures, including evaluations of whether the
organization complies with them.
Assessment process coordination and integration is maximized and disruption of the assessed
work is minimized.
Organizational culture is one of continuous quality improvement, and assessments are accepted
as contributors to the improvement culture.
Management's goal to protect people and the environment from harm is supported by
assessment results.
Quality problems (including safety issues) are identified for resolution by management.
Management takes timely actions to resolve quality problems.
This guide has been developed to assist DOE and contractor line management and personnel,
assessors, and others involved in the assessment process in understanding the philosophy,
requirements, expectations, and benefits of a comprehensive assessment program.
2. APPLICATION
All DOE products/services (and the programs, systems, and processes that deliver them) can be
assessed over their entire life cycles using this Guide. Environment, safety, and health (ES&H),
radiological safety, and safeguards and security activities are also considered products/services and
are subject to assessment (see appendix B). Technical standards and methods that DOE finds
acceptable for meeting the requirements of 10 CFR 830, Subpart A, and DOE O 414.1A, DOE
P 450.4, and DOE P 450.5 are referenced in this Guide. This Guide provides expanded detail on the
assessment criteria discussion in DOE G 414.1-2, Quality Assurance Management System Guide.
The Quality Assurance Management System Guide also describes the relationship of quality
assurance (QA) criteria and Safety Management System (SMS) requirements. The Integrated
Safety Management System Guide, DOE G 450.4-1B (vol. 2, appendixes D and G), describes the
role of assessment in the feedback and improvement safety management function (similar
discussions are included for applying the other eight QA criteria to SMS implementation).
The Department's line managers fulfill their safety responsibilities in part through line management
ES&H oversight and have unfettered access to information and facilities in accordance with safety
and security requirements. The contractor's line managers fulfill their safety responsibilities in part
through the implementation of their self-assessment programs. Contractors are responsible for
establishing robust, rigorous, and credible ES&H assessment programs integrated with their SMSs
(DOE P 450.5).
Assessment programs conducted in accordance with this Guide and appropriately adopted
standards will satisfy the requirements of 10 CFR 830, Subpart A, and DOE O 414.1A, DOE
P 450.4, and DOE P 450.5. (Alternative methods may be acceptable to DOE if the methods are
demonstrated to achieve an adequate level of safety and quality.) This Guide also provides a basis
for determining the adequacy of QA programs and integrated SMS descriptions (DOE P 450.4 and
DOE P 450.5) prepared in response to requirements in the previously noted Policies, Orders, and
regulations.
3. BACKGROUND
3.1 Assessment Program Expectations
The development of an effective assessment and safety management program must focus on
achieving DOE expectations, including the following.
A documented assessment program, defining the systems that will be used to plan, perform, and
follow up on assessments, is in place.
Responsibilities for both performing and responding to assessments are defined.
Management at all levels is responsive to identified issues, regardless of how they are identified.
Actions are taken promptly to correct identified problems and prevent recurrence.
Information can be independently verified.
Feedback is solicited from a variety of sources (e.g. management, workers, independent
evaluations, customers).
Measurable organization goals and objectives have been identified and progress toward those
goals and objectives can be demonstrated.
3.2 Assessment Benefits
The success of an organization depends upon the extent to which its products and services satisfy
customer requirements and expectations. Each member of an organization is responsible for
customer satisfaction. The quality program described in 10 CFR 830, Subpart A, and DOE
O 414.1A (also referred to as "the rule" and "the Order" in this Guide) provides a results-oriented
management system that focuses on the customer's requirements and expectations and embraces
continuous improvement. The assessment component of this management system builds
confidence that organizations can meet customer expectations. Assessments also provide objective
evidence of those areas where improvement is needed to achieve organizational goals and objectives.
Effective internal assessments prepare an organization for third-party, assessments performed by
external governmental and nongovernmental bodies. Third-party assessors typically assess for
performance to and/or conformance with national and international standards (i.e., conformity
assessment). Third-party assessors are neither direct suppliers nor direct customers of the
organization. The organization contracts for the conformity assessment service.
Voluntary third-party conformity assessment bodies include quality/environmental management
system registrars, laboratory accreditors, and product certifiers. By contracting with one of these
assessment bodies, an organization can have its products; laboratory; and/or QA, health and safety,
and/or environmental programs registered/certified as compliant with various national or international
standards. DOE contractors can participate in the Voluntary Protection Program and attain
recognition for excellence in safety and health management. In addition, various national and state
quality awards (e.g., the Presidential Award for Quality and the Malcolm Baldrige National Quality
Award) use comprehensive assessments that focus on integrated management systems and customer
service. All of these assessments provide evaluations of the management systems and
implementation.
Regulatory or oversight bodies such as the Nuclear Regulatory Commission, the Environmental
Protection Agency, or the Occupational Safety and Health Administration measure compliance with
regulatory requirements, standards, and related commitments (involuntary, third-party assessments).
The benefit of an involuntary third-party assessment is the confirmation of compliance with
regulatory requirements or the identification of noncompliance.
Regardless of their main purpose, third-party assessors have at least one common interest:
determining whether an organization has established and implemented an effective assessment
process. In the regulatory arena, an effective assessment process, coupled with prompt
improvements and corrective actions by management, may be considered a mitigating factor in
determining civil and criminal enforcement penalties and in turn may yield reduced monetary fines
and criminal sentences. In the voluntary arena, an effective assessment process can reduce the time
and frequency of third-party assessments. This translates to lower conformity assessment service
costs for the organization.
3.3 Graded Approach
This Guide and the technical standards referenced herein should be applied using a graded
approach. Items, services, or programs that contribute the greatest risk to quality, safety, and
mission are assessed with the greatest rigor and frequency.
4. GENERAL INFORMATION
4.1 Purpose of Assessment
Establishing and implementing an effective assessment program is an integral part of every
management system. Assessment is an important feedback mechanism that provides management
with information to evaluate and improve any aspect of an organization, for example:
organizational progress in reaching strategic goals and objectives;
adequacy and implementation of management programs for mission achievement;
performance capability of SMSs;
products and service quality; and
regulatory and contractual compliance.
Simply stated, an assessment is an opportunity to
identify the gaps between where you are and where you want to be,
identify the reasons for the gaps,
identify the actions that will be taken to close the gaps (corrective actions),
close the gaps between where you are and where you want to be, and
verify that corrective actions have been effective and lasting.
4.2 Types of Assessment
The Department's QA rule and Order establish distinct requirements for two types of assessment:
(1) management assessments and (2) independent assessments. DOE P 450.5 refers to contractor
self-assessment programs that include line and independent evaluations. In this context, self-
assessment is simply the assessments that a contractor conducts on its own ES&H performance.
Management and independent assessments performed in accordance with this Guide will satisfy the
requirements of DOE P 450.5; however, contractors must clearly describe how their self-assessment
programs satisfy the requirement for independent assessment and/or the requirement for
management assessment.
Assessments are tools for improvement. Management and independent assessments may be
performed on the same functions or organizations; however, each has a specific focus, defined by
the QA rule and Order and described below.
"Criterion 9 Assessment/Management Assessment. Ensure managers assess
their management processes and identify and correct problems that hinder the
organization from achieving its objectives." [10 CFR 830.122(i)]
Managers must perform management assessments to comply with the rule and Order. Management
Assessments look at the total picture: how well the management system meets the customer's
requirements; the expectations for safely performing work; and the organizational mission, goals,
and objectives. The emphasis of management assessment is on management issues that affect
performance and related processes such as strategic planning, personnel qualification and training,
staffing and skills mix, communication, and cost control; organizational interfaces; and mission
objectives. The purpose of this type of assessment is to identify the management aspects of
performance and make improvements. Management assessment is an introspective self-analysis to
determine whether the management infrastructure is properly focused on achieving desired results.
Typically, management assessments are performed at a greater frequency than independent
assessments and cover a broader spectrum.
"Criterion 10 Assessment/Independent Assessment. (1) Plan and conduct
independent assessments to measure item and service quality, to measure the
adequacy of work performance, and to promote improvement. (2) Establish
sufficient authority, and freedom from line management, for the group performing
independent assessments. (3) Ensure persons who perform independent
assessments are technically qualified and knowledgeable in the areas to be
assessed." [10 CFR 830.122(j)]
Independent assessments evaluate the performance of work processes with regard to requirements,
compliance, and expectations for safely performing the work and achieving the goals of the
organization. The focus of independent assessments should be the items and services produced and
their associated processes. The purpose is to improve product/service performance and process
effectiveness. Independence is defined as not having direct responsibility for the work being
assessed. Independent assessments typically are performed by personnel from organizations or
work units outside the one being assessed. Thus, management receives an objective view of the
assessed activity. Independent assessments are typically performed less frequently than
management assessments but go into greater depth.
Management is responsible for developing and implementing a coherent plan that balances
management and independent assessments and other forms of feedback and improvement to satisfy
the requirements of the rule and Order.
4.3 Organizational Activity Levels
To shape a comprehensive assessment program that optimizes the application of each assessment
type, it may be helpful to visualize the organization as having three interlinked levels of activity
(figure 1). For this discussion, these levels will be referred to as "process," "system," and
"program." A process is a collection of steps or actions that yield some intermediate outcome; a
system is made up of two or more processes that may operate independently or interdependently
and may yield a complete product or service. A program is the most complex level and consists of
multiple, interdependent systems that often require
many interfaces to provide the desired product or
service. Management and independent assessments
can be applied at all three levels but will examine
different aspects of them.
4.3.1 Process Level Assessments
Process level assessments involve examination of
work controls and verification that they are being
implemented effectively. This level of assessment
is critical for ensuring that the worker, the public,
and the environment are protected from harm.
Process level assessments should also assess the
effectiveness of the processes from a quality and
customer satisfaction perspective.
4.3.2 System Level Assessments
System level assessments focus on whether
appropriate leadership and support systems are
provided to enable the implementation of work processes. These assessments are performed to
ensure human and material resources are being properly used to achieve an organization's mission
and objectives. This level of assessment may range from informal daily oversight of performance to
formal periodic evaluations using established protocols.
4.3.3 Program Level Assessments
Program level assessments are used to determine whether overall organizational programs are
properly established and implemented. They are appropriate for evaluating complex organizations
from several perspectives; consequently, program assessments usually examine the integration of the
many systems designed to achieve organizational goals and customer expectations (with an emphasis
on ES&H factors).
The following paragraphs demonstrate how the different types and levels of assessment could be
applied to an organization's work control methods. Some of the elements within the work control
system are planning, identifying hazards, identifying controls, scheduling, performing,
verifying/testing, and documenting the work performed.
1. At the process level, assessments would be performed by independent assessors to verify
compliance with procedures and to ensure the work-control documents (e.g., procedures,
instructions, radiation surveys, permits, and safety checklists) accurately reflect the task and
associated hazards.
2. At the system level, assessments would be performed to determine whether all the necessary
elements and interfaces are addressed to ensure the system is capable of consistently meeting
requirements and customer expectations. A management assessment of the work control system
might determine the cost and resource allocation issues that impact the system.
3. At the program level, a maintenance management program, which relies on the work control
system, would use results from the process and system level assessments to determine the
effectiveness of the entire maintenance program. This program assessment could be performed
as either a management assessment or an independent assessment. A management assessment
might focus on comparing the strategic goals for maintenance with actual performance to
determine whether the rewards and recognition plan targeted to improve maintenance has had the
desired effect. The independent assessment might compare the program results with contractual
and regulatory commitments or customer requirements.
>From the senior management perspective, the primary focus of management assessments should be
at the program level and concerned with strategic issues. Data on systems performance can be
rolled up from midlevel and first-line management assessments and independent assessments.
Examples of comprehensive assessment models at the program level include the Malcolm Baldrige
National Quality Award, Operational Readiness Reviews, and the Voluntary Protection Program.
Independent assessment emphasis is placed on system performance in support of programs and to
determine its ability to deliver product and services that meet customer expectations. Independent
assessments may also be used to confirm management assessment results where organizational
vulnerability is high (e.g., potential regulatory penalty, ES&H significant hazard).
>From the first-line management perspective, the primary focus of assessments should be the
capability of systems and the processes that support them. To ensure that these systems contribute
to program goals, managers must evaluate system performance based on these goals. Ultimately
management, with support from assessments, is responsible for planning the balance and application
of independent and management assessments to ensure they improve and add value to the
organization.
4.4 Assessing for Compliance, Effectiveness, and Performance
There are three different methods commonly used for accomplishing assessments. These are
usually known as compliance assessment, effectiveness assessment, and performance-based
assessment. While each method has distinct characteristics, a good assessment will use elements of
all three.
4.4.1 Compliance Assessment
Compliance assessment focuses on verifying compliance with requirements through the
implementation of procedures. Compliance assessment begins with a determination of the
contractual and regulatory requirements binding the assessed organization. Assessors then verify
that requirements flow down to implementing documents such as procedures, whose implementation
is in turn verified.
Assessing for compliance alone may not adequately identify higher level systemic or programmatic
problems or determine the effectiveness of the program. In many situations an organization has
written procedures that appear to implement the requirements; however, in practice the intent of the
requirements is not fully achieved because of variables such as the way those procedures are
executed.
4.4.2 Effectiveness Assessment
Effectiveness assessment begins like compliance assessment, looking for implementation of
requirements in procedures and compliance with the procedures in the workplace. This is followed
by a determination whether pure compliance has led to effective implementation of the intent of the
top-level requirements. The assessor is expected to determine whether a noncompliance or series of
noncompliances with procedures could actually result in a failure to satisfy top-level requirements.
The assessor must return to the top-level requirements to determine the program effectiveness.
4.4.3 Performance-Based Assessment
Performance-based assessment takes a different approach by focusing first on the adequacy of the
process that produced a product or service and then the product itself. If problems are found in the
product or work processes, then the assessor evaluates the methods and procedures used to
implement the applicable requirements. This is done to find the failure that led to the problems.
In performance-based assessment, great emphasis is placed on getting the full story on a problem
before coming to a conclusion. If an assessor sees a problem with the execution of a welding
process, he or she should determine the extent of the problem. Is it limited to one welder? Is it
limited to one process? Can the problem be traced to the qualification program for the welder or to
the qualification program for the welding process? Or is there a problem with the weld material
itself, indicating an engineering or procurement problem?
While the assessor must be familiar with requirements and procedures, in performance-based
assessment the assessor's experience and knowledge play an integral part in determining whether
requirements are satisfied. Therefore, participants in performance-based assessments must be
technically competent in the areas they are assessing. For example, if an assessor is evaluating a
welding process, the assessor relies heavily on his or her knowledge of welding codes, welding
processes, and metallurgy, rather than just verifying simple procedure compliance.
Performance-based assessment usually provides the most useful information to management;
however, it requires a much higher level of competence on the part of the assessment team. In
practice, assessments are likely to include elements of all three methods.
5. GUIDELINES
Organizations should establish procedures for planning and performing management and
independent assessments. These should address training and qualification of personnel, planning
the assessment processes, performance protocols and tools, reporting, distributing reports, and
developing and implementing corrective actions and other follow-up activities. The following
guidelines are presented to assist organizations in developing their procedures and protocols.
Management assessments share many procedural and protocol commonalities with independent
assessments. Because of this, the organization must ensure assessment procedures are well defined
and integrated while maintaining the separate focuses of management and independent assessments.
5.1 Assessment Personnel
Assessment personnel facilitate continuous process improvement by identifying ways programs,
systems, and processes can be improved and by providing information to management and process
owners. The assessor should be able to collect performance data through interviews, document
reviews, observation, and inspection. It is very important that the assessor also be able to
communicate effectively, both orally and in writing, and demonstrate effective interpersonal skills.
Both management and independent assessments should be accomplished by qualified individuals
who are knowledgeable of the program, system, or process being assessed and have had training to
ensure an understanding of the assessment processes and the desired outcomes.
Individuals performing independent assessments should not have performed, supervised, or been
directly responsible for performing the activities or in the areas being assessed. Independence is
determined based on the individual not having bias, rather than on organizational affiliation. The
independent assessor must have both the personal and organizational freedom to communicate with
the management of the assessed organizations.
Organizations should establish a formal training and qualification program for independent
assessors, including both assessment team leaders and team members, that reflects both regulatory
and customer requirements. Organizations may adopt third-party personnel qualification programs
such as the American Society for Quality's "Quality Auditor Certification"
(http://www.asq.org/cert/types/index.html) or the Registrar Accreditation Board's certification
program (http://www.rabnet.com/index.shtml). The International Organization for Standardization
and the American Society of Mechanical Engineers provide additional guidance for training and
qualification of assessors (ISO-9001 or ISO-14001 and NQA-1, respectively). At a minimum,
however, training programs should be based on some recognized, relevant standard.
Effective assessments may be accomplished through use of an assessment team with combined
skills and experiences. Training for assessors should address the policies and procedures of the
assessing organization. To enhance assessment performance and capability, new assessment
personnel should participate in on-the-job training with qualified, experienced assessors before being
considered fully trained or receiving a required qualification. Further guidance on assessor training
and qualification is provided in DOE Orders and Guides and the standards in appendix A.
5.2 Assessment Program Planning
5.2.1 Assessment Programs
Assessment programs should be developed to the level of rigor and detail required to ensure
adequate review of programs, systems, and processes. An assessment program is a guide for the
overall process and ensures assessments are conducted in a cost-effective, efficient manner. Items
(figure 2) considered essential for a comprehensive assessment program include the following.
Assessment scheduling, planning approach,
and logic (including how independent and
management assessments are balanced).
Methodology for determining/developing
performance criteria.
Recognition and use of third-party
assessment results (accreditation,
certification, registration, and regulatory).
Assessment ethics and behaviors.
Qualification and training of assessment
team personnel.
Protocols for conduct, including interfaces
and meetings.
Format/review of assessment plans and
agendas.
Reporting methods/procedures for
concerns, findings, observations, and
improvement opportunities, including
distribution, and mechanisms for
addressing imminent danger issues.
Procedure/process for verifying and
following up on concerns.
Assessment records management program, including identification of records that will be
retained, retention periods, and protection.
Assessments should be conducted at a frequency commensurate with the hazards, status, and
importance of the program, system, or work process and should be focused on worker health and
safety, public health and safety, environmental protection, community concern, strategic planning,
organizational resources, compliance and liability, and business efficiency and productivity.
Complexity, reliability, risk, and economic considerations should also be considered when planning
and scheduling assessments. The application of a graded approach using a risk-based decision-
making process will ensure that resources are applied in a manner that provides the greatest benefit
to the assessed organizations and their customers.
5.2.2 Management Assessment Planning
Management assessments should be planned in a systematic manner by the individual managers to
address all areas under their responsibility and to focus on those areas presenting the greatest risk
for failure or potential for improvement or that have not been covered by an independent
assessment. Senior management should retain overall responsibility for the planning and
performance of management assessments. Results of the planning process should be documented
in an assessment plan. Those responsible for planning individual assessments should consider the
following:
descriptive title or name for the assessment area;
brief description of the area or activity to be evaluated;
identification of the assessment team leader and team members;
schedule for the start and completion of the assessment, including issuance of the assessment
report; and
other information related to the actual assessment evaluation (e.g., performance objectives,
management systems, resource availability, efficiency measures, effectiveness measures).
Management assessment planning should include the mechanics of performing the assessment such
as the expectations for time involved, assessment tools that may be used, reporting requirements,
and how areas for improvement will be identified, tracked, and closed.
Management assessments should focus on the identification and resolution of both systemic and
cultural management issues and problems that may contribute to customer expectations not being
met. Assessments should include evaluating conditions such as the state of employee knowledge,
motivation, and morale; communication between employees and management; and the adequacy of
human and material resources.
5.2.3 Independent Assessment Planning
The purpose of independent assessment planning is to ensure assessments efficiently address the
objectives of the assessment program with the least amount of disruption to the facility/organization
being assessed. The level of planning will vary significantly depending on scope, breadth, and
complexity of the system or process being assessed. The planning process should provide for
input from the assessed organizations and consideration of their customers and stakeholders.
Specific budget requirements and required resources and support should be identified as early in the
planning process as possible. Administrative issues such as the need for any review documents
before the actual assessment visit and considerations for travel and clearances should be addressed.
It is important to remember that each assessment needs to characterize a program, system, or
process during a limited time. Effective planning will ensure this occurs.
There are many scheduling and planning approaches to satisfying these requirements. Appendix C
describes tools to aid in assessment planning, and appendix D provides an example of planning and
scheduling independent assessments using an integrated, risk-based approach.
5.2.4 Planning Updates
Assessment planning should be periodically reviewed and modified as new information on the
facility or organization is obtained. An assessment that finds good performance should be used as
justification to reduce the frequency and depth of future assessments. Areas of poor performance
should receive increased attention, especially if there are indications that management is unable to
correct identified problems, because recurring and cumulative deficiencies, even in a low hazard
operation, may decrease the likelihood of its achieving its mission.
5.3 Assessment Integration
Multiple layers of assessment of a single organization or facility do not add value and keep the
organization from accomplishing its mission. Therefore, routine communication and trust among the
various levels of assessment bodies is essential in coordinating plans. This task involves identifying
overlapping and redundant assessments to reduce their negative impact on work performance. Once
identified, assessments can be coordinated and consolidated in several ways.
Change assessment scope to prevent two assessment organizations from performing the same
type of assessment on the same subject.
Combine separate assessment teams to evaluate the subject in a single visit.
Cancel an assessment based on an agreement to share the results of an assessment.
Where significant redundancies exist, as when many contractors perform supplier quality
assessments of analytical laboratory services, recognition and use of accredited, third-party
assessment results and shared information from other contractors should be considered.
(Note: The DOE Supplier Qualification Information Group is one source of shared information.)
Each of these methods requires staff time to develop, implement, and manage; however, they can
result in substantial savings in resources while enhancing the ability of the assessing organization to
cover a greater number of activities.
5.4 Assessment Agendas
The use and detail of an assessment agenda will vary depending on what is being assessed and
whether the assessment is a management assessment or an independent assessment. Agendas are
used to scope and plan individual assessments and should include input not only from the assessed
organizations but from their customers. A documented assessment agenda not only allows
communication of expectations to the assessed organization, but also allows the assessment team to
focus its activities more effectively. The scope of the assessment should be defined in terms related
to the assessed organization's mission and goals so the focus and value of the assessment will be
clearly understood. The extent of detail included in the assessment agenda should be commensurate
with the protocols of both the assessed and assessing organizations. The assessment agenda should
include the following items.
Team members and their qualifications, organizations, and interfaces.
Description of the assessment scope and performance criteria.
Dates of the assessment.
Schedule of assessment meetings preassessment, daily, and postassessment.
List of documents to be provided the assessment team when it arrives.
Request for office space, phone lines, and other administrative support as required.
Request for site safety briefing.
Request for points of contact for each functional area.
The organization being assessed should be contacted and provided an assessment agenda as soon
as possible before the assessment (unless regulations or contracts specify other contact protocols).
Management and internal, independent assessments in smaller organizations may not involve as
much information as that listed above nor the degree of formality required for assessments
performed by external organizations.
5.5 Performance Criteria
Assessments seek to ensure that performance expectations defined by management and process
owners are being met. Assessors should clearly understand the programs, systems, or processes
being assessed, including their goals and associated objectives and requirements for efficient,
effective performance of operations. Performance requirements can be found in the following
source documents.
Source Documents
Federal and State regulatory requirements
Appropriate codes and standards
Contract requirements
DOE Orders, Manuals, and Notices
Implementation plans
Implementation procedures
Facility safety documents
Policy and mission statements
DOE-approved Work Smart Standards
Standards/Requirements Identification Documents (S/RIDs)
Plans and programs
Much information on performance and additional performance requirements may be available to
assessors in documents and reports like the following.
Performance Information
Reports from outside regulators
Facility operations reports
Performance reviews
Previous assessment reports
Internal inspections, reviews, and reports
Corrective action plans and status reports
Concerns and occurrence reports
Performance indicators
Price-Anderson Amendments Act Nonconformance Tracking System reports (at URL
http://tis.eh.doe.gov/enforce/nts/nts.html-ssi)
Requirements contained in these documents are selected based upon impact on the assessed
organization's mission and the relationship to the scope of the assessment. From selected
requirements, objective statements (performance measures) are developed for determining whether
or not a program, system, or process is working efficiently and effectively. From these measures,
the specific performance criteria (based on written programs, DOE Orders, rules, etc.) are
developed and tools selected for conducting the appraisal. In developing performance criteria,
assessment personnel should not reinterpret or redefine requirements specified in the source
documents.
5.6 Assessment Planning Tools
Assessment planning tools such as checklists are an essential element of an effective assessment.
They vary in format, content, and level of detail, but all have one thing in common: they help focus
the assessor on the mission and objectives of the program, system, or process being assessed.
Application of planning tools before an assessment ensures more effective use of time and ensures
the focus of the appraisal is identified and maintained during the course of the assessment.
Assessment planning tools are often used to relate the performance criteria to the established
assessment scope and may include lists of interview questions, major elements of programs, or
detailed process work steps. Similar to a road map, each tool is used to remind the assessor of
where he/she is going and the items likely to be encountered along the way. Planning tools are
extremely useful when the assessment basis is complex or the requirements come from multiple
sources. Typical planning tools include matrices, flowcharts, cause-effect diagrams, tree diagrams,
checklists, and information systems. (See appendix C for examples and further discussion of these
tools.)
5.7 Independent Assessment Process
Sections 5.7 and 5.8 discuss independent assessment and management assessment, respectively, in
detail. As discussed previously, there are many commonalities, so to avoid unnecessary duplication,
Section 5.7 is very detailed, pointing out areas where the two types of assessment differ, and
Section 5.8 is briefer, concentrating on those elements unique to management assessments.
5.7.1 Preassessment Meetings
The effectiveness and efficiency of independent assessments can be improved greatly if
representatives of the assessment team meet informally with key members of the assessed
organization. This should be done at least a week before the assessment fieldwork actually begins.
Preassessment meetings are particularly useful when the assessment team is completely external to
the organization being assessed. These meeting are an opportunity to clarify the assessment team's
agenda and to work out logistical problems. The assessment team should identify and schedule the
individuals they will need to interview so that arrangements can be made to ensure their availability.
Documents that require effort to retrieve can be identified in advance, and the need for assessment
team work space can be resolved.
5.7.2 The Entrance Meeting
An entrance meeting should be conducted immediately before the assessment fieldwork begins.
This is the appropriate place to "set the stage" for the performance of a positive and productive
independent assessment. (Such meetings may not be necessary for management assessments.)
This meeting is held between personnel from the assessing organization and the managers of the
organization being assessed. It usually takes place at the assessed organization's location/facility.
The purpose of this meeting is to allow the assessment team to meet the assessed organization's
managers and to answer any questions they may have about the assessment. This meeting is also
used to establish how concerns involving imminent danger or regulatory noncompliance will be
communicated. The protocols to be followed during the assessment should be clarified during the
meeting, which usually includes a discussion of the following:
purpose and scope of the assessment, including authority for conducting the assessment;
assessed organization's mission, program, systems, and processes;
scheduled length of the assessment;
source documents and performance information that form the basis for the performance criteria
to be used;
knowledgeable individuals from the assessed organization as points of contact for each
assessor;
any restrictions on the collection and/or disposal of assessment notes/records by the assessors;
logistics, including work area, working hours, lunch hours, etc.;
time and location of periodic status meetings; and
time and place of the postassessment meeting.
5.7.3 Performing Independent Assessments
The assessment should be conducted in accordance with established protocols developed by the
assessing organization. Any agendas or specific protocols established during the preassessment
meeting are used to ensure that the assessment is conducted effectively and safely. Assessors
should keep their points of contact informed of their activities to preclude surprises during the
postassessment conference. This may include requests for additional assistance or the
communication of concerns that require immediate action on the part of the assessed organization.
Timely communication, verbal and written, will allow the assessed organization to verify the
accuracy of observations and provide relevant facts and background on the issues. One way to
accomplish this communication is to meet periodically (daily or every second day) with the
organization being assessed to convey questions, concerns, and status.
Daily team meetings provide assessment team leaders with information on the completion status of
the assessment plan and issues requiring additional action (e.g., clearances, access, requests for
personnel or material, and impasse resolution) and may be useful to ensure continuity and overall
focus. These meetings are also the setting for informing other team members of issues that may be
of interest in their assigned scope or for integrating data gathered by the various assessors. The
meetings should be brief so that they do not reduce the team members' time with the processes they
are to assess and the people they are to interview.
It is important that sufficient information is gathered during the assessment to determine whether an
activity meets the performance criteria established. The assessor should be able to clearly state the
criterion impacted by the activity and whether identified results also impact the mission/goals of the
organization. To accomplish this, the assessor may deviate from the assessment agenda to
determine the extent and significance of an issue. Deviations that affect the assessor's ability to
complete the assessment agenda must be made known to the team leader and the organization being
assessed.
5.7.4 Independent Assessment Techniques
Effective assessments use a combination of tools and techniques to maximize the productivity of the
assessment team and resources. Such assessment techniques include document reviews, interviews,
observation, inspection, and performance testing. Use of the planning tools discussed in appendix
C also allows for more complex analysis and systematic coverage of the areas being assessed. In
using these techniques, the assessor should not forget that the objective is to verify accomplishment
of an organization's mission. To save time, the assessor should gather only data and information
relevant to overall program performance and the achievement of program objectives.
It is generally not acceptable to identify suspicions about the adequacy or inadequacy of a program,
system, or process. Investigations should be sufficiently thorough and information gathered with
sufficient diligence that accurate, detailed conclusions and issues can be provided to assist the
organizations that will receive the final report.
In using any of these techniques, assessors should maintain good records of the assessment results.
These may include personal notes or other information to support the assessment and may be
included in the checklist information. These records are useful in writing the report and any
associated findings and recommendations and will become invaluable if questions arise during the
report review process. All classified notes should be disposed of properly in accordance with
established and agreed-upon procedures. A discussion of each of the techniques follows.
5.7.4.1 Document Review. Document review is used extensively during an assessment to
substantiate the information obtained during interviews and observation. During the course of an
assessment, questions may arise concerning what is heard and seen. The review of documents
(including logs, procedures, work orders, and other data) provides a method for answering these
questions and validating the assessment results. The drawback of document review is that the
accuracy of the records cannot be ascertained by review alone. This technique should be combined
with interviews, observation, inspection, and/or performance testing to complete the picture of
performance. Records and documents should be selected carefully to ensure they adequately
characterize the program, system, or process being assessed.
5.7.4.2 Interviews. Interviews provide a means to verify the results of observation, document
review, inspection, and performance testing. In addition, interviews allow the responsible person to
explain and clarify those results. The interview helps to eliminate misunderstandings about program
implementation and provides a venue where apparent conflicts or recent changes can be discussed
and the organization and program expectations can be described. Tools developed during
assessment planning are used to prepare for the interview. Assessors should also prepare questions
in advance to keep the interview focused.
5.7.4.3 Observation. Observation, the viewing of actual work activities, is often considered the
most effective technique for determining whether performance is adequate. Assessors should
understand the effect their presence has on the person being observed and convey an attitude that is
helpful, constructive, positive, and unbiased. The primary goal during observation is to obtain the
most complete picture possible of the performance, which should then be put into perspective
relative to the overall program, system, or process.
Before drawing final conclusions, the assessor should verify the results through at least one other
technique.
5.7.4.4 Inspection. Inspections are performed to verify the adequacy and condition of physical
facilities, systems, equipment, and components. Usually inspections are used to obtain additional
information concerning other items evaluated during the assessment, such as equipment labeling,
configuration control, the status of system lineups, adequacy of construction, or material storage.
Inspections may also be performed to gain information and data for interviews and/or work
observation. While on these inspections, the assessor must heed all security and safety
requirements. It is always a good practice to be accompanied by someone familiar with the facility.
5.7.4.5 Performance Testing. Performance testing is used to observe the response of personnel
or equipment by creating a specific situation and noting performance. This technique is especially
useful when activities of interest would not normally occur during an assessment visit. It is also
used when timeliness and appropriateness of the response are critical to an organization (e.g.,
emergency responses).
5.7.5 The Exit Meeting
Independent assessment programs can gain value from a postassessment exit meeting. This meeting
is used primarily to present the assessment summary and provide the assessed organization an
opportunity to verify the factual accuracy of assessment results. To facilitate this, assessors should
be prepared to provide detailed supporting information for those results (ideally, a draft assessment
report should be available at this time). This meeting also offers an opportunity for the assessed
organization to present its management position and any plans for addressing the results.
Reasonable time should be allowed to discuss any concerns, but this meeting should not be used to
argue the assessment agenda or methodology.
5.7.6 Assessment Reporting
Assessment reports are required to communicate the issues identified during an assessment.
Assessment team leaders have the overall responsibility for preparing the report and obtaining
approval for its release from their senior management. The assessment report may be formal or
informal, depending on the level of assessment performed, but should provide a clear picture of the
results in terms of the programs, systems,
and processes assessed. The report
should be clear and easy to understand and
should include only facts that directly relate
to assessment observations and results. It
should include sufficient information to
enable the assessed organization to check
the report for accuracy (if such a check
was not done during the assessment) and
to develop and implement appropriate
improvement plans. Every effort should
be made to ensure assessment reports are
concise, accurate, and understandable.
For example, summary information may be
clearer or more easily understood if
presented graphically (figure 3). In
preparing the report, authors should also
remember that many people who will read
the report have had no active role in the
assessment and the report may be their
only source of information regarding its
conduct and results. A recognized good
practice is to provide a draft copy of the
report to the assessed organization to allow
the staff to comment on the factual accuracy (if a draft for this purpose was not presented at the
postassessment meeting); however, the review is only to confirm factual accuracy, not to contest or
argue the assessment team's conclusions.
Specific report formats may vary considerably from one organization to the next. In developing a
report format, the assessment organization should solicit input from report recipients to ensure the
report meets their needs. An independent assessment report usually includes the sections described
below. (Note: A management assessment report may be less formal and may only require the
executive summary.)
5.7.6.1 Executive Summary. This summary should be a brief, stand-alone document. It should
describe the programs, systems, and processes assessed and the overall assessment results,
including an evaluation of the effectiveness, efficiency, and adequacy of the area(s) assessed and the
overall results. The executive summary should describe the strengths and weaknesses affecting the
assessed organization, including barriers to performance, so that meaningful action can be taken for
improvement.
5.7.6.2 Observation Section. Each part of this section should focus on the established
assessment scope and the identified organization mission; otherwise, the recipient of the report will
question why a specific area or activity was assessed. The section should include general
background on the assessment, including team members, scope of the assessment, methodology
used, and a summary of the assessment basis and source documents. This section should also
include a detailed discussion of each area assessed, including specific performance criteria used and
summaries of interviews, documents reviewed, observations, and inspections. The summaries
contained in this section should support the specific items discussed under the results section.
Noteworthy practices identified during the assessment should also be documented so that the
assessed organization and other organizations can learn and build upon them.
5.7.6.3 Results Section. This section should list and discuss specific problem areas or
deficiencies, areas needing improvement, or noteworthy practices identified during the assessment.
In addition,
this section should highlight any recurring problems as indicators of ineffective corrective action by
the assessed organization. For each item listed, the report should include a discussion of the
specific performance criteria used and the basis for the nonconformance in sufficient detail to enable
further analysis and action by the responsible organization. The report should also include any
required postassessment actions by the assessed organization. For example, a series of "like"
discrepancies may be symptoms of an underlying system problem. Therefore, a single issue should
be developed that cites the individual discrepancies as evidence of a system breakdown.
Issues should be defined, labeled, and enumerated in a manner that facilitates a response. While this
should be in accordance with the assessing organization's assessment program, the language used
should clearly distinguish objective noncompliances from observations, opinions, and improvement
opportunities.
5.7.6.4 Attachments. Attachments provide supplementary information to validate the assessment
and its methodology. They can be helpful in planning corrective actions and follow-up. Items
frequently included as attachments to assessment reports are the assessment agenda, a list of
persons contacted, a list of documents reviewed, performance criteria, and the tools used to
perform the assessment.
5.7.7 Releasing and Responding to Assessment Reports
While team leaders have overall responsibility for the report, the entire assessment team should have
an opportunity to read and sign the completed report. At a minimum, the final report should be
distributed to the management of both the assessed and assessing organizations. Distribution to
other organizations (e.g., Defense Nuclear Facilities Safety Board or other regulators) should be
defined during the planning phase.
Because the true value of an assessment is the improvement opportunities it identifies, and its value
typically diminishes over time, the best time to release a report is immediately after the
postassessment meeting, which allows the assessed organization to begin improvement actions,
yielding the maximum return from those actions.
The assessment report or transmittal correspondence should clearly indicate what response is
expected from the assessed organization and a reasonable response date.
5.7.8 Corrective Action
Managers responsible for the activities assessed are also responsible for the development of
effective corrective actions for the problem areas or deficiencies discovered during the assessment.
At a minimum, the corrective actions should include the following:
measures to correct each deficiency,
identification of all root causes for significant deficiencies,
determination of the existence of similar deficiencies,
corrective actions to preclude recurrence of like or similar deficiencies,
assignment of corrective action responsibility, and
completion dates for each corrective action.
For independent assessments, the proposed corrective actions should be reviewed for concurrence
by the assessment team leader, with input from the assessment team. The senior line management to
whom the assessed organization is accountable should approve the corrective actions. This will
help ensure that the planned actions will be effective in resolving the problem areas and deficiencies
reported by the assessment team.
Management assessments should be reported in accordance with the organization's management
assessment reporting protocols. Because of the nature of management assessments, corrective
actions may be required by any level of management or by other organizations.
Specific, detailed requirements (DOE O 414.1A, attachment 2) and guidance (DOE G 450.4-1B,
Integrated Safety Management System Guide, vol. 2, appendix G) exist for responding to
assessments conducted by the Department's independent oversight organizations for safety and
emergency management. The requirements and guidance address topics such as line management
responsibility, timeliness, corrective action plans, independent review, tracking, reporting, review,
implementation, verification, and closure. The concepts are based on consensus standards,
compatible with this guide, and may be adapted for use with an organization's internal management
and independent assessment program.
5.7.9 Follow-up
After a reasonable period of time has elapsed, follow-up activities should be performed to verify the
effectiveness of the corrective actions and how they were implemented. The verification should, at a
minimum, sample the corrective actions to determine whether the problem/issue to be addressed has
been resolved. The organization's reporting systems (e.g., noncompliance tracking system,
occurrence reporting and processing system, external oversight reports and regulatory violations,
performance indicators) should be reviewed for evidence of the problem (or a similar problem)
recurring. The same techniques used to conduct assessments may be used for verifying corrective
actions; however, there are several common ways to verify the implementation of corrective actions,
including the following:
reassessment of the deficient areas;
review of new or revised quality-affecting documents such as manuals, procedures, and training
records;
verification during the next scheduled assessment; and
verification by conducting a surveillance covering the areas of concern.
Several DOE directives on special focus assessments also include verification steps: DOE
G 450.4-1B, Integrated Safety Management System Guide, vol. 2, appendix G;
DOE-STD-3006-95, Planning and Conduct of Operational Readiness Reviews (ORR); and Office
of Oversight Environment, Safety, and Health Appraisal Process Protocols, appendix A, "Safety
Management Template,"July 1999.
The key thing to remember is that verification is necessary. A solution to a problem may look good
on paper but may not be readily implemented or long lasting. The failure to adequately identify all
root causes will most likely result in a recurrence of the deficiency. Therefore, an appropriate
amount of follow-up is necessary to ensure the effectiveness of the corrective action process and to
reestablish confidence in the item/service assessed.
5.8 Management Assessment Process
Planning management assessments is an organization-specific effort that should be integrated with
other assessment processes to avoid redundancy and provide the greatest value to the organization.
No one method is appropriate. Organizations are challenged to make the management assessment
process a value-added process that will lead to improvement in organizational performance, safety,
and meeting customer expectations. It is important to remember that while management assessments
share many commonalities with audits, they should not become "managers' audits," but should
instead focus on evaluating organizational performance and identifying barriers that hinder improved
performance. Management assessments should contain an introspective, self-evaluative element
(What/how should things be?) rather than focusing on compliance alone.
5.8.1 Defining the System
The organization should have a written description of the management assessment process. The
description should address all elements of the process, including, but not limited to, the following:
management levels that will be expected to perform assessments,
general goals of the management assessment process,
training or mentoring that will be provided to assessing managers,
overall assessment planning process,
expectations for the number and frequency of assessments to be performed,
expectations and guidelines for management assessment reports,
reporting and follow-up process,
management assessment planning process, and
administration of the process.
5.8.2 Assessment Scheduling
The organization should review and update its management assessment schedule on a regular basis,
either bimonthly or quarterly, to ensure relevance. The review should consider the current
conditions, conclusions of recent management assessments, inputs for independent assessments,
and organizational performance.
Management assessments should be planned with input from all levels of management. Some
organizations have found it beneficial to schedule the number of assessments but leave some of the
topics or areas of assessment to be determined by the performing manager to allow needed
flexibility and to allow the manager the freedom to perform assessments that will result in the greatest
opportunity for improvement.
The planning process may include and take credit for existing management reviews or similar
assessments that routinely occur, such as the following:
program reviews,
strategic planning sessions,
reviews of performance indicators,
organizational goals- and objectives-setting sessions,
financial reviews,
reviews of outputs of improvement teams, and
reviews of independent assessments.
When the above are included in the management assessment system, the requirements for
documenting the assessment and follow-up and tracking of conclusions should be applied (refer to
section 5.7).
Management assessments may include some benchmarking activities, both internal and against other
organizations. The assessment of internal and external performance indicators may also be
beneficial.
There is no fixed number of assessments that must be performed; however, the organization must be
able to show that the management assessment program complies with 10 CFR 830, Subpart A.
5.8.3 Performing Management Assessments
Management assessments may be performed by individual managers or teams of managers. The
primary responsibility for management assessments resides with managers because they are in the
best position to identify barriers to improved performance and to effect changes. While some
aspects of the assessments, such as collecting information, may be delegated to staff, it is the
manager's responsibility to perform the assessment and determine the conclusions. Personal
involvement by the manager will yield the most meaningful information for improving the
performance of the organization.
5.8.4 Assessment Reporting
Management assessments should be reported in accordance with the organization's Management
Assessment program. As with the reports on independent assessments, management assessment
reports should include a concise summary of the topic or area assessed, the conclusions reached,
and follow-up actions that may be required. Reports should be available for use by others and for
future planning. Provisions may have to be made for reports dealing with sensitive areas, proprietary
information, or classified information. Reports identifying potential regulatory compliance issues
should be provided to the appropriate managers for any necessary action or reporting.
5.8.5 Follow-up
Mechanisms should be in place to provide tracking of and follow-up on identified needed actions.
Individuals should be assigned to complete the follow-up actions and provide input back into the
system by specific dates. The management assessment process should provide a means for
coordinating actions when more than one organization's response is required. Follow-up for
management assessments may include another management assessment, inclusion of the assessed
areas in a future independent assessment, and/or evaluation of subsequent performance indicators.
Senior management should review the follow-up actions periodically to ensure that they are
appropriate and that actions are taken in a timely manner.
5.8.6 Feedback
Every system should include a mechanism for feedback to improve the overall system. Management
should periodically request feedback from all levels on the effectiveness and efficiency of the
process, value of the assessments, and actions that should be taken to improve the management
assessment process.
APPENDIX A CONSENSUS STANDARDS AND REFERENCES
A.1 Consensus Standards
The following consensus standards provide methods for implementing the guidance contained
herein. A single standard may not fully implement all elements of the requirements (particularly for
management assessments); therefore, these documents should be used in conjunction with 10 CFR
830, Subpart A, and DOE O 414.1A, Quality Assurance, to develop and implement assessment
processes that meet the DOE assessment requirements. The organization remains responsible for
compliance with 10 CFR 830, Subpart A, and DOE O 414.1A.
1. American National Standards Institute (ANSI)/American Nuclear Society (ANS) 3.2-1994
(R1999), Administrative Controls and Quality Assurance for the Operational Phase of
Nuclear Power Plants.
2. ANSI/ANS 15.8-1995, Quality Assurance Program Requirements for Research Reactors.
3. ANSI/ASQ (American Society for Quality) Z 1.13-1999, Quality Guidelines for Research.
4. ANSI/ASQC (American Society for Quality Control) E4-1994, Quality Systems Requirements
for Environmental Programs, Part A.
5. ANSI/ASQC Q10011-1-1994, Guidelines for Auditing Quality Systems-Auditing, American
Society for Quality.
6. ANSI/ASQC Q10011-2-1994, Guidelines for Auditing Quality Systems- Qualification
Criteria for Quality Systems Auditors, American Society for Quality.
7. ANSI/ASQC Q10011-3-1994, Guidelines for Auditing Quality Systems-Management of
Audit Programs, American Society for Quality.
8. ANSI/ISO (International Organization for Standardization)/ASQ Q9001:2000, Quality
Management Systems: Requirements. (Note: ANSI/ISO/ASQ Q9000 series documents are
recognized as being identical to their ISO 9000 series counterparts.)
9. ASME (American Society of Mechanical Engineers) NQA-1-2000, Quality Assurance
Requirements for Nuclear Facility Applications.
10. DOE-HDBK-1101-96, Process Safety Management for Highly Hazardous Chemicals,
February 1996.
11. DOE-HDBK-3027-99, Integrated Safety Management Systems (ISMS) Verification; Team
Leader's Handbook, June 1999.
12. DOE-EM-STD-5505-96, Operations Assessments, May 1996.
13. DOE-NE-STD-1004-92, Root Cause Analysis Guidance Document, February 1992.
14. DOE-STD-1036-93; Guide to Good Practices for Independent Verification, June 1993 (and
Change Notice 1, dated December 1998).
15. DOE-STD-3006-95, Planning and Conduct of Operational Readiness Reviews (ORR),
November 1995.
16. ISO 9001, see reference 8.
17. ISO 14001.1, Environmental Management Systems Specification with Guidance for Use,
1996.
18. ISO 14010.1, Guidelines for Environmental Auditing General Principles, 1996.
19. ISO 14011/1.1, Guidelines for Environmental Auditing Audit Procedures; Part 1: Auditing
of Environmental Management Systems, 1996.
20. ISO 14012.1, Guidelines for Environmental Auditing Qualification Criteria for
Environmental Auditors, 1996.
21. ISO CD.3 19011, Guidelines on Quality and/or Environmental Management Systems
Auditing. (This guide should be used for information only; it is in the final stage before
approval as an international standard and will replace ISO 10011, parts 1, 2, and 3; ISO 14010;
ISO 14011; and ISO 14012.)
A.2 Other References
The following references provide additional information concerning assessments.
1. DOE G 414.1-2, Quality Assurance Management System Guide for Use with
10 CFR 830.120 and DOE O 414.1, June 1999.
2. DOE/RW-0333P,Quality Assurance Requirements and Description [for the Civilian
Radioactive Waste Management Program], Rev. 10, Section 18.0, "Audits," Office of
Civilian Radioactive Waste Management, April 2000 [http://www.ymp.gov/doclist.htm#q].
3. DOE P 450.1, Environment, Safety and Health Policy for the Department of Energy
Complex, June 1996.
4. DOE G 450.4-1B, Integrated Safety Management System Guide, viols. 1 and 2, March 2001.
5. DOE P 450.5, Line Environment, Safety and Health Oversight, June 1997.
6. DOE, Office of Oversight, Environment, Safety, and Health Appraisal Process Protocols,
July 1999 [http://www.eh.doe.gov/oversight/procedures/9907app/html/toc.htm].
7. U.S. Department of Commerce, National Institute of Standards and Technology, "Malcolm
Baldrige National Quality Award Criteria for Performance Excellence,"
[http://www.quality.nist.gov].
APPENDIX B ASSESSMENT FUNCTIONAL AREAS
This appendix comprises a list of the essential environment, safety, and health (ES&H) and
safeguards and security functional areas/activities that should be included in a comprehensive
assessment program (see next page). The list represents basic ES&H requirements, processes, and
programs found in regulations, DOE policy, and DOE Orders. This appendix should not be
interpreted as limiting application of assessments to only these functional areas that ensure ES&H
protection, nor should it be interpreted as minimizing the importance of assessing product/service
quality and organizational performance. It does, however, illustrate the wide variety of ES&H
programs, systems, and processes currently implemented by DOE and its contractors. These
ES&H functional areas comprise a complex mix of people, hardware, software, and resources, all
impacting on management and performance of activities. An integrated assessment program must
be adaptable to this mix to be responsive to senior management needs and comply with 10 CFR
830, Subpart A, and DOE O 414.1A, Quality Assurance.
Accelerator Safety
Accident, Incident, and Unusual
Occurrence Investigation and Reporting
Process
Aviation Safety
Biological Hazards
Calibration Control
Computer Software Control
Conduct of Operations
Configuration Control
Construction Safety
Criticality Control
Corrective Action
DOE Nuclear Safety Rule Compliance (10
CFR 830)
Document Control and Records
Emergency Preparedness (N.H.)
Employee Concerns System
Engineering Design Processes
Environmental Management Systems
Environmental Protection and NEPA
Compliance
Equipment Modifications
Experimental Programs
Explosives Safety
Facility Operations
Fire Protection
Firearms Safety
Identification & Control of Items
Industrial Hygiene
Industrial Safety
Inspection & Test Control
Integrated Safety Management System
Implementation
Maintenance Management
Motor Carrier/Vehicle Safety
Nonconformance Control
Nuclear Facility Safety
Occupational Medicine
Occurrence Reporting/Trending
Operational Readiness Review Process
Packaging and Transportation
Performance Measures and Indicators
Pollution Prevention
Procurement & Contracts (including supplier
control)
Quality Management Systems
Radiation Protection (10 CFR 835)
Reactor Safety
Safeguards and Security
Safety Management Systems
Safety Analysis Documentation (e.g., Bases
for Interim Operation, hazard analyses, and
safety analysis reports)
Standards/Requirements Identification
Document(s)
Suspect/Counterfeit Items
Technical Safety Requirements
Training of Nuclear Facility Personnel
Unreviewed Safety Questions Process
Voluntary Protection System
Waste Management
Worker Protection/Industrial Safety
APPENDIX C TOOLS FOR ASSESSMENT PLANNING AND CONDUCT
C.1 Checklists
Checklists (example C.1) are lists of assessment objectives and performance criteria. They usually
include a column for the requirements (or references to the requirements) and a column for
recording assessment observations/evidence. Checklists are especially useful for organizing
assessment time by providing a means to list appraisal objectives sequentially. They may also be
structured in a form that can easily be converted into assessment report text.
Example C.1. Laboratory Calibration Program Checklist
Lab/Appraisal Number: Date: Page 1 of
Reference
Criteria
Results
Comments
Sat
Uniat
NL-QAM
1. Is monitoring and data collection
equipment calibrated?
NL-QAM
2. Is equipment calibration traceable to
nationally recognized standards?
NL-QA-5.1
3. Is equipment calibration performed
using approved instructions?
NL-QA-5.1
4. Are calibration records maintained for
each piece of equipment?
NL-QA-5.1
5. Is a use log maintained?
In example C.1, the checklist is used to list the primary elements of a laboratory's calibration
program. The basis or source of each criterion is included in the first column to provide a path
back to the requirements document(s). The "comments" column provides a place for the assessor
to record additional observations as they are discovered during the assessment, which helps to
ensure that important data are not lost.
C.2 Matrices
Matrices (examples C.2 and C.3) are two-dimensional tables showing the relationship between two
sets of information. They can be used to show the logical connecting points between performance
criteria and implementing actions or required actions and personnel responsible for those actions. In
this way, matrices are used to determine what actions and/or personnel have the greatest impact on
an organization's mission. Matrices are especially useful as a way to focus assessment time and
organize assessment conduct.
Example C.2. Organizational Responsibilities Matrix
Program
Development
Deficiency
Tracking
Training
Work
Control
Documents &
Records Retention
Assessment
Director
X
X
Ops Office
X
X
X
Ops Support
X
X
X
Tech Support
X
X
X
X
Admin
X
X
In example C.2, the matrix is used to help the assessor plan the assessment by identifying
organizational responsibilities for the different assessment areas. This type of matrix is used to
maximize use of assessment time during the site visit.
Example C.3. Long-Range Planning Matrix
Administration
Chemistry
Biology
Materials
Building
Services
Engineering
Industrial
Hygiene
A
A
A
A
Radiological
Protection
B
B
B
Fire Protection
A
C
C
C
C
Industrial Safety
A
C
A
Environmental
C
A
C
C
Personnel
Training
B
C
B
C
Conduct of
Operations
C
C
C
Quality
Assurance
A
C
A
C
A = 1st assessment; B = 2nd assessment; C = 3rd assessment
Example C.3 is a much broader use of the matrix that allows the assessor to do the long-range
planning necessary for ensuring proper application of the assessment program. In this example, the
various assessment areas (Y axis) are correlated with the different organizations to be assessed.
C.3 Flowcharts
Flowcharts (figure C.1) illustrate the steps or activities in a process. They provide an excellent tool
for examining how various steps in a process are related to each other and whether or not each
subsequent activity is receiving what it needs from the previous one. Flowcharts are used to help
the assessor understand how a function is being implemented based on written programs and
procedures. Flowcharts also illustrate reporting relationships and indicate whether the handoff of
information or materials is adequate. They are especially useful for locating process bottlenecks,
which may hinder the organization's mission.
In figure C.1, the flowchart diagrams the steps in the design process, which helps identify critical
areas and determine whether an individual step affects the design process output. In addition, this
flowchart may allow the assessor to divide the design assessment between different visits while
ensuring overall coverage.
C.4 Cause-and-Effect Diagrams
Cause-and-effect diagrams (figure C.2) illustrate the relationship between a known "effect" or
outcome and all the "causes" or contributors influencing it. The effect being examined may
represent either a wanted or unwanted outcome. The cause-and-effect diagram is used when the
outcome of a process/program is known but the contributors need to be evaluated further. These
diagrams are especially useful when the contributors stem from different sources across the
organization being assessed.
In this example, the assessor would use the cause-and-effect diagram (figure C.2) to identify all the
program elements that should be in place to prevent worker exposure. This tool can be used in two
ways by the assessor: (1) to verify the effectiveness of individual elements, thereby verifying that the
program is working, and (2) to pinpoint the source of programmatic weaknesses.
C.5 Tree Diagrams
Tree diagrams (figure C.3) are used to map out systematically, in increasing detail, the full range of
controls and tasks needed to achieve a primary goal. They can also be used to map out the barriers
needed to prevent an unwanted event (called "causal analysis" or "barrier analysis" trees). Tree
diagrams may be used by the assessor to verify whether all planned activities are in place to support
a program's objective. They are especially useful for helping the assessor focus on the big picture:
the overall goal of the program, with its supporting subgoals.
In figure C.3, the tree diagram provides a map of the elements needed to support an effective
training and qualification program. Using the diagram, the assessor can plan the assessment to
ensure that the appropriate activities are being performed and to evaluate the training organization's
effectiveness overall. As this tree diagram is used for the assessment, the elements must be
continually rolled up. This means the "Capability/Proficiency Verified" element should be assessed
to determine its impact on "Qualification," which must be assessed to ensure it supports overall
"Training and Qualification."
C.6 Information Systems
Information systems comprise a wide range of different forms and formats. In their simplest form
they may include the weekly and monthly reports of laboratory or organizational performance that
are used to alert the assessment organization of potential assessment areas. In more complex form,
these systems may include computerized databases that link performance to specific performance
objectives or track actions to resolve programmatic weaknesses. In either case, information systems
are important tools for assessors, providing them the data needed to focus assessment activities.
In figure C.4, information on lost-time injuries is displayed in both tabular and graphical form. This
information can be used to focus the assessment on either the location of the injuries or the work
procedures involved to identify any weaknesses in the accident prevention program.
APPENDIX D INDEPENDENT ASSESSMENT PLANNING
Many scheduling and planning approaches satisfy the assessment requirements for integration,
grading, and use of a risk-based approach. The following steps are one approach to detailed
planning for independent assessments. This detailed level of planning should be performed on a
regular basis, such as annually.
D.1 Identifying Assessment Areas
Assessments should be managed to achieve specific objectives. Programs, systems, and processes
(including functional and organizational areas) and their associated performance objectives and
measurements should be reviewed to determine their appropriateness and applicability. In the
following sections we discuss the information that should be compiled and maintained to manage the
assessment process and its objectives after the determination has been made.
D.2 Evaluating Probability and Consequences and Probability (Risk)
Evaluate the program, systems, and processes to identify the various factors and elements that could
result in a failure to achieve the mission or the objective's success. Factors to consider include, but
are not limited to, the following:
worker health and safety (injuries, deaths, exposures);
public health and safety (injuries, deaths, exposures);
community concerns;
regulatory noncompliance and liability;
business efficiency/productivity;
complexity of the involved processes;
reliability of the engineering and administrative controls;
skills and experience of the personnel involved;
maturity of the program, system, or process (developed, mature techniques/processes versus
state-of-the-art or developmental/pilot technology);
changes that may affect performance (including regulatory);
life-cycle phase (new, midlife, closeout of activity);
organizational experience with the program, system, or activity;
economic costs (uncertainty);
schedule/commitment or milestones failure; and
performance measures/indicators, trending downward.
Each of the factors should be evaluated for likelihood of occurrence and severity of consequences.
This is a rough estimate, using professional judgement, and the use of a multidisciplined team for
this process is encouraged.
D.3 Prioritizing Assessment Activities
Rank the program, system, and process risks based on the consequences and likelihood. Those
with high risk or multiple lower risks should be ranked higher. This may identify risks that crosscut
several programs, activities, or organizations, thus benefiting more than one organization.
D.4 Identifying Areas for Further Evaluation
List the programs, systems, or processes and the areas of risk; then further evaluate other factors
such as
time since the last internal, independent assessment;
time since other assessments (external, management, process improvement teams, investigations,
etc.);
opportunities to perform the assessment in conjunction with other organizations (internal or
external);
work schedules (will a lower-ranked program start or complete before a higher-ranked program
or activity);
other scheduled assessments (management assessments, process improvement teams, etc.) that
would be expected to address the area;
availability of assessment personnel, including technical personnel to perform the assessment;
and
certifications, registrations, or other scheduled activities that would be expected to evaluate the
program, system, or process.
D.5 Initiating the Assessment Plan
Develop an assessment plan for the specific assessment including the following elements:
scope of the assessment (i.e., the program, system, process, organization, and/or activity to be
assessed);
objectives of the assessment;
assessment drivers (e.g., the regulatory requirements, contractual agreements, performance
objectives, and/or internal procedures that will be used);
assessment team members, including the lead, supporting assessors, and technical experts (if
appropriate); and
assessment schedule, with start and end dates (final planning, notification, kickoff meetings,
preparation, investigation, closeout meeting, and report issuance).
The assessment plan should establish the depth, scope, and breadth of the assessment. It will
provide a tool for scheduling and information exchange for both the assessment team and the
management of the assessed organizations.
D.6 Allocating Resources
The resources for performing assessments are limited and seldom allow for performing all
assessments. Likewise, the benefits of performing assessments on low risk areas are marginal. As a
result, a realistic estimate of the resources available, including their scheduled availability, should be
developed. Additional factors such as the availability of personnel independent of the areas to be
assessed, budget constraints, management or customer requests, and response/follow-up to
previous external assessments should be considered.
D.7 Developing the Assessment Schedule and Plan
Using the information developed in D.1 through D.6, an assessment schedule and plan can be
developed. The schedule and plan should reflect the areas of greatest risk and the reasonable
allocation of resources. Assessments that fall "below the line" should be retained as "targets of
opportunity" to be performed if resources become available or if one of the planned assessments
changes in risk or schedule. The assessment schedule and plan is a tool that allows management and
customers to understand the basis for the assessments and justifies the allocated resources.
D.8 Maintaining the Assessment Schedule and Plan
Assessment plans should be reviewed periodically and modified as new information on the facility or
organization is obtained that changes the estimated risks or reflects changes in available resources.
These reviews should occur at regular intervals, such as monthly or quarterly. The review can be
used to finalize schedules, team members, etc., for the next period. Assessment areas that have
increased in risk can be moved up in the schedule, while others can be moved down. In some
cases, assessments that were "below the line" should be moved up to reflect changes that have
occurred since the original planning and ranking was performed. The results of assessments, which
identify good performance, should be used to reduce the frequency and depth of future
assessments. Areas of poor performance should receive increased attention, especially if there are
indications that management is unable to correct identified problems. This is because recurring and
cumulative deficiencies, even in a low hazard operation, may decrease the likelihood of achieving its
mission.
D.9 Frequency
Given the likelihood of some or all of these factors being present and the wide variety of Department
of Energy activities, it is impossible to define the "right" assessment frequency in this Guide. Too
few assessments will not keep pace with the changes occurring in the program, system, or process.
Too many assessments will distract the organization from focused attention on the safe conduct of
work and mission accomplishment. Therefore, it is ultimately the responsibility of management,
guided by assessment professionals, to determine the appropriate mix of assessments (independent
and management) to meet customer requirements and ensure mission success.
<>