NAO 216-115: Strengthening NOAA’s Research and Development Enterprise

Issued 07/20/11; Effective 07/20/11


01. This NOAA Administrative Order establishes the principles, policies, and responsibilities for planning, monitoring, evaluating, and reporting research and development (R&D) activities comprising the entire NOAA R&D enterprise. NOAA will use this Order to support strengthening the quality, relevance, and performance of NOAA's R&D portfolio.

02. Development of this Order was directed by the NOAA Senior Science Advisor, acting on behalf of the NOAA Administrator, in a November 2, 2009, memorandum to the NOAA Research Council Chair. The Order responds to the Administrator's call for establishing consistent, Agency-wide peer review and monitoring processes for all NOAA scientific activities, to ensure NOAA science is of the highest quality and is documented in ways that allow the importance and value ofNOAA science to be communicated objectively and accurately.

03. This Order provides a framework that contributes to the strengthening of science across NOAA's entire R&D enterprise. It builds upon existing best practices to promote scientific and technological excellence and enable scientists and science leaders to pursue the R&D necessary to inform NOAA's service and stewardship responsibilities.

04. This Order establishes consistent approaches that enhance NOAA's ability to coordinate across NOAA and with NOAA's partners. It supports exchange of information among scientists and clear communication of the scope and value ofNOAA's R&D to others. This Order provides tools to support NOAA in addressing critical science challenges, particularly those requiring integrated, holistic R&D approaches.

05. This Order applies to all NOAA R&D activities, whether internal or external R&D, and includes R&D conducted by NOAA and sponsored by others.

06. The coordinated efforts described in this Order support the successful transitions of R&D into NOAA applications (operations) as defined in NAO 216-105.

07. The implementation ofthis NAO will be guided by an associated Procedural Handbook.


01. NOAA’s R&D enterprise underpins all that NOAA does. It will:

02. This Order incorporates requirements of the Information Quality Act (2001), the Office of Management and Budget (OMB) Circular A-11 (OMB, 2009a), the Open Government Directive (OMB, 2009b), the GPRA (Government Performance and Results Act) Modernization Act (2010), and other relevant laws and policies. It reiterates established best practices and guiding principles recognized by NOAA.


01. Research and development (R&D) at NOAA is defined consistent with the definitions used by the National Science Foundation (NSF, 2009). Research, development, and R&D plant include all direct, incidental, or related costs resulting from, or necessary to, performance of research and development, and costs of R&D plant as defined below, regardless of whether the research and development are performed by a Federal agency (intramurally) or performed by private individuals and organizations under grant or contract (extramurally). R&D excludes routine product testing, quality control, mapping and surveys, collection of general-purpose statistics, experimental production, and the training of scientific personnel.

02. Customer. A stakeholder that uses NOAA's products and services (NOAA, 2009).

03. Evaluation. A study conducted periodically or on an ad hoc basis to assess how well a program is working. It is often conducted by experts external to the program, either inside or outside the agency, as well as by program managers. They help managers determine if timely adjustments are needed in program design to improve the rate, or quality, of achievement relative to the committed resources (GAO, 2005; DOE, 2009). There are four main types of evaluation, including outcome evaluation (what the program accomplished), impact evaluation (net effect of the program), process evaluation (extent the program is operating as intended), and cost-benefit/cost-effectiveness evaluation (NOAA, 2009).

04. External R&D. R&D conducted by any entity outside of NOAA (e.g., Cooperative Institute, academic scientist, state, other federal agency, etc.).

05. Internal R&D. R&D conducted by NOAA employees.

06. Monitoring. A continuing function that uses systematic collection of data on specified indicators to provide scientific and other staff, management, partners, and stakeholders with indications of the extent of progress, achievement of objectives, and use of allocated funds for a program or project (adapted from OECD, 2002).

07. Objective. An elaboration of a goal statement, developing with greater specificity the intent of the goal, which may provide insight into the strategy for achieving the goal. Used below the Goal level to describe the intended results (NOAA, 2009).

08. Operations. Sustained, systematic, reliable, and robust mission activities with an institutional commitment to deliver appropriate cost-effective products and services (NOAA, 2009).

09. Outcome. An end result, expected and unexpected, of the customer’s use or application of the organization’s outputs. Outcomes may be long-term, mid-term, or short-term in nature (NOAA, 2009).

10. Output. The products or services resulting from a capability (NOAA, 2009).

11. Peer Review. A widely used, time-honored practice in the scientific and engineering community for judging and potentially improving a scientific or technical plan, proposal, activity, program, or work product through documented critical evaluation by individuals or groups with relevant expertise who had no involvement in developing the object under review (NRC, 2000).

12. Performance. A measure of both effectiveness (the ability to achieve useful results) and efficiency (the ability to achieve quality, relevance, and effectiveness in timely fashion and with little waste) (NRC, 2008).

13. Performance Management. The systematic process of monitoring the results of activities; collecting and analyzing performance information to track progress toward planning results; using performance information to inform program decision-making and resource allocation; and communicating results achieved, or not attained, to advance organizational learning and tell the Agency’s story (NOAA, 2009). For science and R&D activities, regular independent peer review is an essential element of performance management.

14. Performance Measure. A structured statement that describes the means by which actual outcomes and outputs are measured against planned outcomes and outputs (NOAA, 2009).

15. Performance Measurement. The ongoing monitoring and reporting of program accomplishments, particularly progress toward pre-established goals (NOAA, 2009).

16. Planning. The formal process to determine both internal and external requirements, including program performance (NOAA, 2009).

17. Portfolio Management. The processes, practices, and specific activities to perform continuous and consistent evaluation, prioritization, budgeting, and finally, selection of investments that provide the greatest value and contribution to the strategic interest of the organization. Through portfolio management, NOAA can explicitly assess the tradeoffs among competing investment opportunities in terms of their benefits, costs, and risks (NOAA, 2009).

18. Program Assessment. A determination, through objective measurement and systematic analysis, of the manner and extent to which Federal programs achieve intended objectives (OMB, 2009a). Independent peer review is an important element of program assessment.

19. Project. Any undertaking of a temporary nature (e.g., research, assessment, prediction, acquisition, or stewardship efforts) designed to create a service, product, system, and/or system upgrade in support of a validated NOAA Mission Requirement. A NOAA project is intended to address a shortfall within defined budget and schedule constraints and terminates when the service, product, or system achieves full operational capability (FOC), when the capability gap is resolved or, in coordination with other programs, when directed by the project manager or higher authority (NOAA, 2009).

20. Quality. A measure of the novelty, soundness, accuracy, and reproducibility of a specific body of research (NRC, 2008).

21. Relevance. A measure of how well a specific body of research support’s NOAA’s mission and the needs of users and the broader society.

22. Reporting. Written or oral summation and/or description regarding research and development. R&D reporting should be repeatable and consistent throughout NOAA.

23. Research Council. A group composed of members from each of the NOAA Lines and Goals, as well as a Chair, a Vice Chair, and ex-officio members who chair the Council’s standing committees. The Council guides research across NOAA, providing corporate oversight and developing policy to ensure that NOAA research activities are of the highest scientific quality, meet long-range societal needs, take advantage of emerging scientific and technological opportunities, shape a forward-looking research agenda, and are accomplished in an efficient and cost effective manner (NOAA, 2009).

24. Result. The expected or unexpected outcome of NOAA’s products and services (NOAA, 2009).

25. Technical Readiness Levels. A systematic metric/measurement system that supports assessments of the maturity of a particular technology and the consistent comparison of maturity between different types of technology (Mankins, 1995). The concept of technical readiness levels is not limited to technology and can be applied to assess and compare the maturity of a variety of research projects.

26. Transition project. The collective set of activities necessary to transfer a research result, or collection of research results, to operational status or to an information service (NOAA, 2009).


01. NOAA’s policy is to use consistent processes across the organization for planning, monitoring, evaluating, and reporting on NOAA’s R&D. Processes will be designed and implemented across the R&D enterprise to ensure that NOAA’s R&D is of the highest quality, relevance, and performance and that its value can be communicated clearly and accurately. Evaluation processes will include regular, independent peer review.

02. Research conducted, funded, and supported by NOAA is a corporate enterprise and therefore should be systematically planned, monitored, and evaluated to ensure it is as effective as possible in supporting NOAA’s mission, goals, and objectives.

03. R&D activities will clearly tie to the NOAA mission, vision, goals, and objectives articulated in NOAA's strategic plans (including 5-year and 20-year research plans, and other relevant documents).

04. Review of R&D activities shall be accomplished regularly and systematically to ensure consistency with NOAA standards on quality, relevance, scientific integrity, transparency, and performance. These reviews shall assess each R&D activity against NOAA’s mission, vision, goals, and objectives. They shall be separate from and not duplicative of existing reviews for grants, cooperative agreements, contracts, purchase orders, interagency agreements, or project agreements.

05. Development and dissemination of R&D products will be consistent with NAO policies and procedures related to peer review, the Open Government Directive (OMB, 2009b), and the Information Quality Act and other legislative mandates and policies. NOAA will ensure and maximize the quality, objectivity, utility, and integrity of R&D information that it disseminates by adhering to the NOAA Information Quality Guidelines (NOAA, 2006). NOAA will ensure that peer review of "influential scientific information" or "highly influential scientific assessments" is conducted in accordance with OMB's Final Information Quality Bulletin for Peer Review (OMB, 2004).

06. To enable the most efficient implementation of this NAO and the associated procedural handbook, and their application will be standardized to the extent possible and allow for updates and exceptions with justification where necessary. The NAO and Handbook will be periodically reviewed and reconfirmed or revised, as needed.

07. Planning

08. Monitoring

09. Evaluation

10. Reporting


0 I. The Under Secretary of Commerce for Oceans and Atmosphere and the NOAA Chief Scientist shall provide top management support for implementation of this Order. Furthermore, the NOAA Chief Scientist will:

02. The NOAA Research Council will assist the NOAA Chief Scientist in execution of this NAO, and the Assistant Administrator for the NOAA Research line office will provide additional support for execution. The duties of the Research Council, with support from the NOAA Research line office, will include but are not limited to:

03. Line Office Assistant Administrators and the Director, Office of Marine and Aviation Operations, will:

04. In accordance with their charter (SAB, 2009), the NOAA SAB will conduct activities which support the intent of this NAO, including:


01. The Research Council will develop and disseminate written procedures, plans, and reports as necessary to implement this Order, including but not limited to:

02. Existing documents referenced in this policy are as follows:



An electronic copy of this Order will be posted on the NOAA Office of the Chief Administrative Officer website under the NOAA Administrative Issuances Website.


Jane Lubchenco
Under Secretary of Commerce
for Oceans and Atmosphere

Office of Primary Interest:
Office of Oceanic and Atmospheric Research