NAO 216-115: Strengthening NOAA’s Research and Development Enterprise
Issued 07/20/11; Effective 07/20/11; Reviewed: 4/29/14
SECTION 1. PURPOSE AND SCOPE.
01. This NOAA Administrative Order establishes the principles, policies, and responsibilities for planning, monitoring, evaluating, and reporting research and development (R&D) activities comprising the entire NOAA R&D enterprise. NOAA will use this Order to support strengthening the quality, relevance, and performance of NOAA's R&D portfolio.
02. Development of this Order was directed by the NOAA Senior Science Advisor, acting on behalf of the NOAA Administrator, in a November 2, 2009, memorandum to the NOAA Research Council Chair. The Order responds to the Administrator's call for establishing consistent, Agency-wide peer review and monitoring processes for all NOAA scientific activities, to ensure NOAA science is of the highest quality and is documented in ways that allow the importance and value ofNOAA science to be communicated objectively and accurately.
03. This Order provides a framework that contributes to the strengthening of science across NOAA's entire R&D enterprise. It builds upon existing best practices to promote scientific and technological excellence and enable scientists and science leaders to pursue the R&D necessary to inform NOAA's service and stewardship responsibilities.
04. This Order establishes consistent approaches that enhance NOAA's ability to coordinate across NOAA and with NOAA's partners. It supports exchange of information among scientists and clear communication of the scope and value ofNOAA's R&D to others. This Order provides tools to support NOAA in addressing critical science challenges, particularly those requiring integrated, holistic R&D approaches.
05. This Order applies to all NOAA R&D activities, whether internal or external R&D, and includes R&D conducted by NOAA and sponsored by others.
06. The coordinated efforts described in this Order support the successful transitions of R&D into NOAA applications (operations) as defined in NAO 216-105.
07. The implementation ofthis NAO will be guided by an associated Procedural Handbook.
SECTION 2. GUIDING PRINCIPLES.
01. NOAA’s R&D enterprise underpins all that NOAA does. It will:
- Provide essential information, understanding, and tools to enable NOAA to accomplish its diverse missions and meet the needs of our customers through service, innovation, and creativity;
- Continue to provide significant benefits to our Nation, including protecting lives and property, creating jobs and economic value through scientific output and innovation, and enabling understanding and stewardship of our earth system;
- Be relevant to the overall mission and goals of NOAA;
- Support scientists and science managers by operating transparently and efficiently and promoting collaboration to address critical science challenges;
- Be planned, monitored, evaluated (including regular peer review), and reported on a regular and consistent basis to ensure that the Nation obtains a sustained return on its investment pursuant to NOAA's strategic goals and objectives;
- Include an optimized mix of near-term, mid-term, and long-term R&D (SAB, 2004). NOAA will foster a supportive culture and commitment to quantifiable investments in high-risk, high-reward research. Such research may not have a near-term operational application but provides the cutting-edge solutions for future applications (SAB, 2004). The diverse character of different research categories must be fully recognized to develop suitable evaluation criteria for each (NRC, 1999), and all will be subjected to an appropriate form of independent peer review on a regular basis;
- Foster and sustain strategic partnerships with the external research community, whose valuable contributions are critical to meeting NOAA's mission; and
- Be of the highest quality and documented in ways that allow NOAA to communicate objectively and accurately the importance and value of NOAA science to the Administration, the Department of Commerce (DOC), the Congress, academia, regulated and user communities, and the public at large.
02. This Order incorporates requirements of the Information Quality Act (2001), the Office of Management and Budget (OMB) Circular A-11 (OMB, 2009a), the Open Government Directive (OMB, 2009b), the GPRA (Government Performance and Results Act) Modernization Act (2010), and other relevant laws and policies. It reiterates established best practices and guiding principles recognized by NOAA.
SECTION 3. DEFINITIONS.
01. Research and development (R&D) at NOAA is defined consistent with the definitions used by the National Science Foundation (NSF, 2009). Research, development, and R&D plant include all direct, incidental, or related costs resulting from, or necessary to, performance of research and development, and costs of R&D plant as defined below, regardless of whether the research and development are performed by a Federal agency (intramurally) or performed by private individuals and organizations under grant or contract (extramurally). R&D excludes routine product testing, quality control, mapping and surveys, collection of general-purpose statistics, experimental production, and the training of scientific personnel.
- Research is systematic study directed toward fuller scientific knowledge or understanding of the subject studied. Research is classified as either basic or applied according to the objectives of the sponsoring agency.
- In basic research the objective of the sponsoring agency is to gain fuller knowledge or understanding ofthe fundamental aspects of phenomena and of observable facts without specific applications towards processes or products in mind.
- In applied research the objective ofthe sponsoring agency is to gain knowledge or understanding necessary for determining the means by which a recognized and specific need may be met.
- Development is systematic use of the knowledge or understanding gained from research, directed toward the production of useful materials, devices, and systems or methods, including design, development, and improvement of prototypes and new processes. It excludes quality control, routine product testing, and production.
- Demonstration activities that are part of research or development (i.e., that are intended to prove or to test whether a technology or method does, in fact, work) should be included. Demonstrations intended primarily to make information available about new technologies or methods should not be included.
- R&D plant (R&D facilities and fixed equipment, such as reactors, wind tunnels, and particle accelerators) includes acquisition of, construction of, major repairs to, or alterations in structures, works, equipment, facilities, or land for use in R&D activities at Federal or non-Federal installations. Excluded from the R&D plant category are expendable or movable equipment (e.g., spectrometers, microscopes) and office furniture and equipment. Also excluded are the costs of pre-design studies (e.g., those undertaken before commitment to a specific facility). These excluded costs should be reported under total conduct of research and development. Obligations for foreign R&D plant are limited to Federal funds for facilities located abroad and used in support of foreign research and development.
02. Customer. A stakeholder that uses NOAA's products and services (NOAA, 2009).
03. Evaluation. A study conducted periodically or on an ad hoc basis to assess how well a program is working. It is often conducted by experts external to the program, either inside or outside the agency, as well as by program managers. They help managers determine if timely adjustments are needed in program design to improve the rate, or quality, of achievement relative to the committed resources (GAO, 2005; DOE, 2009). There are four main types of evaluation, including outcome evaluation (what the program accomplished), impact evaluation (net effect of the program), process evaluation (extent the program is operating as intended), and cost-benefit/cost-effectiveness evaluation (NOAA, 2009).
04. External R&D. R&D conducted by any entity outside of NOAA (e.g., Cooperative Institute, academic scientist, state, other federal agency, etc.).
05. Internal R&D. R&D conducted by NOAA employees.
06. Monitoring. A continuing function that uses systematic collection of data on specified indicators to provide scientific and other staff, management, partners, and stakeholders with indications of the extent of progress, achievement of objectives, and use of allocated funds for a program or project (adapted from OECD, 2002).
07. Objective. An elaboration of a goal statement, developing with greater specificity the intent of the goal, which may provide insight into the strategy for achieving the goal. Used below the Goal level to describe the intended results (NOAA, 2009).
08. Operations. Sustained, systematic, reliable, and robust mission activities with an institutional commitment to deliver appropriate cost-effective products and services (NOAA, 2009).
09. Outcome. An end result, expected and unexpected, of the customer’s use or application of the organization’s outputs. Outcomes may be long-term, mid-term, or short-term in nature (NOAA, 2009).
10. Output. The products or services resulting from a capability (NOAA, 2009).
11. Peer Review. A widely used, time-honored practice in the scientific and engineering community for judging and potentially improving a scientific or technical plan, proposal, activity, program, or work product through documented critical evaluation by individuals or groups with relevant expertise who had no involvement in developing the object under review (NRC, 2000).
12. Performance. A measure of both effectiveness (the ability to achieve useful results) and efficiency (the ability to achieve quality, relevance, and effectiveness in timely fashion and with little waste) (NRC, 2008).
13. Performance Management. The systematic process of monitoring the results of activities; collecting and analyzing performance information to track progress toward planning results; using performance information to inform program decision-making and resource allocation; and communicating results achieved, or not attained, to advance organizational learning and tell the Agency’s story (NOAA, 2009). For science and R&D activities, regular independent peer review is an essential element of performance management.
14. Performance Measure. A structured statement that describes the means by which actual outcomes and outputs are measured against planned outcomes and outputs (NOAA, 2009).
15. Performance Measurement. The ongoing monitoring and reporting of program accomplishments, particularly progress toward pre-established goals (NOAA, 2009).
16. Planning. The formal process to determine both internal and external requirements, including program performance (NOAA, 2009).
17. Portfolio Management. The processes, practices, and specific activities to perform continuous and consistent evaluation, prioritization, budgeting, and finally, selection of investments that provide the greatest value and contribution to the strategic interest of the organization. Through portfolio management, NOAA can explicitly assess the tradeoffs among competing investment opportunities in terms of their benefits, costs, and risks (NOAA, 2009).
18. Program Assessment. A determination, through objective measurement and systematic analysis, of the manner and extent to which Federal programs achieve intended objectives (OMB, 2009a). Independent peer review is an important element of program assessment.
19. Project. Any undertaking of a temporary nature (e.g., research, assessment, prediction, acquisition, or stewardship efforts) designed to create a service, product, system, and/or system upgrade in support of a validated NOAA Mission Requirement. A NOAA project is intended to address a shortfall within defined budget and schedule constraints and terminates when the service, product, or system achieves full operational capability (FOC), when the capability gap is resolved or, in coordination with other programs, when directed by the project manager or higher authority (NOAA, 2009).
20. Quality. A measure of the novelty, soundness, accuracy, and reproducibility of a specific body of research (NRC, 2008).
21. Relevance. A measure of how well a specific body of research support’s NOAA’s mission and the needs of users and the broader society.
22. Reporting. Written or oral summation and/or description regarding research and development. R&D reporting should be repeatable and consistent throughout NOAA.
23. Research Council. A group composed of members from each of the NOAA Lines and Goals, as well as a Chair, a Vice Chair, and ex-officio members who chair the Council’s standing committees. The Council guides research across NOAA, providing corporate oversight and developing policy to ensure that NOAA research activities are of the highest scientific quality, meet long-range societal needs, take advantage of emerging scientific and technological opportunities, shape a forward-looking research agenda, and are accomplished in an efficient and cost effective manner (NOAA, 2009).
24. Result. The expected or unexpected outcome of NOAA’s products and services (NOAA, 2009).
25. Technical Readiness Levels. A systematic metric/measurement system that supports assessments of the maturity of a particular technology and the consistent comparison of maturity between different types of technology (Mankins, 1995). The concept of technical readiness levels is not limited to technology and can be applied to assess and compare the maturity of a variety of research projects.
26. Transition project. The collective set of activities necessary to transfer a research result, or collection of research results, to operational status or to an information service (NOAA, 2009).
SECTION 4. POLICY.
01. NOAA’s policy is to use consistent processes across the organization for planning, monitoring, evaluating, and reporting on NOAA’s R&D. Processes will be designed and implemented across the R&D enterprise to ensure that NOAA’s R&D is of the highest quality, relevance, and performance and that its value can be communicated clearly and accurately. Evaluation processes will include regular, independent peer review.
02. Research conducted, funded, and supported by NOAA is a corporate enterprise and therefore should be systematically planned, monitored, and evaluated to ensure it is as effective as possible in supporting NOAA’s mission, goals, and objectives.
03. R&D activities will clearly tie to the NOAA mission, vision, goals, and objectives articulated in NOAA's strategic plans (including 5-year and 20-year research plans, and other relevant documents).
04. Review of R&D activities shall be accomplished regularly and systematically to ensure consistency with NOAA standards on quality, relevance, scientific integrity, transparency, and performance. These reviews shall assess each R&D activity against NOAA’s mission, vision, goals, and objectives. They shall be separate from and not duplicative of existing reviews for grants, cooperative agreements, contracts, purchase orders, interagency agreements, or project agreements.
05. Development and dissemination of R&D products will be consistent with NAO policies and procedures related to peer review, the Open Government Directive (OMB, 2009b), and the Information Quality Act and other legislative mandates and policies. NOAA will ensure and maximize the quality, objectivity, utility, and integrity of R&D information that it disseminates by adhering to the NOAA Information Quality Guidelines (NOAA, 2006). NOAA will ensure that peer review of "influential scientific information" or "highly influential scientific assessments" is conducted in accordance with OMB's Final Information Quality Bulletin for Peer Review (OMB, 2004).
06. To enable the most efficient implementation of this NAO and the associated procedural handbook, and their application will be standardized to the extent possible and allow for updates and exceptions with justification where necessary. The NAO and Handbook will be periodically reviewed and reconfirmed or revised, as needed.
- R&D in support of the organization’s mission must cover the full spectrum of time frames, from short-term to long-term (SAB, 2004).
- Planning will ensure that the R&D enterprise is relevant to and optimally aligned with current and future generations of the NOAA Strategic Plan and other relevant documents as appropriate (e.g., Annual Guidance Memoranda, Annual Operating Plans, the Department of Commerce Strategic Plan, court orders, Executive Orders, appropriations, and statutory requirements).
- To support this effort, NOAA will implement enterprise R&D planning procedures in accordance with the Planning chapter of the Procedural Handbook.
- Components of the enterprise R&D planning will include:
- A “20-Year Research Vision” to provide overarching direction for the agency’s research based on future environmental challenges;
- A “5-Year Research Plan” to describe the research activities needed to achieve specific milestones during the 5-year period;
- Annual lists of R&D integrated planning and execution priorities to highlight emerging R&D opportunities, issues, and needs to support NOAA’s mission; to identify R&D priorities prior to each planning year; and to be an asset in the preparation of the Annual Guidance Memorandum, proposed alternatives, and other activities.
- As needed, individual line offices, laboratories, programs, etc. should prepare supplemental R&D plans that align with the NOAA Strategic Plan and the enterprise R&D plans.
- Collecting, tracking, and analyzing NOAA’s R&D funding, project, and performance data are essential to managing NOAA’s research portfolio, as well as the transition portfolio managed by the Line Office Transition Managers (see NAO 216-105). Systematic monitoring of NOAA’s R&D enterprise enables NOAA to make informed investment decisions, optimize the project portfolio, and track advancements in quality, relevance, and performance. R&D monitoring activities will provide a valuable tool that will support and enable NOAA’s Policy on Transition of Research to Application (NOAA, 2005).
- To facilitate systematic monitoring, NOAA will implement an enterprise R&D monitoring database system in accordance with requirements and guidance provided in the Monitoring chapter of the Procedural Handbook.
- Project Database: The enterprise R&D monitoring database system will:
- Result in a net increase in the efficiency and effectiveness of NOAA’s R&D enterprise through improved communication, coordination, and planning of R&D across NOAA;
- Include costs, major outputs, performance metrics, transition milestones, and other appropriate categories;
- Capture trends in R&D investment;
- Identify and track transition projects using NOAA-adopted technical readiness levels (as recommended by SAB, 2004);
- Track execution and evaluation; and
- Support strategic portfolio analysis.
- The enterprise R&D monitoring database system will have:
- An accountable lead, with a milestone included in the Annual Operating Plan (AOP), identified for each line office and/or respective sub-organizational entity that is responsible for populating the database; and
- A life-cycle business plan, including designation of the entity responsible for database operation and maintenance.
- Evaluations are critical for determining program success in achieving intended outcomes. Evaluations are performance management tools used to inform strategic planning as well as decision-making regarding execution of future R&D activities. Rigorous independent evaluations are a key resource in determining whether government programs are achieving their intended outcomes and will enable policy makers and agency managers to strengthen the science enterprise (OMB, 2009c). NOAA's evaluations comply with Administration (including OMB), Congressional, DOC, NOAA, Line Office, and other requirements for evaluations at all levels of execution..
- Evaluations shall be completed in accordance with the Evaluation and Performance Measure chapters of the Procedural Handbook.
- Expert review will be used for evaluations and will include experts in relevant science (independent peer reviews) and service (for portfolio reviews) fields.
- To the extent practicable, consistent evaluation procedures will be used for both internal and external R&D activities.
- To promote transparency, results from evaluations will be made publicly available (OMB, 2009c).
- For the purposes of this policy, the categories of evaluation include, but are not limited to:
- Program/Laboratory/science center reviews. Reviews will address the quality, relevance and performance of individual programs, laboratories and science centers. Each Line Office will arrange for reviews of its programs/ laboratories/science centers by expert panels on a regular and recurring basis (not less frequently than every five years).
- NOAA portfolio reviews. Reviews will address the extent to which areas of NOAA's R&D are relevant to its mission and societal needs. The Research Council will conduct portfolio reviews.
- Benchmarking. Benchmarking reviews will consider NOAA research in comparison with similar organizations nationally and internationally. The Science Advisory Board (SAB) will conduct benchmarking reviews consistent with its charter.
- Assessing R&D against performance measures/milestones. Performance measures will be vetted annually with the Research Council to enable portfolio management and ensure that the measures are appropriately outcome-oriented and ambitious. Performance measures may be elements of the evaluations described in sections i-iii above, and they also may be used for other purposes.
- NOAA will conduct standardized, representative reporting for its R&D enterprise to document the current state of the enterprise, highlight strategic R&D investment needs for the future, and communicate the return-on-investment and overall benefits to society derived from its current R&D portfolio.
- The Reporting chapter of the Procedural Handbook will provide requirements and guidance for conducting R&D reporting and will identify the minimum data needed to develop a representative snapshot of the R&D enterprise.
- The State of NOAA Research report will be issued annually and will describe the status of the NOAA R&D enterprise.
SECTION 5. RESPONSIBILITIES.
0 I. The Under Secretary of Commerce for Oceans and Atmosphere and the NOAA Chief Scientist shall provide top management support for implementation of this Order. Furthermore, the NOAA Chief Scientist will:
- Drive policy and program direction for science and technology priorities for the NOAA R&D enterprise;
- Lead the implementation of this NAO to continually strengthen and optimize the quality, relevance, and performance of NOAA’s R&D portfolio;
- Be accountable to the Undersecretary for maintaining an optimized R&D portfolio;
- Provide direction to the Research Council and Line Offices regarding execution of this NAO;
- Champion the NOAA R&D enterprise in interactions with NOAA, DOC, OMB, the Congress, other Federal agencies, other parties, and the public.
02. The NOAA Research Council will assist the NOAA Chief Scientist in execution of this NAO, and the Assistant Administrator for the NOAA Research line office will provide additional support for execution. The duties of the Research Council, with support from the NOAA Research line office, will include but are not limited to:
- Development, review, approval, and promulgation of the Procedural Handbook, enterprise plans, and reports identified in this NAO;
- Regular review of the NAO and Procedural Handbook after promulgation, and revisions to the documents if and when necessary;
- Maintenance of a website to provide access to the Procedural Handbook and relevant R&D plans, performance measures, evaluation reports, and enterprise reports;
- Review of annual R&D performance measures to ensure they are appropriate measures and to track progress;
- Regular review and analysis of the R&D portfolio and provision of advice to the Chief Scientist regarding the status and needs of NOAA’s corporate R&D enterprise; and
- Regular monitoring to ensure the intent of this NAO is being fulfilled.
03. Line Office Assistant Administrators and the Director, Office of Marine and Aviation Operations, will:
- Ensure R&D activities comply with the intent of this NAO;
- Provide input to the Research Council for plans and reports;
- Assign a lead scientist to represent the LO perspective in NOAA R&D planning and to be accountable for populating the enterprise R&D monitoring database in an accurate and timely manner;
- Ensure evaluations are conducted and reported in accordance with procedural handbooks; and,
- Track and report performance measures/milestones as required.
04. In accordance with their charter (SAB, 2009), the NOAA SAB will conduct activities which support the intent of this NAO, including:
- Advise the Under Secretary on short- and long-range strategies for research and application of science to resource management and environmental assessment and prediction;
- Submit reports to the Under Secretary; and
- Conduct or participate in laboratory, science center, program, cooperative institute, and benchmarking reviews as requested.
SECTION 6. REFERENCES.
01. The Research Council will develop and disseminate written procedures, plans, and reports as necessary to implement this Order, including but not limited to:
- A Procedural Handbook covering the following topics
- R&D Planning
- Enterprise R&D Monitoring Database
- Program/Laboratory/Science Center Reviews
- Portfolio Reviews
- Benchmarking Reviews
- Performance Measures
- R&D Reporting
- R&D Plans
- 20-Year Research Vision
- 5-Year Research Plan
- Annual R&D Integrated Priority List for Planning
- Annual R&D Integrated Priority List for Execution
- R&D Reports
- Annual State of NOAA Research Report
02. Existing documents referenced in this policy are as follows:
- Department of Energy (DOE), 2009. Energy Efficiency and Renewable Energy – Glossary. Department of Energy.
- Government Accountability Office (GAO), 2005. A Glossary of Terms used in the Federal Budget Process. United States Government Accountability.
- GPRA Modernization Act of 2010, Public Law 111-352, 2010.
- Information Quality Act (IQA). Public Law 106-554, Section 515. 2001. (See also "Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies" (67 FR 8452).)
- Mankins, JC, 1995. Technical Readiness Levels: A White Paper. Advanced Concepts Office, Office of Space Access and Technology, National Aeronautics and Space Administration.
- National Research Council (NRC), 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. The National Academies Press, Washington, DC. http://www.nap.edu/openbook.php?record_id=12150
- NRC, 2000. Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. National Academy Press, Washington, DC. http://www.nap.edu/catalog.php?record_id=9882
- NRC, 1999. Evaluating Federal Research Programs: Research and the Government Performance and Results Act. National Academy Press, Washington, DC. http://www.nap.edu/openbook.php?record_id=6416
- National Science Foundation (NSF), Division of Science Resources Statistics. 2009. Federal Funds for Research and Development: Fiscal Years 2006-08. Detailed Statistical Tables NSF 10-303. Arlington, VA.
- National Oceanic and Atmospheric Administration (NOAA), 2009. Business Operations Manual, September 2009, version 5.2. National Oceanic and Atmospheric Administration. http://www.ppi.noaa.gov//PPI_Capabilities/Documents/BOM.pdf
- k. NOAA, 2006. National Oceanic and Atmospheric Administration Information Quality Guidelines.
- NOAA, 2005. Policy on Transition of Research to Application. NAO 216-105, effective 05/17/05. http://www.corporateservices.noaa.gov/ames/administrative_orders/chapter_216/216-105.html
- Office of Management and Budget (OMB), 2009a. Circular No. A-11, Preparation, Submission, and Execution of the Budget. Section 84 Character Classification. Executive Office of the President, Office of Management and Budget.
- OMB, 2009b. Open Government Directive. Memorandum for the Heads of Executive Departments and Agencies from Peter R. Orzag, Director of the Office of Management and Budget (M-10-06).
- OMB, 2009c. Increased Emphasis on Program Evaluations. Memorandum for the Heads of Executive Departments and Agencies from Peter R. Orzag, Director of the Office of Management and Budget (M-10-01).
- OMB, 2004. Final Information Quality Bulletin for Peer Review. Memorandum for the Heads of Executive Departments and Agencies from Joshua B. Bolten.
- Organisation for Economic Co-operation and Development (OECD), 2002. Glossary of Key Terms in Evaluation and Results Based Management. OECD Publications, Paris, France. http://www.oecd.org/dataoecd/29/21/2754804.pdf
- Science Advisory Board (SAB), 2009. U.S. Department of Commerce, Charter of the NOAA Science Advisory Board.
http://www.sab.noaa.gov/Charter/SAB charter, 7-8-11_signed_FINAL.pdf
- SAB, 2004. Review of the Organization and Management of Research in NOAA. A Report to the NOAA Science Advisory Board. The Research Review Team.
SECTION 7. EFFECT ON OTHER ISSUANCES.
An electronic copy of this Order will be posted on the NOAA Office of the Chief Administrative Officer website under the NOAA Administrative Issuances Website.
Under Secretary of Commerce
for Oceans and Atmosphere
Office of Primary Interest:
Office of Oceanic and Atmospheric Research