BOC graduate results are well-documented on our Success Stories page. In addition, comprehensive evaluations are completed by third-party evaluators approximately every two years. The evaluation teams assess the level of satisfaction with the training program from the perspective of students and key stakeholders like employers and utilities.
2017 BOC Program Dataset Analysis (PDF 948 k) This memo documents Research Into Action’s analysis of the Northwest Energy Efficiency Council (NEEC) Building Operator Certification (BOC) program dataset as of 2017. The primary goal of this analysis was to describe the 2017 new BOC certificants and update the count of active BOC certificants in the four Northwest (“Northwest”) states as of January 29, 2018. The certificants counted in this memo include those in target groups of NEEA’s BOC Expansion (BOC-E) Initiative, established in 2012 to accelerate adoption of BOC and increase its market penetration in the Northwest. Those groups are: operators in underserved areas, 1 Federal employees, and members of the International Union of Operating Engineers (IUOE). Although BOC-E targets the above specific groups, NEEA’s support for BOC is not limited to those groups.
BOC Energy Savings Summary and FAQ (PDF 350k) The Building Operator Certification (BOC®) program has consistently produced positive documented energy savings and has proved to be cost effective. Since 2000, a number of BOC program sponsors have engaged independent third-party evaluators to assess and document the BOC’s energy savings impacts. This factsheet (FAQ) summarizes the electricity and fossil fuel savings from the studies published in this body of work. With increased reliance on energy efficiency as a resource and more utilities claiming energy savings for their BOC programs, the energy savings continue to be rigorously scrutinized.
2016 BOC Program Data Analysis (PDF 312k) This memorandum documents Research Into Action’s analysis of the Northwest Energy Efficiency Council Building Operator Certification (BOC) program dataset as of 2016. A review of the BOC dataset of certified operators took place during January and February 2017. This report provides an updated count of the number of active certified building operators for 2016 energy savings reporting. In 2016, the BOC program registered 157 new certificates. Also in this report, the reader will find a brief update on the program’s recent alignment with the ISO 17024 standard, the process used to determine the number of active BOC certificants, and a year-by-year count of new, retired and total active certificants. An additional section has been included this year on opportunities for further research.
2015 Impact Evaluation – Energy Efficiency and Demand by Public Service of New Mexico (PDF 2.8MB) This report represents findings from an impact evaluation of the 2015 Energy Efficiency and Demand Side Management (DSM) programs by the Public Service of New Mexico (PNM). The methodology used in the evaluation provided net impact results, program feedback and recommendations, and program cost effectiveness. In this report, BOC is a sub component of the Building Tune-Up program. BOC participants and instructors were surveyed on program awareness, the importance of tuition rebate provided by PNM, and energy efficiency improvements made before and after completing a BOC course.
Comprehensive Review of Non-Residential Training and Education Programs, with a Focus on Building Operator Certification (PDF 1.2MB) June 2015 – This evaluation provides a comprehensive review of education and training programs that may produce savings for the eight Massachusetts program administrators (PAs). The focus was on Building Operator Certification (BOC), and highlights how to increase the number of and share of MA customers who achieve BOC certification with assistance of Program Administrator-provided training subsidies, develop a better understanding of the factors influencing BOC savings, and updated the MA TRM savings estimate for BOC.
BOC-Expansion Initiative Market Progress Evaluation Report #2 (PDF 822k) February 2015 – This is the second Market Progress Evaluation Report (MPER) of the Building Operator Certification Expansion (BOC-E) Initiative. The report concludes that BOC-E is progressing toward its goals, especially those supporting expanded outreach to large employers and utility sponsors, market penetration, and energy savings from certified operators. Market penetration in the 4-state Northwest region is 18% which suggests opportunity for further expansion. Non-credentialed building operators are interested in BOC but employer support is critical. BOC certificants, compared to a control group of non-certificants, are saving energy for their companies to the tune of 2% of electricity use and 3.5% of natural gas use. Finally, five regional utilities recommend or require BOC certification as a criterion for participation in energy efficiency programs.
Focus on Energy MEEA Training Program Evaluation (PDF 703k) January 2015 – This report summarizes the findings from a process and impact evaluation study conducted with Wisconsin building operators participating in BOC training in the period 2011-14. The evaluation showed high participant satisfaction with the courses and presents details on factors that led participants to take the trainings. Additionally, the evaluation assessed savings for the retrofit activities using the Wisconsin Technical Reference Manual (TRM) and other common industry sources for energy consumption data by building type and end use. The BOC savings reflect the energy savings from projects training participants completed after their training course, excluding projects rebated by Wisconsin Focus on Energy programs. The report identifies savings metrics per participant of 84,900 kWh/year for electricity and 36 therms/year for natural gas.
BOC-Expansion Initiative Market Progress Evaluation Report #1 (PDF 2.6MB) April 2014 – This is the first Market Progress Evaluation Report (MPER) of the Building Operator Certification Expansion (BOC-E) Initiative. The report concludes that BOC-E is progressing toward its goals, especially those supporting expanded outreach. To reach Federal employees, NEEC/BOC has taken actions to become a U.S. General Services Administration (GSA) training provider and continues to seek American National Standards Institute (ANSI) accreditation. The number of active BOC certificants is growing, but the operator market is larger, and thus BOC penetration is lower than previously assumed. Between 2011 and 2012, the Northwest experienced a net gain of 98 active BOC certificants (those who received or renewed certification within the previous five years). However, previous research underestimated total market size because it overestimated the mean facility square footage per operator. We estimate the total NW market to be about 10,000 building operators. With about 2,000 currently employed BOC operators, market penetration is about 20%.
The survey found that certain categories of operations staff are relatively more common among certain employer types: custodial staff in K-12 schools, management staff in colleges, professional staff in health care and property management firms, and general technical/mechanical staff in government and municipally owned facilities. This finding suggests multiple market sub-sectors, which may have implications for marketing, outreach, recruitment, and/or course planning. BOC-attributed savings comprise approximately 2% of electricity use, 1.8% of fossil fuel use, or 1.9% of BTU consumption from both electricity and fossil fuels. Certification-related savings may persist beyond five years. We found no evidence that savings percentages decreased after certification had lapsed five or more years, suggesting possible persistence of savings beyond the currently assumed five-year measure life. However, low statistical power prevents drawing firm conclusions.
Impact Evaluation of the California Statewide Building Operator Certification Program (PDF 2.6MB) Feb. 2014 – This report presents findings from an impact evaluation of the 2010-2012 statewide Building Operator Certification (BOC) program, which is a series of classes on various energy-saving equipment, measures and operations for building operators and/or facilities personnel. The program is funded by California Investor-Owned Utilities (IOUs) and administered by the Northwest Energy Efficiency Council (NEEC). The study was prepared by Opinion Dynamics Corporation under contract with Itron Inc. and California Public Utilities Commission. This study sought to better understand the energy saving potential and whether there are methods that would allow for a higher level of rigor when estimating energy savings from training programs. The study was conducted by means of a multi-step process built on existing program databases, course information, participant surveys and on-site visits. The study explored gross savings, attribution issues and double-counting with resource programs. Results indicate that performing on-site verifications in addition to surveys, allow for more accurate and comprehensive collection of data. The study also highlights the growing trend of low response rates to these types of studies and how that can impact the certainty of energy savings.
Impact and Process Evaluation of NorthWestern Energy 2007–2011 DSM Programs (BOC Chapter) (PDF 1.2MB) This report presents the methodology, findings and recommendations based on an impact and process evaluation of the NorthWestern Energy (NWE) Demand Side Management (DSM)portfolio, including Building Operator Certification, BOC. The conclusions that were reached from the process evaluation of this program were that energy savings for each year were reflected at a rate of UES values of 0.24 kWh/ft2 and 0.0013 dkt/ft2. Eighty percent of respondents reported that no area of the course needed to be improved. Nearly all (29 of 30) surveyed trainees reported that they would recommend the BOC trainingto their colleagues. Almost all surveyed trainees (28 of 30) indicated that they had applied the concepts or methods from the training at their facilities or shared the concepts with their coworkers (29 of 30) and most trainees reported that their facilities had started or completed at least one project aimed at increasing energy efficiency since they received the training. Recommendations for the BOC program from this evaluation were to consider ways to increase the use of internet tools to facilitate participation and to develop written program and process plans to improve consistency of objectives/ goals and strategies / tactics for the BOC.
ACEEE 2012 Estimating Savings from BOC Training, (PDF 293k) This report summarizes the methodology used to determine energy savings tied to Building Operator Certification. Explains how the behavior based training model can translate into real dollar savings through energy management in facilities.
Program Year 3 DCEO Building Operator Certification (BOC) Program Evaluation, May 2012 (PDF 1.3 MB) In this report to the Illinois Department of Commerce and Economic Opportunity (DCEO), an outside consulting firm, Navigant Consulting, examined the key findings that validate the benefits of completing the Building Operator Certification training course by analyzing and modeling student feedback. They found that net savings were 181 MWh and 37 kW per BOC participant. In depth detail about savings per participant, per square foot, and by various energy measures can be found in the report in Table E-2. Participant satisfaction with the course is very high- no respondent provided either the lowest rating of “fair” in the final course evaluations or the lowest ratings of “somewhat satisfied” or “not at all satisfied” in the participant telephone survey. Consistent with a high satisfaction rating, 81% of all participants surveyed responded that they had already recommended the BOC training program to colleagues. Feedback regarding the course approach was positive overall, while the study did outline several areas of opportunity to meet enrollees scheduling and logistics needs. Students enrolled in the program also had high remarks for the program instructors and marketing.
Northwest Energy Efficiency Alliance (NEEA) Long-Term Monitoring and Tracking Report on 2011 Activities, July 2012 (PDF 2.5 MB) In this report, Navigant Consulting examined the post-funding market effects of five initiatives: Verdiem, Residential Windows, Drive Power, Building Operator Certification and Commissioning. Per the logic of most market transformation efforts, the bulk of any energy savings tends to occur in these post-funding periods. Navigant estimates these projects have yielded an incremental 6.6 aMW in 2011 with the most significant savings associated with Commissioning (2.4 aMW), followed by Building Operator Certification (2.0 aMW). Drive Power reports 1.4 aMW in incremental savings, while Residential Windows resulted in 0.8 aMW. Verdiem showed 0.02 aMW in incremental energy savings.
Summary of Building Operator Certification Program Evaluations for Consumers Energy,Nov. 2011 (PDF 1.5 MB) To support a decision to expand its BOC pilot program in the future, Consumers Energy sought input from its evaluation team to gain a better understanding of the potential for deeming measurable savings based on the results of prior evaluations of BOC programs. A review of six reports led the team to recommend that Consumers Energy conduct the BOC program and claim savings based on prior research.
Northwest Energy Efficiency Alliance (NEEA) Long-Term Monitoring and Tracking Report on 2010 Activities, June 2011 (PDF 2.1 MB) Third-party evaluator, Navigant Consulting, examined the post-funding market effects of five projects: BacGen, Building Operator Certification, Drive Power, ENERGY STAR Residential Windows, and the SAV-AIR Initiative. Navigant estimates these projects yielded an incremental 2.9 aMW in 2010 with the most significant savings associated with Building Operator Certification (1.5 aMW).
Evaluation of Minnesota Building Operator Certification Training Prepared for Midwest Energy Efficiency Alliance (MEEA) and Minnesota Office of Energy Security, March 2011 (1.5 MB) Long time partners Midwest Energy Efficiency Alliance, the Minnesota Department of Commerce and Minnesota Power have certified over 230 Building Operators since 2005. Third party Navigant Consulting concluded that participant satisfaction with the course is quite high, communication between instructors, participants and MEEA is excellent.
The Evaluation of Kansas City Power and Light’s BOC Program, Opinion Dynamics Corporation, 2009 (PDF 487k), notes that, in the trainings under the evaluation team’s review, there was an average savings of 43,600 kWh per graduate. Participant satisfaction rating was very high, with 93% rating the overall program very good or excellent.
BOC Evaluation Report from the California Workforce Education & Training Program evaluation (PDF 123k) Conducted on 2006-08 activities. The report concludes “Overall, all BOC participants generally agree that they have been able to reduce energy usage, enhance comfort and save money at their facility as a result of the BOC training program.”
American Council for an Energy Efficient Economy’s (ACEEE) 2006 BOC Summer Study(PDF 182k) concludes that “…it is clear that building operators are saying the BOC changes their on-the-job behaviors.” One supervisor interviewed said that by sending his employee for BOC training, “We made a dangerous man out of him. Now all he wants to do is save money!” BOC is a recipient of ACEEE’s exemplary program award.
California Evaluation Study Brief: Nonresidential Sector Process Evaluation of the 2004-2005 Statewide Building Operator Certification & Training Program (PDF 180k) revealed another high satisfaction rate with the program, even among highly experienced operators. Two-thirds of the participants from Level I indicated that they wanted to continue on to Level II training.
The Impact and Process Evaluation of the Building Operator Training and Certification Program, June 2005 (PDF 1.6mb) noted in its study that persistence was 114.2% for electric energy savings, 108.5% for oil and gas savings, and 100% in water savings over a five year period for the participants under review.
Within the Retrospective Assessment of the Northwest Energy Efficiency Alliance, 2003 (PDF 467k), the effectiveness of a number of NEEA programs is examined, including BOC, for their cost effectiveness.
The Evaluation of the 2002 Statewide Building Operator Certification and Training Program, 2003 (PDF 1.32MB) found that 93% of participants and 90% of their supervisors either have or would recommend BOC training to others in the field.
BOC Market Progress Evaluation Report series prepared by the Northwest Energy Efficiency Alliance marks the progress of the BOC training program from 1997 through 2001. The final report (September 2001) estimates the BOC saved the region (at that time, the Pacific Northwest and California) 172,000 MWh annually.