Jane S. Peters Advising

The advent of AMI (aka. Smart Meters) has brought a welcome evolution in the ability to prove project savings. Several new measurement and verification (M&V) software products associated with M&V 2.0 have entered the energy efficiency world with some Open Source and some proprietary though all with substantial transparency so that the service providers and their customers can each see the savings resulting from the project.

The problem of determining when savings have occurred and when they have not is affected by baselines, by productivity, by changing building use, occupancy and other factors besides weather and equipment changes. When performance payment programs and projects first appeared in the 1980s as demand side bidding, shared savings, and standard performance contracting programs; customers had reason to be concerned. Could you really see the savings on the meter?

Energy service companies (ESCOs) devised methods to assure their customers that savings present along with contractual processes for resolving disagreements. An early technical tool was the International Performance Measurement and Verification Protocol (IPMVP). With the development of the IPMVP, energy services companies and other energy project providers had a better communication process and guidelines for selecting the methods to use for determining savings. Yet transparency was less than satisfactory as the customer and the services provider each needed their own engineers to implement the IPMVP and agree. Agreement was not guaranteed.

Making matters more complicated is that in some cases the energy services project used utility rate payer funds to pay for part of the project. If the savings are provided to the utility for a financial return, then the utility and their regulator want certainty as well. It is at this point that evaluation, specifically program evaluation comes into play. Program evaluation is an unwieldy means for determining savings on a project-by-project basis. Program evaluation inherently means the overall assessment of whether a program (a collection of individual projects completed following a common set of rules and guidelines) has had its intended effects, it is not intended to assess project specific savings. Though methods are similar, and in some cases, projects are sampled to provide evidence for program effects, evaluation is programmatic.

Looking at the solutions targeting pay for performance efforts, it is easy to be confused by the rhetoric, that implies that these M&V 2.0 solutions can replace evaluation. There should be no confusion. These tools are project and building specific, they are not program evaluation tools. One might think of these tools providing the customer and energy services team the confidence needed to establish payment, a first tier for gaining confidence in the project. But not enough to address the question of "is the program performing as intended." The good news is that these project specific tools should make the evaluators job easier especially when Open Source solutions are used. The research questions of a program evaluation and the sampling to address the questions remain necessary and fundamental to assessing program effectiveness.