Lessons Learned from a Smart Home Demonstration in New York City

April 01, 2019
Spring 2019
A version of this article appears in the Spring 2019 issue of Home Energy Magazine.
Click here to read more articles about

While the concept of a smart home may still appear to be confined to the niche market, there now is a wide range of commercially mature, consumer-facing smart home devices that can improve comfort and convenience for a home’s occupants, and that also have the potential to save energy. These products all fall under a broad category of technology known as home energy management systems (HEMS). In recent years, many HEMS products have proven to be cost-effective and useful but the public has been slow to adopt them, especially when compared to the rapid adoption of smartphones, voice assistants, and other Internet of Things (IOT) devices.

This article describes a HEMS demonstration effort conducted in 2017–18 that was funded by the New York State Energy Research and Development Agency (NYSERDA). The National Renewable Energy Laboratory (NREL) collaborated with Consolidated Edison (Con Edison), which provided project cost share, to conduct an end-to-end demonstration of practical HEMS packages that can be deployed at scale, using technologies that are commercially available. The main objectives of this effort were to implement simple HEMS solutions for lighting, space conditioning, and plug loads in single-family homes (SFHs) and multifamily units (MFUs) in Con Edison’s service territory, and to develop technology transfer strategies based on the insights gained during the implementation process. A particular focus was placed on stakeholder education to help promote the benefits, including the energy savings potential, of HEMS technologies across New York State, and to achieve wide-scale adoption of proven products and strategies.

Most of the multifamily homes and two single-family homes in the project were given an Amazon Echo Dot. While voice assistants are not a conventional energy-management tool, consumers are adopting them at a stunning pace. (Courtesy of Amazon.com, Inc.)

This emphasis on stakeholder engagement was based on the hypothesis that the public has been slow to adopt HEMS not because they aren’t available but because people are not aware of the benefits they can provide. The surge in HEMS startup companies and product offerings over the past decade coincides with several notable technology and market trends. These include

  • growth in web-based cloud computing applications that make possible low-cost home energy data storage, analytics, and display;
  • ubiquitous access to simple user interfaces through broad market penetration of inexpensive touch screens for smartphones and tablets;
  • increase in demand for teleworking capabilities, expanding the need for secure home area networks; and
  • increase in embedded sensors and controls capabilities in smart home appliances, including internet-ready appliances, multimode and variable-speed controls, fault diagnostics for space-conditioning equipment, and sophisticated control cycles for major appliances.

These trends suggest that intelligence in residential building systems should be commonplace; but at present there is little or no market for HEMS devices. A key barrier to market transformation, then, may be a lack of robust consumer engagement programs that would encourage homeowners to install HEMS. The demonstration described here was designed to address this need.

Demonstration Design

The project leveraged NREL’s expertise in HEMS technology and Con Edison’s customer network and relationships, as well as its experience with conducting measurement and verification. NREL was primarily responsible for the overall demonstration design; for selecting, procuring, and installing the HEMS devices; for developing the homeowner engagement strategies and ensuring that participants received enough support to make the most of their smart home experiences; and for data analysis and reporting. Con Edison and its subcontractor, Energy & Resource Solutions (ERS), conducted participant recruitment and all of the energy-monitoring activities.

We considered for deployment a range of products targeting lighting, space conditioning, and plug loads. For homes with central heating and cooling, a smart thermostat offers several attractive advantages over a conventional programmable or nonprogrammable thermostat. Occupancy detection can help reduce space-conditioning energy use when the home is vacant. All smart thermostats are connected, which means that they can be remotely accessed via smartphones and tablets for scheduling and set point adjustments. Some devices are equipped with additional wireless remote sensors to improve comfort throughout the home, not just where the thermostat is located. Past program results demonstrate savings of up to 12.5% and 16% for gas (heating) and electric (heating and cooling) respectively, and studies conducted by thermostat manufacturers indicate that smart thermostats can potentially save anywhere from 10% to 23% on heating and cooling costs.

Smart lighting can be incorporated either by changing out individual lightbulbs with smart, connected lightbulbs, such as the TP-Link Smart WiFi LED bulb (pictured), or installing smart switches that have wireless communication, such as the TP-Link Smart WiFi Plug Mini.

Many homes, especially MFUs, have point-source heating and/or cooling systems, such as window-mounted air conditioners or mini-split heat pumps (MSHPs). There are third-party controllers (which function as thermostats but are very different from smart thermostats used on central systems) that are designed to add smart and connected functionalities to point-source equipment. These devices use an infrared (IR) signal to control the window air conditioner or MSHP and also connect to Wi-Fi so they can be remotely adjusted. Schedulers allow for setbacks so that equipment is off while people are away but can still ensure that the apartment is comfortable when people return home.

There is a wide array of smart home lighting products on the market, all aimed at saving energy through the use of light-emitting diode (LED) bulbs combined with scheduling features. Convenience is also a powerful driver for smart lighting. Users can schedule lights to turn on before they come home or rely on “house-sitting” security features that can mimic occupancy by cycling lights at random intervals when occupants are away. Smart lighting can be incorporated either by installing smart switches that have wireless communication, or by changing out individual lightbulbs with smart, connected lightbulbs, and there are pros and cons to each approach.

Switches are the preferred solution for light fixtures with numerous bulbs or with nonstandard bulb types, where replacing each bulb with a smart bulb may be cost prohibitive or impractical because of the limited available selection of smart bulbs. A possible complication is that most smart switches require connection to the neutral wire, and older homes with legacy electrical wiring may not be set up with a neutral. On the other hand, smart bulbs offer a less-expensive entry into smart lighting and are easy to install. A key difference is that for smart bulbs to operate on a schedule or to be controlled via a wireless app, the wall switch needs to remain in the “on” position. Depending on the user and use case, this could present an undesirable limitation.

There are several plug-load controllers with connected capabilities. Some are plug pass-through devices for controlling individual loads, while others, called advanced power strips (APS), double as power strips and are designed to facilitate convenient energy savings in the home office and entertainment center, where the concentration of consumer electronics is typically high. Pass-through plug controllers can be remotely turned on and off, and schedules can be set up using a smartphone app or website. APSs are typically not connected but employ a variety of smart algorithms to manage active and standby power loss.

The particular devices used for this demonstration project were selected based on features, prices, availability, and ease of use (and thus likely consumer acceptance). The products and key functionalities are summarized in Table 1.

Table 1. Selected HEMS devices

Device Make and Model







Sensibo Sky (Gen. 2)




TP-Link Smart WiFi LED bulbs



Plug loads

TP-Link Smart WiFi Plug Mini



Embertec Emberstrip AV



Voice assistant

Amazon Dot



* Embertec power strip can connect to local phone via Bluetooth for additional data and control.

Consumer engagement and acceptance were key goals for this demonstration, so voice-based artificial intelligence (a “voice assistant”) was included in the HEMS package. As shown in Figure 1, all of the HEMS devices employed in this demonstration except the APS work with Amazon’s Alexa. The Amazon system was selected over other popular options from Google, Apple, and others because Alexa is built into the ecobee4, so that the thermostat functions as a smart speaker. Given the rising popularity of smart speakers, we leveraged this opportunity to introduce voice command options for the other smart home products included in the demonstration. Most of the MFUs and two of the SFHs did not receive an ecobee thermostat; those homes received an Amazon Dot smart speaker, which is a more basic version of the Amazon Echo, without the premium speakers. While voice assistants are not a conventional energy-management tool, consumers are adopting them at a stunning pace. According to a 2017 Pew Research Center survey, nearly half of all Americans use some form of digital voice assistant on a regular basis. While most interactions are with smartphone assistants, and most people do not have stand-alone smart speakers (such as Amazon Echo or Google Home) in their homes, this is changing rapidly, as more and more manufacturers of consumer-connected devices look to leverage popular voice platforms.

Figure 1. All HEMS devices used in this project (except APS) integrate with Amazon Alexa.

Participant Recruitment Challenges

Con Edison and its subcontractor, ERS, conducted recruitment. They emailed a recruitment flyer to prospective participants who had previously received rebates for new central air-conditioning systems or new window air conditioners, but who were not already enrolled in Con Edison’s smart thermostat or smart window air-conditioning program. Interested customers were invited to fill out a brief online survey to help us determine their eligibility for this study. Once people completed the survey, ERS contacted them for a follow-on conversation to determine eligibility and confirm interest. Initially the target pool included only Brooklyn and Queens, but because it was difficult to recruit enough participants from these areas, the target pool was expanded to include Westchester County.

Recruitment proved to be a major challenge. A total of 5,818 solicitation e-mails were sent to prospective participants. The incentive offered for participation was that the HEMS devices installed would be free to keep. Depending on the home, this offer was equivalent to a roughly $500 value, an amount that we anticipated would draw many more people to respond with interest. After ERS conducted phone screens, we were able to recruit a total of 19 prospective participating households—a few short of our initial goal of 24.

There were four opt-outs over the course of the demonstration, leaving a final total of 15 participating households. Two participants dropped out after learning that their space-conditioning systems were incompatible with the smart thermostat. One participant became irritated with the connectivity issues encountered during the smart bulb installation and opted out of the program immediately. One household had the pre-HEMS monitoring equipment installed on the first site visit, but didn’t respond to the project team’s efforts to contact them to schedule a HEMS installation visit.


While hard energy data were difficult to obtain, in a demonstration consisting of only 15 homes quantitative savings conclusions would be anecdotal at best. The most useful results from this demonstration are derived from the observations and challenges we encountered during the actual site visits, and during subsequent homeowner interactions. These results have led us to make to the following recommendations.

Smart thermostats are commonly marketed as do-it-yourself, or DIY, products, but they are not practical DIY installations for many people. The most common issue is the lack of a common, or C, wire, which makes it hard for do-it-yourselfers to install them. But there are many other situations where a smart thermostat cannot be installed. Even within this small demonstration, we encountered a variety of configurations that were not immediately compatible with the ecobee smart thermostat. These included

  • lack of a C wire;
  • wireless thermostat proprietary to existing air-conditioning unit;
  • master/slave thermostats;
  • multiple zones; and
  • proprietary thermostat wiring.

In some of these cases, a service technician was able to follow up and run new wires, install a device to mimic the C wire, or replace the control board on the air handler to make the smart thermostat compatible, but these solutions can hardly be considered DIY for most people.

Because installation challenges were significant, future programs may wish to include installation credit as an option, and work with local trades to have a list of approved and recommended installers. For this project, it was difficult and time consuming to identify service technicians willing to visit the sites and complete the ecobee installations.

The ecobee4 comes wired with Alexa, allowing the thermostat to function as a smart speaker. (Courtesy of ecobee)

Window A/C (or MSHP) controllers posed fewer compatibility problems than the smart thermostats, but there were notable installation hiccups. The Sensibo requires a remote control to the air conditioner, but many window air conditioners do not have remote control. Furthermore, the remote is actually required for initial setup. This proved problematic for some installations where the remote was lost, or its battery had expired. The controller needs to be plugged in and mounted in a location with a direct line of sight to the window air conditioner or MSHP IR sensor. The Sensibo controller had a fairly short power cord, which often made installation challenging, especially in older homes with few power outlets. These quirks notwithstanding, window air-conditioning controllers were some of the easier devices to set up. The app was basic but usable. For utility service areas that have many window air conditioners, we recommend incentivizing window air-conditioning controllers in their programs.

Convenience is a key driver, and lack of convenience is a major turnoff. Many people are willing to purchase a product and learn how to use it if they believe it will make their lives easier. Once the introductory barrier was overcome, many people really embraced the convenience features.

Conversely, if the audience perceives that the learning or commissioning process is too cumbersome, that audience may be lost forever. One skeptical would-be participant was somewhat reluctantly going along with the HEMS site visit (that his partner had signed up for), but as soon as one smart bulb failed to connect via Wi-Fi on the first try, he became convinced that these technologies were a waste of his time. This was not the only household that was sensitive to installation time.

Voice assistants appear to be here to stay. While it is not itself an energy-saving device, Alexa (and other voice assistants like it) can be a powerful hook to get people to embrace smart controls. Although most participants reported initially that they were motivated primarily by the chance to upgrade to a smart thermostat, it often became clear during the HEMS visit that the smart speaker was in fact the upgrade that excited people the most.

Installation difficulties were numerous and significant. DIY systems are often not as straightforward as advertised. Wi-Fi signal in homes is often unreliable, making it difficult to set up and demonstrate new devices, and we found that most homes (even small condominiums) had extenders and/or multiple networks. In addition, many people do not necessarily know, or have easy access to, their Wi-Fi passwords, which were required to connect devices to their home Wi-Fi networks.

Smart bulbs save significant energy if they are installed to replace non-LED bulbs, but otherwise the savings depend on how people use the remote and schedule features, and whether they use the dimming function. Our data showed that homeowners used the smart bulbs more than they used the bulbs they replaced, possibly because they could set up schedules that operated the lights more often than was possible with the regular bulbs. Homeowners were generally excited about the controllability of the smart bulbs, so from a consumer engagement standpoint that enthusiasm may be worth the higher cost, and potentially increased usage, of the smart bulbs. Smart bulbs are generally easy to install, so they are a good candidate for DIY measures. Making sure that existing rebates for LEDs can be used for smart bulbs, and possibly offering higher rebates for the more expensive smart bulbs, should encourage more people to buy them. Programs may wish to consider packaging smart bulbs with a voice assistant.

Schedules almost always improve convenience, but they may increase or reduce energy use, depending on how they are used. Scheduling setbacks when residents are not home is the main way that smart thermostats and window air-conditioning controllers can save energy, but that is not necessarily how the scheduling function was used. There were cases where energy use went up after smart controllers were installed. This was probably because the homeowners were scheduling their heating system to heat the house when they were not at home so that it would be warmer when they got home. The improved comfort would benefit consumers even with an energy penalty. It is possible that providing homeowners with an easier way to adjust their thermostat (such as from a smartphone app) enabled them to find their preferred set point more easily. In the case of lighting, being able to set up a schedule or use the voice assistant to turn on lights may have made it more convenient to use some lights than it was before.

Plug-load controllers should be purchased to address specific-use cases, since it was difficult to find plug loads that were suitable to use with plug-load controllers. A few good examples were holiday lights, lamps with nonstandard bulbs, and diffuser candles. This may be an opportunity for further research or improved education to point people toward effective applications, perhaps by sharing a list of examples. Programs could offer rebates (and possibly bundle with a voice assistant) but should try to give guidance so plug-load controllers are useful.

learn more

For more information, look for the full NYSERDA report titled “Home Energy Management Systems (HEMS) Demonstrations In New York City and Westchester Residences."

Advanced power strips can be challenging for consumers to adopt if they interfere with the way people normally operate their electronics. Of all of the HEMS devices employed in this demonstration, the APS was the only product where the feedback was entirely negative. This was in stark contrast to the other devices, which, although there were occasional glitches or minor irritations, generally improved convenience for users. The APS was difficult to use and interrupted people’s use of their entertainment center. It is possible that with more user training some of the obstacles may be overcome, but the threshold seems too high to be practical. A simpler solution—such as a collection of controlled outlets that can be scheduled or turned off via a voice assistant—might be more acceptable to users.

Smart home features and functionalities are still very new to most people, and the importance of homeowner interactions cannot be overstated. At first many participants seemed overwhelmed by the confusing array of features and app functions presented to them, on top of the basic functions of the new HEMS devices. The participants in this demonstration largely report positive experiences with the smart home devices, but it took a lot of user support and continued engagement to obtain their buy-in.

Bethany Sparn and Lieko Earle are researchers at NREL.

The authors thank NYSERDA for funding this work.

  • 1
  • NEXT
  • LAST
Click here to view this article on a single page.
© Home Energy Magazine 2023, all rights reserved. For permission to reprint, please send an e-mail to contact@homeenergy.org.
Home Energy Pros
Discuss this article with other home performance professionals at Home Energy Pros!.

Add a new article comment!

Enter your comments in the box below:

(Please note that all comments are subject to review prior to posting.)


While we will do our best to monitor all comments and blog posts for accuracy and relevancy, Home Energy is not responsible for content posted by our readers or third parties. Home Energy reserves the right to edit or remove comments or blog posts that do not meet our community guidelines.

Related Articles
SPONSORED CONTENT What is Home Performance? Learn about the largest association dedicated to home performance and weatherization contractors. Learn more! Watch Video