In a new book titled, Insurance and Behavioral Economics: Improving Decisions in the Most Misunderstood Industry, authors Howard Kunreuther and Mark Pauly, both professors at Wharton, and Urban Institute researcher Stacey McMorrow analyze the behavior of individuals, insurance industry leaders and policy makers, all of whom bring certain behavioral biases to their understanding of the insurance industry. The result, say the authors, is an overall failure to grasp how insurance can fulfill the roles it is designed to play: reducing future losses and financially protecting those at risk.

In the following essay, Kunreuther and Pauly elaborate on some of the key findings presented in their book.

Insurance is an extraordinarily useful tool to manage risk. When it works as intended, it provides financial protection to individuals and firms who pay insurers a relatively small premium to protect themselves against a large loss. But insurance is broadly misunderstood by consumers, insurance executives and regulators.

Many consumers do not voluntarily buy coverage against potentially risky and serious losses. Case in point: Fewer than half the residents in flood and hurricane-prone areas were insured against water damage from Hurricane Katrina and Hurricane Sandy. And a significant fraction of the population does not have health insurance today, despite the large premium subsidies currently offered in the form of Medicare and Medicaid and tax breaks for employment-based health insurance. A principal reason for this is that many people tend to view insurance as an investment rather than a protective measure. If, after several years, one doesn’t make a claim, there is a feeling that one’s premium has been wasted.

Insurance firms also behave strangely. After they suffer a severe loss, they may decide that a risk is completely uninsurable rather than determining whether they should increase their premium. For example, prior to 9/11, insurers did not price terrorism risk when providing coverage against damage to commercial property. After 9/11, most carriers refused to offer terrorism insurance because they feared catastrophic losses from future attacks.

State regulators often constrain insurance premiums because they are concerned that insurance will not be “affordable,” especially to those who are at higher risk. In Florida, the state set up its own insurance company called “Citizens” that offers highly subsidized premiums to residents in hurricane-prone areas. Private insurers could not compete against these prices, and Citizens became the largest insurer of homeowners’ coverage in the state. All taxpayers in Florida will be required to help pay for Citizens’ losses, should the state be hit by a devastating hurricane.

Similarly, the Affordable Care Act (ACA) health reform legislation requires sellers of individual and small group insurance to sell coverage to all comers at premiums that do not take into account the buyer’s medical risk, given age and local prices for health services. These policies assist those in the high risk category but impose additional costs on lower risks in the form of higher medical premiums.

Why do consumers, insurance firms and regulators behave as they do?

There is a tendency for those at risk to assume that disaster losses or major health related expenses will not happen to them. Given this view, they feel no need to purchase insurance protection. Only after suffering a loss will consumers voluntarily buy insurance. After a disaster, insurers may decide to restrict coverage, and state regulators are likely to prevent private insurers from charging premiums that reflect the actual risk.

Behavior of this kind defeats the three principal purposes of insurance: to provide information via premiums as to how serious your risk is; to provide motivation for undertaking financial protection against an event that could produce a significant loss but has a low probability of occurrence; and to offer incentives in the form of premium reductions to reward people who invest in risk-reducing measures.

Incentives, rules and institutions that encourage a constructive role for insurance will ultimately improve individual and social welfare. Several recent pieces of legislation have set the tone for appropriately dealing with risk.

In light of the private insurance industry’s refusal to provide sufficient amounts of terrorism coverage following 9/11, Congress passed the Terrorism Risk Insurance Act (TRIA) in 2002. It provided taxpayer-backed protection to insurers against catastrophic losses from future terrorist attacks if they agreed to make coverage widely available. As a result, businesses are now able to purchase reasonably priced terrorism coverage. To date, there has been no need to call on taxpayers to fund the guarantee. TRIA is up for renewal in 2014, and there is an opportunity to re-examine the appropriate roles of the private sector and the federal government in providing coverage.

The Biggert-Waters Act, passed in July 2012, proposed major reforms to the National Flood Insurance Program (NFIP) over the next five years. Future premiums will reflect risk (tied both to specific location and expected climate change) so individuals are aware of the hazard they face. They can also be rewarded with lower insurance rates if they undertake protective measures. FEMA is in the process of developing more accurate flood maps to set these rates. The Act authorizes $400 billion per year for this purpose over fiscal years 2013 – 2017.

The ACA requires insurers to offer insurance to all residents in the United States who do not currently have coverage through either their job or a public plan. It also levees a tax penalty on those who choose to be uninsured. To deal with the affordability issue, premiums are to be subsidized for some low- and middle-income households. However, with the exception of offering premium discounts for those who engage in a limited set of less risky behaviors (such as not smoking), premiums after 2014 no longer reflect individual medical risk factors. There is thus some concern that the penalties specified by the ACA may not be enough to encourage low risk individuals to buy insurance because of the high premiums they will have to pay.

What can be done to make insurance a better policy tool and to avoid adverse side effects of the well-intentioned programs already in place?

One way to convince people of the long-term benefits of insurance is to stretch the time horizon over which the event can occur. Studies have shown that people are much more likely to buy insurance or invest in protective measures if an event, such as a hurricane, that has a one in 100 chance of occurring next year is presented as having a greater than one in five chance of happening at least once in the next 25 years. And if the disaster does not happen – well, the truth is that the best return on an insurance policy is no return at all. One should celebrate not having a major loss!

Insurers should construct worst-case scenarios for rare events. They can then determine a premium that reflects their best estimate of their expected future risks, factoring in the uncertainty of the event’s happening. Insurers could also consider offering multi-year policies if state regulators allow them to price coverage that reflects risk over that period. A multi-year insurance policy with risk-based premiums coupled with a multi-year home-improvement loan to pay for risk-reducing measures may enable policyholders to reduce their overall costs.

State insurance regulators should be appointed rather than elected so they are less prone to being influenced by special interest groups and lobbyists. Regulatory decisions should make transparent who stands to benefit from a subsidized insurance program, and who will be paying part of that cost to protect others. State insurance programs, such as Citizens in Florida, should indicate to all residents in the state that property insurance on homes near the ocean (including second homes) is likely to be highly subsidized, and those living elsewhere may bear the expenses of the clean-up following the next severe hurricane.

These concepts, if followed, will increase the chances that insurance is better understood so it can fulfill the roles it is designed to play: reducing future losses and financially protecting those at risk.