A growing number of employers are requiring workers who smoke to pay more for their health care costs, The New York Times reports.

According to a survey by the benefits consultant Towers Watson, 40 percent of employers with at least 1,000 workers have instituted a health insurance premium surcharge for smokers. Recently, Wal-Mart announced it is significantly raising health insurance premiums for many employees who smoke—up to $2,000 in additional charges per year.

Many companies adding surcharges for employees who smoke or who are overweight say they are making up for higher health care costs, and some say they want employees to take more responsibility for their health.

Some experts say such policies punish people for health problems that are not entirely under their control. They point out nicotine addiction makes it difficult for smokers to quit.

A number of health groups including the American Cancer Society and American Heart Association sent a letter to federal officials earlier this year asking that wellness programs be prohibited altogether from varying an individual’s insurance premiums or cost-sharing amounts due to a health factor.