While momentum to legalize marijuana at the state level is growing, a number of towns and counties are moving to ban legal sales of the drug, The New York Times reports.

In Colorado and Washington state, where recreational marijuana is now legal for adults 21 and older, localities’ decision to ban marijuana sales could sharply reduce tax revenues, the newspaper notes. Supporters of legalization have promised marijuana sales could bring in hundreds of millions of dollars in tax revenues.

If many towns and counties ban marijuana sales, legal marijuana could end up being sold in a patchwork system, with most sales occurring in urban areas.

“In some ways I think the best thing that could have happened to the anti-legalization movement was legalization, because I think it shows people the ugly side,” said Kevin A. Sabet, a former drug policy adviser to President Obama and the Executive Director and co-founder of the anti-legalization group Smart Approaches to Marijuana. The group supports removing criminal penalties for using marijuana.

Yakima County, Washington plans to outlaw marijuana businesses in unincorporated areas outside Yakima city. Clark County, Washington is considering banning recreational sales, while Pierce County, just south of Seattle, has said it will ban recreational marijuana businesses from opening. In Colorado, dozens of local governments have banned marijuana commerce, including Colorado Springs, the state’s second-largest city.

In California, where medical marijuana is legal, a state appeals court ruled last year that local governments could ban the growing of medical marijuana. Fresno County became the first county in the state to do so. In Oregon, legislators are considering a measure that would permit municipalities to restrict or ban medical marijuana.