Wealthy buyers are pulling back from some of the most expensive housing markets in the U.S., the latest sign that sky-high prices and fears of a recession are weighing on a key sector of the economy.

Horsham-based Toll Brothers Inc., the nation’s largest publicly traded luxury-home builder, said late Tuesday in announcing its third-quarter earnings that purchase agreements fell 3% from a year earlier, worse than a decline of less than 1% that was expected by a Bloomberg survey of six analysts. The company’s orders in California, home to some of the priciest markets in the country, tumbled 36% from a year earlier.

The shares closed down $1.65 or 4.47% in Wednesday trading to $35.26 after earlier falling as much as 6.3%. It was the worst performance in an S&P index of homebuilders, which rose 0.8%.

Toll’s results underscore a shift taking place in the U.S. housing market. The return of low mortgage rates is heating up competition for starter homes and fueling steep price gains in cities that have long been more affordable. Meanwhile, expensive markets, such as San Jose, Calif., and Seattle, as well as the luxury homes that Toll builds, have seen a drop-off in demand.

Purchases of previously owned U.S. homes, which cost less than half of Toll’s offerings, on average, rose to a five-month high in July, the National Association of Realtors said Wednesday.

Toll’s buyers “are a little more sensitive to what’s going on in the broader economy,” said Drew Reading, an analyst at Bloomberg Intelligence. “They’re paying more attention to the stock market.”

Slowdown signs

Wall Street and Washington have been buzzing this month about the potential for an economic slowdown after U.S. equities suffered one of the deepest sell-offs of the year on Aug. 14 and a key portion of the U.S. Treasury yield curve inverted for the first time in 12 years. New-home sales also were weaker than expected in June.

Homebuilders that cater to entry-level buyers are better positioned to weather shifting demand, Reading said. Shares of D.R. Horton Inc., which focuses on starter homes, have climbed 41% this year through Tuesday, while Toll Brothers was up 12%.

The luxury builder’s biggest challenge may be its concentration in California, where its homes under contract had an average price of $1.74 million in the quarter. Chinese buyers have pulled back there, and the federal tax overhaul limited deductions for property levies and mortgage interest.

As at other homebuilders, Toll’s profits have gotten squeezed by the need to drive sales with buyer incentives. Fewer California deals, with their relatively fat margins, also didn’t help. Toll’s full-year guidance for adjusted gross margin of 23% was slightly lower than the consensus of 23.2%, based on a Bloomberg survey of analysts.

‘Tailwinds’ intact

Toll is off to a good start in August, chief executive officer Doug Yearley said on a call with analysts. The company was able to increase prices in about half its markets, he said.

“With demographics improving, low interest rates, record-low unemployment, continued wage growth, and limited new and resale inventory in many markets, we are optimistic about the opportunities ahead,” Yearley said.

The company said the weakness in California was primarily in the northern part of the state, and that order growth will improve in subsequent quarters because the year-over-year comparisons will be with periods that were softer. The sales slowdown for new homes took hold late last summer.

“It’s no secret that lower prices are a better place to be right now,” Jack Micenko, an analyst with Susquehanna International Group, said after the call. “But I think the numbers this quarter were very much a function of how good things were well into the summer last year, and then they fell off a cliff.”

The company has been looking to scale up its business in other parts of the country and build slightly less expensive homes for wealthy millennials.

“They are diversifying the type of product they’re building, but it’s not a transformation that happens overnight,” said Alex Barron, an analyst with the Housing Research Center in El Paso, Texas. “Toll Brothers is luxury. They’re not going to go from being a Nordstrom to being Walmart.”