Chef ‘Bobby Chez,’ the king of crab cakes, is selling his New Jersey mansion. Asking price: $4.4 million.

Veteran chef Robert Sliwowski has been up and he’s been down, and although he has put his South Jersey mansion on the market, he is certainly not out.

The listing, with an asking price of about $4.4 million, just means that Sliwowski — who calls himself Bobby Chez, same as his mini-chain of food shops — says he is downsizing. He and his wife, Linda, own properties in Arizona and upstate New York.

In 2005, when the couple had the 9,000-square-foot rancher built on four acres in Moorestown, his mother, Phyllis, lived with them. Now, “it’s just me and Linda and it’s time to move on,” said Sliwowski, 71.

Sliwowski brushed off a question about retirement. After making a name in South Jersey with his long-ago destination restaurant Chez Robert — which he lost in 1994 over allegations of violating the Fair Labor Standards Act — he rose to prominence in 1996 with his family crab cake recipe and an idea for takeout food. He now operates four Bobby Chez locations in South Jersey. Its first franchise opened recently in Newtown, Bucks County.

Sliwowski’s house on Bridgeboro Road, with five bedrooms, four full baths, and two half baths, also had its ups and downs. It hosted the couple’s 2007 wedding (the second for both), whose guest list included actor James Gandolfini, with whom Sliwowski owned a racehorse. The pony was named Uncle Junior, after a Sopranos character as well as Sliwowski’s uncle, former Camden Mayor Angelo Errichetti, who was jailed in the Abscam FBI sting. For years, the Sliwowskis hosted a Christmas Eve party at which the highlight was Santa Claus arriving by helicopter.

The manse also attracted negative attention from Moorestown Township, which fined Sliwowski in April 2006 because he moved before the occupancy permit was issued. (Sliwowski explained that he needed a place for his ailing mother, who died that summer.) Sliwowski also faced a decade’s worth of litigation over a pond and a berm that he had installed. A lawsuit claimed that the berm forced storm water onto a neighbor’s farm field and onto a second neighbor’s property; the issue was remediated.

Melissa Young, the couple’s Realtor, describes the house as “truly a work of art.” Sliwowski said he dislikes sharp edges, so every corner and molding is rounded. Its features include a slate roof, an aboveground basement design with surround sound music, radiant heated floors, 10-foot ceilings, custom architectural millwork, five gas/wood fireplaces, and copper gutters.

The Clive Christian kitchen has an eight-burner range, four ovens, three Sub-Zero refrigerators, an oversize walk-in pantry, a chopping block, and an island that Young says “seems to stretch indefinitely.” There’s a handcrafted wooden wet bar complete with wine refrigerator, sink, dishwasher, and two Sub-Zero ice machines and three built-in televisions. There are four bedrooms on the first floor — the main one has what she calls a large spa bathroom retreat with two water closets for privacy; it overlooks a private terrace, landscaped gardens, pond with fountain, and an Olympic-size pool. There’s a fifth bedroom with a full bath upstairs.

“It was just a labor of love and we’ve enjoyed 18 years there,” Sliwowski said.

Robert Sliwowski grew up in Haddon Heights and Cherry Hill. His father, Edward, ran a restaurant in Maple Shade. As a kid, Bobby helped his mother run the hot dog stand at Woodcrest Country Club in Cherry Hill.

Though his upbringing was modest, he never thought small. In his first marriage, he lived on a 103-acre horse farm in Marlton.

The Moorestown house, with its over-the-top decorations, could be described as a residential version of Sliwowski’s former French restaurant, Chez Robert. It opened in 1976 in the Coliseum in Voorhees but moved to Haddon Avenue in Westmont (now the site of Giumarello’s) in 1981. The formal dining room had champagne-colored walls accented by brass twin lamp sconces with pleated shades and champagne-colored tablecloths and napkins. Tables were set with Limoges porcelain, brass candlesticks, and finger bowls. Outside the dining room was a garden and large tanks with tropical fish.

Few restaurateurs might buy two peacocks to stroll their garden for effect, as Sliwowski did in 1983. (This did not end well, incidentally. The birds shrieked at nights, upsetting neighbors. After Sliwowski reported that one bird had been killed — possibly by a neighbor — state officials told him that he would be fined if he didn’t get rid of the remaining bird.)

The Labor Department action forced Sliwowski’s hand in 1993 and 1994. “It was the biggest lawsuit ever against a restaurant — $2.7 million,” Sliwowski said. As he retooled Chez Robert, turning it into a casual restaurant for several months, it closed in late 1994. He lost the business, his house, and his marriage. “I went from driving a brand-new Mercedes to a 10-year-old Taurus with a broken window,” he said.

“I had to start all over and my friend said, ‘What are you going to do now?’ I said, ‘I don’t know.’ He said, ‘Why don’t you sell crab cakes?’ I said, ‘Crab cakes? I’m a master chef. I don’t sell crab cakes. We had them on specials and for parties.’”

“We started from nothing,” he said. “It was very humbling and I’m going around hustling crab cakes and people are laughing at me. But I had to do something. I had to stay afloat. I had to worry about my mother. She was always there for me and she worked hard, too.”

In 1996, they started making crab cakes in her house “and started going from one store to the next. I had no really substantial money, but there was a place near us in Voorhees called Wrestlers Pizza that was available. I went to a loan shark and I borrowed $6,000. He said, ‘Listen. I can give you $6,000. But if you don’t give us $660 a week for 10 weeks, you know what’s going to happen?’ I said, ‘I know, sir.’ Well, fortunately I was able to overcome that.”