SEPTA bus riders were treated this fall to unintentional free rides that potentially numbered in the thousands due to what appears to be a software problem with the Key, the transit agency's smart-card fare system.
Buses reported failures with the onboard equipment that scans and records digital fare payments about 1,500 times from August to Oct. 12, SEPTA data show. While some of those failures may have been the result of problems as simple as a rider not having enough money on a card to pay for a bus trip, SEPTA officials said, a significant number of the failures resulted from card readers freezing. Once that happened, bus drivers were directed to board all Key-carrying passengers for free until the problem was fixed, which in some cases was not possible until after a bus finished its route.
So far, SEPTA doesn't have a count of how many free rides were given.
While determining an exact number is difficult, the problem has cost SEPTA at a time when transit ridership is shrinking. The agency will conduct a cost analysis and may attempt to recoup its lost revenue from Conduent, the company that installed and administers the Key system, said Andrew Busch, a SEPTA spokesperson.
Conduent is a spin-off of Xerox, which began the Key project. As of July, SEPTA had paid $162 million to Xerox and Conduent for their contributions to the smart-fare card. That's about $40 million more than the contract was supposed to cost when it was approved in 2011.
"We continue to work with our client to quickly address and resolve any identified issues," said Neil Franz, a Conduent spokesman. "We are committed to providing the high-quality service that SEPTA and its riders deserve."
Failing to record all riders could also lead to issues with state funding, since ridership counts are one tool used to determine how much money SEPTA receives from Pennsylvania.
SEPTA and Conduent are still trying to figure out exactly what caused the problem. Card readers have frozen on SEPTA's subways and trolley systems, too, but the issue has not been as acute as on its buses, officials said.
The freezes happened most frequently between 7 and 10 a.m., officials said. The problem first cropped up in August, coinciding with a larger-than-usual software upgrade that included security fixes, and continued happening through early October. SEPTA officials identified a "memory leak" in the software that handles transactions.
"A piece of code overloaded the affected servers at certain times and under certain conditions, which caused the validators to be stuck for an extended time with the 'processing' message,' " Busch said.
SEPTA officials say the problem is resolved, but they still aren't sure what the cause was. If the software dump was the catalyst, it could have been a bug in the new software, or a problem in the way the new data interacted with older coding, SEPTA officials said. Another potential factor could have been that buses were turned off while receiving the new software.
To find the exact cause, millions of lines of code are being sifted through, Busch said. SEPTA has 1,400 buses and has processed 195 million trips on Key cards since the system debuted.
A Temple University expert said such issues may be the new normal as SEPTA nears the completion of its transition from older fare systems to the digital fare card.
"It's not unheard of, and we're sort of living in a world where software gets rolled out and then upgraded periodically," said David Schuff, professor of management information systems at Temple University's Fox School of Business. "Each time you change something, you run the risk of breaking something else."
Whatever caused the problem, about 1.4 percent of buses in service in August reported card-reader failures, slightly more than what's typical in a month. September was worse, with 2.4 percent of buses reporting problems, according to data provided by SEPTA. As of Oct. 12, the failure rate this month has been 2.1 percent.
There are about 300,000 Key transactions a work day on SEPTA buses.
It was almost exactly a year ago that a different problem with SEPTA Key caused card validators on buses to fail at a similar rate.
The card failures aren't costing SEPTA a lot, officials say, since the majority of riders use one of the prepaid monthly or weekly fare plans. But according to a recent ridership report, about 21 percent of city transit riders, which includes subway and trolley users along with bus passengers, pay per ride with their Key card or tokens.
The transit agency lost 18 million bus trips between 2016 and 2017. And that recent ridership report recorded July and August's revenue from surface modes of travel was lower than the budget projected.
Schuff noted that virtually all digital systems are now subject to frequent software upgrades and patches. SEPTA does them monthly, officials there said, and Schuff compared what was done to the Key system to a software update for a smartphone. Sometimes, the new software can create glitches.