It turns out writing “URGENT” or “Critical!!!!” in a help desk request doesn’t actually help — at least not at the National Oceanic and Atmospheric Administration, where even some of the most desperate pleas for aid with a faulty new finance system were labeled “low priority.”
Despite Commerce Secretary Gina Raimondo’s assurance to Congress in May that “the system is now working,” the parent agency of the National Weather Service continued to be plagued by an inability to pay bills, inaccurate alerts, and incorrect information, according to a database of 17,000 employee complaints obtained by Bloomberg Government.
The complaints, filed from the system’s launch in October 2023 through June 2024, are speckled with all caps and exclamation points, revealing the breadth of contracts affected and the outrage among employees tasked with tracking the agency’s finances.
The $341 million Business Applications Solution system, built by Accenture Federal Services, caused NOAA to not pay the electric bills at some of its facilities, which could have endangered lives if not for a stretch of good weather in tornado-prone areas. It has also stalled employee travel reimbursements, prompting a union grievance. Further internal agency documents obtained by Bloomberg Government show that officials knew many components of the BAS weren’t working before they launched the system, and that employees told leaders in a survey that the new software is worse than the old patchwork system.
Commerce Department officials have only reluctantly publicly acknowledged the problems. Pressed by senators about delays in distributing disaster aid for fisheries in their states, Raimondo said on May 15 that while officials were “not totally out of the BAS woods,” the “software is working.”
In June, employees flagged 1,438 incidents, an improvement from the peak of the chaos in November 2023, when employees submitted 2,749 incidents. But there were still 1,156 unresolved tickets as of July 1, according to the document. Of those, 1,099 were marked “low priority.” Some complaints had languished for months, including 381 that were opened before April. A total of 1,809 employees filed complaints or questions from October through June.
“BAS is still not fixed, and it’s coming up on a year,” said JoAnn Becker, president of the National Weather Service’s employee union.
The documents highlight a combination of technical and human errors that have plagued NOAA since October 2023. Server outages, mismatched data, and confused employees all caused hold-ups in conducting the everyday business of government. A spokesman for NOAA declined to answer questions on the documents, the problems with the BAS system, and the effects on employee morale. Commerce and Accenture spokespeople also declined to comment.
Missing Invoices, Overpayments
Over the nine-month period of complaints recorded in the document, employees reported that Verizon invoices were missing. AT&T invoices were duplicated. A Lockheed Martin subsidiary was mislabeled on a contract to provide tools to measure ocean temperature. “Comms and utility vendors” were “getting invoices of varying amounts.” A finance official struggled to “pay the electric bill for our National Weather Service Annex building.”
There were unapproved invoices for a janitorial company and an overpayment for the installation of diesel engines on boats. A software company couldn’t enter invoices into the system. Money was missing when it came time to pay a Buffalo, N.Y., snowplow company. The user ID for the Silver Spring, Md., DoubleTree Hilton’s LLC expired prematurely.
“****URGENT****Invoice# 13856704 is not being picked up in the payment runs,” one employee wrote on May 22. “This invoice needs to be paid ASAP.”
That note was labeled a low priority and had yet to get a response as of July 1.
Among unfinished requests listed in the July 1 document, plenty include pleas for urgent action, which were usually labeled low priority. Some detailed pressing problems, while others simply highlighted employees’ exasperation with the finance system.
“URGENT: Validation error of insufficent funds for project code 1402BFR3VR in PRISM. Helpdesk said to submit to EBS.Budget EDW report shows $120k available.”
“Critical!!!! Wrong ASAP ID showing in ASAP Extension for FED24DEN0G0202 (PO 133090442)”
“URGENT! Invoice Stuck in EBS - Invoice # 13788598-2-V1424009467-1 has been entered in EBS and remains unpaid. Please advise how to fix”
“Please expedite! Please unlock project 1403BR1GWC.”
There may have been some improvement, but the system still doesn’t work properly and employees “see no end in sight,” Becker said in a phone interview. At least three employees are still waiting on travel reimbursements that have been delayed, she said.
National Weather Service Director Ken Graham has conceded that the financial chaos has hurt morale at an agency that’s suffered years of burnout and attrition. The National Weather Service had 16% fewer non-management employees in May 2024 than it did in September 2010, according to figures tracked by its union.
“It’s incredibly challenging right now,” Graham said during a June town hall discussion with employees. “But I want to thank the team of admin folks across this country. I know it’s just been hard. It’s added to the burnout.”
1.63 Out Of 5
A survey of NOAA employees who use the BAS finance system garnered overwhelmingly negative results. Asked to rate their satisfaction with the system between 1, for very dissatisfied, and 5, for very satisfied, 546 employees gave an average score of 1.63, agency officials said in a July 3 internal newsletter obtained by Bloomberg Government.
While the BAS system is supposed to give employees a new, centralized finance system with data on invoices, property management, and more, most employees said the old system was better. Asked to compare BAS to the patchwork of legacy systems on a 1 (much worse) to 5 (much better) scale, they gave an average answer of 1.55.
Many of the problems shouldn’t have surprised NOAA officials. In a report created a month before the BAS launch, officials warned that many of its features weren’t ready — or even close to ready. Bloomberg Government obtained the Sept. 1, 2023, report in a FOIA request.
Employees compiled a list of 28 aspects of the system, from lease payments to invoicing. Of those 28, 11 were evaluated as being 25% complete and two were labeled 0%. Only one component was deemed completely ready for use.
One of the most troublesome features was one that was supposed to send financial information to Treasury Department officials through a system called GTAS. Treasury officials use GTAS reports — short for Governmentwide Treasury Account Symbol Adjusted Trial Balance System — to analyze agency budget data and send accurate information to the White House and Congress.
Two and a half months before the system launch, the BAS team demonstrated that it could create a GTAS file. But “NOAA did not have access to the files to validate that all required fields were included,” the report says of a July 13, 2023, meeting. NOAA warned that its employees “will need access to running these GTAS Reports” and others “if it owns any portion of the financial reporting process.” The document also said the BAS team didn’t demonstrate any tools “that would assist in the validation of these reports.”
That confusion would haunt NOAA officials for months after the system launched, at least into mid-April 2024. A series of documents detailing problems with the BAS system consistently noted problems with GTAS reports and the broader ability to send financial documents to the Treasury Department, according to the records obtained by FOIA requests.
Other problems emerged before the launch as the BAS team walked NOAA officials through demonstrations. When employees tested out a system to track construction work, agency officials noted that there was no information indicating “Estimated Placed In Service Date” or “Estimated Cost,” the document says. That surprised NOAA officials, who expected that information to be available, the document says.
The BAS team “agreed to redesign the process” to track construction work, “and discussed a theoretical potential solution” in a follow-up discussion in late July, the document says.
“NOAA will need to fully understand and verify the process before Go‐Live as this activity has significant dollar balances and volume of transactions,” officials wrote in the Sept. 1 document, one month before the system launched.
Spinning Wheel
Nine months later, the BAS help desk was still getting more than 1,400 tickets a month.
“Now the system is totally stalled,” reads a June 2024 complaint.
A follow-up complaint offered a sense of frustration, though little detail: “Another crash.”
A third described “30 minutes of watching a spinning wheel.”
“Another crash this morning,” wrote the same employee, who declined to speak to Bloomberg Government about the issue, which the help desk would attribute to intermittent server errors.
“Another crash.”
“Crash again.”
Several more messages came through, all on the same thread, including a description of “almost two hours of trying and watching the spinning wheel.”
Finally, a 10th message simply read, “Tired of this.”
Bloomberg Government analyst Timothy Yeaney contributed to this report. An earlier version of this story corrected the sourcing in the second paragraph.
Near-weekly documents detailing problems with the BAS system, published Jan. 11 through April 16 and obtained via a Freedom of Information Act request, can be accessed here:
To contact the reporter on this story:
To contact the editors responsible for this story:
Learn more about Bloomberg Law or Log In to keep reading:
Learn About Bloomberg Law
AI-powered legal analytics, workflow tools and premium legal & business news.
Already a subscriber?
Log in to keep reading or access research tools.