Attacks/Breaches
10/5/2012
01:17 PM
Connect Directly
RSS
E-Mail
50%
50%

Exclusive: Anatomy Of A Brokerage IT Meltdown

Regulators last year issued the SEC's first-ever privacy fine against broker-dealer GunnAllen for failing to protect customer data. But former IT staffers say regulators didn’t seem to know half of this cautionary tale of outsourcing and oversight gone wrong.

The network slowdown was one of the first clues that something was amiss at GunnAllen Financial, a now defunct broker-dealer whose IT problems were only a symptom of widespread mismanagement and deeper misconduct at the firm.

It was the spring of 2005. Over a period of roughly seven business days, traffic had slowed to a crawl at the Tampa, Fla.-based firm, which had outsourced its IT department to The Revere Group. GunnAllen's acting CIO, a Revere Group partner, asked a member of the IT team to investigate.

Dan Saccavino, a former Revere Group employee who at the time served at GunnAllen as the IT manager in charge of the help desk, laptops, and desktops, says he and another network engineer eventually pinpointed the cause of the slowdown: A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem. As a result, none of the company's trades, emails, or phone calls were being archived, in violation of Securities and Exchange Commission regulations.

Despite the fact that at least five people at The Revere Group knew about the engineer's action, it's unclear whether it was reported at the time to GunnAllen or regulators. The SEC didn't reference the incident in a subsequent announcement about a settlement with GunnAllen for unrelated privacy and data security violations, and interviews with former Revere Group employees reveal that regulators may have known about only a fraction of the data security failures at the firm.

What follows is a chronicle of one firm's myriad IT and other missteps over a period of at least four years, as related by former employees and various official documents. It's a cautionary tale of what happens when a company tosses all IT responsibility over a wall and rarely peeks back. It also reveals what happens when an IT outsourcing vendor gets in over its head, and it points to the failures of regulators to identify and clean up a corporate mess on a grand scale.

While these missteps go back as far as seven years, they have continuing relevance today in the context of how businesses oversee outsourcing, information security, regulatory, and employee matters.

Rogue Home Router

Why would a network engineer route all of his employer's traffic through his home RoadRunner cable modem? "You can direct where your traffic is going, and we found out that he'd sent the traffic home to ensure that his routing patterns at work were correct," Saccavino told InformationWeek in a recent interview. But after a week, Saccavino said, he'd forgotten to turn it off.

During the week or so in 2005 that all brokerage traffic was being piped through the home router, the data being sent by GunnAllen's 200 or so employees included bank routing information, account balances, account and social security numbers, and customers' home addresses and driver's license numbers, says Roger Sago, a former Revere Group SQL Server database administrator who was working at the GunnAllen offices at the time. Sago was in charge of defining the data stream to and from Pershing (a unit of Bank of New York Mellon that provides prime brokerage and other services to financial services organizations), which involved thousands of transactions per day. "They transmitted it over the system, online, to the clearinghouse, and if anyone had access to that data ... the ramifications would be huge," Sago said. "There's enough data there that a person could run off and live forever off of what they found."

Sago contacted InformationWeek, saying that the SEC's 2011 settlement announcement relating to prior information security and privacy failures at GunnAllen had failed to mention additional security breaches at the firm. By way of background, Sago filed a civil action--since settled--against The Revere Group and GunnAllen in December 2008, alleging that he'd been unfairly laid off. During the course of that lawsuit, Sago says he learned about the undisclosed breaches from other former employees. Because such security breaches must be reported to the relevant authorities, Sago says he brought them to the attention of The Revere Group and GunnAllen lawyers involved in his case and asked them to respond within 30 days--and preferably, to report the incidents to the relevant authorities.

When neither responded, according to Sago, he says he then alerted the Federal Trade Commission, the Financial Industry Regulatory Authority (FINRA), the SEC, and attorneys general in the 42 states where GunnAllen had conducted business.

Negligence, Incompetence, or Sabotage?

Other former IT staffers, in interviews with InformationWeek, confirmed Sago's assertions, saying the home router incident was indicative of a pattern of either security negligence or incompetence--or possibly sabotage--at GunnAllen, much of which could be traced to the previously mentioned senior network engineer. "The network would get screwy over the weekend ... then [he] would show up, and five minutes in on a Monday, he'd fix the problem," Saccavino said.

It's the opinion of Thomas Lynott, a former senior systems engineer at Revere Group who worked at GunnAllen at the time, that the network engineer's actions suggested a pattern of sabotage. "He'd purposefully break things, then come in in the morning and be the hero," Lynott claimed. "I ended up key-logging all the servers, and I logged him logging in from home at 2:30 in the morning, logging on to BlackBerry servers and breaking them."

After the router incident was brought to the attention of the acting CIO, the offending network engineer received a "written warning and corrective action plan" from his manager, Jerome DiMarzio, the Revere Group IT operations manager assigned to GunnAllen. DiMarzio reported to the acting CIO.

DiMarzio's "confidential memorandum" to the network engineer--dated August 24, 2005, and copied to the acting GunnAllen CIO as well as a Revere Group HR official--outlines episodes involving "insubordination and/or indifference" as well as "dereliction of duty," including failure to obtain formal change control permission for undertaking BlackBerry server maintenance, rebooting the Cisco Call Manager, rebooting the domain controller "without ensuring it had fully recovered," and "changing the default gateway for Exchange."

The memorandum, which DiMarzio confirmed as legitimate, also accused the network engineer of "purposely pulling a cable out of a production environment in order that you would not have to travel to Jacksonville to attend an HP event at the request of the CIO." It also accused him of failing to identify the root cause of problems, including a Microsoft Exchange "data store corruption" and "BlackBerry server MAPI profile loss," and failing to note that logging had been disabled on the company's WatchGuard firewalls.

Officials from The Revere Group, including president and COO Todd Miller and CEO Michael Parks, didn't respond to multiple email requests from InformationWeek to comment on the episodes detailed by the former employees. Our multiple calls to The Revere Group seeking comment also weren't returned.

Keep Quiet

Lynott said he'd been brought in to clean up one case in which the engineer pulled tables from a server to crash it so he could skip an offsite meeting. "But if you pull tables out, you can corrupt data, transactions, all sorts of stuff," Lynott said. "I don't know whether or not it was intentionally turned off or due to incompetence, but the end result was when I brought it to their attention, I was told to turn it on and not tell anybody. We were told on so many occasions not to tell anyone anything."

Another alleged incident at GunnAllen involved a database that had been set to disable email logging, though SEC regulations require broker-dealers to retain copies of all emails for seven years. "For email, they did all the transaction logging, where they'd send all mail incoming and outgoing and they'd log it all offsite," Lynott said. "There was a point in time for probably two months where no one's email was logged. I brought it up in a meeting once and was told to shut up [by the acting CIO]," he said.

"The protocol from the CIO was to shut your mouth, don't say anything, and just brush it under the rug," Lynott said.

The former acting CIO of GunnAllen didn't respond to our requests for comment, sent via LinkedIn. Revere Group officials didn't respond to our request for the former CIO's current contact information.

Microsoft Threatens Shutdown

Not all of GunnAllen's alleged IT missteps had SEC implications. One incident detailed by two former employees involved unpaid Microsoft SQL Server licenses. The Revere Group had been receiving from Microsoft license-renewal bills for GunnAllen, which the acting CIO had ignored, according to the two former employees. Ultimately, Microsoft issued a final warning with a bill for about $20,000, saying it would disable the license at a specified date and time. "It was like an hour or two before the deadline--before Microsoft shuts the SQL servers down, which would bring GunnAllen to its knees," Saccavino recalled.

That ultimatum led one of DiMarzio's employees to contact Microsoft and share GunnAllen's licensing details. But two former employees say the acting CIO at the time, when given the licensing news and bill by the employee, threatened to fire the employee if he spoke of the matter again.

In another incident, DiMarzio relates that an internal network project plan he first delivered in the spring of 2005 to help GunnAllen comply with the Sarbanes-Oxley Act was dismissed by the acting CIO. When, in January 2006, DiMarzio heard that he was to be replaced, he reached out to a GunnAllen executive for perspective on the matter and was allegedly told that the acting CIO's incentive plan included a bonus tied to the longevity of the SOX project. DiMarzio says he also heard at the time that multiple GunnAllen executives, fed up with the pace of the SOX work, were calling for The Revere Group's contract to be canceled.

DiMarzio said he immediately held a conference call with a Revere Group HR official, as well as the acting CIO's boss at The Revere Group, to bring the concerns to their attention. He also requested that the meeting participants not identify him to the acting CIO as the source of the information. But the next week, DiMarzio said, he was called into the acting CIO's office and told to sign a resignation letter in exchange for receiving severance benefits. DiMarzio said he signed the letter and never looked back.

In September 2008, Sago said, he too was dismissed by The Revere Group. Revere ascribed the layoff to declining market conditions, though Sago said that after strong work reviews during his 11 years with the company, he was offered only two weeks of severance pay instead of the standard two weeks per year of employment. Sago rejected the severance offer, filed a civil action for harassment, retaliation, and unfair treatment, and entered into arbitration with his former employer in October 2009, at which time he says he learned about the home router incident, among other IT incidents. Ultimately, Sago and The Revere Group settled out of court.

Previous
1 of 3
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
RobPreston
50%
50%
RobPreston,
User Rank: Apprentice
1/4/2013 | 2:27:29 PM
re: Exclusive: Anatomy Of A Brokerage IT Meltdown
You're misrepresenting the story, queuester. The story doesn't "tie the demise of GunnAllen to the actions of Revere." In the very first paragraph, the story states that GunnAllen's "IT problems were only a symptom of widespread mismanagement and deeper misconduct at the firm." The facts laid out in the story support that thesis.
queuester
50%
50%
queuester,
User Rank: Apprentice
11/17/2012 | 11:55:43 PM
re: Exclusive: Anatomy Of A Brokerage IT Meltdown
This article at best was a one sided and inaccurate accounting of the IT staff that worked for the company after Revere was shown the door.Trying to tie the demise of GunnAllen to the actions of Revere is the same as trying to tie mother's milk to heroin addiction. There is no doubt that Revere was a drag on GunnAllen and did nothing in the interest of their client. That changed when GAF appointed their own CTO who subsequently rid the company of this incompetent and self serving consultancy. To place so much weight on the quotes of Revere help desk manager whose greatest contribution was writing poems about eating donuts doesn't really seem to be great investigative journalism. I was there as an employee of GAF during the time and worked for the CTO who was a very competent technologist as were many of the people who were kept on. I was also there as we were forced to decommission all of the systems at the behest of FINRA who also displayed an amazing amount of indifference and incompetency during the process. GAF is shut down for a cash reserve deficiency of $100k while the SEC and FINRA allowed MF Global and John Corzine to "misplace" $1.2 billion of investor money. They (the SEC and FINRA) were only successful at dragging the name of one of the only ethical members of the executive management team throught the mud. Maybe a little more research might help next time as the only parties that really were hurt were the customers and that was done by FINRA not the company.
MyW0r1d
50%
50%
MyW0r1d,
User Rank: Apprentice
10/9/2012 | 8:01:34 PM
re: Exclusive: Anatomy Of A Brokerage IT Meltdown
Reads like a company I had experience with and yes, it was a calculated plan on the part of the IT "engineer". Maybe more akin to the doctor/nurse who causes a patient's ills to be seen as the hero for relieving them or a fireman who starts fires to put them out. In the case I was familiar with, the engineer calculated that management would look favorably on him for saving them and unfavorably on anyone who would attack him as being jealous of his expertise rather than invest to independently investigate and perhaps uncover his intentional staging of the cases. He was right. The company fired two of his superiors for harassing the engineer who had "saved" the company.
Recognizing that RevereGroup and GunnAllen are not islands in this respect, there are still more than a few questions surrounding the validity of Sago's accusations (he did work there for what looks to be an extended period before being let go at the height of the 2008 financial crisis). A little vendictiveness? Some of these IT informants seem to share a little responsibility themselves if nothing else for complacency (why didn't DiMarzio take care of RG personnel problems internally without relating full details to GunnAllen?).
Allaun
50%
50%
Allaun,
User Rank: Apprentice
10/9/2012 | 5:34:48 PM
re: Exclusive: Anatomy Of A Brokerage IT Meltdown
The words that come to mind are Malicious, incompetent and hubris. I can understand not liking your job. I can understand having a bad day. But by the great FSM! I have never read about a company that seems so eager to destroy itself. Not even when MCI was around, did I ever see such cavalier disregard for both customer data.
jabadie
50%
50%
jabadie,
User Rank: Apprentice
10/9/2012 | 1:11:02 PM
re: Exclusive: Anatomy Of A Brokerage IT Meltdown
I have always felt embarrassed when I come in on monday and something needs fixing, let alone a trivial 5 minute fix. That guy was a dishonorable idiot.

Interesting article. Read like a horror story.
FireRose
50%
50%
FireRose,
User Rank: Apprentice
10/9/2012 | 3:23:22 AM
re: Exclusive: Anatomy Of A Brokerage IT Meltdown
I worked at GunnAllen in the IT dept for 13 months - 2004/2005 - and one of the "urban legends" from prior to 2004 was that a senior IT programmer was fired for running a porn site on unused space on the web servers. I don't know the truth about this, but it was interesting to hear. I was "downsized" after making the GunnAllen CIO and staff unhappy during the planning of the national convention - no big loss for me, in hindsight!
rlawson346
50%
50%
rlawson346,
User Rank: Apprentice
10/8/2012 | 8:50:42 PM
re: Exclusive: Anatomy Of A Brokerage IT Meltdown
It almost reminds me of the type of behavior seen in arsonists. It's as if the guy enjoys "starting fires", in the IT sense. Also seems like passive-aggressive behavior... but more aggressive than passive. Like he "forgot" to change the settings back.

Really strange. Was it incompetence or sabotage?
Register for Dark Reading Newsletters
White Papers
Flash Poll
Current Issue
Cartoon
Video
Slideshows
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2013-6335
Published: 2014-08-26
The Backup-Archive client in IBM Tivoli Storage Manager (TSM) for Space Management 5.x and 6.x before 6.2.5.3, 6.3.x before 6.3.2, 6.4.x before 6.4.2, and 7.1.x before 7.1.0.3 on Linux and AIX, and 5.x and 6.x before 6.1.5.6 on Solaris and HP-UX, does not preserve file permissions across backup and ...

CVE-2014-0480
Published: 2014-08-26
The core.urlresolvers.reverse function in Django before 1.4.14, 1.5.x before 1.5.9, 1.6.x before 1.6.6, and 1.7 before release candidate 3 does not properly validate URLs, which allows remote attackers to conduct phishing attacks via a // (slash slash) in a URL, which triggers a scheme-relative URL ...

CVE-2014-0481
Published: 2014-08-26
The default configuration for the file upload handling system in Django before 1.4.14, 1.5.x before 1.5.9, 1.6.x before 1.6.6, and 1.7 before release candidate 3 uses a sequential file name generation process when a file with a conflicting name is uploaded, which allows remote attackers to cause a d...

CVE-2014-0482
Published: 2014-08-26
The contrib.auth.middleware.RemoteUserMiddleware middleware in Django before 1.4.14, 1.5.x before 1.5.9, 1.6.x before 1.6.6, and 1.7 before release candidate 3, when using the contrib.auth.backends.RemoteUserBackend backend, allows remote authenticated users to hijack web sessions via vectors relate...

CVE-2014-0483
Published: 2014-08-26
The administrative interface (contrib.admin) in Django before 1.4.14, 1.5.x before 1.5.9, 1.6.x before 1.6.6, and 1.7 before release candidate 3 does not check if a field represents a relationship between models, which allows remote authenticated users to obtain sensitive information via a to_field ...

Best of the Web
Dark Reading Radio
Archived Dark Reading Radio
This episode of Dark Reading Radio looks at infosec security from the big enterprise POV with interviews featuring Ron Plesco, Cyber Investigations, Intelligence & Analytics at KPMG; and Chris Inglis & Chris Bell of Securonix.