Exclusive: Anatomy Of A Brokerage IT Meltdown

The network slowdown was one of the first clues that something was amiss at GunnAllen Financial, a now defunct broker-dealer whose IT problems were only a symptom of widespread mismanagement and deeper misconduct at the firm.

It was the spring of 2005. Over a period of roughly seven business days, traffic had slowed to a crawl at the Tampa, Fla.-based firm, which had outsourced its IT department to The Revere Group. GunnAllen's acting CIO, a Revere Group partner, asked a member of the IT team to investigate.

Dan Saccavino, a former Revere Group employee who at the time served at GunnAllen as the IT manager in charge of the help desk, laptops, and desktops, says he and another network engineer eventually pinpointed the cause of the slowdown: A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem. As a result, none of the company's trades, emails, or phone calls were being archived, in violation of Securities and Exchange Commission regulations.

Despite the fact that at least five people at The Revere Group knew about the engineer's action, it's unclear whether it was reported at the time to GunnAllen or regulators. The SEC didn't reference the incident in a subsequent announcement about a settlement with GunnAllen for unrelated privacy and data security violations, and interviews with former Revere Group employees reveal that regulators may have known about only a fraction of the data security failures at the firm.

What follows is a chronicle of one firm's myriad IT and other missteps over a period of at least four years, as related by former employees and various official documents. It's a cautionary tale of what happens when a company tosses all IT responsibility over a wall and rarely peeks back. It also reveals what happens when an IT outsourcing vendor gets in over its head, and it points to the failures of regulators to identify and clean up a corporate mess on a grand scale.

While these missteps go back as far as seven years, they have continuing relevance today in the context of how businesses oversee outsourcing, information security, regulatory, and employee matters.

Rogue Home Router

Why would a network engineer route all of his employer's traffic through his home RoadRunner cable modem? "You can direct where your traffic is going, and we found out that he'd sent the traffic home to ensure that his routing patterns at work were correct," Saccavino told InformationWeek in a recent interview. But after a week, Saccavino said, he'd forgotten to turn it off.

During the week or so in 2005 that all brokerage traffic was being piped through the home router, the data being sent by GunnAllen's 200 or so employees included bank routing information, account balances, account and social security numbers, and customers' home addresses and driver's license numbers, says Roger Sago, a former Revere Group SQL Server database administrator who was working at the GunnAllen offices at the time. Sago was in charge of defining the data stream to and from Pershing (a unit of Bank of New York Mellon that provides prime brokerage and other services to financial services organizations), which involved thousands of transactions per day. "They transmitted it over the system, online, to the clearinghouse, and if anyone had access to that data ... the ramifications would be huge," Sago said. "There's enough data there that a person could run off and live forever off of what they found."

Sago contacted InformationWeek, saying that the SEC's 2011 settlement announcement relating to prior information security and privacy failures at GunnAllen had failed to mention additional security breaches at the firm. By way of background, Sago filed a civil action--since settled--against The Revere Group and GunnAllen in December 2008, alleging that he'd been unfairly laid off. During the course of that lawsuit, Sago says he learned about the undisclosed breaches from other former employees. Because such security breaches must be reported to the relevant authorities, Sago says he brought them to the attention of The Revere Group and GunnAllen lawyers involved in his case and asked them to respond within 30 days--and preferably, to report the incidents to the relevant authorities.

When neither responded, according to Sago, he says he then alerted the Federal Trade Commission, the Financial Industry Regulatory Authority (FINRA), the SEC, and attorneys general in the 42 states where GunnAllen had conducted business.

Negligence, Incompetence, or Sabotage?

Other former IT staffers, in interviews with InformationWeek, confirmed Sago's assertions, saying the home router incident was indicative of a pattern of either security negligence or incompetence--or possibly sabotage--at GunnAllen, much of which could be traced to the previously mentioned senior network engineer. "The network would get screwy over the weekend ... then [he] would show up, and five minutes in on a Monday, he'd fix the problem," Saccavino said.

It's the opinion of Thomas Lynott, a former senior systems engineer at Revere Group who worked at GunnAllen at the time, that the network engineer's actions suggested a pattern of sabotage. "He'd purposefully break things, then come in in the morning and be the hero," Lynott claimed. "I ended up key-logging all the servers, and I logged him logging in from home at 2:30 in the morning, logging on to BlackBerry servers and breaking them."

After the router incident was brought to the attention of the acting CIO, the offending network engineer received a "written warning and corrective action plan" from his manager, Jerome DiMarzio, the Revere Group IT operations manager assigned to GunnAllen. DiMarzio reported to the acting CIO.

DiMarzio's "confidential memorandum" to the network engineer--dated August 24, 2005, and copied to the acting GunnAllen CIO as well as a Revere Group HR official--outlines episodes involving "insubordination and/or indifference" as well as "dereliction of duty," including failure to obtain formal change control permission for undertaking BlackBerry server maintenance, rebooting the Cisco Call Manager, rebooting the domain controller "without ensuring it had fully recovered," and "changing the default gateway for Exchange."

The memorandum, which DiMarzio confirmed as legitimate, also accused the network engineer of "purposely pulling a cable out of a production environment in order that you would not have to travel to Jacksonville to attend an HP event at the request of the CIO." It also accused him of failing to identify the root cause of problems, including a Microsoft Exchange "data store corruption" and "BlackBerry server MAPI profile loss," and failing to note that logging had been disabled on the company's WatchGuard firewalls.

Officials from The Revere Group, including president and COO Todd Miller and CEO Michael Parks, didn't respond to multiple email requests from InformationWeek to comment on the episodes detailed by the former employees. Our multiple calls to The Revere Group seeking comment also weren't returned.

Keep Quiet

Lynott said he'd been brought in to clean up one case in which the engineer pulled tables from a server to crash it so he could skip an offsite meeting. "But if you pull tables out, you can corrupt data, transactions, all sorts of stuff," Lynott said. "I don't know whether or not it was intentionally turned off or due to incompetence, but the end result was when I brought it to their attention, I was told to turn it on and not tell anybody. We were told on so many occasions not to tell anyone anything."

Another alleged incident at GunnAllen involved a database that had been set to disable email logging, though SEC regulations require broker-dealers to retain copies of all emails for seven years. "For email, they did all the transaction logging, where they'd send all mail incoming and outgoing and they'd log it all offsite," Lynott said. "There was a point in time for probably two months where no one's email was logged. I brought it up in a meeting once and was told to shut up [by the acting CIO]," he said.

"The protocol from the CIO was to shut your mouth, don't say anything, and just brush it under the rug," Lynott said.

The former acting CIO of GunnAllen didn't respond to our requests for comment, sent via LinkedIn. Revere Group officials didn't respond to our request for the former CIO's current contact information.

Microsoft Threatens Shutdown

Not all of GunnAllen's alleged IT missteps had SEC implications. One incident detailed by two former employees involved unpaid Microsoft SQL Server licenses. The Revere Group had been receiving from Microsoft license-renewal bills for GunnAllen, which the acting CIO had ignored, according to the two former employees. Ultimately, Microsoft issued a final warning with a bill for about $20,000, saying it would disable the license at a specified date and time. "It was like an hour or two before the deadline--before Microsoft shuts the SQL servers down, which would bring GunnAllen to its knees," Saccavino recalled.

That ultimatum led one of DiMarzio's employees to contact Microsoft and share GunnAllen's licensing details. But two former employees say the acting CIO at the time, when given the licensing news and bill by the employee, threatened to fire the employee if he spoke of the matter again.

In another incident, DiMarzio relates that an internal network project plan he first delivered in the spring of 2005 to help GunnAllen comply with the Sarbanes-Oxley Act was dismissed by the acting CIO. When, in January 2006, DiMarzio heard that he was to be replaced, he reached out to a GunnAllen executive for perspective on the matter and was allegedly told that the acting CIO's incentive plan included a bonus tied to the longevity of the SOX project. DiMarzio says he also heard at the time that multiple GunnAllen executives, fed up with the pace of the SOX work, were calling for The Revere Group's contract to be canceled.

DiMarzio said he immediately held a conference call with a Revere Group HR official, as well as the acting CIO's boss at The Revere Group, to bring the concerns to their attention. He also requested that the meeting participants not identify him to the acting CIO as the source of the information. But the next week, DiMarzio said, he was called into the acting CIO's office and told to sign a resignation letter in exchange for receiving severance benefits. DiMarzio said he signed the letter and never looked back.

In September 2008, Sago said, he too was dismissed by The Revere Group. Revere ascribed the layoff to declining market conditions, though Sago said that after strong work reviews during his 11 years with the company, he was offered only two weeks of severance pay instead of the standard two weeks per year of employment. Sago rejected the severance offer, filed a civil action for harassment, retaliation, and unfair treatment, and entered into arbitration with his former employer in October 2009, at which time he says he learned about the home router incident, among other IT incidents. Ultimately, Sago and The Revere Group settled out of court. Regulatory Sanctions

GunnAllen's IT failures paralleled larger business problems. Formerly known as Napex Financial Corp., GunnAllen was founded in 1996 by Donald Gunn and Richard "Allen" Frueh. GunnAllen provided a place for brokers and dealers, who must be associated with a FINRA member firm in order to trade, to hang their shingle. But by 2008, senior members of the firm had come under fire for not properly vetting those brokers or monitoring what they were doing in the name of GunnAllen.

Notably, 2008 was when FINRA fined GunnAllen $750,000 for a "trade allocation scheme" conducted by former head trader Alexis J. Rivera. "In 2002 and 2003, the firm, acting through Rivera, engaged in a 'cherry picking' scheme in which Rivera allocated profitable stock trades to his wife's personal account instead of to the accounts of firm customers," according to FINRA. "Rivera garnered improper profits of more than $270,000 through this misconduct, which violated the anti-fraud provisions of the federal securities laws and FINRA rules. Rivera was barred in December 2006."

FINRA accused GunnAllen's investment division of doing business with companies, then failing to inform the broker-dealer's own compliance department that those companies should be placed on a restricted or watch list for investments, as is required by the agency. FINRA also said the brokerage failed to safeguard non-public information in its investment division, meaning that other employees could have profited from insider information. Finally, FINRA accused GunnAllen of "failing to preserve emails and instant messages."

A lack of top-down oversight of Michigan-based GunnAllen broker Frank Bluestein ultimately led to the firm's demise. Bluestein resold investments on behalf of Ed May, who FINRA said "created and marketed unregistered investments" to an estimated 1,500 investors under the company he ran, E-M Management Co., LLC. In 2007, the SEC charged May with fraud, for allegedy running a Ponzi scheme focused on a fictitious Las Vegas casino and fake telecommunications equipment and leasing deals that took in more than $250 million before being discovered and stopped.

In 2009, the SEC also charged Bluestein with fraud. According to the SEC complaint, from 2002 to 2007 Bluestein ran seminars that "lured elderly investors into refinancing the mortgages on their homes," ultimately recruiting about 800 investors and securing $74 million in investments.

In April 2011, May plead guilty to 59 counts of mail fraud, received a 16-year prison sentence, and was ordered to pay a $250,000 fine. Bluestein, however, denied all knowledge of the Ponzi scheme, citing in his defense that he'd personally purchased the investments being sold by May.

Regardless, GunnAllen faced a volley of investor lawsuits after the SEC's 2009 allegations. By March 2010, FINRA found that GunnAllen no longer had sufficient net capital to trade and closed the firm, leading to the layoff of 400 employees. By November 2010, GunnAllen had been liquidated.

First-Ever Standalone SEC Privacy Fine

Although GunnAllen went bankrupt, regulators weren't done with it. The SEC in 2011 accused two former employees--president Frederick O. Kraus and national sales manager David C. Levine--of having inappropriately used GunnAllen customer data, and it fined them each $20,000. The SEC also slammed GunnAllen's former chief compliance officer, Mark A. Ellis, for having failed to put in place or enforce proper policies and procedures for protecting customer information. It fined Ellis $15,000. The agency noted that the broker-dealer's written policies were "vague" and turned out to be little more than a rewording of the actual SEC regulations.

As for the alleged security breaches related to InformationWeek by the former Revere Group employees, a 2010 SEC enforcement action against former GunnAllen executives detailed multiple security incidents, but not the full extent of the breaches alleged by the former employees, which included at least one missing laptop containing financial information. Likewise, the home router incident didn't even come to light until 2009, one year after FINRA fined GunnAllen.

New SEC Violations Emerge

In June 2011, Sago detailed the additional security violations in a six-page letter to the SEC's Miami office, which had conducted the GunnAllen investigation. The agency's associate director of enforcement in Miami, who was in charge of the investigation, didn't respond to multiple calls and emails seeking comment on Sago's allegations, whether the investigation was still open, or whether the additional revelations might lead to any new fines or sanctions against current or former employees of GunnAllen or The Revere Group. A spokeswoman for the SEC, reached by phone, declined to comment on any of those questions. In the bigger picture, it's unclear where the SEC was during all of this activity. "How is it that GunnAllen was an examined entity and they had no security policy?" said independent privacy expert Andrew M. Smith, an attorney at Morrison & Foerster. "Say you're 25 years old, recently graduated college, you're an SEC inspector, what's the first thing you're going to do? You're going to ask for their policies and procedures, and when you see that it takes up less than a quarter of a page, there's going to be something wrong."

Of course, that perspective assumes that the SEC or FINRA had in fact audited GunnAllen's compliance. "Is it possible that they never examined this broker-dealer? If so, that's fair enough," Smith says. In fact, it's not clear if FINRA or the SEC ever audited GunnAllen's policies before they began their relevant enforcement actions, or whether the additional security violation revelations detailed by Sago in mid-2011 might lead the agencies to reopen their investigation.

Officials at both FINRA and the SEC declined to comment on any examinations or audits their agencies may have conducted of GunnAllen. But FINRA's publicly accessible records for GunnAllen make no mention of the agency having audited or examined the company before evidence of the Ponzi scheme emerged.

What could have been done to help the SEC spot brokerages with poor IT policies? In 2008, the agency proposed amendments to Regulation S-P, also known as the Safeguard Rule, to increase customer data protection requirements for the businesses it regulates. According to Chris Wolf, an attorney who directs law firm Hogan Lovells' privacy and information management practice, these include requiring "a written security program, identification of specific employees to run it, identification of documentation for reasonably foreseeable security risks, as well as implementation of safeguards for managing those risks, as well as training, oversight, and so on, including for providers." Wolf added, "It would also have a data breach notification obligation, which currently does not exist."

But those proposed amendments have remained stalled since they were first proposed in March 2008. An SEC spokeswoman declined to comment on the status of the proposed Reg S-P amendments, or whether the agency is still backing them.

Life After GunnAllen

Knowing what they now know, would the Revere Group IT employees who worked at GunnAllen have done anything differently? "Things probably should have been told directly to GunnAllen, but we were in such fear of keeping our jobs," Lynott said. "Looking back and thinking back now, I probably would have gone back and told the GunnAllen people. But they may already have known."

Ultimately, Lynott said, he quit The Revere Group. "I got to the point where I morally couldn't go to work anymore," he said. One week after he left, he heard that the network engineer who'd allegedly sabotaged the IT systems was fired.

Saccavino, meanwhile, said he suspects GunnAllen had no idea what was happening in the IT department. "They weren't told the whole truth, and I don't think they were told even part of the truth," he said. "Shame on them for not having a check and balance in place, but you can't blame them for being the victim."

Smith, the privacy expert, offered four takeaways for any company that outsources its IT department: "One, you need to do your due diligence up front so you know that your service provider can keep this safe. Two, you need to have contractual obligations that allow you to keep this data safe, and audit that. Three, monitor so you know it's safe. And four, if there's unauthorized access, have your service provider notify you promptly."

Benchmarking normal activity and then monitoring for users who stray from that norm is an essential strategy for getting ahead of potential data and system breaches. But choosing the right tools is only part of the effort. Without sufficient training, efficient deployment and a good response plan, attackers could gain the upper hand. Download our Fundamentals Of User Activity Monitoring report. (Free registration required.)