Risk
9/3/2013
01:30 PM
Commentary
Commentary
Commentary
50%
50%

Software Patches Eat Government IT's Lunch

The software industry's publish-now, update-later approach exacts a huge toll on government IT leaders like Robert Jack, CIO of the U.S. Marine Corps.

Netscape co-founder and prominent tech investor Marc Andreessen famously noted that "software is eating the world." Unfortunately, it's also eating the lunch of most enterprises, including federal agencies.

For all the talk about wasteful government IT spending, little is said about the costs agencies pay to patch buggy software, a consequence of the industry's predisposition to release their wares now and fix them later. For Robert Jack, CIO of the U.S. Marine Corps, those costs aren't incidental.

"We have roughly 300,000 people, of which a third have day-to-day access to the enterprise network," Jack said at a recent forum on cybersecurity. "I have to defend the network at the desktop or end-user device. I have over 450 registered systems that are regressed to 10 significant versions. When we get a patch from a vendor, we have to go out and test that against all that."

He continued, "Think about the labor hours where I have to touch [and administer patches on] all those devices. And that's just for one patch." Every week, dozens of new vulnerabilities are catalogued by US-CERT, the government's computer emergency readiness team, offering a glimpse of the headaches Jack and CIOs like him face.

Speaking to the software industry at large, Jack said bluntly, "You're killing me."

[ As cloud and mobile proliferates, federal IT leaders should take more data-centric approach to security. Read Secure Data, Not Devices. ]

The staggering cost of software bugs is hard to nail down. However, a Cambridge University study released earlier this year estimated that finding and fixing coding problems costs software makers and the global economy $312 billion a year. That doesn't reflect what customers must also spend to patch and maintain that software on their networks.

The problem, however, goes well beyond the mechanics of software and system maintenance. It also goes to the heart of network security and the growing risks associated with unknown software vulnerabilities, Jack said. Having spent 40 years in charge of command, control, communications, computers and cyber operations for the Air Force, the Defense Department and now the Marine Corps, Jack knows the problems as well as anyone.

Software by its nature is a work in progress. While vendors can't anticipate every problem, some of which are spawned when software interacts with other software on a network, vendors are making too many calculated compromises in order to ram their products and updates into production, Jack said. But worse, they're exposing organizations and their executives to growing liabilities if something goes wrong.

Jack pointed to recent reports, which he didn't specify, indicating that 25% of hospital operating room liability lawsuits are now tied to software coding problems. Lawsuits based on software failures are also becoming a big concern for the auto industry, he said, and the issue has prompted high-level discussions within the Defense Department.

It's only a matter of time before the high-profile enterprises become targets for liability lawyers looking to exploit software mishaps, Jack warned, adding that those in positions of authority ought to consider "looking for some big-time insurance."

Previous
1 of 2
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
WKash
50%
50%
WKash,
User Rank: Apprentice
9/17/2013 | 12:50:08 PM
re: Software Patches Eat Government IT's Lunch
There's an interesting story and commentary on the importance of, and the balance between, software developers who crank out code and product managers who are responsible for the customer experience. See "Why your software development process is broken" at: http://www.informationweek.com...
WKash
50%
50%
WKash,
User Rank: Apprentice
9/17/2013 | 12:38:19 PM
re: Software Patches Eat Government IT's Lunch
This misses the point of what the Marine Corps and other large scale organizations are facing. First, the Marine Corps can't just software A and switch to software B because software A is costing more to update/fix. Second, we're talking about high costs because of the large number of users and devices that exist. The Marine Corps works with the US Navy. Together they have more than 800,000 users on the network. Simple bug fixes and version updates may not be a big deal for an organization of 1000 workers. It's a much different and more expensive matter for large organizations.
blowenth
50%
50%
blowenth,
User Rank: Apprentice
9/7/2013 | 3:04:49 PM
re: Software Patches Eat Government IT's Lunch
I find it interesting when people complain that they are losing
money as a result of applying software fixes.



The simple answer for organizations that are losing money applying
software fixes is for the organizations to discontinue using the
software. Then they would they recover those "losses" -- right?



But, of course, they will nearly always "lose" even more money if
they discontinue using the software. Adoption of nearly all
business software products results in cost reduction of
actions that can already be performed. There is often very quick
adoption of new products that provide significant cost reductions
because the people that purchase the products can get their money
back in less than a year and then go on to get significant cost
reductions year over year. However, people still gripe because
they don't save even more because of product defects.



For example, Larry Ellison of Oracle said that Oracle saved a Billion dollars a year when it
adopted Internet technology products for HR and other internal processes. -- I believe
it. Now Oracle could have saved even more that a Billion dollars
if there were no product defects. Lets suppose Oracle saved
$1,000,000,000 but if there were no product defects they would have
saved $1,010,000,000. Now Larry could say that Oracle lost
$10,000,000 by adoption of Internet technology products --- and it
would be true using the logic of the per "Software Patches Eat
Government IT's Lunch" story.



Software, unlike almost all other products, is peculiar because
customers pay to get fixes to defects in delivered products. Why
is this? Why do customers purchase such products and then have to
pay to have design defects repaired. Lets consider to companies
building the same type of software products.


Lets say Company A gets the product out in 2010 where it is
purchased by the Bank of America. Lets stay that Company A's
product has many software defects costing BoA $10,000,000 during
the next two years. Lets say Bank of America, saves
$500,000,000 per year because of the adoption.

Lets say Company B gets the product out in 2012 at the same
price as Company A and because of the extra two years of
testing, software defects cost $0 per year and the Bank of
America starts saving $500,000,000 per year at that point.
What's the better deal for the Bank of America?

Did Bank of America lose $10,000,000 by adopting Company A's
product or did it lose $990,000,000 by waiting to adopt Company
B's bug free product?



Of course there are tradeoffs here. If the products have too many
bugs then they become unfit for use. But competition typically
sorts this out.



Anyway, the bottom line here is that when you hear about
organizations losing money because of the cost of applying fixes for
software defects, its probably not the case that the organization
lost money by adopting the use of that software. They just want to save even more -- and that is good. However, sometimes it's
good to remind people to consider all the costs and benefits.
WKash
50%
50%
WKash,
User Rank: Apprentice
9/5/2013 | 6:58:06 PM
re: Software Patches Eat Government IT's Lunch
Government gets a lot criticism for its bureaucratic ways, but it does offer a template, courtesy of the National Institute of Standards and Technology, for how software code can be certified to meet certain operating standards. Enterprises would be better served if there were standards software developers could agree to that, as you say, using an automated process, would give software the equivalent of a clean bill of health. That won't address the need for updates, as you say, but it would give customers more confidence in what they're buying. And that hopefully would led to marketplace choices that would reward the more responsible developers.
moarsauce123
50%
50%
moarsauce123,
User Rank: Apprentice
9/5/2013 | 5:04:24 PM
re: Software Patches Eat Government IT's Lunch
A bug is anything that negatively impacts user experience. That means it does not need to be a coding error. The code can be all well, but if using the application is so complicated that it will lead to many mistakes the software is buggy and needs fixing.
A big problem these days are indeed the ridiculously short times to market as well as the 'agile' methods that reward not committing to anything, no planning, no documentation, and rapid releases of something. I think we all would better served by spending a few months up front to design applications and make up our minds as to what we really want, then build that. Sure, flexibility is always needed as some aspects come up while doing the development work, but it just doesn't pan out when doing everything on the fly.
MarciaNWC
50%
50%
MarciaNWC,
User Rank: Apprentice
9/4/2013 | 9:07:41 PM
re: Software Patches Eat Government IT's Lunch
Would be interesting to see those reports Jack cites indicating that 25% of hospital operating room liability
lawsuits are now tied to software coding problems.
Lorna Garey
50%
50%
Lorna Garey,
User Rank: Ninja
9/4/2013 | 2:40:33 PM
re: Software Patches Eat Government IT's Lunch
Excellent analysis. Patches have always been painful, and seem to be getting more so. The "goods vs. service" distinction is a factor those choosing between SaaS and COTS should consider.
cbabcock
50%
50%
cbabcock,
User Rank: Apprentice
9/3/2013 | 7:30:47 PM
re: Software Patches Eat Government IT's Lunch
Software patches are often updates, not bugs in the sense of poor code needing corrections, to bring an application into line with updates elsewhere in itself or to keep it compatible with a wider set of software. We need to get to the point where software is built in such a standard fashion that updating it is a standard, and automated, process. This is where cloud vendors start to draw away in terms of efficiency from the one-of-everything enterprise data center.
Register for Dark Reading Newsletters
White Papers
Cartoon
Current Issue
Dark Reading December Tech Digest
Experts weigh in on the pros and cons of end-user security training.
Flash Poll
Video
Slideshows
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2014-7178
Published: 2014-11-28
Enalean Tuleap before 7.5.99.6 allows remote attackers to execute arbitrary commands via the User-Agent header, which is provided to the passthru PHP function.

CVE-2014-7850
Published: 2014-11-28
Cross-site scripting (XSS) vulnerability in the Web UI in FreeIPA 4.x before 4.1.2 allows remote attackers to inject arbitrary web script or HTML via vectors related to breadcrumb navigation.

CVE-2014-8423
Published: 2014-11-28
Unspecified vulnerability in the management portal in ARRIS VAP2500 before FW08.41 allows remote attackers to execute arbitrary commands via unknown vectors.

CVE-2014-8424
Published: 2014-11-28
ARRIS VAP2500 before FW08.41 does not properly validate passwords, which allows remote attackers to bypass authentication.

CVE-2014-8425
Published: 2014-11-28
The management portal in ARRIS VAP2500 before FW08.41 allows remote attackers to obtain credentials by reading the configuration files.

Best of the Web
Dark Reading Radio
Archived Dark Reading Radio
Now that the holiday season is about to begin both online and in stores, will this be yet another season of nonstop gifting to cybercriminals?