Stressed out from stress testing

Tuesday Oct 26th 1999 by Rich Levin
Share:

Computing in Web time is making IT organizations think and act more like ISVs, driving greater use of automated testing technologies.


How much has the Web changed the face of enterprise application development? Let us count the ways. Start with HTML, pervasive clients, fat servers, distributed architectures, and application servers.

Move to object messaging, server-side components, decoupled frameworks, and enterprise information portals. And let's not forget the year's big buzzes: eXtensible Markup Language (XML), enterprise application integration, electronic data interchange (EDI), and global supply chain optimization.

And that's just scratching the surface. Call it what you will: e-business, e-commerce, e-service, e-tailing, e-tc.; from IT's perspective, it's all e-ssential. Indeed, to say the Web has redefined IT, and done so overnight, would not be an understatement.

Yet for all its impact, the Web's most fundamental change to the fabric and essence of enterprise development remains largely unnoticed. That change is the transformation of IT from a plodding corporate function to a fast running independent software vendor--an ISV.

"The moment you put a dynamic application on the Net, you've shifted from an IT organization that builds solutions for internal consumption, to a company that develops software for external consumers," says Larry Freed, director of the e-commerce practice at Compuware Corp., in Farmington Hills, Mich.

It's a shift that also requires adjusting application development priorities. Technologies many IT organizations have long ignored, such as automated GUI, function, and load-testing tools, suddenly acquire strategic importance.

The reason: When an application system is pointed at the Web, the world is your enterprise stage. Bugs, slow performance, or collapsing under heavy Web loads can mean the difference between brilliant success and bankrupt e-business efforts, Freed says.

Walk like an ISV

Growing numbers of IT leaders, consultants, and vendors agree with Freed's assessment: IT is taking on the attributes of ISVs. To succeed today takes more than migrating sales, marketing, purchasing, and customer services to the Net; and more than adopting new architectures and platforms to keep pace technologically.

It means, experts say, thinking and acting like an ISV, and adopting proven practices and development methodologies that have long separated successful vendors of shrink-wrapped apps from abject market failures.

In fact, IT organizations that ignore the best practices employed by leading ISVs do so at their own peril. Consider this stunning prediction issued by analysts from the Gartner Group, of Stamford, Conn., in September 1999: 75% of e-business efforts will fail. This is due in large part to lack of adherence to best development practices and methodologies.

"You need to look at some of the best development practices that [ISVs] have used for years to be sure their products make the grade," says Mickey Lutz, VP of IT at PHH Vehicle Management Services, in Hunt Valley, Md. "For most IT [shops], that usually means one thing: They have to get serious about software testing for the first time."

Lutz should know. As head of the information technology arm of Avis Rent-A-Car, his PHH group was charged with rapidly evolving Avis' UNIX-based client/server environment to the latest bleeding-edge distributed architecture.

The new requirements called for transforming Avis' legacy enterprise architecture from one that served a handful of internal agents, to a publicly accessible fleet management system that thousands of Avis clients could access live, direct from their desktop PCs.

Ambitious from the start, Avis aimed to do more than just surface marketing and reporting schemes on the Web. It would touch literally every customer, every driver, and every one of the 350,000 vehicles in the company's corporate rental fleet.

The first application targeted: A huge vehicle fleet maintenance solution, where clients could go online and manage vehicle histories, repair costs and operational expenses, driver profiles, and safety training reports, as well as analyze accident records.

It was a high-velocity U-turn for Avis. The company historically relied on its customer call centers to support client queries by telephone, with monthly reports generated by computer and delivered to fleet managers by snail mail.

"When we started this effort, we were a principal vendor, but not a principal app on the fleet manager's desktop," Lutz says. "Today when they manage their fleet, we are the principle application they use. We're no different from Microsoft [Corp.] or Sun [Microsystems Inc.] in that regard. We have become a mission-critical software provider."


Critical mission, critical testing

Lutz's team decided the shift from mission-critical vendor to mission-critical ISV called for mission-critical testing. It was a fortuitous call. It turns out that, had the company not applied automated stress-testing technologies, the system would have collapsed on its first day online.

"The application just wouldn't work under load," Lutz recounts. "It ran great with a small team of QA testers exercising it, but when we applied the load-testing software, it wouldn't work."

Using the LoadRunner load-testing software from Mercury Interactive Corp. of Sunnyvale, Calif., Lutz's team was able to simulate 10,000 concurrent users banging away on the app. The problem was traced to a bug in the ColdFusion app server from Allaire Corp. of Cambridge, Mass.

After Allaire issued a patch, the system was again subjected to a round of load testing. This time, it passed. As the development process ensued, Lutz's team brought more customer data online, eventually facing Avis' entire 600 gigabyte data warehouse to the Web.

Each step of the way, the team repeatedly threw app modules into a load-testing pressure cooker. Lutz says that, at the time, without Mercury's LoadRunner product, this extreme level of load testing would have been beyond the realm of possibility.

That's because most load-testing products required pools of PCs. The server-based LoadRunner required only one central server.

"Typically with client/server or Web stress testing, you have to drive it from multiple PCs," Lutz explains. "I would have had to commandeer entire buildings of PCs. It would have been impossible to test the kinds of numbers we're talking about."

In terms of predictability, the testing results have been "right on," Lutz says. The Avis site zoomed to an average of 90,000 hits per day soon after it was deployed, with a one-day peak of 141,000.

"We've had no problems due to load," Lutz says. Now, as the site gets set to scale again with a new development phase--opening the app up to more online customers, i.e., "scaling up"--Avis' PHH group has purchased an additional LoadRunner license to enable scalability testing beyond 30,000 simulated users.

"No architecture can scale infinitely," says Lutz. "Every time you add or change something, you have to test. We want to take it up further, and stay ahead of the curve as far as numbers of users. Load testing lets us do that."

On the razor's edge

Staying ahead of the curve means relentlessly updating, upgrading, revving, improving, adding features, embracing trends, and technologically innovating within the context of the software development process, under extreme market pressures.

It's the hallmark of any successful ISV that needs to ship shrink-wrapped product to a fickle enterprise marketplace; the aggressive release mentality one would expect to find at Microsoft, Sun, or Red Hat Inc.

Not the kind of thinking normally associated with IT organizations, which historically deal with internally driven business requirements that take months to refine, development cycles that are measured in years, and application systems that are built to span decades.

Examine the cycle time of any e-business shop, though, and you'll find development lifecycles pegged to monthly, weekly, daily, even hourly release builds.

"IT used to operate under the notion that you write perfect requirements, and everything flows from there," says Sam Guckenheimer, the Lexington, Mass.-based senior director of automated testing products for Rational Software Corp., in Cupertino, Calif. "But the Web is iterative. The requirements change daily. We've seen some dot-com organizations with six-hour release cycles."

E-business developers agree, and add that compressed application lifecycles aren't the only challenge in maintaining high-quality customer-facing e-apps. The architecture of Web-based systems is inherently complex, heterogeneous, and fragile.

"We might do five releases in three months," says William Flow, software quality manager at Frontier Corp., in Rochester, N.Y. "And it's not just the Web client we're revving. We have to test all the stuff--integration, databases, other Web apps we hit, e-mail systems--it can be hundreds of things. It's become impossible to do manually."

Frontier personifies the diverse community of platforms many enterprise organizations struggle to integrate as they fight their way to the Web. The company's architecture is a mix of mainframes, with Solaris UNIX and Linux IP servers running on Intel, SPARC, and UltraSPARC machines.


Automating quality

Born over 100 years ago as Rochester Telephone, a small local telco, the company started buying up small Baby Bells after the AT&T breakup. A long string of acquisitions later, Frontier emerged as the country's #5 domestic long-distance carrier.

Under the leadership of CEO Joe Clayton, Frontier's IT organization has, for the past two years, been charged with aggressively moving the firm's entire computing infrastructure to the Web. The initiative is dubbed TMN, for Telco Management Network.

The e-business programming team at Frontier is turning to automated testing technologies to maintain the highest service levels of the firm's customer-facing apps. The reason: With literally all of the company's systems headed for the Web, a failure of any company system means wasting money.

"There was a time when application downtime didn't directly impact revenue," Flow says. "Today the revenue for our company is based on these Web-based apps. If any aspect is down or not doing its job properly, I'm losing revenue."

The laundry list of Web apps under development at Frontier runs the gamut from order-entry systems, to inventory control, to customer care and billing, and everything in between. To cope with the varied testing requirements demanded by these increasingly interdependent Web systems, Flow says he's had to redefine the role of QA engineer.

"My QA engineers have a dual role," Flow explains. "They play unit testing, and they play integration QA." Flow says he integrates QA engineers directly into the development process from day one. This is so they can understand the user requirements and application specs, and ensure the code delivers.

Once the development team declares a unit "code complete," Flow has his QA testers switch gears. "When [the developers] say something's complete, my QA engineer has to change his hat from a unit tester to an integration tester. That's where the automated tools come in."

Multiple points of failure

For example, Frontier's Inventory Management System (IMS) is the needle's eye through which five other enterprise Web systems are threaded. Virtually every conceivable interaction between these business-critical, codependent application systems must be rigorously tested.

IMS manages the company's total inventory and, as such, is depended upon by virtually all the other major application systems. There are five different Web-based apps that rely on it, including the company's product configurator, order entry system, work-flow engines, billing system, and more.

"We used to just test the GUI," Flow says. "Does the app ask the right questions, do the forms work, is the data saved, and so on. But now we have to make sure every app hits IMS, and that the data interacts properly with other apps in the dependency chain. This is where integration testing takes over, and why we had to find an automated tool."

Because the complexity of the integration testing was beyond human means, Flow's group turned to automation; specifically, Compuware's QA Director. The product allows multiple applications and databases to be scripted and executed simultaneously--a key feature for integration testing.

Flow's team uses QA Director to hit upon all the applications and their databases and to generate reports that flag errors in system interactions. "Application A touches application B, and B hits C and D, while E might hit A," Flow says. "QA Director can actually manage this kind of elaborate test."

Now, whenever Frontier prepares to issue a new release, it is first subjected to an entire integration regression test suite, managed by scripts running under QA Director. This ensures previously tested functionality is unchanged, and validates the accuracy of new features.

Risk avoidance

Flow says that, without the availability of Web-savvy integration testing tools, Frontier's entire application portfolio would be at risk. "If we didn't have these automated tools, we simply couldn't do the testing," he says. "We'd be in a world of hurt right now."

Certainly automated testing tools can ease the pain of integration testing and help ensure a site's ability to withstand heavy user loads, the likes of which no legacy IT app has ever been asked to sustain.

But not one of the automated testing tools available today can replace the need to beta test, using qualified users culled from the application's target audience. It's the only engineering process known that can isolate bad user interfaces.

It's also a process that ISVs live and die by, and that IT has historically sworn off. But that too is changing as IT organizations rise to meet the challenges, and reap the rewards, of today's wired world.

"As a [testing tool] vendor, I hate to say that our tools can't perform a certain function, but the truth is, usability testing is the one thing no automated tool can do," says Diane Hagglund, senior manager for e-business product marketing at Mercury Interactive.

Hagglund says usability testing might never be automated, because it has to do with responding to human emotions--something that has yet to be computerized. "We're seeing more and more traditional IT shops doing what ISVs would call beta testing, under the guise of usability testing," she says.

Banking on the beta

That's exactly what's happening at Acentris Wireless Communications, a telco services reseller in Seattle. There the beta-test process has been integrated into the overall development lifecycle, with a core group of developers, internal users, and customers comprising Acentris' beta test team.

The company recently migrated from its legacy VB4 client/server system to a fully distributed platform. The new system is built in VB6 and leverages several beta technologies itself, including a COM+ framework and Windows 2000 Beta 3 RC1 servers.

"We prototyped the Web UI first, and sent it out to a small group of customers and internal users for beta testing," says Acentris VP Darren Lang. "That gave us a huge head start, because we were able to fine-tune the user experience and hand the UI off to the programmers early in the development process."

Acentris' development team was then free to focus on the migration's nuts and bolts, and use automated tools to stress and regression test the application architecture, knowing usability was already in hand.

"The reputation of the IT department no longer rides on how well they manage the printers, back up the servers, or get a new PC on your desk," says Michael Marquardt, president of Internet Operations Center Inc., an e-commerce application hosting company in Southfield, Mich. "It's now the software development arm of the business, and that means we need to think and act more like ISVs, and less like islands of technology." IJ

About the author:

Rich Levin covers IT for CBS Radio and the Coast to Coast Radio Network. He can be reached at RBLevin@RBLevin.net, hit his Web site at http://www.RBLevin.net, or visit his BBS at http://www.RBLevin.net/BBS.

Lessons learned about automated testing technologies
  • Think like an ISV. Recognize that development in the e-business age requires IT organizations to conceptualize systems as commercial applications. To maximize application system quality, adopt proven best practices long used by successful ISVs, such as heavy use of automated testing tools, technologies, and development methodologies.

  • Test early and often. Integrate your QA testing team into the development process from day one. The more the testers know about the application requirements, coding, and resulting app, the better they'll be able to devise test scenarios and regression scripts.

  • Prototype the user interface first. Send the resulting UI demo out to customers, partners, and other users for early feedback. By defining the UI early, the IT team eliminates the possibility of show-stopping usability bugs late in the development lifecycle.

  • Don't overlook load testing. Applications that perform well under low to moderate loads can behave unpredictably, or even implode, when stress scales up. Use an automated load-testing tool to stress application architectures two or three times beyond what you expect the actual user load to be.

  • Test the application's technology as well as its logic. Your application code might be robust, but a single bug in your vendor's app server or OS can bring your organization to its knees.

  • Build a multidisciplinary QA team. Software quality assurance today involves far more than GUI regression and load testing. Web applications are inherently complex, leveraging multiple new technologies and heterogeneous legacy systems. QA teams need to test integration points, database integrity, object messages and communications, and more.

  • Evaluate all the major test tool product lines. Automated testing tools have come a long way in a short time. Most vendors have completely updated their offerings to address the requirements of e-business development efforts.

  • Have contingency plans. Even applications subjected to the most rigorous QA processes can fail. Be sure your IT team is prepared for a major system failure, and have a disaster plan in place to bring systems back up as quickly as possible. Test the contingency plans often with drills.

A glossary of Web terminology
  • Application servers: The software that serves up an application in a three-tier distributed system. Essentially the runtime platform for distributed apps, the app server lives on the "application tier" (the other two tiers being the database and the client-side presentation). First and foremost, the app server executes source code and provides traditional middleware services such as TP monitoring; pooling of connections, objects, and data. They may also provide or otherwise support fault tolerance through clustering, failover, etc.
  • Decoupled frameworks: An application architecture built with reusable, fully encapsulated components. These components can be snapped in or out at will, making upgrades and maintenance far easier compared to traditional monolithic applications, where much of the code is tightly integrated and interdependent.
  • Distributed architectures: Application systems that run across multiple computers, also known as "tiers." Tier 1 is the presentation tier, most often a Web browser or a desktop PC. Tier 2 is the application tier, sometimes referred to as the middle, business logic, business rules, or business components tier. Tier 3 is the database. While the architecture consists of three virtual tiers, in reality, the actual implementation of a distributed "three-tier" architecture might comprise hundreds of application servers, dozens of database servers, and millions of clients.
  • Electronic Data Interchange (EDI): An industry-standard means of formatting a data file so that it can be shared among multiple computer systems.
  • Enterprise application integration: The process of tying multiple enterprise software programs, systems, or databases together, so they can seamlessly share data or objects.
  • Enterprise information portals: A Web site that provides enterprise knowledge workers with an information "dashboard" as a home page. For example, the page might include the company's latest stock price; human resources links; a menu of the company's master applications for sales, accounting, customer service, etc.; a company address and telephone book; a company directory; maps to various office locations; the latest company news, and so on. It's an employee's one-stop-shop for company news, information, communication, and software applications.
  • eXtensible Markup Language (XML): A "markup" language that allows programmers to define ways to share information between applications. XML is to data what HTML (HyperText Markup Language) is to documents. Just as HTML defined a standard way for documents to be exchanged through Web browsers, XML allows IT shops to define how enterprise data is shared among applications and across platforms.
  • Fat servers: Servers that house all or most of the database and app logic, leaving only or mostly the user interface logic on the thin client.
  • Global supply chain optimization: Improving the efficiency of an organization's end-to-end supply chain through application of software algorithms.
  • HyperText Markup Language (HTML): The tag-based programming language of the World Wide Web. HTML defines how documents are formatted and displayed. It is interpreted by a Web browser at run time, which converts the text-based HTML document into the lovely display you're looking at now.
  • Independent Software Vendor (ISV): A company that designs, develops, markets, and supports software products. Microsoft is an ISV.
  • Integration tester: A member of an application development team who validates that two or more software components are communicating properly and without error.
  • Object messaging: Using software objects or components to communicate inside of or among programs, systems, or processes.
  • Pervasive clients: What some believe is the next great era of computing, where computer clients will be everywhere, in many shapes, sizes, colors, and form factors. This includes wearable computers (such as Star Trek style communicators), Internet appliances, palm-sized devices, smart cellular telephones, and interactive TVs, in addition to traditional desktop, laptop, and server PCs.
  • Server-side components: Software objects that reside exclusively on a server.
  • Software algorithms: A series of software commands that, when executed in sequence, solve a problem and produce a result. Algorithms may contain other algorithms. In fact, a complete program is a collection of algorithms, and itself can be considered a large algorithm.
  • Unit tester: A member of an application development team who validates that software components are performing according to spec, behaving properly, and executing without error.

Share:
Mobile Site | Full Site
Copyright 2017 © QuinStreet Inc. All Rights Reserved