The day we found Performance Center's ceiling
One of the biggest engagements of my career. A global bank, WebSphere over IBM AIX, and the front-end load testing that kept the eCommerce platform standing.
The scale was the thing
Two environments, Site A and Site B, and before every single run we went through a setup checklist that took longer than the test itself. CPU counters, memory counters, every server profiled before a single virtual user hit the door. Introscope for monitoring, PsyScope when we needed to dig into the tripas of the JVM.
LoadRunner, Performance Center, and the hardware wall
We ran LoadRunner through Performance Center, and here is the thing nobody tells you: Performance Center has a ceiling, and that ceiling is your hardware. Our load generators were these massive HP ProLiant boxes, and the amount of load we could push was literally bound by how many of them we could stand up. We found that limit the hard way, in the middle of a validation window.
Two findings that still stick with me
First, a mailbox component with a slow memory leak that only surfaced under sustained load. The kind of thing that looks fine for an hour and then eats your heap by hour six.
Second, the AIX security patches. We started seeing tests degrade for no reason we could explain, and it took us a while to realize the OS patching team had rolled security fixes that quietly changed how the servers handled connections. Patches that protect you can also slow you down, and finding that out late is how you learn to ask about OS changes before every run.
Front-end only, and it worked
No back-end testing required on this engagement. We proved the front could take it, and the tests came out strong.