Zombie Crystal Reports, part 1

I've struggled with Crystal Reports and the PeopleSoft report server over the years. Back when we were on PeopleTools 8.42, it took me several weeks to get them running smoothly. I had to carefully limit the number that would run simultaneously on any server, or they'd lock up on us. It took a lot of experimentation to find the right limit that would minimize zombie reports, yet still give us the throughput we required. I ended up adding another server just to handle the load.

Another thing I did, which was an ugly but effective hack, was to run a VBscript that would find and kill any Crystal that had been running for more than an hour. I know nothing about VBScript, but I found a generic process-killing script somewhere, and modified it for my purposes. I scheduled it to run periodically on each report server, and it did what it was supposed to do.

On our last PeopleSoft upgrade, we deployed all new Windows servers, including the report servers. I didn't put those VBScripts on the new servers, and so far on PeopleTools 8.49, I haven't seen many Crystal zombies.

On Tuesday of this week, I found a zombie that had been running over the three-day holiday weekend. I'd received complaints about Crystals sitting in queue for a long time. Part if it was just a tremendous number of report requests, but that one stuck report was apparently taking up a slot in the report server queue, so we were running one less report at a time than we normally do.

The PeopleSoft process request table (PSPRCSRQST) and the Oracle V$SESSION table showed me the thing had been running for a few days, but the Windows task manager showed three pscrrun process. The task manager doesn't show a start time. Which was the zombie?

Well, one clue was that one of them was using a crazy amount of memory -- I believe it was close to 1 GB. I was fairly sure that was the culprit, but to be extra sure, I put that report server in "suspend" mode, which completes any current processes, but doesn't start any new ones. Sure enough, the other two Crystals finished within a minute. I killed the zombie using the Windows task manager, and un-suspended the report server.

Had I scheduled that VBScript, that Crystal would have been long gone. I considered doing just that, but it seemed like a good opportunity to learn a little PowerShell.

Next, in part 2: finding long-running reports.