Resonate and Siebel load balancing

Load balancing for web applications is a critical piece of most companies infrastructure, and pretty well established in terms of techniques and products. Load balancing for Enterprise applications such as Siebel is much more of a special science. As this article shows, to monitor areas that actually matter takes quite some effort. Resonate is going away in the newer versions of Siebel, replaced with a Siebel provided load balancer. But Resonate-Siebel is still used in thousands of Siebel sites and needs a further look.

Siebel has generally relied on Resonate Load Balancing software. Resonate is kind of a black box to most of us in the Siebel community. The theory is relatively simple: requests come from a Siebel Client, it is routed to the Web Server (sometimes with HTTP load balancers prior to the web server). Then the request goes to the Siebel Gateway server into Resonate, and Resonate decides which Siebel Application server to route the request to.

Internally, Resonate uses some logic to decide which Siebel Application server to route requests to. Resonate refers to the information it stores about the Siebel application servers and processes as Rules. It stores one Resonate Rule per Siebel Process that is load balanced. For example, a typical Siebel Component, such as eCommObjMgr may have 4 processes , depending on the configuration of the Siebel server.

So, how would your monitor this system? Well, there are a couple of points. One, you want to make sure your Resonate Service is running. That's pretty standard monitoring. Where the real application specific stuff comes is in parsing the Resonate Agent Rules files. What you're basically monitoring for is "stale rules" in Resonate. A stale rule is when Resonate has a process listed in its rules file, but that process doesn't actually exist in the Siebel server. You also want to monitor that your Siebel component is Avalible, and that is not so easy.

So, it becomes kind of complicated to monitor Resonate, and compare the number or rules with the number of Siebel processes. It is hard because there isn't standardized source of information for any of these points important application points. As is endemic in Enterpise software applications, each system stores it's operation information in proprietary silo's, not designed for monitoring. That is where VA2 comes in. Basically, VA2 tracks all the processes and Components for the Siebel servers, and in adition it tracks the number of processes per Siebel component. Here is a screen shot of Siebel Server information stored in VA2. There is a column called Running MTS Procs in the Siebel Server, this tells you how many actual processes are currently running for that Siebel Component. Resonate stores its rules in a format that looks like this. So the challenge for VA2 was to parse and agregrate the number of rules, one per process, and store it in tables that could represent per Siebel Server the number of resonate rules. When said and done, this is VA2 capture of Resonate rules, seen through the MMC.

So, once the data of Resonate is captured, in terms of how many Rules there are, and VA2 already stores information about specific Siebel Servers and how many processes there are, comparing the two together isn't too hard using VA2. It isn't too easy either! The main issue is that Resonate stores it's Siebel Server information with IP addresses, where as Siebel stores it's server information with Siebel server names. On top of that, the components that Resonate stores in it's Agent-Rules file are lower case, whereas Siebel Component names are upper case. However, none of these problems were insurmountable, and after writing some libraries, users of VA2 can easily monitor Resonate to compare in real time the number of Resonate rules with the number of Siebel processes. And of course features such as monitoring the NT Services and Siebel component availiblity comes built in.

Here are some of the methods provide by VA2 to do that. The main one is:
my @listofcomps = ("srbroker","ecommobjmgr");
$resonaterror = $resonate->rulecomparison_nomap($siebsrvr,@listofcomps);

This one will return errors for any components that the number of Siebel process do not match the number of resonate rules for that component in the Siebel enterprise. Running that as a VA2 Analyiss Rule is a usefull tool for monitoring Resonates behaviour. Below are a list of some of the methods that can be used independently to track and monitor Resonate performance.

Resonate based:
totalrules(gateway) #returns total # for rules for gateway
rulesforcomponent(component) #returns number of rules per component accross all resonate servers
rulesforserver(ipaddress) #returns number of rules for ipaddress
rulesforservercomponent(ipaddress,component) #returns number of rules for ip/component combo

Siebel Server based:
activeforappsrvrcomp(siebelserver,component) #returns the active process component for siebelserver
activeprocsforservercomplist(siebelserver,@listofcomponents) # returns active processes for list of components for appserver
activeprocsenterprisecomplist(@listofcomponenents) # returns total number of active processes for all appservers in siebel enterprise for the listofcomponents

So, to make VA2 compatible with Resonate, a custom table was added to store Resonate