We have a stored procedure which calculates daily success rate (Jobs) for 5000 clients. The report runs at 8am and 4pm. The 8am report fails every alternate day. However, the 4pm report with same sql stored procedure runs successfully. I have taken the log when the 8am report failed as shown below. All best practices have been followed as suggested by Symantec (adjusting heap, cache, disk defrag etc). My question is: If the 4pm report runs succesfully , then why not at 8am? Is it something to do with long running jobs, clients at 8am?
LOG 147
Apr 11, 2013 10:22:58 AM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/15 config=null
Apr 11, 2013 10:22:58 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 12362 ms
Error fetching domain list
Error fetching domain list
Error fetching domain list
04/11/2013, 10:45:50 :: CServiceWrapper[Exit]::ServiceHandlerApr 11, 2013 10:50:53 AM org.apache.jasper.runtime.JspFactoryImpl internalGetPageContext
SEVERE: Exception initializing page context
java.lang.IllegalStateException: Cannot create a session after the response has been committed
at org.apache.catalina.connector.Request.doGetSession(Request.java:2437)
at org.apache.catalina.connector.Request.getSession(Request.java:2157)
LOG 148
156.id AND a156.masterServerId = nb_JobAttemptArchive.masterserverId and a156.clientName = nb_JobAttemptArchive.clientName and a156.id = nb_JobAttemptArchive.jobId inner join domain_Policy domain_Policy on domain_Policy.masterServerId = domain_JobArchive.masterServerId and domain_Policy.name = domain_JobArchive.policyName and domain_Policy.policyDomainName = domain_JobArchive.policyDomainName and domain_Policy.versionNo = domain_JobArchive.policyVersionNo inner join domain_MasterServer domain_MasterServer on domain_MasterServer.id = domain_JobArchive.masterServerId inner join domain_Client domain_Client on domain_Client.masterServerId = domain_JobArchive.masterServerId and domain_Client.name = domain_JobArchive.clientName,domain_Entity, ( select * from domain_entity)#entity1 where ( (domain_MasterServer.id = domain_Entity.id) ) AND ( (domain_Client.id = #entity1.id) ) AND ( ( (domain_JobArchive.isValid = 1) ) AND ( ( (domain_JobArchive.endTime BETWEEN 135815994000000000 AND 135816012000000000) ) ) ) ORDER BY UPPER("domain_Entity"."name" ) ASC , UPPER("#entity1.name") ASC , "domain_JobArchive.startTime" ASC,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2434,1365518206174,0,71,0:,56:Converting query result for report Tabular Backup Report,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2435,1365518206174,0,71,0:,70:Time taken to execute report (in millis) Tabular Backup Report : 2090,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2436,1365518239886,0,71,0:,42:fetching report definition for report -511,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2437,1365518239886,0,71,0:,63:loading canned report definition using JobSummaryByExitCode.xml,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2438,1365518239886,0,66,0:,42:fetching report definition for report -503,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2439,1365518239886,0,66,0:,65:loading canned report definition using JobSummaryByExitStatus.xml,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2440,1365518239886,0,86,0:,42:fetching report definition for report -506,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2441,1365518239886,0,86,0:,65:loading canned report definition using AlertSummaryBySeverity.xml,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2442,1365518239886,0,85,0:,42:fetching report definition for report -505,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2443,1365518239886,0,85,0:,60:loading canned report definition using JobSummaryByState.xml,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2444,1365518239886,0,83,0:,42:fetching report definition for report -501,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2445,1365518239886,0,83,0:,63:loading canned report definition using topPolicyByFailedJob.xml,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2446,1365518239886,0,71,0:,42:running report Top 7 Job Error Log Summary,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2447,1365518239886,0,66,0:,40:running report Job Summary by Job Status,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2448,1365518239886,0,86,0:,40:running report Alert Summary by Severity,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2449,1365518239886,0,85,0:,35:running report Job Summary by State,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2450,1365518239886,0,83,0:,44:running report Top 7 Policies by Failed Jobs,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2451,1365518239902,0,71,0:,54:building query for report Top 7 Job Error Log Summary,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2452,1365518239917,0,66,0:,52:building query for report Job Summary by Job Status,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2453,1365518239917,0,86,0:,90:StoredProcQueryBuilder building stored proc with parameters for getAlertSummaryBySeverity,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2454,1365518239917,0,85,0:,47:building query for report Job Summary by State,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2455,1365518239917,0,83,0:,56:building query for report Top 7 Policies by Failed Jobs,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2456,1365518239933,0,71,0:,599:Query generated is select TOP 7 START AT 1 COUNT(domain_Job.id) as "domain_Job.id",domain_Job.statusCode as "domain_Job.statusCode" from domain_Job where ( (domain_Job.statusCode NOT IN (0, 1 )) AND (domain_Job.state = 3) AND (domain_Job.isValid = 1) ) AND ( ( (domain_Job.startTime BETWEEN 135847246399020000 AND 135848110399020000) OR (domain_Job.endTime BETWEEN 135847246399020000 AND 135848110399020000) ) AND ( (domain_Job.clientId IN (209,210,211,221,223,224,226,217,218,225,214,213,215,216,219,220,222 )) ) ) GROUP BY "domain_Job.statusCode" ORDER BY "domain_Job.id" DESC,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2457,1365518239933,0,71,0:,580:Query generated is select COUNT(domain_Job.id) as "domain_Job.id",domain_Job.statusCode as "domain_Job.statusCode" from domain_Job where ( (domain_Job.statusCode NOT IN (0, 1 )) AND (domain_Job.state = 3) AND (domain_Job.isValid = 1) ) AND ( ( (domain_Job.startTime BETWEEN 135847246399020000 AND 135848110399020000) OR (domain_Job.endTime BETWEEN 135847246399020000 AND 135848110399020000) ) AND ( (domain_Job.clientId IN (209,210,211,221,223,224,226,217,218,225,214,213,215,216,219,220,222 )) ) ) GROUP BY "domain_Job.statusCode" ORDER BY "domain_Job.id" DESC,40:com.symantec.nbu.nom.util.JavaLogger.app,0,(0|)
2,58330,423,423,2458,1365518239933,0,66,0:,951:Query generated is select COUNT(domain_Job.id) as "domain_Job.id",(case when domain_Job.state =106 then 3 when domain_Job.state =3 and domain_Job.statusCode =0 then 0 when domain_Job.state =3 and domain_Job.statusCode =1 then 1 else 2 end) as "jobMonitotExitStatus",lookup_JobStatus.name as "lookup_JobStatus.name" from domain_Job , lookup_JobStatus where ( (domain_Job.state = 3) AND (domain_Job.isValid = 1) ) AND ( ( ( (domain_Job.startTime BETWEEN 135847246399020000 AND 135848110399020000) OR (domain_Job.endTime BETWEEN 135847246399020000 AND 135848110399020000) ) OR ( (domain_Job.startTime < 135848110399020000) AND (d