Geir Willy Rogde

Problem description
oc4j_soa instance consuming 100% cpu.
ESB message processing running in an infinite loop where JMS adapter message consumers throws exception. When an MDB responds to a message by throwing an exception, the message is not considered delivered and it may be
redelivered. If the MDB might always respond to a given message by throwing an exception, and MaxDeliveryCnt is set to 0 to prevent the message from ever being discarded, the result may be an MDB stuck in an “infinite loop” – failing to process the same message over and over again.) For an MDB this is fine. For a JMS adapter this is more troublesome.
The problem can be seen in the application log $ORACLE_HOME/j2ee/oc4j_soa/oc4j_soa_default_group_1/oc4j/log.xml as continous flow of “Global transaction get” “global transaction begin”, infinite transaction loop, even if no load on system.
So the oraesb schema is basically corrupted.

Error message of type can be seen in container log:
Designtime cache has not been initialized Please look in logs for following signs of failure.
Fix them and restart. (a) Database access errors (b) ESB Bootstrap errors
(c) OC4J class load errors (d) Product installation errors
(e) Export ESB params and verify if host and port parameters are correct.
Please contact Oracle Support if unable to fix the issue.

reset oraesb schema

Of course, Oracle Support was contacted, but they tend to take some time responding, so meanwhile a qucik check on some of the trusted oracle bloggers around gave result:
http://www.thisisahmed.com/tia/soa/esbdesigntimecacheerror.html

So now the reset of oraesb was performed, but still there is an issue with the instances view:

ESB instances view:
“Could not retrieve the messages due to the error : ORA-00904: “SEQ”: invalid identifier ”

“Could not retrieve the messages due to the error : ORA-00904: “SEQ”: invalid identifier ”

This was caused by the table ORAESB.ESB_ACTIVITY did not contain the columns SEQ and SUB_FLOW_SEQ. Happened on reset of ORAESB.
The correct DDL was executed, and voila, suddenly the ESB instances view was behaving as expected.

The ESB_ACTIVITY DDL

drop table oraesb.esb_activity;
CREATE TABLE "ORAESB"."ESB_ACTIVITY"
( "ID" NUMBER NOT NULL ENABLE,
"FLOW_ID" VARCHAR2(256 BYTE) NOT NULL ENABLE,
"SUB_FLOW_ID" VARCHAR2(48 BYTE),
"SEQ" NUMBER,
"SUB_FLOW_SEQ" NUMBER(3,0),
"BATCH_ID" VARCHAR2(48 BYTE),
"SOURCE" VARCHAR2(48 BYTE),
"OPERATION_GUID" VARCHAR2(48 BYTE),
"TIMESTAMP" NUMBER NOT NULL ENABLE,
"TYPE" NUMBER(2,0) NOT NULL ENABLE,
"RR_OUTPUT_STATUS" NUMBER(2,0),
"ADDI_INFO" VARCHAR2(500 BYTE),
"IS_STALE" VARCHAR2(1 BYTE)
) PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255 NOCOMPRESS LOGGING
STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT)
TABLESPACE "USERS" ;
commit;


Introduction

Viewing the payload of an ESB instance within Oracle SOA suite is not available by default, at least not for successfully executed instances. But during development payload logging has proven extremely useful. And in production the concept of trackable fields have given insight into the core parts of the payload, even for successfully executed instances.
I will first explain how to enable payload logging, through a ESBLoggingInterceptor. And then how to utilize the trackable fields concept.

ESBLoggingInterceptor

Starting from Oracle Application Server release 10.1.3.4, ESB logging capability has been enhanced. To take advantage of this enhanced logging feature, you should perform the following steps:

• implement the oracle.tip.esb.server.common.interceptor.IEsbMessageInterceptor interface.
• Add the jar into the ESB classpath, for example: ORACLE_HOME/j2ee/oc4j_soa/server.xml, in block esb.common.
• Add the following configuration variables in the esb_config.ini file:

inboundMessageInterceptor = no.gwr.esb.ESBLoggingInterceptor
outboundMessageInterceptor = no.gwr.esb.ESBLoggingInterceptor

package no.gwr.esb;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.w3c.dom.Element;

import oracle.tip.esb.monitor.FlowId;
import oracle.tip.esb.server.common.interceptor.IEsbMessageInterceptor;

public class ESBLoggingInterceptor implements IEsbMessageInterceptor {

	private static final Logger log = LoggerFactory.getLogger(ESBLoggingInterceptor.class);
	private static String tab = "\t";

	public void invoke(FlowId flowId, String instanceId, Element payload, Element header, String serviceName, String systemName, Element response, Throwable throwable, boolean isOutbound,	boolean isResponse) {
		StringBuffer sb = new StringBuffer();
		sb.append("flowId: ").append(flowId).append(tab);
		sb.append("instanceId: ").append(instanceId).append(tab);
		sb.append("serviceName: ").append(serviceName).append(tab);
		sb.append("systemName: ").append(systemName).append(tab);
		if (header!=null)
			sb.append("headerContent: ").append(header.getTextContent()).append(tab);
		if (payload!=null)
		  sb.append("payloadContent: ").append(payload.getTextContent()).append(tab);
		if (response!=null)
			sb.append("response: ").append(response.getTextContent()).append(tab);
			System.out.println(sb.toString());//goes into default log
			log.info(sb.toString());//goes into the configured separate log
	}
}

As you notice sl4j is used for logging. This could be skipped, if you are happy with the default log for payload output. We separated this as we actually do payload logging also in production. This is to be able to scan historically executed payload content, as an audit log. A sample logback.xml could be:

<configuration>
 	<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
		<File>ESBInterceptor.log</File>
		<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
		  <FileNamePattern>ESBInterceptor.%d{yyyy-MM-dd}.log</FileNamePattern>
		   <MaxHistory>30</MaxHistory>
		</rollingPolicy>
		<layout class="ch.qos.logback.classic.PatternLayout">
			<Pattern>%d{HH:mm:ss.SSS} %-5level %logger{36} - %msg%n</Pattern>
		</layout>
	</appender>
   <turboFilter class="ch.qos.logback.classic.turbo.MarkerFilter">
    <Marker>PROFILER</Marker>
    <OnMatch>ALLOW</OnMatch><!--DENY-->
  </turboFilter>
  <root level="debug">
    <appender-ref ref="FILE" />
  </root>
</configuration>

Now payload is available in the ESBInterceptor.%d{yyyy-MM-dd}.log

Trackable fields within ESB

The ESB trackable fields concept can be used to execute xpath-expressions. The way to do it is to deploy a ESB service, configure the trackable fields on the server, and then pull back the file *.esbsvc containing the relevant trackable fields expressions. This is to get the correct syntax of how to define trackable fields. Note that you must connect the trackable field to the correct operation guid, found elsewhere in the *.esbsvc-file.

The xpath is executed, and the trackable fields values are viewable in ESB console, in lower part of pane tracking.

...
  <trackableFields>
      <trackableField name="sourceSystem" status="ENABLED" bindTo="REQUEST">
         <operationRef guid="C31378D0FFAA11DEBFA619588F2E21CD" qname="ConsumerServices.EventsListener.EventsListener_v1_0.JmsAdapterIn_RS.Consume_Message"/>
         <expression>{/tns:eventsMessage/tns:messageBody/tns:sourceSystem};{ namespace tns=http://soa.gwr.no/shipping/v1.0/shipmentevents }</expression>
      </trackableField>
...

Summary

Now both the full payload is available in server logs, and a set of chosen payload values are available in ESB console.

References

@see Oracle bug 6995195
@see Oracle® Application Server Release Notes and New Features
10g Release 3 (10.1.3.4)

Introduction

XML is the ubiquitous message format in SOA. To decouple the components exchanging messages, a mapping layer can be introduced, thus isolating the knowledge on message format within this mapping layer. The obvious choice of transforming from the XML format over the wire, from the XML format within component, is XSLT.

Decoupling

The figure illustrates the basic principles of decoupling message producer from message consumer. Using a mapping layer for XSLT, and data binding, will isolate the business logic in message consumer from the xml message format. Whether a mapping layer is introduced on the message producer side, depends on design.
But the mapping layer on producer side will by the same arguments as on the consumer side introduce decoupling between business logic message format and the exchange message format.

package no.gwr.mdb.samples;

@MessageDriven(activationConfig = {
    @ActivationConfigProperty(propertyName = "connectionFactoryJndiName", propertyValue = "jms/TopicConnectionFactory"),
    @ActivationConfigProperty(propertyName = "destinationName", propertyValue = "jms/MyTopic"),
    @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Topic"),
    @ActivationConfigProperty(propertyName = "SubscriptionDurability", propertyValue = "Durable"),
    @ActivationConfigProperty(propertyName = "SubscriptionName", propertyValue = "TopicListener")
})
public class OrganizationEventsListenerMDBBean implements MessageDrivenBean, MessageListener {
...

  public void onMessage(Message message) {
    try {
      if (message instanceof TextMessage) {
        TextMessage txtMsg = (TextMessage) message;
        String txt = txtMsg.getText();
        String result = transform(txt);
        Organization organization = unmarshal(result);
        MessageConsumerAdapterImpl handler = new MessageConsumerAdapterImpl();
        handler.updateOrganization(organization);
        message.acknowledge();
      }
    } catch (Throwable ex) {
      ex.printStackTrace();//TODO introduce sl4j logging framework, and log to application.log
    }
  }
	/**using XSL (xalan-2.7.1) to transform XML message*/
  public String transform(String txt) throws Exception {
    javax.xml.transform.TransformerFactory tf =  javax.xml.transform.TransformerFactory.newInstance();
    javax.xml.transform.sax.SAXTransformerFactory stf = (javax.xml.transform.sax.SAXTransformerFactory)tf;
    javax.xml.transform.stream.StreamSource xsl = new javax.xml.transform.stream.StreamSource(getClass().getResourceAsStream(XSL_FILE));
    javax.xml.transform.Templates templates = stf.newTemplates(xsl);
    javax.xml.transform.Transformer transformer = templates.newTransformer();
    javax.xml.transform.stream.StreamSource source = new javax.xml.transform.stream.StreamSource(new ByteArrayInputStream(txt.getBytes()));
    javax.xml.transform.stream.StreamResult res =  new javax.xml.transform.stream.StreamResult(new StringWriter());
    transformer.transform(source, res);
    String xmlResult = res.getWriter().toString();
    return xmlResult;
  }
  /**using JAXB (jaxb-2.2) to do the data binding*/
  @SuppressWarnings("unchecked")
  public Organization unmarshal(String result) throws JAXBException {
    Organization ret=null;
    JAXBContext jc = JAXBContext.newInstance(TNS);
    Unmarshaller u = jc.createUnmarshaller();
    JAXBElement element = (JAXBElement)u.unmarshal(new ByteArrayInputStream(result.getBytes()));
    ret=element.getValue();
    return ret;
  }
}

Summary

When doing xslt and jaxb inside a container you will most certainly run into classloading issues. But all containers have ways of controlling the application classpath to avoid some old container libraries being used from a parent classloader.

References

http://xml.apache.org/xalan-j/ for XSLT
https://jaxb.dev.java.net/ for data binding
Make sure you use JAXB 2.x,it has several improvements to 1.0. full XML schema support, generating pojos with annotations and javadoc etc.

As part of the Apache Myfaces Trinidad project there is a maven JDeveloper plugin developed. This is made available to easily move from the maven project structure to JDeveloper setup. The maven plugin is generating the *.jpr and *.jws files. The command for generating the JDeveloper metafiles is not straight forward, but when you type it correct it will save you a lot of time.

mvn -Djdev.release=10.1.3.0.4 org.apache.myfaces.trinidadbuild:maven-jdev-plugin:jdev

Note that there is only project.xml templates for JDeveloper releases 10.1.3.0.2, 10.1.3.03, 10.1.0.4, 11.1.1.0.0, 11.1.1.1.0. But that is fine. Just go to the a release number before the one actual used, because when opening in JDeveloper, the migration to correct version will happen. Now the next thing would be to have some deploy profiles created, but this could more effectively be done using core maven.

of course mvn eclipse:eclipse will generate .classpath .project to make the project available in eclipse.
So letting maven produce the JDeveloper metafiles and Eclipse metafiles make it easy to switch between the two environments. Something is faster to do in JDev and something is faster to do in Eclipse. Freedom of choice.

See also:

http://www.oracle.com/technology/products/jdev/howtos/1013/maven/maven.html

http://repo1.maven.org/maven2/org/apache/myfaces/trinidadbuild/maven-jdev-plugin/

Introduction

When working in a heterogeneous SOA environment there is usually a range of business components implemented using different technologies. The development tools will vary by technology, and by developer preference. Supporting a wide range of development tools will enable developer efficiency.
And there are development tools geared towards the different target platforms, like Oracle JDeveloper, Mule IDE, Netbeans, xml/wsdl generators etc., not to mention the .net side.

deployable unit

But when a deployable unit is prepared it is time to feed it into the release process. This is of course to keep manual intervention, and developer tools, away from the production environment.

Only deployable units that have been run through test, and QA, are processed through to staging and production.

release process

The release process requires a clear cut between development process and production environment. Did I mention ITIL?
And manual intervention should be limited, all the manual steps are error-prone. Putting effort into a standardized way of generating deployable units, and a well-defined process of certifying these deployable units is the way to go.

maven to produce deployable units

We have chosen maven to produce deployable units. Stay away from ant if you can. Still the developer can use the best available tool for efficiency on producing the deployable unit content, but final DU (deployable unit) is generated using maven. Maven is the obvious choice for this task. Wide range of plugins, and possibility to extend with self-made plugins. We had to create some maven plugins on esb and bpel-suitcase generation. This was to handle the different endpoints for different environments (DEV, INTTEST, SYSTEST, STAGING, PROD).

For coding components, like java ws, the different environments is handled by logic. So the component is self-responsible for resolving its environment, and corresponding properties.
The build time knowledge of environment properties is only relevant for configuration components, (esb/bpel).

script to support release process

To support the operational team on release process, a small set of scripts, adapted to the target enviroment is created. When Weblogic was target platform we used wlst, when Oracle SOA suite is target platform, we use the admin-client.jar library, combined with shell-scripts. All platforms will support command-line deployment!

It does take some preparation, but confidence on deployable units is mandatory in a SOA environment

Summary

Spend time establishing a release process within a SOA environment. Prepare to spend some time on establishing the final touch, suitable for your organizations needs, but remember that most is done in advance. You will find it in your target platforms libraries and documentation. Pull it out, twist it, and use it as you like.