continuous integration an unit tests

The standard should be that tests are executed as part of build, and that a failing test results in failing build, meaning no deployable unit or artifact is uploaded to nexus/artifactory. But every now and then there is a need to build the projects, even if tests are failing. Then you can allow for unstable build temporarily.

While effort is put into fixing the test env and tests, we can use maven properties to relax the regression test regime temporarily.
The relaxed regression test regime should only be allowed for  hours or maybe days, but not for weeks, or months!
The first thing to look for when coming to a new customer is the CI-server. You can figure out commit activity, dependencies, test execution, other tools and plugins used by CI-server.
I will never forget coming in to the “Unnamed” customer, and the chief architect was happy to show me the CI-server. But the CI-server was a mess. Failing jobs, and had been failing for months. A lot of disabled jobs, mainly to avoid annoying mails. This was clearly not a well-maintained CI-server.
And then when going into more detail of jobs, the tests was completely skipped for several of them, meaning they where not even compiled.
It was even deployed into Dev-env and SysTest-env without compiling a single test. But then when moving on further to AccTest, someone asked about test execution, and they did not even compile.

Disabling tests because they become to much hassle indicates you write the wrong type of tests. Even if the functional developer is busy producing functionality, it is the wrong priority if they are not able to produce one single test that can be rerun automatically, as part of regression testing from CI-server.
Do not maintain a large number of tests. Reduce the test set to cover the functionality being delivered from component. Test actual business value. If it is a data-driven component, include database access in the test setup.

-Dmaven.test.failure.ignore=true will ignore any failures occurred during test execution, will execute tests, but failing tests will not stop the build, it will be marked as unstable. This one is a good choice
-DskipTests=true would compile the test classes but skip test execution entirely, make sure you enable test execution as soon as temporary problem is resolved. Not so good choice
-Dmaven.test.skip=true would not even compile the tests, should only be used if the skipTests is still causing trouble, meaning the tests are not even compiling…This is bad, do you really have to do this

refer to for details

Using SSL certificates for web service communication

web service communication

In a SOA integration the web service communication may need to cross network boundaries between integration partners. To secure this communication a digital certificate can be used.

Generate SSL certificate

1. First we generate a privatekey publickey pair, using openssl.
the output says generating private key. But it is actually generating a pair of private key and public key.
the key pair is saved in an encoded format called PEM. The private key should not be shared, and it is generally recommended to store the RSA keys in an encrypted form.

$ openssl.exe genrsa -out server_rsa.pem 2048
Generating RSA private key, 2048 bit long modulus

2. Create a certificate signing request, if you intend to have the SSL certificate signed from a proper certificate vendor/CA like Verisign, Thawte, Go Daddy…

Note that from within a Oracle ESB environment it is the JRE validating the connection, so the certificate will be validated by the JRE, and a valid chain of certificates need to be available in the JRE keystore, $ORACLE_HOME/jdk/jre/lib/security/cacerts.

If the SSL certificate is signed from a proper certificate vendor/CA there will be no need for further config, as these certificate vendors already are available in the root keystore (within the cacerts).
Very often, especially during development, it is more convenient to work with self-signed certificates.
1. One can then import the newly created certificate into the default certificate store, within the ORACLE_HOME for the Application server.
Note that a restart is required after updating the keystore with the self-signed certificate.
keytool -list -keystore $JAVA_HOME/jre/lib/security/cacerts will list the certificates in keystore.
(default password: changeit)
$home\gwr>keytool -list -keystore c:\Java\jdk1.6.0_21
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 75 entries

digicertassuredidrootca, 07-Jan-2008, trustedCertEntry,
Certificate fingerprint (MD5): 87:CE:0B:7B:2A:0E:49:00:E1:58:71:9B:37:A8:93:72
trustcenterclass2caii, 07-Jan-2008, trustedCertEntry,
Certificate fingerprint (MD5): CE:78:33:5C:59:78:01:6E:18:EA:B9:36:A0:B9:2E:23

$keytool -list -alias digicertassuredidrootca -keystore c:\Java\jdk1.6.0_21\jre\lib\security\cacerts
Enter keystore password:
digicertassuredidrootca, 07-Jan-2008, trustedCertEntry,
Certificate fingerprint (MD5): 87:CE:0B:7B:2A:0E:49:00:E1:58:71:9B:37:A8:93:72

$keytool -import -alias oslsoahad1 -file cert_oslsoahad1.pem -keystore c:\product\\OracleAS_1\jdk\jre\lib\security\cacerts
Enter keystore password:
Do you still want to add it? [no]: yes
Certificate was added to keystore



keytool -list -keystore cacerts

keytool -printcert -file rootcacert.pem

keytool -import -file rootcacert.pem -alias Cisco_CesiumRoot -keystore cacerts

2. Add a keystore location using jvm argument
Inside the keystore there will be a “cacerts” file representing a system-wide keystore with CA certificates.

For the verification to pass, the keystore should contain the actual certificate as well as the root and any intermediate certificates.

Certificate missing in keystore

Message send failed: PKIX path building failed: unable to find valid certification path to requested target

exception during SOAP invoke: Server was unable to process request. —> Object reference not set to an instance of an object.; nested exception is: javax.xml.rpc.soap.SOAPFaultException: Server was unable to process request. —> Object reference not set to an instance of an object.


Cryptography Tutorials – Herong’s Tutorial ExamplesPublic key certificate
Regarding ESB and certificates
java keytool


one thing to consider is to put the keystore outside default location, as the keystore might be replaced during patching.

OC4J will not send ONS ProcReadyPort messages to opmn, or ORABPEL-12165 ERRJMS_PROVIDER_ERR


After a hard shutdown of server, due to network issues, the SOA suite would not restart properly.
In the opmn log there are messages of type:

WARNING: OC4J will not send ONS ProcReadyPort messages to opmn for service: OC4JServiceInfo id: null protocol: jms hostname: null port: 12603 description: null

null hostname was null

Could not produce message due to JMS provider error.


1. stop soa suite
2. Go to $ORACLE_HOME/j2ee//persistence/ and remove jms.state and *.lock files.
3. start soa suite

if this doesn’t do the trick you could try removing all content of folder $ORACLE_HOME/j2ee//persistence before starting soa suite.



Receiving error ORABPEL-05215 while deploying BPEL process.
This turned out to be caused by mismatch in client side deployment libraries and server side version.

To be clear, my server was, and my client side maven deployment libraries were
This would also be the error code if deployment done from JDeveloper

Error message

The process domain encountered the following errors while loading the process “ProcessName” (revision “1.0″): null.
If you have installed a patch to the server, please check that the bpelcClasspath domain property includes the patch classes.


Error while loading process.
The process domain encountered the following errors while loading the process “ProcessName” (revision “1.0″): null.


Make sure the maven deployment libraries and server side version match. I had to switch back to deployment libraries. And switch back to Jdeveloper
My SYSTEST environment is still version, and my DEV env is moved to, so I need to handle this intermediary state when deploying services, until also SYSTEST, and eventually PROD is upgraded to
best of luck to you…

WSM performance statistics


Oracle Web Services Manager is used to secure web services, and enable security of service-oriented architecture. In addition the WSM database schema contains performance statistics that is not easily accessible through user interface.
In the following some sql’s are documented to give an idea of what information can be pulled out of WSM database.

WSM request statistics

–connect and alter session
sqlplus orawsm/[password]@(description=(address_list=(address=(protocol=tcp)(host=[hostname])(port=[port])))(connect_data=(sid=[sid])))
alter session set nls_date_format=’YYYY/MM/DD hh24:mi:ss’;

  • –selecting number of requests per hour for a given service and operation
    select trunc(invoke_time,’hh’)”hour”, count(*)”requests”
    from messagelogs, log_objects
    where message_destination=’SID0003006′ –serviceid,change
    and invoke_time >= to_date(’2011/03/16 09:00:00′)
    and invoke_time < to_date('2011/03/18 09:00:00')
    and messagelogs.logid = log_objects.logid
    and UTL_RAW.CAST_TO_VARCHAR2(DBMS_LOB.SUBSTR(log_message,2000,1)) like '%operationName%'
    group by trunc(invoke_time,'hh');
  • –selecting number of requests per minute for a given service and operation
    select trunc(invoke_time,’mi’)”minute”, count(*)”requests”
    from messagelogs, log_objects
    where message_destination=’SID0003006′ –serviceid,change
    and invoke_time >= to_date(’2011/03/16 09:00:00′)
    and invoke_time < to_date('2011/03/18 09:00:00')
    and messagelogs.logid = log_objects.logid
    and UTL_RAW.CAST_TO_VARCHAR2(DBMS_LOB.SUBSTR(log_message,2000,1)) like '%operationName%'
    group by trunc(invoke_time,'mi');
  • –selecting request, and request size, for a given interval
    select invoke_time,
    dbms_lob.getlength(log_message),UTL_RAW.CAST_TO_VARCHAR2(DBMS_LOB.SUBSTR(log_message,2000,1))|| UTL_RAW.CAST_TO_VARCHAR2(DBMS_LOB.SUBSTR(log_message,2000,2001))
    from messagelogs, log_objects
    where message_destination=’SID0003006′ –serviceid, change
    and invoke_time >= to_date(’2011/03/16 00:00:00′)
    and invoke_time < to_date('2011/03/16 00:00:05')
    and messagelogs.logid = log_objects.logid
    and UTL_RAW.CAST_TO_VARCHAR2(DBMS_LOB.SUBSTR(log_message,2000,1)) like '%operationName%';

–if size of request is > 4000 characters (varchar2 limit) you will have to handle the blob as bytes in your favourite tool.
–java sample
byte[] requestAsBytes = requestAsBlob.getBytes(1, length);
String request = new String(requestAsBytes);

purge ESB instances


Within SOA suite there is console to view and manage ESB instances.
When trying to purge instances the process might take a long time, and the log could contain entries like:

11/03/16 10:37:39 at oracle.tip.esb.monitor.manager.database.DBActivityMessageStore.purge(
11/03/16 10:37:39 … 20 more
11/03/16 10:37:57 oracle.tip.esb.console.exception.ConsoleException: An unhandled exception has been thrown in the ESB system. The exception reported is: “oracle.tip.esb.monitor.MonitorException: Could not purge the messages due to the error : ORA-01795: maximum number of expressions in a list is 1000


Oracle provides sql-procedures for this, in folder $ORACLE_HOME/integration/esb/sql/other.


  1. $ORACLE_HOME\integration\esb\sql\other> sqlplus oraesb/[password]@(description=(address_list=(address=(protocol=tcp)(host=[hostname])(port=[port])))(connect_data=(sid=[sid])))
  2. sql>@purge_by_date.sql
  3. sql>alter session set nls_date_format=’YYYY-MM-DD’;
  4. sql>exec purge_by_date_package.purge_by_date(to_date(’2010-10-19′));
  5. sql>commit;

The date_format could also just be YYYY-MM-DD, or any valid date format.

sql>alter session set nls_date_format=’YYYY/MM/DD hh24:mi:ss’;
sql>exec purge_by_date_package.purge_by_date(to_date(’2010/10/19 00:00:00′));

If you hit the error:
ORA-30036: unable to extend segment by 8 in undo tablespace ‘UNDOTBS1′
try to delete using smaller date window, like deleting month by month.

SOA Suite ESB console > Instances

When working on SOA suite and looking at ESB instances instances console the default number of instances shown are 100.
Message at the top of Instances pane would be: “Search returned more than 100 instances, please modify your search criteria”

It is possible to specify search criteria , but only to increase the window from now and backwards meaning the limitation of 100 instances will still be a problem if you are looking for historical instance number 101 or above.
This limitation is controlled by parameter MaxInstanceCount in table ORAESB.ESB_PARAMETER.

insert into oraesb.esb_parameter(PARAM_NAME, PARAM_VALUE) VALUES ('MaxInstanceCount', '1000');

By increasing the MaxInstanceCount=1000 you will be able to view instances from a wider timespan also in ESB instances console.

If going beyond 1000, let’s say 10000, you might run into a db limitation:
Caused by: oracle.tip.esb.monitor.MonitorException: Could not retrieve the messages due to the error : ORA-01795: maximum number of expressions in a list is 1000

Another option would of course be to execute sql in database directly. The instances are stored in table ORAESB.ESB_ACTIVITY.

Java decompiler

highly recommended java decompiler gui frontend can be found here:

JDBC connection pool and firewall

Connection closed

When a jdbc connection is going through a firewall you might experience connection closed issues. What happens is that the firewall detects idle TCP connections and kills them. This is good stuff, both for security reasons and to free unused resources.
But the application is not always aware of the pretty brutal TCP connection killing done by the firewall. And would end up trying to use a connection already killed by the firewall. This could give all kinds of unexpected results, but bottom line is that you don’t get through to your database.
It would be much better if the application shut down idle and unused connections before the firewall detects it and kills it. If you have written your own jdbc connection handling there are ways to set the jdbc connection timeout.
Firewalls typically has a session timeout of 30 min, before killing idle TCP connections, but it could easily be as low as 10 min. Ask you network administrator, or do a test.

Application server

Another thing is that even if the application server connection pool is configured to remove idle connections after a certain period of time which is below the firewall session timeout, we still experienced connection issues.
The reason was that the minimum size of connection pool was configured to be larger than 0, as suggested by some best practices tutorials.
Note that IDLE_TIMEOUT will not remove conn if it result in connpool smaller than minimum.
I repeat: database connection pool idle timeout value will not remove connections from pool if this result in a pool smaller than minimum.
And what do you think will happen when the application server decide to maintain a minimum size of connections in the pool? Well, the firewall comes along and kills them. The firewall will not respect your minimum number of connections configuration!


Connections should timeout if idle for a certain period of time, to reduce unnecessary overheads, and to avoid open connections left open across a firewall. The brutal killing of idle TCP connections performed by the firewall can cause all kinds of problems to the application itself.
Keeping a minimum size will again allow the firewall to detect them as idle TCP connections.
As well as timing out, make sure you allow your pool to shrink to size of 0.

OWSM – changing gateway.repository.password

When the DBA changed the passwords of the orawsm schema user, the OWSM gateway was not able to connect anymore.

The procedure for updating owsm connect password is:

  1. edit the to reflect the new password in cleartext.
  2. execute./ encodePasswords ../config/gateway/ gateway.repository.password
  3. now redeploy the gateway to load the new password: ./ deploy gateway
  4. restart gateway

If you get this error message:
scripts/oc4j.xml:15: Unable to locate a valid OC4J URI. The server is probably down
make sure you type correct oc4jadmin password.