WebServices Performance Tuning

Recently I have been working on high traffic and highly available web service which is hosted by Siebel. I was asked to improve performance of it. I checked all the places for any performance degradation clues, there was hardly any, all queries were on indexed columns and thin BC's were used, there was no fat whatsoever.

With all the logs and spool generation my focus went to these two problems. 

1. Before every web service call Siebel was requesting file system to write read user preference file which was taking fair bit of time due to contention at file system. You might see following in logs, if user preference file is corrupted.

ObjMgrLog Debug 5 0 2016-06-29 06:50:56 SharedFileReader: \SiebelFS\userpref\EAIUSER&Siebel Sales Enterprise.spf: Open iteration 0 failed. errcode = 2.

This file read was absolutely unnecessary, luckily there was a way to turn off by setting "OM - Save Preferences" to FALSE for EAI object manager. This will make all logins to avoid looking up for user preference file.

2. Another useless transaction I noticed was with database. It was when Siebel tries to update the last login date on S_USER record. 

In logs you can see queries like these:

UPDATE SIEBEL.S_USER SET
LAST_LOGIN_TS = :7,
MODIFICATION_NUM = :8,
LAST_UPD_BY = :9,
LAST_UPD = :10,
DB_LAST_UPD = current_date,
DB_LAST_UPD_SRC = :11
WHERE
ROW_ID = :12 AND MODIFICATION_NUM = :14;

This can also be turned off in recent Siebel versions by setting DisableLastLoginTS = TRUE 

Its a new parameter introduced by Oracle, you can read more about this parameter on oracle support 1665762.1

Hope it helps. If you have come across any such parameter which can improve performance then please share it in comments below.

[Download] EAI JSON Converter v2

Pleased to bring you the complete EAI JSON Converter v2. It happened a lot sooner than I expected.

Siebel EAI JSON Converter

Below is the code which converts JSON string to property set. The heart of the code is eval statement, which converts the JSON string into eScript Object. 

Siebel eval statement

Remaining code is just the recursive traversing to convert object into property set. I have tested the code for couple JSON strings, it passed everytime with flying colours every time. Feel free to change the code as required. 

You can download the complete sif from git hub. https://github.com/Jimjson/eScript-based-EAI-JSON-Converter-

EAI JSON Converter Demo: 

EAI JSON Converter - v2 (No Java)

I published EAI JSON Converter business service based on Siebel Java business service back in 2013, and many people seemed to like it and implemented it. See the old post here. It was great to see that code getting implemented in production by many people. Surprisingly equal number of people faced difficulty in implementing the Java business service which is required to get EAI JSON Converter working.

Thus I started working towards removing dependency on JAVA business service and created complete eScript based solution as ver2 of EAI JSON Converter business service.

Generate JSON in Siebel without Java

This service will have the same methods and arguments, and will outputs similar JSON string and property set as previous version did but without the need of JDK. It is complete plug and play service, just copy paste the code and start using the service.

function CreateJSON(Inputs){
/*********
Created By: Jim
Support: http://howtosiebel.blogspot.com
*********/
    var str,j;
    var propName = "";
    var propVal = "";
   
    str = "{";
   
    propName = Inputs.GetFirstProperty();   
   
    //Type and value
    if(Inputs.GetValue() != null && Inputs.GetValue() != ""){
        str = str + '"' + Inputs.GetType()  + '"' + ":"  + '"' + Inputs.GetValue() + '"';   
        if(propName != "") str= str + ",";
    }
   
   
    //Properties
    while (propName != "") {
        propVal = Inputs.GetProperty(propName);
        str = str + '"' + propName +  '"' + ":" + '"' +propVal + '"';
        propName = Inputs.GetNextProperty();
        if(propName != "") str = str + ",";
       
    }
    propName = Inputs.GetFirstProperty();
    if(propName != "" && Inputs.GetChildCount()> 0)str = str + ",";
   
    //Loop for child
    if(Inputs.GetChildCount()> 0){
    if(Inputs.GetChildCount()> 1)str = str + '"Object"' + ":" + "{";
    for (var i = 0; i < Inputs.GetChildCount(); i++){
            j = Inputs.GetChild(i);
            str = str + '"' + j.GetType() + i + '"' + ":" + CreateJSON(j);// + ",";
            if(i+1<Inputs.GetChildCount()) str =str + ",";
    }
    if(Inputs.GetChildCount()> 1)str = str + "}";
    }
    str = str + "}";
    return str;   
}

Hope the code is self explanatory. Feel free to edit it for your implementation. I have tested the output and validated for RFC4627.

Creating JSON string from property set was easy I traversed through the property set using out of the box Siebel methods and created the string. Creating propset from JSON proved difficult, I am still working on solution for interpreting JSON and converting it into property set, I have made some progress will post it as soon as it is working satisfactorily.

If you want to contribute in nobel cause then contact me via comments below and don't forget to plus one this post on Google. All comments are welcome.

Siebel - Twitter Integration - Part 1: Authentication

This is post is second in series of Siebel Twitter Integration and will talk about logging in to twitter as an application.

Twitter has support two form of authentication, O-Auth Authentication and Application only authentication. For pulling tweets in Siebel we need Application only authentication, luckily which is technically easier than O-Auth and have more generous rate limits.






Before starting with Siebel configuration follow these steps:

1. Create account on http://dev.twitter.com by agreeing to terms and conditions.
2. Request to create your access token, and copy your Consumer Key and Consumer Secret.

Now copy paste following code to your repository, I prefer client side scripting as it easy to change.

StringToBase64 encoder

http://www.siebel-tech.com/2013/07/escript-base64-encoder/
This is piece of code is required for encoding the Client Key and Client Secret to Base64 before sending it to twitter. Developers at siebel-tech.com has done wonderful work in converting String to Base64, which saved me lot of time. Thanks Iain.

function StringToBase64(Inputs, Outputs)
{
  // *************************************************************************
  // Purpose: Encodes to a Base64 string
  // Author: Iain Ollerenshaw
  // Date: 30-Jul-2013
  //
  // Inputs: InString - string to be encoded
  // Outputs: Base64String - string in Base64
  //
  // Modification History
  //
  // Date          By                Details
  // 25-Jul-2013   Iain Ollerenshaw   Created
  // 2-Feb-2014    Jim  Updated
  // **************************************************************************

  try
  {

    // Define the Base64 codex
    var sCodex = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";
    var sOutput = "";
    var sInput = Inputs.GetProperty("InString");
    var iLen = sInput.length;
    // Parse input string
    var iPos = 0;
    while (iPos < iLen)
    {
      var sChr1 = sInput.charCodeAt(iPos);
      iPos++;
      var sChr2 = sInput.charCodeAt(iPos);
      iPos++;
      var sChr3 = sInput.charCodeAt(iPos);
      // Shift bytes
      var iEnc1 = sChr1 >> 2;
      var iEnc2 = ((sChr1 & 3) << 4) | (sChr2 >> 4);
      var iEnc3 = ((sChr2 & 15) << 2) | (sChr3 >> 6);
      var iEnc4 = sChr3 & 63;
      if (isNaN(sChr2))
      {
        iEnc3 = iEnc4 = 64;
      }
      else if (isNaN(sChr3))
      {
        iEnc4 = 64;
      }
      sOutput +=(sCodex.charAt(iEnc1) + sCodex.charAt(iEnc2)+ sCodex.charAt(iEnc3) + sCodex.charAt(iEnc4));
      iPos++;
    }
    Outputs.SetProperty("Base64String", sOutput);  }
  catch(e)
  {    throw(e);  }
}

Get Description / Set Description

This is a small piece of code which helps to get the description of any LOV value, I have used LOV as scratch pad to store access token in this example, I will explain to store an maintain these in custom Twitter dashboard. I thought of saving them in system preferences first, but it only has 100 char limit thus had to switch over to LOV Description column.

function GetDescription (sType,sName)
{
var boListOfVal = TheApplication().GetBusObject("List Of Values");
var bcListOfVal = boListOfVal.GetBusComp("List Of Values");
bcListOfVal.ClearToQuery();   
bcListOfVal.ActivateField("Description");
bcListOfVal.SetSearchSpec("Name",sName);
bcListOfVal.SetSearchSpec("Type",sType);
bcListOfVal.ExecuteQuery();
if(bcListOfVal.FirstRecord()){
return(bcListOfVal.GetFieldValue("Description"));
}else return("none");
}

function SetDescription (sType,sName,sDesc)
{
var boListOfVal = TheApplication().GetBusObject("List Of Values");
var bcListOfVal = boListOfVal.GetBusComp("List Of Values");
bcListOfVal.ClearToQuery();   
bcListOfVal.ActivateField("Description");
bcListOfVal.SetSearchSpec("Name",sName);
bcListOfVal.SetSearchSpec("Type",sType);
bcListOfVal.ExecuteQuery();
if(bcListOfVal.FirstRecord()){
bcListOfVal.SetFieldValue("Description",sDesc);
bcListOfVal.WriteRecord();
}
}

LOVs look like:

Main code for Twitter login is as follows:

//Get Consumer Key and consumer Secret from LOV Description
var sConsumerKey = GetDescription("TWITTER_TOKEN","ConsumerKey");
var sConsumerSecret = GetDescription("TWITTER_TOKEN","ConsumerSecret");

//Encode Key and Secret into Base64
var inp = TheApplication().NewPropertySet();
var op = TheApplication().NewPropertySet();
inp.SetProperty("InString",sConsumerKey + ":" + sConsumerSecret);
StringToBase64(inp,op);


//Use EAI HTTP Transport to call Twitter API to login.
var httpSvc= TheApplication().GetService("EAI HTTP Transport");
var httpIn = TheApplication().NewPropertySet();
httpIn.SetProperty("HTTPRequestURLTemplate","https://api.twitter.com/oauth2/token?grant_type=client_credentials");
httpIn.SetProperty("HTTPRequestMethod","POST");
httpIn.SetProperty("HTTPIsSecureConn","TRUE");
httpIn.SetProperty("HTTPContentType","application/x-www-form-urlencoded;charset=UTF-8");
httpIn.SetProperty("HDR.Authorization","Basic " + op.GetProperty("Base64String"));
httpIn.SetProperty("HDR.Accept-Encoding","identity");
httpIn.SetProperty("HDR.User-Agent","something");
httpSvc.InvokeMethod("SendReceive", httpIn, Outputs);


//Transcode the JSON response into UTF-8
var oTransService = TheApplication().GetService("Transcode Service");
var oTransOutputs = TheApplication().NewPropertySet();
Outputs.SetProperty("ConversionMode", "EncodingToString");
Outputs.SetProperty("TargetEncoding", "UTF-16");
Outputs.SetProperty("SourceEncoding", "UTF-8");
oTransService.InvokeMethod("Convert", Outputs, oTransOutputs);
var sResponse = oTransOutputs.GetValue();


//Convert the JSON response to property set
var oJSONConverter = TheApplication().GetService("EAI JSON Converter");
oJSONConverter.InvokeMethod("JSONToPropSet",oTransOutputs,Outputs);

//extract access_token and clip
var token = Outputs.GetChild(0).GetProperty("access_token");
token = token.substring(1,token.length-1);


//Save the token for future use
SetDescription("TWITTER_TOKEN","EncodedTokenCredentials",token);

This access token will be used in all the future communications with twitter as authorization code.
Post version: Draft :) keep checking for more updates on explanations over the HTTP transport.

Siebel Twitter Integration - Overview

What is Siebel?

Siebel is Oracle's CRM(Customer Relationship Management) product suite, which offer to manage customer relationships through various touchpoints. Siebel suite is designed mostly for all industry verticals and horizontals, and help business to manage customer data, order information, partner information, sales force automation, customer self service and many more customer interactions.

With its powerful integration capabilities Siebel can interact with other enterprise systems (like billing, back office systesms) on many protocols(like MQ Series, JMS, HTTP, Webservices, database connections, COM, Java, DLL and more..).

What is Twitter?

Does anybody need introduction? Twitter is first and biggest microblogging platform having billions of users accross the world. Some features like verified accounts, hash tagging and trending has made it one of the biggest social network platform.

User can vent their spleen out to the world lietrally about any topic and can follow any user or topic accross the world. From couple of years people have started venting about the poor customer service and problems on twitter. Business are forced to act on the complaints and provide response to keep the brand image. Recently angry customer's tweet about his lost baggage forced British airways to conduct a massive search operation, and helped passenger to get his lost baggage.

Does Siebel Provides Out of the box way to integrate with twitter? No!
Siebel recommends a third party product Buzzinet to handle the twiiter interaction and provide processed information back to Siebel. Which is a very tacttical solution and provides some benefits too. Thus it comes with premium licensing cost and service agreement.

Siebel Twitter Integration

In this series we will discuss a solution which can be implemented to integrate Siebel with twitter without the need of any another enterprise application or middleware.
Twitter only support REST API 1.1 for integration with other applications, current Twiiter API only accepts and provide response in JSON strings. That does not mean Siebel can not integrate Twitter. By using Java based EAI JSON Converter Siebel can create JSON strings and convert back string to data. Follow these links to understand more about EAI JSON Converter..

Twitter REST API v1.1 (@twitterapi) have three major ways of pulling and posting tweets into the cloud.

1. O-Auth User authentication

This method is used by mobile application and third party websites to allows users to acccess twiter features without leaving their environment.

2. Applicaton Only authentication

This type of authentication allows application to execute search queries on twitter and follow topics. With this type of authentication Twitter provides little more generous rate limits for querying twitter, and are best suited for CRM applications.

3. Streaming API

This is firehose of twitter tweets, and it streams live tweets to the applcation using long lived open HTTP connections. This feature should only be implemented if some serious research of user analytics need to be performed.


For the purpose of this series we will demonstrate a solutions which will help organizations having, to follow topics on twitter and post replies to the tweets all by using Siebel on premise application.

Later in this series:

• How to implement Twitter -Application only authentication in Siebel?
• Implementing Twitter search API in Siebel & How to import tweets in Siebel?
• Posting replies to tweets.
• Twitter - Administration Console

 Stay tuned for coming up posts.

EAI JSON Converter

View new eScript based solution to create JSON Strings in Siebel.

Siebel does not provide any out of the box functionality to convert Siebel data into JSON string or vice versa. Developers end up with string manipulations or giving up on integration. I have created custom EAI JSON Converter business service which uses Java capabilities and Siebel-Java integration EAI Java Business Service to produce fully compliant JSON strings and convert the external JSON into Siebel Property Sets.

The service built on the lines of methods of EAI XML Converter which converts Property set to XML Doc and XML Doc to property set.
Methods available in EAI JSON Converter are:

1. JSONToPropSet
2. PropSetToJSON

Input data to this service is sent to EAIJSONConvert java class using Siebel API(Siebel.jar) which converts the data in required format using GSON 2.2.4 Java API . The Gson library was originally developed for internal purposes of Google, and Version 1.0 was later released on May 22, 2008 under the terms of Apache License 2.0. The latest version, 2.2.4, is used for the service.

How to set up:

1. Download and place java files in the CLASSES folder of Siebel Client the binaries 

Download location: GitHub Project(https://github.com/Jimjson/Siebel-EAI-JSON-Converter/) for: 

Files:

EAIJsonConverter.class

gson-2.2.4.jar

Siebel.jar

2. Download the sif file for EAI JSON Converter from github

3. Update the cfg file to include gson-2.2.4.jar file. My cfg file look like:

[JAVA]
DLL = C:\Program Files\java\jre6\bin\client\jvm.dll
CLASSPATH = C:\Siebel\8.1\Tools_1\CLASSES\Siebel.jar;C:\Siebel\8.1\Tools_1\CLASSES\SiebelJI_enu.jar;C:\Siebel\8.1\Tools_1\CLASSES\gson-2.2.4.jar;C:\Siebel\8.1\Tools_1\CLASSES\.
VMOPTIONS = -Xrs -Djava.compiler=NONE -Djms.log=C:\logs\

4. Simulate the service

How to use:

Service can simply be used in place of EAI XML converter to convert the Siebel message into JSON or convert back JSON to property set.

Please feel free to use comments for any questions or to post any difficulty faced in using the service.

Hope it helps.

-Jim

Whats wrong with EAI Java Business Service?

What is wrong with EAI Java Business Service?? This was my reaction after struggling with EAI Java Business Service and trying to execute small Java code from Siebel. Actually there is nothing wrong with the service, it is our lack of knowledge and less experience with Java development which makes it difficult to configure.

I am going to list down steps required to get the Sample bookshelf code working with Siebel web client.



1. Siebel is not compatible with Java 1.7.
If you compile from Java 1.7 or above you will get the following error while calling service from Siebel

Class name incorrect or does not extend SiebelBusinessService : AddBusinessService -- JVM Exception:java.lang.UnsupportedClassVersionError: AddBusinessService : Unsupported major.minor version 51.0(SBL-EAI-05010)

For workaround Siebel Tools comes with a older version of jdk which can help you to compile and get rid of the above error.

JSON and Siebel

View new eScript based solution to create JSON Strings in Siebel.

JSON(JavaScript Object Notation) is a new lightweight integration message format which is replacing XML in thin client integrations. There are no meta tags and data description in JSON as in XML, it is just a string representation of JavaScript Object, it is designed to help javascript code to convert data to string and string to data without considerable overhead.

JSON vs XML

Many companies have started replacing XML with JSON in APIs. All the APIs of Google (Maps,Mail,Calendar, plus etc are now serving JSON), Twitter API,  Facebook apps and interfaces are built using JSON.
Read more on JSON on : http://www.json.org/

Siebel lacks in support of JSON on server side. Oracle support recommends use of workarounds such as using custom scripting to parse JSON response received by Siebel.

So how to interpret json in Siebel? Answer lies in the escript engine of Siebel. Siebel eScript as we know is actually ECMA script which was basis of Javascript. That is why the syntax of Javascript and eScript closely matches each other.

ECMA adopted JSON before it became standard in javascript. Thus our well known Siebel eScript also have some support.