DataStage job aborted due to ulimits - Need to be able to open at least 16 files; please check your ulimits settings for number of file descriptors

I am running a job on AIX 64-bit with DataStage V8.1 installed. The DataStage job failed with following error messages (refer to screen attached):

Fatal Error: Need to be able to open at least 16 files; please check your ulimits settings for number of file descriptors

Solutions: 

1. Check the value for ulimits in dsenv is set to 102400 or higher. If not, change the value of ulimits and do a restart on DataStage engine. Command to restart DataStage engine as below:
          • cd $DSHOME/
          • . ./dsenv
          • bin/uv admin –stop
          • bin/uv admin -start
2. Login using root and check /etc/security/limits. Look for value of nofiles and nofiles_hard in that file. If the value is lower than 102400 then modify it to 102400 for both of the entries. Add the entries if there are not in the limits file.

To resolve the problem, both of the steps must be applied in dsenv and root level.

References
http://www-01.ibm.com/support/docview.wss?uid=swg21322302
http://www-01.ibm.com/support/docview.wss?uid=swg21385849

Enable BASIC transformer in DataStage

When you are developing a Parallel job in DataStage, we are able to call a server routine in our Parallel job provided that we make use of BASIC transformer.

BASIC transformer is not visible in the Proccesing palette in Parallel job. Hence we would need to bring in the BASIC transformer to our design canvas following the steps stated below:
Go to Repository tree -> Stages Types -> Parallel ->  Processing -> BASIC Transformer.
Drag and drop the BASIC transformer stage to our design canvas.

With BASIC transformer, we are able to perform various server type data transformation in parallel job. For example, calling a server routine, link variables, etcs.

Check Services Running in AIX

In AIX, we can run the following command to verify what are the services that are currently running on the system.
$lssrc -a

Non-Wire protocol drivers VS Wire protocol drivers

Basically there are two type of ODBC drivers being used in DataStage:

1. non-wire protocol drivers - require  database client software to be installed on the DataStage server (the drivers use the API supplied by the database client)
2. wire protocol drivers - do not require database client software (they communicate with the database directly)
   

Technorati Tags: Non-Wire VS Wire protocol drivers

Connection to Teradata database via ODBC stage in DataStage

Before we can connect to the Teradata database using ODBC stage in DataStage, we need to define the Data Source on the DataStage server.

In Windows environment, we can defined the Data Source by going to Control Panel -> Administrative Tools -> Data Sources (ODBC) . Add a system DSN.

Note: DataStage cannot work on user DSN.

In Unix environment, three files need to be edited for ODBC configuration.

1. dsenv file - contains the environment variables to connect to different databases
2. .odbc.ini file - contains the names and the data sources available to the Unix users.
3. uvodbc.config file - contains the DSNs name that connect through ODBC

Example of dsenv file

#Oracle 9i
ORACLE_HOME=/space/oracle9i
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib:$ORACLE_HOME/rdbms/lib;export
LD_LIBRARY_PATH
ORACLE_SID=WSMK5
export ORACLE_HOME ORACLE_SID<o:p></o:p></span></p>

Example of an .odbc.ini file

[ODBC]
InstallDir=/usr/odbc
Trace=0
TraceDll=/usr/odbc/lib/odbctrac.so
TraceFile=/usr/odbcusr/joe/trace.log
TraceAutoStop=1

[ODBC Data Sources]
financial=tdata.so

[financial]
Driver=/usr/odbc/drivers/tdata.so
Description=NCR 3600 running Teradata V2R5.1
DBCName=123.45.67.10
DBCName2=123.45.67.11
DBCName3=123.45.67.12
Username=odbcadm
Password=
Database=
DefaultDatabase=sales

Example of uvodbc.config

[ODBC DATA SOURCES]

<localuv>
DBMSTYPE = UNIVERSE
network = TCP/IP
service = uvserver
host = 127.0.0.1

<TEST_DSN>
DBMSTYPE = ODBC

Once we are done with the configuration, we should do a test to the ODBC driver connectivity.
Step 1. Change to $DSHOME directory. Then type in . ./dsenv
Step 2. Run the DataStage server shell by type in ./bin/dssh
Step 3. Log on to the project using LOGTO your_project_name
Step 4. Use DS_CONNECT to get the list of DSNs available.
Step 5. Type DS_CONNECT dsn_name to test the connectivity.

References
http://publib.boulder.ibm.com/infocenter/iisinfsv/v8r0/index.jsp?topic=/com.ibm.swg.im.iis.productization.iisinfsv.install.doc/tasks/wsisinst_Third_Party_ODBC_Drivers.html