Tuesday 7 October 2014

Oracle data Integrator Tutorial --- Part 2 -- (Repositories)

To create a work repository right clicks on Work Repositories under Repositories tree in Topology navigator and select New Work Repository.
A Create Work Repository window is opened and put the following details –
In Step 1 of 2:
Technology : Technology used for creating the repositories that means the type of database e.g. oracle, sql server etc.
JDBC Driver : JDBC driver used in specified technology. In Oracle technology it is oracle.jdbc.OracleDriver.
JDBC Url: JDBC url to mention the connectivity of that server. In Oracle technology select jdbc:oracle.thin@<host><port><sid> from the dropdown list and change <host>,<port> and <sid> parameters accordingly.
User: user/schema for master repository. [odiw]
Password: password for that user. [odiw]
Press Test Connection button to check the connectivity. Press OK in Information window. Click Next.
 
 
In Step 2 of 2:
Id: Work repository id. By default the first repository under master repository is 1. You can change it.
Name: Name of the work repository.
Password: Password for work repository.
Work Repository Type: Select the nature of work repository .it is Development or Testing or Production.
Click Finish.
 
 
To connect to specified work repository you click on Connect To Repository… in Topology navigator and select the necessary login name from the drop down list. After putting the proper password press OK.
 

Topology:
After logging into ODI your first objective is to create the necessary topologies used for your ODI tasks.
In the topology navigator you have to define the detailed information including technologies, server definitions, schemas, contexts, languages etc.
First of all you define the data server responsible for your source and target database. Expand the technology tree under Physical Architecture and select Oracle. Right click on it and select New Data Server.
 
 

In the detailed information page under Definition tab write down the following information-
Name:  name of the data server.
Instance/dblink (data server): name of the database instance.
User: name of the user of database (except sys/system).
Password: password of that user
Array fetch size and batch update size are the parameters required for row based operation performed by some KM.
You can increase the size for better performance (by default it is 30)
 
In the JDBC tab put the following information –
JDBC Driver:  Select oracle.jdbc.OracleDdriver from the dropdown list.
JDBC Url: Select jdbc:oracle.thin@<host><port><sid> from the dropdown list and replace <host>, <port>, <sid> with the actual information shown in the below picture.
Press save button afterwards.

Under Definition tab put the following details-
Schema (Schema) – Oracle schema which is the candidate of either source or target schema.
Schema (Work Schema) – Oracle schema where all kind of work table to be stored .
Check the box against Default to make it default physical schema.
Press save button in toolbar.

In the Definition tab under logical schema put the following details –
Name: Name of the logical schema.
Select the appropriate physical schema for the Global context (as there is no other context yet to be created)
Press save button in the toolbar.
After creation of all the required topologies your next step is to create the necessary models for the source as well as target database.
Under Designer navigator in the Model tab select the right hand icon and after clicking select New Model.
In Definition tab you write the following details-
Name: name of the model.
Code: by default same as model name. You can explicitly change the code.
Technology: the technology used in the data source for which model to be created.
Logical schema: name of the logical schema for which model to be created.
In the reverse engineer tab by default ‘Standard’ mode is selected. If you want to do customized reverse engineer then select ‘Customized’ radio button.
By default Global as ‘context’ and Table as ‘Type of objects to reverse engineer’ selected. You can change them explicitly. You keep the other properties as it is.
Press save button in toolbar.
If you want to selective reverse engineering then under selective reverse-engineering tab check the boxes against ‘selective reverse engineering’, ‘new datastores’ and ‘existing datastores’. If you only particular objects to be reverse engineered then check the ‘objects to reverse engineer’. After populating all the objects to be resided in said logical schema you explicitly select the required objects to be used in reverse engineering.
Press save button.
In  the Model tree you do right click on the newly created model and select reverse engineer while you want to perform the reverse engineering for all the objects resided in same logical schema .After successful reverse engineering you see the objects while expanding the said Model tree.
 
 
Project:
Project is the logical repository in terms of ODI where all the specific activities for a particular ODI login user working on.
For creating a new project click on the following icon in Projects tab and select “New Project”.
After creating a project your next step is to identify the knowledge module to be used for the specified task. There are lots of knowledge modules being used in the different steps of any task e.g. loading, integration, check, journalization, reverse-engineering and service. You have the option to import the appropriate knowledge module from the ODI repository stored in xml-reference folder. Select the proper knowledge module and press ok. A pop-up window is open saying the detailed information of that knowledge module. If you want to save it as your future reference click Save otherwise press Cancel.
 
You can see the detailed information of any knowledge module after clicking on the same knowledge module in the necessary knowledge module category.
 

 
Interface:
Interface is the lowest component used in ODI for data loading from source to target area.
For creating new interface select Interfaces under the First Folder tree in the Projects tab and after right clicking on select New Interface.
Write thefollowing details in Definition tab.
Name: name of the interface
Optimization context: global by deafult.
If you want the staging area is different from target area then check the box and select the appropriate staging area  from the below drop down list.
In the mapping editor drag the required objects in source as well as target area.  If you want ODI itself do the mapping instead of you then select the automatic mapping as Yes while this message is being popped up during the objects dragging at target side.
When you select a field under the target area a property inspector is displayed at bottom. The following information populated for each and every field.
Active mapping: If the column is used in mapping then check the box.
In the implementation tab you see the mapping expression. Generally it is the column name at source end. If any customized mapping is required then expression editor is used by clicking on  .
Execute on: It mentions the area where this mapping is being executed. There are three areas – source, staging and target.
Insert: check the box if you want that said mapping is used during insert.
Update: the box if you want that said mapping is used during insert.
Key: If this column used as an unique key to identify the existence of any row of that object, check the box.
Check not null (flow control only): If you want to check the not null on the said column of target area, check the box.
In the expression editor there are three parts –
1) Left hand part shows you the object details.
2) Right hand lower part shows you the different functions of ODI.
3) Right hand upper part shows you a window where you write the customized expression.
You can double click on any column of any table to reflect column name in expression window. In same way you can also double click on any command to reflect it in same window. After completion of expression you press Ok and then apply button. The control backs to go in mapping editor. You select  (tick)  to validate the expression.
In the quick edit tab there are certain extra mappings are used on top of default mapping. These are joins, filters and lookup.
In the Join subheading click on   to add any join mapping. A join wizard is opened. Select the proper tables as the right and left source and click the Join button afterwards. If you need any functions to be intact with any column click on   to add the functions appended on any column. Next properly select the Execution Location and the Join Type parameters.
If you want to use any lookup tables then click on   under the lookup subheading to create any lookup condition. A lookup wizard is opened and you properly select the driving table and lookup table and press Next button. In next window properly select the columns from those two tables and press Join button. Afterwards properly select the suitable options on Execute on and Lookup type parameters.
 

If you want to perform any filetering condition on any source tables need to add any filtering condition through the filter wizard. The wizard is opened after clicking on the   under filter subheading. You select the proper source table and press the ‘Add to filter’ button.In the expression window you may write your customized expression by own or by using expression editor.Select the proper execution location as either source or staging.

Now you see all kind of mapping condition in the quick-edit tab.
 
 

In the Flow tab you generally select the appropriate knowledge modules to be used in different stages. While clicking on the SrcSet0 on the source area a property window is shown in below and select the required knowledge module from drop-down list. After selecting the knowledge module you can change the ‘value’ of different options being used in that KM. In same way you select the Staging Area as well as Target to determine the exact KM to be used.
 
In the Control tab you explicitly call the Check KM to isolate invalid data during integration phase.

Press the save button from the toolbar. To see the code of interface and execute the interface do right click on the exact interface and select Execute.


A window is opened with the following information.
Context: Name of the context
Logical agent: Name of the logical agent by which you run the interface. This may be Local( No Agent) or any defined Agent.
Log level: select the appropriate log level. By default it is 5.
Simulation: Check it to see the codes generated by ODI internally for that interface.
Press OK.
If you want to save the code for future reference click Save. Otherwise press Close.
To execute the interface uncheck the Simulation parameter of execution window.
You see the execution log details for the interface execution in Execution tree under Operator navigator
 
 

 
 
For further details on Oracle Data Integrator please mail me to recieve training on demand..
 
 
Regards,
Akash Pramanik

Monday 6 October 2014

Oracle data Integrator Tutorial --- Part 1

Overview :
Oracle Data Integrator is an integration platform. Simply put, it is used to move and transform information across the information system.
Oracle Data Integrator is also a development platform for integration processes. It is unique in two respects:
• It uses an approach driven by business rules. In this approach, you focus your effort on the business side of integration, and not on the technical aspects.
• It uses the E-LT approach. Oracle Data Integrator does not execute the integration processes itself at run time, but orchestrates a process which leverages existing systems.
Oracle Data Integrator is based on metadata. That is, descriptive information about the information system and its contents. This metadata is stored in a centralized metadata repository.
---------------------------------------------------------------------------


Above picture shows the overall Oracle Data Integrator Architecture.
The central component of the architecture is the repository. This stores configuration information about the IT infrastructure, the metadata for all applications, projects, scenarios, and execution logs. Repositories can be installed on an OLTP relational database. The repository also contains information about the Oracle Data Integrator infrastructure, defined by the administrators.
Administrators, developers, and operators use different Oracle Data Integrator Graphical User Interfaces to access the repositories. Security and Topology are used for administering the infrastructure, Designer is used for reverse engineering metadata and developing projects, and Operator is used for scheduling and operating run-time operations.
At design time, developers work in a repository to define metadata and business rules. The resulting processing jobs are executed by the Agent, which orchestrates the execution by leveraging existing systems. It connects to available servers and requests them to execute the code. It then stores all return codes and messages into the repository. It also stores statistics such as the number of records processed, the elapsed time, and so on.
Several different repositories can coexist in a single IT infrastructure. In the graphic in the previous page, two repositories are represented: one for the development environment, and another one for the production environment. The developers release their projects in the form of scenarios that are sent to production.
In production, these scenarios are scheduled and executed on a Scheduler Agent which also stores all its information in the repository. Operators have access to this information and are able to monitor the integration processes in real time.
Business users, as well as developers, administrators and operators, can get Web-based read access to the repository. The Metadata Navigator application server links the Oracle Data Integrator Repository to any Web browser, such as Firefox or Internet Explorer.
------------------------------------------------------------------




The four Oracle Data Integrator GUIs—Designer, Operator, Topology , and Security , are based on Java.
Designer is the GUI for defining metadata, and rules for transformation and data quality. It uses these to generate scenarios for production, and is where all project development takes place. It is the core module for developers and metadata administrators.
Operator is used to manage and monitor Oracle Data Integrator in production. It is designed for production operators and shows the execution logs with errors counts, the number of rows processed, execution statistics, and so on. At design time, developers use Operator for debugging purposes.
Topology manages the physical and logical architecture of the infrastructure. Servers, schemas, and agents are registered here in the Oracle Data Integrator Master Repository. This module is usually used by the administrators of the infrastructure.
Security manages users and their privileges in Oracle Data Integrator. It can be used to give profiles and users access rights to Oracle Data Integrator objects and features. This module is usually used by security administrators
Repository :
First of all you should know how to create the Master and Work repository used in ODI manually.You can create them during before ODI installtion by using Repository creation utility (RCU).
After successful installtion of ODI you create the master as well as work repository manually.To perform this you first creare the tablespace and schema/user dedicated for each repository and give the necessary privileges to those users.
SQL> create tablespace odim_tab add datafile “d:\oracle\dbf\odim01.dbf” size 1G;
SQL> create user odim identified by odim default tablespace odim_tab temporary tablespace temp;
SQL>grant resource,connect to odim;
SQL> create tablespace odiw_tab add datafile “d:\oracle\dbf\odiw01.dbf” size 1G;
SQL> create user odiw identified by odiw default tablespace odiw_tab temporary tablespace temp;
SQL>grant resource,connect to odiw;
Afterwards go to the File menu and select New.In the new gallery select Master Repository Creation Wizard.Press OK.
----------------------------------------------------------------

In Repository Connection  menu Following details would be required to fill it up.
Technology : Technology used for creating the repositories that means the type of database e.g. oracle, sql server etc.
JDBC Driver : JDBC driver used in specified technology. In Oracle technology it is oracle.jdbc.OracleDriver.
JDBC Url: JDBC url to mention the connectivity of that server. In Oracle technology select jdbc:oracle.thin@<host><port><sid> from the dropdown list and change <host>,<port> and <sid> parameters accordingly.
User: user/schema for master repository. [odim]
Password: password for that user. [odim]
Press Test Connection button to check the connectivity. Press OK in Information window. Click Next.
-------------------------------------------------------

In Authentication menu put the password details for ODI supervisor user SUPERVISOR. You cannot change the ODI supervisor user. Click Next.
In Password Storage select ‘Internal Password Storage’ option and click Finish.
-------------------------------------------------------

After creation of master repository in ODI you have to create the ODI repository login id for master repository. You may have more than one ODI repository login id for a single master repository.
To create ODI repository login id again click New under File menu and select Create a New ODI Repository Login. Then click OK.
-------------------------------------------------------------
A Repository Connection Information window is opened.You have to put  the relevant information in each properties.
Oracle Data Integrator Connetion :
Login name:  user defined name as login name.
User: ODI supervisor user SUPERVISOR. (It is case sensitive and you cannot change it).
Password: password of ODI supervisor user.
Database Connection (Master Repository) :
User: database user of master repository.
Password: database password of above user .
Driver List: select Oracel JDBC Driver from drop-down list.
Driver Name: automatically populate according to driver list selected.
Url: select jdbc:oracle.thin@<host><port><sid> from the dropdown list and change <host>,<port> and <sid> parameters accordingly.
Work Repository :
Select Master Repository Only radio button as you create the ODI login id for master repository.
Click Test button to check the connectivity.Press OK afterwards.
------------------------------------------------------------------------
 
 
 Finally we are able to connect to our Master repository.
---------------------------------------------------------------------------------

Please stay in touch for further updates on ODI. For any information in Database visit "Apps DBA Corner" --- akashpramanik.blogspot.com