Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to use ENRICHMENT component in Data pipeline?
#1
As we know that ,All component configurations are classified broadly into 3 sections-
1)basic
2)Meta_info 
3)Resource-configuration

*Basically Enrichment  component helps users to enrich the incoming data from in-event by querying lookup table in RDBMS and MongoDB and currently it supports 5 drivers :-
1)MySql
2)Ms-SQL
3)MongoDB
4)Postgres
5)Oracle

It has to be configure in a way that-suppose we are reading a data from mongo collection through pymongo reader and joining with one more table of mysql by Enrichment component,so in both Pymongo reader & Enrichment component we need to specify database name and table name which we are gonna join.

*During the Component configuration,There is one option in Meta_info, Conditions: Select conditions type (Remove or Blank option)

Remove [works like an inner join]– This option completely removes the row If any row of incoming data is not matching with query condition of the lookup table. Only passes the matching records.

Blank – This option sets blank for the cell of a row if complete incoming data from the in-event and matching rows from the lookup table.

"@data." is used to refer to the data coming from the previous event. For e.g.: If the data coming from the previous event is department_id and the column in the master table is dept_id then in the where clause we write as

Query which has to be entered in a format:-

SELECT department_name,location_id FROM departments WHERE department_id = @data.department_id     = Valid

Asterisk (*) is not valid in the select statement
SELECT * FROM departments WHERE department_id = @data.department_id       = Invalid
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)