+- Forums (https://bdn.bdb.ai)
+-- Forum: BDB Knowledge Base (https://bdn.bdb.ai/forumdisplay.php?fid=13)
+--- Forum: BDB Data Pipeline (https://bdn.bdb.ai/forumdisplay.php?fid=48)
+---- Forum: BDB Data Pipeline Q & A (https://bdn.bdb.ai/forumdisplay.php?fid=17)
+---- Thread: How to use ENRICHMENT component in Data pipeline? (/showthread.php?tid=351)
How to use ENRICHMENT component in Data pipeline? - mohd.gulam - 12-21-2022
As we know that ,All component configurations are classified broadly into 3 sections-
1)basic
2)Meta_info
3)Resource-configuration
*Basically Enrichment component helps users to enrich the incoming data from in-event by querying lookup table in RDBMS and MongoDB and currently it supports 5 drivers :-
1)MySql
2)Ms-SQL
3)MongoDB
4)Postgres
5)Oracle
It has to be configure in a way that-suppose we are reading a data from mongo collection through pymongo reader and joining with one more table of mysql by Enrichment component,so in both Pymongo reader & Enrichment component we need to specify database name and table name which we are gonna join.
*During the Component configuration,There is one option in Meta_info, Conditions: Select conditions type (Remove or Blank option)
Remove [works like an inner join]– This option completely removes the row If any row of incoming data is not matching with query condition of the lookup table. Only passes the matching records.
Blank – This option sets blank for the cell of a row if complete incoming data from the in-event and matching rows from the lookup table.
"@data." is used to refer to the data coming from the previous event. For e.g.: If the data coming from the previous event is department_id and the column in the master table is dept_id then in the where clause we write as
Query which has to be entered in a format:-
SELECT department_name,location_id FROM departments WHERE department_id = @data.department_id = Valid
Asterisk (*) is not valid in the select statement
SELECT * FROM departments WHERE department_id = @data.department_id = Invalid