Welcome, Guest
You have to register before you can post on our site.

Username/Email:
  

Password
  





Search Forums

(Advanced Search)

Forum Statistics
» Members: 58,736
» Latest member: MarianaKra
» Forum threads: 389
» Forum posts: 395

Full Statistics

Latest Threads
What is the maximum numbe...
Forum: BDB Designer Q & A
Last Post: sariga.vr@bdb.ai
12-28-2022, 07:59 AM
» Replies: 0
» Views: 829
Inbuilt Capability of VC...
Forum: BDB - Platform
Last Post: shivani.jaipuria
12-27-2022, 05:23 AM
» Replies: 0
» Views: 599
Can dataset/cube refresh...
Forum: BDB - Platform
Last Post: shivani.jaipuria
12-27-2022, 05:08 AM
» Replies: 0
» Views: 566
How to load business stor...
Forum: BDB Designer Q & A
Last Post: sariga.vr@bdb.ai
12-26-2022, 04:47 PM
» Replies: 0
» Views: 730
How to load business stor...
Forum: BDB Designer Q & A
Last Post: sariga.vr@bdb.ai
12-26-2022, 04:46 PM
» Replies: 0
» Views: 741
How to load business stor...
Forum: BDB Designer Q & A
Last Post: sariga.vr@bdb.ai
12-26-2022, 04:45 PM
» Replies: 0
» Views: 690
How to load business stor...
Forum: BDB Designer Q & A
Last Post: sariga.vr@bdb.ai
12-26-2022, 04:44 PM
» Replies: 0
» Views: 674
Data Preparation operati...
Forum: BDB-Data Prep & ETL
Last Post: shivani.jaipuria
12-26-2022, 10:09 AM
» Replies: 0
» Views: 579
Plugability Feature of B...
Forum: BDB Platform Q & A
Last Post: shivani.jaipuria
12-26-2022, 08:32 AM
» Replies: 0
» Views: 572
How to use environment va...
Forum: BDB Platform Q & A
Last Post: archana
12-26-2022, 05:57 AM
» Replies: 0
» Views: 592

 
  How to use ENRICHMENT component in Data pipeline?
Posted by: mohd.gulam - 12-21-2022, 07:13 AM - Forum: BDB Data Pipeline Q & A - No Replies

As we know that ,All component configurations are classified broadly into 3 sections-
1)basic
2)Meta_info 
3)Resource-configuration

*Basically Enrichment  component helps users to enrich the incoming data from in-event by querying lookup table in RDBMS and MongoDB and currently it supports 5 drivers :-
1)MySql
2)Ms-SQL
3)MongoDB
4)Postgres
5)Oracle

It has to be configure in a way that-suppose we are reading a data from mongo collection through pymongo reader and joining with one more table of mysql by Enrichment component,so in both Pymongo reader & Enrichment component we need to specify database name and table name which we are gonna join.

*During the Component configuration,There is one option in Meta_info, Conditions: Select conditions type (Remove or Blank option)

Remove [works like an inner join]– This option completely removes the row If any row of incoming data is not matching with query condition of the lookup table. Only passes the matching records.

Blank – This option sets blank for the cell of a row if complete incoming data from the in-event and matching rows from the lookup table.

"@data." is used to refer to the data coming from the previous event. For e.g.: If the data coming from the previous event is department_id and the column in the master table is dept_id then in the where clause we write as

Query which has to be entered in a format:-

SELECT department_name,location_id FROM departments WHERE department_id = @data.department_id     = Valid

Asterisk (*) is not valid in the select statement
SELECT * FROM departments WHERE department_id = @data.department_id       = Invalid

Print this item

  Script in connection and Components
Posted by: neeraja.pattathil@bdb.ai - 12-21-2022, 06:19 AM - Forum: BDB Designer Q & A - No Replies

What is the difference between writing a script in connection and components?


If the script is in connection, then it gets triggered when the connection is called.

If the script is in components, it will get executed only by click or interaction with the component.

Print this item

  How to read data from web url in pipeline using python?
Posted by: shreekantgosavi - 12-21-2022, 05:30 AM - Forum: BDB Data Pipeline Q & A - No Replies

Use python script component.
Paste below script in meta information script section.
Provide URL where csv file is located.

import pandas as pd
def read_data():
   url = 'http://example.com/path/to/file.csv'
   df = pd.read_csv(url)
   return df

Print this item

  How to write data to a sql table using python?
Posted by: shreekantgosavi - 12-21-2022, 05:27 AM - Forum: BDB Data Pipeline Q & A - No Replies

Use python script component.
Paste below script in meta information script section.
Provide connection credentials.

import pandas as pd
from sqlalchemy import create_engine
def sql_write(data):

   # Connect to the database
   engine = create_engine('mysql+pymysql://user:password@host/database')

   # Store the data in the database
   data.to_sql('table_name', engine, if_exists='replace')

Print this item

  Write data to a MongoDB collection using python
Posted by: shreekantgosavi - 12-21-2022, 05:15 AM - Forum: BDB Data Pipeline Q & A - No Replies

Use python script component.
Paste below script in meta information script section.

import pandas as pd
from pymongo import MongoClient

def myconnect(data):
  df = data
  client = MongoClient("mongodb://localhost:27017/") #connection_string
  mydatabase = client.db_name #dbname
  mycollection = mydatabase.collection_name #collection_name
 
  mycollection.insert_many(df.to_dict('records'))
  client.close()

Print this item

  How to read a csv file from SFTP using python?
Posted by: shreekantgosavi - 12-21-2022, 05:10 AM - Forum: BDB Data Pipeline Q & A - No Replies

Using python script component.
Paste below script in meta information script section.
Provide connection credentials.

import pandas as pd
import paramiko
def myconnect():
   ssh = paramiko.SSHClient()
   ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
   ssh.connect('localhost',port=22,username='user',password='123456')
   #print("connected suceessfully to SFTP Server")
   sftp_client = ssh.open_sftp()
   sftp_client.chdir('/home/user/folder_name')
   df= pd.read_csv(sftp_client.open('/home/user/folder_name/file_name.csv'))
   sftp_client.close()
   ssh.close()
   return df

Print this item

  How to read a MongoDB collection using python in pipeline?
Posted by: shreekantgosavi - 12-21-2022, 04:57 AM - Forum: BDB Data Pipeline Q & A - No Replies

Use python script component.
Under meta information paste below script in script section.
Provide required credentials.
import pandas as pd
from pymongo import MongoClient
def myread():
    client = MongoClient("mongodb://localhost:27017/"#connection_string
    mydatabase = client.db_name#cdatabase_name
    mycollection = mydatabase.collection_name #collection_name
    cursor = mycollection.aggregate([ { '$project': { '_id': 0}}]) #aggregate_query
    df = pd.DataFrame(list(cursor))
    client.close()
    return df
myread()

Print this item

  How to read a sql table using python in pipeline?
Posted by: shreekantgosavi - 12-21-2022, 04:50 AM - Forum: BDB Data Pipeline Q & A - No Replies

Paste below script in meta information script section.

Provide required credentials.

import pandas as pd

from sqlalchemy import create_engine
def read_data():
   engine = create_engine('mysql://root:root@localhost/db')
                       #dbsystem://username:password@host/db
   df = pd.read_sql('SELECT * from table_name', con=engine)
   return df 
 

Print this item

  Notify GV Update
Posted by: neeraja.pattathil@bdb.ai - 12-20-2022, 01:15 PM - Forum: BDB Designer Q & A - No Replies

What happens when we check Notify GV Update in the filter?


It executes the script written inside the components

Print this item

  Update Global variable of component
Posted by: neeraja.pattathil@bdb.ai - 12-20-2022, 01:13 PM - Forum: BDB Designer Q & A - No Replies

What happens when we click on the check box of Auto Update GV in the filter?


It updates the global variables of the selected index value

Print this item