Welcome, Guest |
You have to register before you can post on our site.
|
Latest Threads |
What is the maximum numbe...
Forum: BDB Designer Q & A
Last Post: sariga.vr@bdb.ai
12-28-2022, 07:59 AM
» Replies: 0
» Views: 8,004
|
Inbuilt Capability of VC...
Forum: BDB - Platform
Last Post: shivani.jaipuria
12-27-2022, 05:23 AM
» Replies: 0
» Views: 1,179
|
Can dataset/cube refresh...
Forum: BDB - Platform
Last Post: shivani.jaipuria
12-27-2022, 05:08 AM
» Replies: 0
» Views: 1,216
|
How to load business stor...
Forum: BDB Designer Q & A
Last Post: sariga.vr@bdb.ai
12-26-2022, 04:47 PM
» Replies: 0
» Views: 3,167
|
How to load business stor...
Forum: BDB Designer Q & A
Last Post: sariga.vr@bdb.ai
12-26-2022, 04:46 PM
» Replies: 0
» Views: 3,225
|
How to load business stor...
Forum: BDB Designer Q & A
Last Post: sariga.vr@bdb.ai
12-26-2022, 04:45 PM
» Replies: 0
» Views: 2,244
|
How to load business stor...
Forum: BDB Designer Q & A
Last Post: sariga.vr@bdb.ai
12-26-2022, 04:44 PM
» Replies: 0
» Views: 2,222
|
Data Preparation operati...
Forum: BDB-Data Prep & ETL
Last Post: shivani.jaipuria
12-26-2022, 10:09 AM
» Replies: 0
» Views: 1,197
|
Plugability Feature of B...
Forum: BDB Platform Q & A
Last Post: shivani.jaipuria
12-26-2022, 08:32 AM
» Replies: 0
» Views: 1,079
|
How to use environment va...
Forum: BDB Platform Q & A
Last Post: archana
12-26-2022, 05:57 AM
» Replies: 0
» Views: 1,080
|
|
|
How to use ENRICHMENT component in Data pipeline? |
Posted by: mohd.gulam - 12-21-2022, 07:13 AM - Forum: BDB Data Pipeline Q & A
- No Replies
|
|
As we know that ,All component configurations are classified broadly into 3 sections-
1)basic
2)Meta_info
3)Resource-configuration
*Basically Enrichment component helps users to enrich the incoming data from in-event by querying lookup table in RDBMS and MongoDB and currently it supports 5 drivers :-
1)MySql
2)Ms-SQL
3)MongoDB
4)Postgres
5)Oracle
It has to be configure in a way that-suppose we are reading a data from mongo collection through pymongo reader and joining with one more table of mysql by Enrichment component,so in both Pymongo reader & Enrichment component we need to specify database name and table name which we are gonna join.
*During the Component configuration,There is one option in Meta_info, Conditions: Select conditions type (Remove or Blank option)
Remove [works like an inner join]– This option completely removes the row If any row of incoming data is not matching with query condition of the lookup table. Only passes the matching records.
Blank – This option sets blank for the cell of a row if complete incoming data from the in-event and matching rows from the lookup table.
"@data." is used to refer to the data coming from the previous event. For e.g.: If the data coming from the previous event is department_id and the column in the master table is dept_id then in the where clause we write as
Query which has to be entered in a format:-
SELECT department_name,location_id FROM departments WHERE department_id = @data.department_id = Valid
Asterisk (*) is not valid in the select statement
SELECT * FROM departments WHERE department_id = @data.department_id = Invalid
|
|
|
Script in connection and Components |
Posted by: neeraja.pattathil@bdb.ai - 12-21-2022, 06:19 AM - Forum: BDB Designer Q & A
- No Replies
|
|
What is the difference between writing a script in connection and components?
If the script is in connection, then it gets triggered when the connection is called.
If the script is in components, it will get executed only by click or interaction with the component.
|
|
|
How to write data to a sql table using python? |
Posted by: shreekantgosavi - 12-21-2022, 05:27 AM - Forum: BDB Data Pipeline Q & A
- No Replies
|
|
Use python script component.
Paste below script in meta information script section.
Provide connection credentials.
import pandas as pd
from sqlalchemy import create_engine
def sql_write(data):
# Connect to the database
engine = create_engine('mysql+pymysql://user:password@host/database')
# Store the data in the database
data.to_sql('table_name', engine, if_exists='replace')
|
|
|
Write data to a MongoDB collection using python |
Posted by: shreekantgosavi - 12-21-2022, 05:15 AM - Forum: BDB Data Pipeline Q & A
- No Replies
|
|
Use python script component.
Paste below script in meta information script section.
import pandas as pd
from pymongo import MongoClient
def myconnect(data):
df = data
client = MongoClient("mongodb://localhost:27017/") #connection_string
mydatabase = client.db_name #dbname
mycollection = mydatabase.collection_name #collection_name
mycollection.insert_many(df.to_dict('records'))
client.close()
|
|
|
How to read a csv file from SFTP using python? |
Posted by: shreekantgosavi - 12-21-2022, 05:10 AM - Forum: BDB Data Pipeline Q & A
- No Replies
|
|
Using python script component.
Paste below script in meta information script section.
Provide connection credentials.
import pandas as pd
import paramiko
def myconnect():
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('localhost',port=22,username='user',password='123456')
#print("connected suceessfully to SFTP Server")
sftp_client = ssh.open_sftp()
sftp_client.chdir('/home/user/folder_name')
df= pd.read_csv(sftp_client.open('/home/user/folder_name/file_name.csv'))
sftp_client.close()
ssh.close()
return df
|
|
|
How to read a MongoDB collection using python in pipeline? |
Posted by: shreekantgosavi - 12-21-2022, 04:57 AM - Forum: BDB Data Pipeline Q & A
- No Replies
|
|
Use python script component.
Under meta information paste below script in script section.
Provide required credentials.
import pandas as pd
from pymongo import MongoClient
def myread():
client = MongoClient("mongodb://localhost:27017/") #connection_string
mydatabase = client.db_name#cdatabase_name
mycollection = mydatabase.collection_name #collection_name
cursor = mycollection.aggregate([ { '$project': { '_id': 0}}]) #aggregate_query
df = pd.DataFrame(list(cursor))
client.close()
return df
myread()
|
|
|
How to read a sql table using python in pipeline? |
Posted by: shreekantgosavi - 12-21-2022, 04:50 AM - Forum: BDB Data Pipeline Q & A
- No Replies
|
|
Paste below script in meta information script section.
Provide required credentials.
import pandas as pd
from sqlalchemy import create_engine
def read_data():
engine = create_engine('mysql://root:root@localhost/db')
#dbsystem://username:password@host/db
df = pd.read_sql('SELECT * from table_name', con=engine)
return df
|
|
|
|