Data Engineering - ETL, Web Scraping,Big Data,SQL,Power BI

A common problem that organizations face is how to gathering data from multiple sources, in multiple formats, and move it to one or more data stores. The destination may not be the same type of data store as the source, and often the format is different, or the data needs to be shaped or cleaned before loading it into its final destination. Extract, transform, and load (ETL) is a data pipeline used to collect data from various sources, transform the data according to business rules, and load it into a destination data store.

SQL Server Integration Services (SSIS) is a useful and powerful Business Intelligence Tool. It is best suited to work with SQL Server Database. It is added to SQL Server Database when you install SQL Server Data Tools (SSDT)which adds the Business Intelligence Templates to Visual studio that is used to create Integration projects.

SSIS can be used for: Data Integration Data Transformation Providing solutions to complex Business problems Updating data warehouses Cleaning data Mining data Managing SQL Server objects and data Extracting data from a variety of sources Loading data into one or several destinationsWeb scraping is the process of automatically downloading a web page's data and extracting specific information from it. The extracted information can be stored in a database or as various file types. Web scraping software tools may access the World Wide Web directly using the Hypertext Transfer Protocol, or through a web browser.

While web scraping can be done manually by a software user, the term typically refers to automated processes implemented using a bot or web crawler. It is a form of copying, in which specific data is gathered and copied from the web, typically into a central local database or spreadsheet, for later retrieval or analysis. Scraping a web page involves fetching it and extracting from it.