Copy Data tool makes it very easy to bring your data to Azure Synapse. Over the last few decades, data has been the backbone of many of the world’s most successful businesses. You can now import data from a variety of sources into Azure Synapse using a variety of methods and begin analyzing your data right away.
Prerequisites
There are a few requirements that must be met before we can start.
- You should have your Azure subscription or contributor-level access to any other subscription.
- With this subscription, you can create your Synapse workspace.
- You must have SQL Server installed on your machine.
Bring Your Data to Azure Synapse Using Copy Data tool
Data ingestion is one of the most important components of data analytics, and there are numerous data-moving tools available. The task at hand is to determine which tool is most effective in your particular situation.
Under the Orchestrate tab, there are two options: the first is Pipeline, which may be used for data ingestion as well as adding transformation logic. The Copy Data tool, on the other hand, allows you to simply relocate data without creating any data transformation logic.
In a summary, if you only need a pipeline for data intake, you can use Copy Data, but if you need to add business logic to your data, you should utilize Pipeline.
Steps To Bring Your Data to Azure Synapse
1) As seen in the image, select the Copy Data tool. This will create a new window where you must enter the connection details for the source and destination.
2) Select copy data task type and configure task schedule. You may either run this pipeline once or plan it to run on a regular basis. We’re going to schedule our pipeline to run once now in this example.
3) You must make a new connection for your data source on the next screen. To make a new connection, click + Create new connection:
4) We’ll utilize an on-premises SQL Server as our data source (SQL Server installed on your local machine or on-premises server), so choose SQL Server from the list of potential sources. You can, however, choose the supplier based on your business needs. Find and pick the appropriate data source, then click Continue.
5) You must now supply information in order to construct a linked service for the data source. Give the associated service a meaningful name and a description.
6) To connect the pipeline to the on-premises SQL Server, you must first set up an integration runtime.
The compute infrastructure utilized by Azure Data Factory or Synapse pipelines to enable data movement, data flow, activity dispatch, and SQL Server Integration Services (SSIS) package execution capabilities across various network settings is known as the integration runtime.
Self-Hosted and Azure integration runtimes are the two types of integration runtimes. We’ll use the self-hosted integration runtime in this part, which is mostly used for conducting activities in an on-premises or private network. Select +New:
7) Click Continue after selecting Self-Hosted. This will bring up a new form where you can give the integration runtime a name and a description:
8) Under the Integration runtime setup window, give it a name and a description, then click Continue:
9) After that, you may choose the integration runtime setup. In this phase, we will choose a Manual setup. You can, however, go with option 1 for a quick setup:
10) Option 2: Click on the Download and install integration runtime link. This will redirect you to a new URL where you can download the integration runtime.
11) Double-click the file after it has been downloaded to launch the installation procedure on your server. To finish the setup, follow the wizard’s instructions:
12) Before your integration runtime setup is complete, this setup process will take a few minutes. Then copy Key1 and Key2 and paste them into the integration runtime box to finish the configuration. An integration runtime can be shown running on a local PC in the following snapshot, where we’ve pasted Key1 from the Synapse pipeline:
13) Return to the Copy Data tool to provide connection information for your on-premises SQL Server. Fill in the information for the server name, database name, and database credentials, then click Create.
14) To proceed to the next screen, click Next.
15) Select a table/view from the dropdown to replicate the data, or use your custom query and click Next:
16) On the following screen, you can add a filter to your dataset, but we’ll skip it and continue on to the next stage, which is to define the target’s connection:
17) For the target, you must now construct linked services. Click Next after searching for Synapse in the search window and selecting Synapse Analytics from the results.
18) Provide the server name, database name, username, and password for the SQL pool in your Azure Synapse version. After you’ve filled in all of the information, click Create.
19) To create a table mapping between the source and target, click Next. If your target doesn’t have a table, click the Auto-create a destination table with the source schema link and then Next for column mapping:
20) All of the columns from the source are mapped to the columns at the destination. You’ll see that the column mappings populate automatically if the source and destination schemas are identical.
21) Under Performance settings, select the Bulk insert radio button, leave the other fields at their default values, and then select Next to go to the Summary page.
22) On the Summary page, review all of the details before clicking Next to start deploying your pipeline. Your pipeline will be constructed in a matter of minutes, and you’ll be able to use it to send data to Synapse.
23) Result(Deployment Complete).
Despite the fact that there isn’t much of a difference between Synapse and Data Factory pipelines, it’s still worth looking into.
Related/References
- Microsoft Certified Azure Data Engineer Associate | DP 203 | Step By Step Activity Guides (Hands-On Labs)
- Exam DP-203: Data Engineering on Microsoft Azure
- Azure Data Lake For Beginners: All you Need To Know
- Batch Processing Vs Stream Processing: All you Need To Know
- Introduction to Big Data and Big Data Architectures
Next Task For You
In our Azure Data Engineer training program, we will cover all the exam objectives, 27 Hands-On Labs, and practice tests. If you want to begin your journey towards becoming a Microsoft Certified: Azure Data Engineer Associate by checking our FREE CLASS.
The post Bring Your Data to Azure Synapse | Data Engineering on Azure appeared first on Cloud Training Program.