In this article we are going to use YAML pipelines for doing the deployment of synapse code along with customization of input parameters which can help you create the deployment dynamic.
We will also display how to start and stop the triggers which will play important role in your synapse deployment.
Linked services
A workspace can contain any number of Linked service, essentially connection strings that define the connection information needed for the workspace to connect to external resources.
Synapse SQL
Synapse SQL is the ability to do T-SQL based analytics in Synapse workspace. Synapse SQL has two consumption models: dedicated and server less. For the dedicated model, use dedicated SQL pools. A workspace can have any number of these pools. To use the server less model, use the server less SQL pools. Every workspace has one of these pools. Inside Synapse Studio, you can work with SQL pools by running SQL scripts.
Apache Spark for Synapse
To use Spark analytics, create and use server less Apache Spark pools in your Synapse workspace. When you start using a Spark pool, the workspaces creates a spark session to handle the resources associated with that session.
Pipelines
Pipelines are how Azure Synapse provides Data Integration - allowing you to move data between services and orchestrate activities.
Source: Please refer this link to know more https://docs.microsoft.com/en-us/azure/synapse-analytics/overview-terminology
Git Integration in Synapse Workspace (Continuous Integration):
For this section you can refer to an existing tech blog from here.
Pre-requisites before Release to higher environments:
1.Make sure you have the 'Synapse Workspace Deployment' extension installed from visual studio marketplace in the organizational settings.
2.Make sure appropriate permissions are given to service connection (used for Azure DevOps Deployment Pipelines) in the Synapse Workspace as Synapse Administrator. (Refer below screenshot)
Stopping and Starting Pipeline Triggers before and after the Deployment:
#starting the triggers
- task: AzurePowerShell@5
displayName: Stop Triggers
inputs:
azureSubscription: '$(azureSubscription)'
ScriptType: 'InlineScript'
Inline: |
Install-Module -Name "Az.Synapse" -Confirm:$false -Scope CurrentUser -Force;
$triggersSynapse = Get-AzSynapseTrigger -WorkspaceName "$(DeployWorkspaceName)" ;
$triggersSynapse | ForEach-Object { Stop-AzSynapseTrigger -WorkspaceName "$(DeployWorkspaceName)" -Name $_.name }
azurePowerShellVersion: 'LatestVersion'
#stopping the triggers
- task: AzurePowerShell@5
displayName: Restart Triggers
inputs:
azureSubscription: '$(azureSubscription)'
ScriptType: 'InlineScript'
Inline: '$triggersSynapse = Get-AzSynapseTrigger -WorkspaceName "$(DeployWorkspaceName)" ; $triggersSynapse | ForEach-Object { Start-AzSynapseTrigger -WorkspaceName "$(DeployWorkspaceName)" -Name $_.name }'
azurePowerShellVersion: 'LatestVersion'
The following are some guidelines for creating the custom parameters file:
{
"Microsoft.Synapse/workspaces/notebooks": {
"properties":{
"bigDataPool":{
"referenceName": "="
}
}
},
"Microsoft.Synapse/workspaces/sqlscripts": {
"properties": {
"content":{
"currentConnection":{
"*":"-"
}
}
}
},
"Microsoft.Synapse/workspaces/pipelines": {
"properties": {
"activities": [{
"typeProperties": {
"waitTimeInSeconds": "-::int",
"headers": "=::object"
}
}]
}
},
"Microsoft.Synapse/workspaces/integrationRuntimes": {
"properties": {
"typeProperties": {
"*": "="
}
}
},
"Microsoft.Synapse/workspaces/triggers": {
"properties": {
"typeProperties": {
"recurrence": {
"*": "=",
"interval": "=:triggerSuffix:int",
"frequency": "=:-freq"
},
"maxConcurrency": "="
}
}
},
"Microsoft.Synapse/workspaces/linkedServices": {
"*": {
"properties": {
"typeProperties": {
"*": "="
}
}
},
"AzureDataLakeStore": {
"properties": {
"typeProperties": {
"dataLakeStoreUri": "="
}
}
}
},
"Microsoft.Synapse/workspaces/datasets": {
"properties": {
"typeProperties": {
"*": "="
}
}
}
}
Here's an explanation of how the preceding template is constructed, broken down by resource type.
-<parameter-overridden> : <value-to-be-overridden>
-<parameter-overridden> : <value-to-be-overridden>
CD YAML Code
name: Release-$(rev:r)
trigger:
branches:
include:
- workspace_publish
paths:
include:
- '<source workspace name>/*'
resources:
repositories:
- repository: <repo name>
type: git
name: <repo name>
ref: workspace_publish
variables:
- name: azureSubscription
value: '<name of service connection>'
- name: vmImageName
value: 'windows-2019'
- name: KeyVaultName
value: '<Kv Name>'
- name: SourceWorkspaceName
value: '<Source workspace name>'
- name: DeployWorkspaceName
value: '<Deployment Workspace name>'
- name: DeploymentResourceGroupName
value: '<Deployment Workspace Name>'
stages:
- stage: Release
displayName: Release stage
jobs:
- job: Release
displayName: Release job
pool:
vmImage: $(vmImageName)
steps:
- task: AzureKeyVault@1
inputs:
azureSubscription: '$(azureSubscription)'
KeyVaultName: $(KeyVaultName)
SecretsFilter: '<Secrets needed to override template parameters>'
- checkout: <repo name>
- task: AzurePowerShell@5
displayName: Stop Triggers
inputs:
azureSubscription: '$(azureSubscription)'
ScriptType: 'InlineScript'
Inline: "Install-Module -Name \"Az.Synapse\" -Confirm:$false -Scope CurrentUser -Force;\n$triggersSynapse = Get-AzSynapseTrigger -WorkspaceName \"$(DeployWorkspaceName)\" ; \n$triggersSynapse | ForEach-Object { Stop-AzSynapseTrigger -WorkspaceName \"$(DeployWorkspaceName)\" -Name $_.name }\n"
azurePowerShellVersion: 'LatestVersion'
- task: Synapse workspace deployment@1
inputs:
TemplateFile: '$(System.DefaultWorkingDirectory)/$(SourceWorkspaceName)/TemplateForWorkspace.json'
ParametersFile: '$(System.DefaultWorkingDirectory)/$(SourceWorkspaceName)/TemplateParametersForWorkspace.json'
azureSubscription: '$(azureSubscription)'
ResourceGroupName: '$(DeploymentResourceGroupName)'
TargetWorkspaceName: '$(DeployWorkspaceName)'
DeleteArtifactsNotInTemplate: true
OverrideArmParameters: |
workspaceName: $(DeployWorkspaceName)
#<parameter-overridden> : <value-to-be-overridden> there are parameters in arm template
#<parameter-overridden> : <value-to-be-overridden>
Environment: 'prod'
- task: AzurePowerShell@5
displayName: Restart Triggers
inputs:
azureSubscription: '$(azureSubscription)'
ScriptType: 'InlineScript'
Inline: '$triggersSynapse = Get-AzSynapseTrigger -WorkspaceName "$(DeployWorkspaceName)" ; $triggersSynapse | ForEach-Object { Start-AzSynapseTrigger -WorkspaceName "$(DeployWorkspaceName)" -Name $_.name }'
azurePowerShellVersion: 'LatestVersion'
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.