diff --git a/docs/180-microsoft-integration/020-purview.md b/docs/180-microsoft-integration/020-purview.md index ba69c5ad8..bf4b6af45 100644 --- a/docs/180-microsoft-integration/020-purview.md +++ b/docs/180-microsoft-integration/020-purview.md @@ -25,6 +25,7 @@ The following table provides a list of integration features. | Synchronize Purview Glossary Terms To CluedIn Vocabularies | Create Purview glossary terms from CluedIn vocabulary. | Synchronize Streams | Create and update "Cluedin Entity", "Cluedin Stream Process", and "Cluedin Organization Provider" entities on Purview lineages that link CluedIn streams and connectors. | Synchronize Crawlers And Enrichers | Create and update "Cluedin Organization Provider", "Cluedin Crawl Process", "Cluedin Enrich Process", "Cluedin Ingest Process", "Cluedin Dataset", "Cluedin Map Process", and "Cluedin Entity" entities on Purview lineages that link Purview assets to CluedIn data sets and matching CluedIn Entity Types. | +| Synchronize Data Products | Create CluedIn DataSources from Purview Data Products and Data Assets. | We have the following **assumptions** about the customers' Microsoft Azure setup: diff --git a/docs/180-microsoft-integration/purview/010-intro.md b/docs/180-microsoft-integration/purview/010-intro.md deleted file mode 100644 index b280244bc..000000000 --- a/docs/180-microsoft-integration/purview/010-intro.md +++ /dev/null @@ -1,100 +0,0 @@ ---- -layout: cluedin -title: Features -parent: Microsoft Purview Integration -grand_parent: Microsoft Integration -permalink: /microsoft-integration/purview/introduction -nav_order: 020 -has_children: false -tags: ["integration", "microsoft", "azure", "purview"] ---- -## On this page -{: .no_toc .text-delta } -- TOC -{:toc} - -This article describes the Microsoft Purview integration features. - -## Sync CluedIn Data Sources - -When this feature is enabled, CluedIn fetches all Microsoft Purview asset entities from Purview to create data source groups and their respective data sources. The data source groups can be viewed in **Integrations** > **Data Sources**. The CluedIn Purview integration components create Purview assets under a single root collection. - -CluedIn creates a lineage when one or more data sets are created within a data source associated with a Purview asset entity previously created via the **Sync CluedIn Data Sources** feature. The **Ingest Data** process displays data flow from the Microsoft Purview asset entity to a newly created or updated Microsoft Purview data set entity. The Microsoft Purview data set entity represents the CluedIn data set with its populated column names. - -Data sets in CluedIn with a mapping of at least one property to a CluedIn entity type results in both the CluedIn entity and the **Map to Entity** process being created in Purview. The **Map to Entity** process connects the CluedIn data set to the CluedIn entity type under the assets lineage tab. - -The following image shows an example of a data set lineage. - -![Example of a Data Set lineage](./media/dataset_lineage.png) - -Background processes in CluedIn detect changes in CluedIn data sets and their respective mapping. These changes are synchronized with the existing Microsoft Purview data set assets. - -## Auto-Map CluedIn Data Sets - -This feature auto-maps data sets to a vocabulary matching the Purview asset's glossary term. The feature applies to data sets tied to Purview asset and has Purview glossary terms assigned to either the Purview asset itself or the schema columns. - -The Purview glossary terms used by the Purview assets must first be added as CluedIn vocabularies. If the vocabulary (made from Purview glossary terms) is available, the data set is automatically mapped to the right vocabulary. - -If the CluedIn data set column name matches an existing vocabulary key of the given vocabulary, the vocabulary key data type is used instead of the Purview entity's schema. If the CluedIn data set column name does not match any existing vocabulary key of the given vocabulary, it will create a new vocabulary key for the vocabulary. The data type of the vocabulary key is determined from the Purview entity's schema. - -The default data type is text when no suitable data type in CluedIn is found. - -Sync interval applies to this feature. - -## Poll CluedIn Data Sources - -This feature differs from **Sync CluedIn Data Sources** by updating the existing data set entities on Microsoft Purview lineages without having to sync new data sources on Microsoft Purview back to CluedIn. Data quality metrics for the associated data sources in CluedIn are also synced back to Microsoft Purview lineages this way. - -## Sync CluedIn Streams - -A background processing in CluedIn synchronizes streams and their respective export connector as assets in Purview. These assets show outbound lineage from CluedIn entity types to the export target. - -## Monitor Events - -CluedIn updates Microsoft Purview entities when specific actions are carried out by a user in CluedIn. The supported actions are as follows: - -- Adding or removing a CluedIn data source or enrichers. - -- Adding, updating, reprocessing or removing a CluedIn stream. - -- Adding, updating or removing a CluedIn export target. - -## Sync Purview glossaries to CluedIn vocabularies - -This synchronization feature allows the import of Microsoft Purview glossaries as CluedIn vocabularies. If there are matching CluedIn vocabularies, they will be updated; otherwise, new CluedIn vocabularies are created for the incoming Microsoft Purview glossaries. - -## Sync CluedIn Vocabularies to Purview Glossary Terms - -![Purview glossary terms created under the root glossary term "CluedIn" + [Your machine name], ex. "CluedIn-Dell"](./media/vocab_to_glossary.png) - -Purview glossary terms are created under the root glossary term "CluedIn" + [Your machine name] (e,g. "CluedIn-Dell"). - -By default, a CluedIn root glossary term is created in Purview glossaries. New glossary terms are created under the root glossary term. - -If a glossary term is deleted but the vocabulary or vocabulary key still exists, the sync re-creates the glossary term. If a vocabulary or vocabulary key is deleted, the linked glossary term is removed, except for glossary terms with assigned entities. If a newly created vocabulary or vocabulary key matches an existing glossary term, the name of the new glossary term is appended with numbers. - -## Sync Purview Glossary Terms to CluedIn Glossary Terms - -When this feature is enabled, a CluedIn glossary category with the default name **Purview** is automatically created. The glossary category name is configurable in **Organization** > **Settings**. All Purview glossary terms (excluding names containg "CluedIn-" prefix) are synced under this glossary category. New glossary terms created under this glossary category are ignored. - -## Sync CluedIn Crawlers and Enrichers - -This feature will create or update existing crawlers and enricher lineages in Purview. The **DataSource** provider types (data imported via files, endpoints, or databases) are handled by the **Sync CluedIn Data Sources** feature. - -The following image shows an example of a crawler lineage. - -![Example of a Crawler lineage](./media/crawler_lineage.png) - -When a Crawler imports clues into CluedIn, this feature creates a lineage from the Crawler provider to the entity types of the CluedIn entities via the **Crawl** process. - -The following image shows an example of an enricher lineage. - -![Example of a Enricher lineage](./media/enricher_lineage.png) - -When an enricher enriches an entity, this feature creates a lineage from the enricher provider to the entity types of the CluedIn entities via the **Enrich** process. - -## Azure Data Factory pipeline Automation - -This job will automatically create your ADF pipeline and execution to ingest data from Purview asset source to CluedIn ingestion endpoint (CluedIn ADF Copy Data Automation). - -![CluedIn ADF Copy Data Automation Dataset](./media/adf_copy.png) diff --git a/docs/180-microsoft-integration/purview/010-setup-credentials.md b/docs/180-microsoft-integration/purview/010-setup-credentials.md new file mode 100644 index 000000000..858176f04 --- /dev/null +++ b/docs/180-microsoft-integration/purview/010-setup-credentials.md @@ -0,0 +1,35 @@ +--- +layout: cluedin +title: Setup credentials +parent: Microsoft Purview Integration +grand_parent: Microsoft Integration +permalink: /microsoft-integration/purview/setup-credentials +nav_order: 010 +has_children: false +tags: ["integration", "microsoft", "azure", "purview", credentials] +--- +## On this page +{: .no_toc .text-delta } +- TOC +{:toc} + +## Credentials + +To connect CluedIn to Microsoft Purview, you need to provide the Microsoft Purview account and Service Principal account information. + +In CluedIn, on the navigation pane, go to **Administration** > **Settings**, and then scroll down to find the **Purview** section. + +### Enter Microsoft Purview credentials: + +- **Base URL** – `https://{accountName}.purview.azure.com` where `accountName` is your Purview subscription name. For example, if your Purview subscription name is ContosoPurview, then the base URL would be `https://contosopurview.purview.azure.com`. For more details about Purview subscription, see [Microsoft documentation](https://docs.microsoft.com/en-us/azure/purview/create-catalog-portal#open-the-microsoft-purview-governance-portal). +- Another way to find the Base URL value is by navigating to your Purview Account home page and viewing the **Resource JSON** + ![Input Microsoft Purview credentials](./media/purview-resource-json.png) +- **Client ID** – can be obtained from **Application (client) ID** on the **Overview** tab of the app registration that the organization has dedicated to access Purview account on behalf of CluedIn. + +- **Client Secret** – can be obtained from the **Certificates & secrets** of the app registration that the organization has dedicated to access Purview account on behalf of CluedIn. + +- **Tenant ID** – can be obtained from **Directory (tenant) ID** on the **Overview** tab of the app registration that the organization has dedicated to access Purview account on behalf of CluedIn. + + ![Input Microsoft Purview credentials](./media/settings.png) + + For more information on where to find the values for client ID and tenant ID, see [Microsoft documentation](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app#register-an-application). diff --git a/docs/180-microsoft-integration/purview/020-features/010-sync-datasources.md b/docs/180-microsoft-integration/purview/020-features/010-sync-datasources.md new file mode 100644 index 000000000..c2cd14e41 --- /dev/null +++ b/docs/180-microsoft-integration/purview/020-features/010-sync-datasources.md @@ -0,0 +1,39 @@ +--- +layout: cluedin +title: Sync Datasources +parent: Microsoft Purview Integration +grand_parent: Microsoft Integration +permalink: /microsoft-integration/purview/features/sync-datasources +nav_order: 010 +has_children: false +tags: ["integration", "microsoft", "azure", "purview", "collection", "sync", "datasources"] +--- +## On this page +{: .no_toc .text-delta } +- TOC +{:toc} + +## Sync Purview Assets as CluedIn DataSources + +![Settings Sync Datasources](../media/settings-sync-datasources.png) + +When this feature is enabled, CluedIn fetches all Microsoft Purview asset entities from Purview to create data source groups and their respective data sources. The data source groups can be viewed in **Integrations** > **Data Sources**. The CluedIn Purview integration components create Purview assets under a single root collection. + +![Sync Datasources to CluedIn](../media/sync-datasources-to-cluedin.png) + +The `Keywords` used from the above setting is the Glossary Term Name which is tagged into the Assets. + +![Sync Datasources Glossary Term](../media/sync-datasources-glossary-term.png) +![Sync Datasources Assets](../media/sync-datasources-assets.png) + +## Dataset Lineage + +CluedIn creates a lineage when one or more data sets are created within a data source associated with a Purview asset entity previously created via the **Sync CluedIn Data Sources** feature. The **Ingest Data** process displays data flow from the Microsoft Purview asset entity to a newly created or updated Microsoft Purview data set entity. The Microsoft Purview data set entity represents the CluedIn data set with its populated column names. + +Data sets in CluedIn with a mapping of at least one property to a CluedIn entity type result in both the CluedIn entity and the **Map to Entity** process being created in Purview. The **Map to Entity** process connects the CluedIn data set to the CluedIn entity type under the assets lineage tab. + +The following image shows an example of a data set lineage. + +![Example of a Data Set lineage](../media/dataset_lineage.png) + +Background processes in CluedIn detect changes in CluedIn data sets and their respective mapping. These changes are synchronized with the existing Microsoft Purview data set assets. diff --git a/docs/180-microsoft-integration/purview/020-features/020-auto-map.md b/docs/180-microsoft-integration/purview/020-features/020-auto-map.md new file mode 100644 index 000000000..dd9704e61 --- /dev/null +++ b/docs/180-microsoft-integration/purview/020-features/020-auto-map.md @@ -0,0 +1,30 @@ +--- +layout: cluedin +title: Auto-Map CluedIn DataSets +parent: Microsoft Purview Integration +grand_parent: Microsoft Integration +permalink: /microsoft-integration/purview/features/auto-map-datasets +nav_order: 020 +has_children: false +tags: ["integration", "microsoft", "azure", "purview", "collection", "auto-map", "dataset"] +--- +## On this page +{: .no_toc .text-delta } +- TOC +{:toc} + +## Auto-Map CluedIn Data Sets + +![Settings Auto-Map Datasets](../media/settings-auto-map-datasets.png) + +This feature auto-maps data sets to a vocabulary matching the Purview asset's glossary term. The feature applies to data sets tied to the Purview asset and has Purview glossary terms assigned to either the Purview asset itself or the schema columns. + +The Purview glossary terms used by the Purview assets must first be added as CluedIn vocabularies. If the vocabulary (made from Purview glossary terms) is available, the data set is automatically mapped to the right vocabulary. + +If the CluedIn data set column name matches an existing vocabulary key of the given vocabulary, the vocabulary key data type is used instead of the Purview entity's schema. If the CluedIn data set column name does not match any existing vocabulary key of the given vocabulary, it will create a new vocabulary key for the vocabulary. The data type of the vocabulary key is determined from the Purview entity's schema. + +The default data type is text when no suitable data type in CluedIn is found. + +Sync interval applies to this feature. + +## Requirements diff --git a/docs/180-microsoft-integration/purview/020-features/030-poll-datasources.md b/docs/180-microsoft-integration/purview/020-features/030-poll-datasources.md new file mode 100644 index 000000000..a1ec6cbc4 --- /dev/null +++ b/docs/180-microsoft-integration/purview/020-features/030-poll-datasources.md @@ -0,0 +1,21 @@ +--- +layout: cluedin +title: Poll Datasources +parent: Microsoft Purview Integration +grand_parent: Microsoft Integration +permalink: /microsoft-integration/purview/features/poll-datasources +nav_order: 030 +has_children: false +tags: ["integration", "microsoft", "azure", "purview", "collection", "poll", "datasources"] +--- +## On this page +{: .no_toc .text-delta } +- TOC +{:toc} + + +## Poll CluedIn Data Sources + +![Settings Poll Datasources](../media/settings-poll-datasources.png) + +This feature differs from **Sync CluedIn Data Sources** by updating the existing data set entities on Microsoft Purview lineages without having to sync new data sources on Microsoft Purview back to CluedIn. Data quality metrics for the associated data sources in CluedIn are also synced back to Microsoft Purview lineages this way. diff --git a/docs/180-microsoft-integration/purview/020-features/040-sync-crawlers-and-enrichers.md b/docs/180-microsoft-integration/purview/020-features/040-sync-crawlers-and-enrichers.md new file mode 100644 index 000000000..1eb588261 --- /dev/null +++ b/docs/180-microsoft-integration/purview/020-features/040-sync-crawlers-and-enrichers.md @@ -0,0 +1,34 @@ +--- +layout: cluedin +title: Sync CluedIn Crawlers and Enrichers +parent: Microsoft Purview Integration +grand_parent: Microsoft Integration +permalink: /microsoft-integration/purview/features/sync-crawlers-and-enrichers +nav_order: 040 +has_children: false +tags: ["integration", "microsoft", "azure", "purview", "collection", "sync", "crawlers", "enrichers"] +--- +## On this page +{: .no_toc .text-delta } +- TOC +{:toc} + + +## Sync CluedIn Crawlers and Enrichers + +![Settings Sync Crawlers and Enrichers](../media/settings-sync-crawlers-and-enrichers.png) + +This feature will create or update existing crawlers and enricher lineages in Purview. The **DataSource** provider types (data imported via files, endpoints, or databases) are handled by the **Sync CluedIn Data Sources** feature. + +The following image shows an example of a crawler lineage. + +![Example of a Crawler lineage](../media/crawler_lineage.png) + +When a Crawler imports clues into CluedIn, this feature creates a lineage from the Crawler provider to the entity types of the CluedIn entities via the **Crawl** process. + +The following image shows an example of an enricher lineage. + +![Example of an Enricher lineage](../media/enricher_lineage.png) + +When an enricher enriches an entity, this feature creates a lineage from the enricher provider to the entity types of the CluedIn entities via the **Enrich** process. + diff --git a/docs/180-microsoft-integration/purview/020-features/050-sync-streams.md b/docs/180-microsoft-integration/purview/020-features/050-sync-streams.md new file mode 100644 index 000000000..b1319c573 --- /dev/null +++ b/docs/180-microsoft-integration/purview/020-features/050-sync-streams.md @@ -0,0 +1,32 @@ +--- +layout: cluedin +title: Sync CluedIn Streams +parent: Microsoft Purview Integration +grand_parent: Microsoft Integration +permalink: /microsoft-integration/purview/features/sync-streams +nav_order: 050 +has_children: false +tags: ["integration", "microsoft", "azure", "purview", "collection", "sync", "streams"] +--- +## On this page +{: .no_toc .text-delta } +- TOC +{:toc} + +## Sync CluedIn Streams + +![Settings Sync Streams](../media/settings-sync-streams.png) + +A background processing in CluedIn synchronizes streams and their respective export connector as assets in Purview. These assets show outbound lineage from CluedIn entity types to the export target. + +Details of CluedIn Streams named as `Customer Golden Record` with Export Target configured with Target name `Customer` + +![CluedIn Streams](../media/cluedin-stream.png) + +Full Lineage of Purview Asset to CluedIn Stream + +![Full Lineage](../media/sync-streams-asset-lineage.png) + +CluedIn Stream created as a Process Asset and Export Target as the Entity Asset. + +![Stream and Export Target Asset](../media/sync-streams-process-and-export-target.png) diff --git a/docs/180-microsoft-integration/purview/020-features/060-sync-purview-glossaries-to-vocab.md b/docs/180-microsoft-integration/purview/020-features/060-sync-purview-glossaries-to-vocab.md new file mode 100644 index 000000000..c7e3b6666 --- /dev/null +++ b/docs/180-microsoft-integration/purview/020-features/060-sync-purview-glossaries-to-vocab.md @@ -0,0 +1,64 @@ +--- +layout: cluedin +title: Sync Purview Glossaries to CluedIn Vocabularies +parent: Microsoft Purview Integration +grand_parent: Microsoft Integration +permalink: /microsoft-integration/purview/features/sync-purview-glossaries-to-vocab +nav_order: 050 +has_children: false +tags: ["integration", "microsoft", "azure", "purview", "collection", "sync", "glossary", "vocabulary"] +--- +## On this page +{: .no_toc .text-delta } +- TOC +{:toc} + +This synchronization feature allows the import of Microsoft Purview glossaries as CluedIn vocabularies. If there are matching CluedIn vocabularies, they will be updated; otherwise, new CluedIn vocabularies are created for the incoming Microsoft Purview glossaries. + +## Syncronization Settings + +![Settings Sync Purview Glossaries to CluedIn Vocabularies](../media/settings-sync-purview-glossaries-to-vocab.png) + +#### **Sync Purview glossaries to CluedIn vocabularies** +When this feature is enabled, the job will start to look in Purview Glossaries and Create a CluedIn Vocabularies if the requirements have been met. + +#### **Glossary To Vocabulary Attribute Filter** + +Key Value pair of Property Name and Value to filter the Glossary Terms. `CluedInVocab` is an additional property from `CluedIn Template` Template. + +![Purview Glossary Term Setup](../media/purview-glossary-term-setup.png) + +#### **Glossary To Vocabulary Term Pattern** + +Regex Pattern value used to filter the Glossary Terms by Name or NickName + +#### **Glossary To Vocabulary Template Name Pattern** + +Regex Pattern value used to filter the Glossary Terms by Template Name + +#### **Vocabulary Prefix** + +Assign a custom value for the Vocabulary Key prefix. + +![Vocabulary Configuration Page - Prefix](../media/cluedin-vocabulary-configuration.png) + +#### **EntityType Prefix** + +Assign a custom value for the EntityType prefix for identifying the EntityType if it's created under Purview Integration. + +![EntityType Configuration Page - Prefix](../media/cluedin-entitytype-configuration.png) + +## Sample Previews +Created a Glossary Term `Customer` with Children Glossary Terms `Name`, `HouseholdId`, `Age`, `Id`, `Phone` and `Notes` + +![Settings Sync Purview Glossaries to CluedIn Vocabularies](../media/purview-child-glossary-terms.png) + +Assigned the Children Glossary Term to Purview Asset Schema + +![Settings Sync Purview Glossaries to CluedIn Vocabularies](../media/asset-glossary-term-mapping.png) + +When Synchronization is done, notice that `Customer` Vocabulary has been created with `Name` only as its Vocabulary Key. + +![Settings Sync Purview Glossaries to CluedIn Vocabularies](../media/cluedin-vocab-from-glossary-terms.png) + +The reason for this is that Glossary Terms `Customer` and `Name` are the only terms that have been assigned with `CluedInVocab=true` value in its property. \ No newline at end of file diff --git a/docs/180-microsoft-integration/purview/020-features/070-sync-vocab-to-purview-glossaries.md b/docs/180-microsoft-integration/purview/020-features/070-sync-vocab-to-purview-glossaries.md new file mode 100644 index 000000000..759f63076 --- /dev/null +++ b/docs/180-microsoft-integration/purview/020-features/070-sync-vocab-to-purview-glossaries.md @@ -0,0 +1,26 @@ +--- +layout: cluedin +title: Sync CluedIn Vocabularies to Purview Glossaries +parent: Microsoft Purview Integration +grand_parent: Microsoft Integration +permalink: /microsoft-integration/purview/features/sync-vocab-to-purview-glossaries +nav_order: 050 +has_children: false +tags: ["integration", "microsoft", "azure", "purview", "collection", "sync", "glossary", "vocabulary"] +--- +## On this page +{: .no_toc .text-delta } +- TOC +{:toc} + +## Sync CluedIn Vocabularies to Purview Glossaries + +![Settings Sync CluedIn Vocabularies to Purview Glossaries](../media/settings-sync-vocab-to-purview-glossaries.png) + +![Purview glossary terms created under the root glossary term "CluedIn" + [Your machine name], ex. "CluedIn-Dell"](../media/vocab_to_glossary.png) + +Purview glossary terms are created under the root glossary term "CluedIn" + [Your machine name] (e,g. "CluedIn-Dell"). + +By default, a CluedIn root glossary term is created in Purview glossaries. New glossary terms are created under the root glossary term. + +If a glossary term is deleted but the vocabulary or vocabulary key still exists, the sync re-creates the glossary term. If a vocabulary or vocabulary key is deleted, the linked glossary term is removed, except for glossary terms with assigned entities. If a newly created vocabulary or vocabulary key matches an existing glossary term, the name of the new glossary term is appended with numbers. diff --git a/docs/180-microsoft-integration/purview/020-features/080-sync-purview-glossaries-to-cluedin-glossaries.md b/docs/180-microsoft-integration/purview/020-features/080-sync-purview-glossaries-to-cluedin-glossaries.md new file mode 100644 index 000000000..e577c7824 --- /dev/null +++ b/docs/180-microsoft-integration/purview/020-features/080-sync-purview-glossaries-to-cluedin-glossaries.md @@ -0,0 +1,28 @@ +--- +layout: cluedin +title: Sync Purview Glossaries to CluedIn Glossaries +parent: Microsoft Purview Integration +grand_parent: Microsoft Integration +permalink: /microsoft-integration/purview/features/sync-purview-glossaries-to-cluedin-glossaries +nav_order: 080 +has_children: false +tags: ["integration", "microsoft", "azure", "purview", "collection", "sync", "glossary"] +--- +## On this page +{: .no_toc .text-delta } +- TOC +{:toc} + +## Sync Purview Glossaries to CluedIn Glossaries + +![Settings Sync Purview Glossaries to CluedIn Glossaries](../media/settings-sync-purview-glossaries-to-cluedin-glossaries.png) + +When this feature is enabled, a CluedIn glossary category with the default name **Purview** is automatically created. The glossary category name is configurable in **Organization** > **Settings**. All Purview glossary terms (excluding names containing "CluedIn-" prefix) are synced under this glossary category. New glossary terms created under this glossary category are ignored. + +### Purview Glossaries + +![Purview Glossaries](../media/purview-glossaries.png) + +### CluedIn Glossaries after Sync + +![Purview Glossaries to CluedIn Glossaries](../media/purview-glossaries-to-cluedin-glossaries.png) \ No newline at end of file diff --git a/docs/180-microsoft-integration/purview/020-features/090-sync-dataproducts.md b/docs/180-microsoft-integration/purview/020-features/090-sync-dataproducts.md new file mode 100644 index 000000000..2b3f3f7f2 --- /dev/null +++ b/docs/180-microsoft-integration/purview/020-features/090-sync-dataproducts.md @@ -0,0 +1,75 @@ +--- +layout: cluedin +title: Sync Data Products +parent: Microsoft Purview Integration +grand_parent: Microsoft Integration +permalink: /microsoft-integration/purview/features/sync-dataproducts +nav_order: 090 +has_children: false +tags: ["integration", "microsoft", "azure", "purview", "collection", "sync", "dataproduct", "dataasset"] +--- +## On this page +{: .no_toc .text-delta } +- TOC +{:toc} + +## Sync Data Products Settings + +This feature will enable you to sync the Purview Data Products and Data Assets into CluedIn DataSources. The data source groups can be viewed in **Integrations** > **Data Sources**. + +![Settings Sync Data Products](../media/settings-sync-dataproducts.png) + +## Sync Data Products by Glossary Term +There are 2 ways to select the Glossary Term that you want to sync. + +![Glossary Term Setup](../media/data-product-glossary-term-setup.png) + +1. By providing the Glossary Term ID, which can be found in the Url path. + +1. By providing the Glossary Term Name. In case you implemented a multiple glossary term with the same name. + + +#### Link Glossary Term to Data Product + +In the Glossary Term Page, you can see a `Link `data product` on the right-hand side of the screen. + +![Glossary Term Setup](../media/assign-glossary-term-to-dataproduct.png) + +## Purview Data Product Preview + +Let's break down the details we can see in the Purview Data Product Page + +- Data Product name - **Customer Journey** +- Data Product Type - **Master data and reference data** +- Data Assets Count - **10** +- Linked Glossary Term - **Sync to CluedIn** + +![Purview Data Products](../media/purview-dataproduct.png) + +Navigating to the list of associated Data Assets to the Data Product + +![Purview Data Products Data Assets](../media/purview-dataproduct-dataaassets.png) + +## CluedIn Syncronization + +When the CluedIn Job executes and synchronizes the Purview Data Products to CluedIn. New `DataSources` will be created. + +![Data Product DataSources](../media/sync-dataproducts-to-cluedin-dataSets.png) + +![Data Product DataSources](../media/sync-dataassets-to-cluedin-datasources.png) + +Requirements to Sync +- Data Product Status must be **_Published_** +- Data Product Type must be **_Master data and Reference data_** +- Data Product must be in a valid Glossary Term _(see next section below)_ + + +## Append Asset to Data Product + +As we write back to Purview the CluedIn Assets, we are now able to easily see the Assets we are creating by adding this into the associated Data Product that we synchronize. + +![Data Product DataSources](../media/add-dataasset-to-dataproduct.png) + +Requirements to Sync + +- `Service Principal` must be in the Business Domain Role **Data Product Owner**. Please see the _Setup Permissions_ document. \ No newline at end of file diff --git a/docs/180-microsoft-integration/purview/020-features/100-adf-pipeline-automation.md b/docs/180-microsoft-integration/purview/020-features/100-adf-pipeline-automation.md new file mode 100644 index 000000000..bb2cb73ca --- /dev/null +++ b/docs/180-microsoft-integration/purview/020-features/100-adf-pipeline-automation.md @@ -0,0 +1,75 @@ +--- +layout: cluedin +title: Azure Data Factory Pipeline Automation +parent: Microsoft Purview Integration +grand_parent: Microsoft Integration +permalink: /microsoft-integration/purview/features/adf-pipeline-automation +nav_order: 100 +has_children: false +tags: ["integration", "microsoft", "azure", "purview", "collection", "sync", "adf", "data factory"] +--- +## On this page +{: .no_toc .text-delta } +- TOC +{:toc} + +## Azure Data Factory setup + +Azure Data Factory integration is enabled on the **Settings** page in CluedIn. When the fields are left empty, no synchronization is performed with Azure Data Factory. + +### Role Assignment + +To enable the automation of ADF Pipeline, we need to assign a _**Reader**_ Role for the Service Principal. Please refer to this [link](https://learn.microsoft.com/en-us/purview/troubleshoot-connections) for more details. + +## To connect CluedIn to Azure Data Factory + +1. In CluedIn, go to **Administration** > **Settings**, and then scroll down to find the **Purview** section. + + ![Input Microsoft Azure Data Factory credentials](../media/adf_settings.png) + +1. Enter [Microsoft Azure Data Factory Credentials](https://learn.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory): + + - **Base URL** – `https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.DataFactory/factories/{factoryName}/`. You can get the resource ID value by navigating to your Azure Data Factory resource and checking the value through JSON View. + + ![json-view](../media/json-view.png) + + - **Client ID**, **Client Secret**, **Tenant ID** – you can get these values by navigating to your Azure Active Directory > App registration. For more information, see [Microsoft documentation](https://learn.microsoft.com/en-us/power-apps/developer/data-platform/walkthrough-register-app-azure-active-directory). + +## Automation Feature +1. Enable the **Azure Data Factory Pipeline Automation** feature. + +2. Enter the **ADF Pipeline Automation Term Pattern** to filter the asset that you want to automate. + + ![Input Microsoft Azure Data Factory credentials](../media/adf_settings2.png) + + The same procedure with `Sync Datasources`, you also need to assign a Glossary Term to the asset to identify which asset you want to automate the pipeline creation to ingest to CluedIn DataSet. + + ![ADF Glossary Term](../media/adf-automation-glossary-term.png) + + ![ADF Automation Asset List](../media/adf-automation-asset-list.png) + + But for the Data Product synchronization, as this is already grouped and managed within the Business Domain, no need to assign the Glossary Term and the automation of the pipeline will kick in immediately. + +3. Once the ADF Automation is done, the data will start to come forward to our ingestion endpoint. Some examples below are the creations of CluedIn Ingestion Endpoint for Assets. + + ![ADF Automation Datasets](../media/adf-automation-datasets.png) + + With Data ingested via ADF Pipeline + + ![ADF Automation Dataset with Data](../media/adf-automation-dataset-with-data.png) + + Data Product - Data Assets creation of Ingestion Endpoint + + ![ADF Automation DataProduct Datasets](../media/adf-automation-dataassets-datasets.png) + +## Azure Data Factory automation coverage + +- Azure SQL Server + +- Azure Data Lake gen 2 (Avro, DelimitedText, Excel, JSON, ORC, Parquet, XML) + +- Azure File (Avro, DelimitedText, Excel, JSON, ORC, Parquet, XML) + +- Snowflakes + +- SQL Server \ No newline at end of file diff --git a/docs/180-microsoft-integration/purview/020-setup-permissions.md b/docs/180-microsoft-integration/purview/020-setup-permissions.md index 73e5b97ee..7f0c81a83 100644 --- a/docs/180-microsoft-integration/purview/020-setup-permissions.md +++ b/docs/180-microsoft-integration/purview/020-setup-permissions.md @@ -1,12 +1,12 @@ --- layout: cluedin -title: Setup credentials and permissions +title: Setup Permissions parent: Microsoft Purview Integration grand_parent: Microsoft Integration -permalink: /microsoft-integration/purview/setup -nav_order: 010 +permalink: /microsoft-integration/purview/setup-permissions +nav_order: 020 has_children: false -tags: ["integration", "microsoft", "azure", "purview"] +tags: ["integration", "microsoft", "azure", "purview", "permissions"] --- ## On this page {: .no_toc .text-delta } @@ -21,38 +21,6 @@ Microsoft Purview integration is enabled on the **Settings** page in CluedIn. Wh 1. In CluedIn, go to **Administration** > **Settings**, and then scroll down to find the **Purview** section. -1. Enter Microsoft Purview credentials: - - - **Base URL** – `https://{accountName}.purview.azure.com` where `accountName` is your Purview subscription name. For example, if your Purview subscription name is ContosoPurview, then base URL would be `https://contosopurview.purview.azure.com`. For more details about Purview subscription, see [Microsoft documentation](https://docs.microsoft.com/en-us/azure/purview/create-catalog-portal#open-the-microsoft-purview-governance-portal). - - - **Client ID** – can be obtained from **Application (client) ID** on the **Overview** tab of the app registration that the organization has dedicated to access Purview account on behalf of CluedIn. - - - **Client Secret** – can be obtained from the **Certificates & secrets** of the app registration that the organization has dedicated to access Purview account on behalf of CluedIn. - - - **Tenant ID** – can be obtained from **Directory (tenant) ID** on the **Overview** tab of the app registration that the organization has dedicated to access Purview account on behalf of CluedIn. - - ![Input Microsoft Purview credentials](./media/settings.png) - - For more information on where to find the values for client ID and tenant ID, see [Microsoft documentation](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app#register-an-application). - -2. In **Collection Name**, enter one of the following: - - - The ID of the collection in Purview. You can find this ID in the Purview collection's URL. - - ![collection-id.png](./media/collection-id.png) - - Based on the screenshot above, the collection name would be "hkkib1". - - ![settings-collection-name.png](./media/settings-collection-name.png) - - - The path from the topmost collection to the intended collection in Purview. The example pattern is as follows: root_collection_name/sub_collection_1/sub_collection_2/target_collection_name. - - ![collection-id.png](./media/settings-collection-name-purview.png) - - Based on the screenshot above, the collection name would be "Presalesdemo/Demo". - - ![settings-collection-name-path.png](./media/settings-collection-name-path.png) - 3. Choose to enable one or more [features](/microsoft-integration/purview/introduction) for your use cases: - Sync CluedIn Data Sources @@ -68,18 +36,21 @@ Microsoft Purview integration is enabled on the **Settings** page in CluedIn. Wh ## Minimum permission requirements The following table lists the Purview roles ([permissions](https://learn.microsoft.com/en-us/azure/purview/catalog-permissions)) that CluedIn requires per integration feature. Roles assignments can be found under the **Role assignments** tab of each collection in Purview. + + ![Role Assignments](./media/role-assignments.png) | Integration feature | Role | Collection Level | | ---- | ------ | ------- | -| Polling Data Sources | Data Curator | Target Collection | -| Synchronize CluedIn Vocabularies to Purview Glossary Terms | Data Curator | Target Collection | | Synchronize Data Sources | Data Curator | Target Collection | | Synchronize Data Sources | Data Reader | Root Collection | +| Polling Data Sources | Data Curator | Target Collection | +| Synchronize CluedIn Vocabularies to Purview Glossary Terms | Data Curator | Target Collection | | Synchronize Purview Glossaries Terms To CluedIn Glossary Terms | Data Reader | Root Collection | | Synchronize Purview Glossary Terms To CluedIn Vocabularies | Data Reader | Root Collection | | Synchronize Streams | Data Curator | Target Collection | | Synchronize Streams | Data Reader | Root Collection | | Synchronize Crawlers And Enrichers | Data Curator | Target Collection | +| Azure Data Factory Pipeline Automation | Data source admins | Root Collection | Collection levels: @@ -87,32 +58,23 @@ Collection levels: - **Target-collection** – refers to the **Collection Name** specified in CluedIn's organization settings. -## Azure Data Factory setup - -Azure Data Factory integration is enabled on the **Settings** page in CluedIn. When the fields are left empty, no synchronization is performed with Azure Data Factory. - -**To connect CluedIn to Azure Data Factory** - -1. In CluedIn, go to **Administration** > **Settings**, and then scroll down to find the **Purview** section. - -1. Enter [Microsoft Azure Data Factory Credentials](https://learn.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory): - - - **Base URL** – `https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.DataFactory/factories/{factoryName}/`. You can get the resource ID value by navigating to your Azure Data Factory resource and checking the value through JSON View. +## Microsoft Purview (Preview) - ![json-view](./media/json-view.png) - - - **Client ID**, **Client Secret**, **Tenant ID** – you can get these values by navigating to your Azure Active Directory > App registration. For more information, see [Microsoft documentation](https://learn.microsoft.com/en-us/power-apps/developer/data-platform/walkthrough-register-app-azure-active-directory). +### Data Catalog - Roles and permissions -1. Enable the **Azure Data Factory Pipeline Automation** feature. +![Role - Data Product Owner](./media/role-groups-and-app-permissions.png) -1. Enter the **ADF Pipeline Automation Term Pattern** to filter the asset that you want to automate. +**Business Domain Preparations** +- As we navigate within the Business Domain of the Data Catalog, assigning the right permissions helps the integration do its job right. +- Assign your `Service Principal` to the **Business Domain Creators**. - ![Input Microsoft Azure Data Factory credentials](./media/adf_settings2.png) +### Business Domain - Roles -## Azure Data Factory automation coverage +![Business Domain Roles](./media/business-domain-roles.png) -- Azure SQL Server +### Append Asset to Data Product -- Azure Data Lake gen 2 +- The Integration is writing back to Purview the Data Assets for CluedIn. +- Assign your `Service Principal` to the **Data Product Owner**. -- Azure File (in progress) \ No newline at end of file + ![Role - Data Product Owner](./media/role-data-product-owner.png) diff --git a/docs/180-microsoft-integration/purview/030-preparations.md b/docs/180-microsoft-integration/purview/030-preparations.md new file mode 100644 index 000000000..e7b158ea8 --- /dev/null +++ b/docs/180-microsoft-integration/purview/030-preparations.md @@ -0,0 +1,55 @@ +--- +layout: cluedin +title: Preparations +parent: Microsoft Purview Integration +grand_parent: Microsoft Integration +permalink: /microsoft-integration/purview/preparations +nav_order: 030 +has_children: false +tags: ["integration", "microsoft", "azure", "purview", "collection", "business", "domain"] +--- +## On this page +{: .no_toc .text-delta } +- TOC +{:toc} + +## Collection Name + +Setting up the Collection name in CluedIn-Purview Integration is required as this will be the destination of all the `Asset`, `Process` and `Lineage` created by CluedIn. + +### Administration Settings + +In CluedIn, go to **Administration** > **Settings**, and then scroll down to find the **Purview** section. + +In **Collection Name**, enter one of the following: + +- The ID of the collection in Purview. You can find this ID in the Purview collection's URL. + + ![collection-id.png](./media/collection-id.png) + + Based on the screenshot above, the collection name would be "hkkib1". + + ![settings-collection-name.png](./media/settings-collection-name.png) + +- The path from the topmost collection to the intended collection in Purview. The example pattern is as follows: root_collection_name/sub_collection_1/sub_collection_2/target_collection_name. + + ![collection-id.png](./media/settings-collection-name-purview.png) + + Based on the screenshot above, the collection name would be "Presalesdemo/Demo". + + ![settings-collection-name-path.png](./media/settings-collection-name-path.png) + +### Required Permission + +As we are trying to read, validate the collection value/path and write back the CluedIn Purview objects, we need to setup the permission of the `Service Principal` in the `Target Collection`. To do this, +- In **Purview Data Map** => Domains, navigate to the collection you want to use in the integration +- Go to **Role assignment** tab +- Add the `Service Principal` used in the **Collection admins** + + ![settings-collection-name-path.png](./media/assign-collection-admin.png) + +## Business Domain + +A business domain is a new concept in Microsoft Purview that provides context for your data assets and makes it easier to scale data governance practices. + +We need to give the `Service Principal` account full access in this field as we are going to navigate in this module. Add the `Service Principal` account to `Business Domain Creators`. See the Setup Permission page for more info. diff --git a/docs/180-microsoft-integration/purview/040-features.md b/docs/180-microsoft-integration/purview/040-features.md new file mode 100644 index 000000000..ed32fc846 --- /dev/null +++ b/docs/180-microsoft-integration/purview/040-features.md @@ -0,0 +1,14 @@ +--- +layout: cluedin +title: Features +parent: Microsoft Purview Integration +grand_parent: Microsoft Integration +permalink: /microsoft-integration/purview/features +nav_order: 040 +has_children: true +tags: ["integration", "microsoft", "azure", "purview"] +--- +## On this page +{: .no_toc .text-delta } +- TOC +{:toc} diff --git a/docs/180-microsoft-integration/purview/media/add-dataasset-to-dataproduct.png b/docs/180-microsoft-integration/purview/media/add-dataasset-to-dataproduct.png new file mode 100644 index 000000000..ff057666d Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/add-dataasset-to-dataproduct.png differ diff --git a/docs/180-microsoft-integration/purview/media/adf-automation-asset-list.png b/docs/180-microsoft-integration/purview/media/adf-automation-asset-list.png new file mode 100644 index 000000000..f10a26ad3 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/adf-automation-asset-list.png differ diff --git a/docs/180-microsoft-integration/purview/media/adf-automation-dataassets-datasets.png b/docs/180-microsoft-integration/purview/media/adf-automation-dataassets-datasets.png new file mode 100644 index 000000000..9751e0df7 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/adf-automation-dataassets-datasets.png differ diff --git a/docs/180-microsoft-integration/purview/media/adf-automation-dataset-with-data.png b/docs/180-microsoft-integration/purview/media/adf-automation-dataset-with-data.png new file mode 100644 index 000000000..d0cdd47ed Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/adf-automation-dataset-with-data.png differ diff --git a/docs/180-microsoft-integration/purview/media/adf-automation-datasets.png b/docs/180-microsoft-integration/purview/media/adf-automation-datasets.png new file mode 100644 index 000000000..f6cef3e51 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/adf-automation-datasets.png differ diff --git a/docs/180-microsoft-integration/purview/media/adf-automation-glossary-term.png b/docs/180-microsoft-integration/purview/media/adf-automation-glossary-term.png new file mode 100644 index 000000000..3b8a0d59e Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/adf-automation-glossary-term.png differ diff --git a/docs/180-microsoft-integration/purview/media/adf_settings.png b/docs/180-microsoft-integration/purview/media/adf_settings.png index 4a5f43b12..6dcf5f17e 100644 Binary files a/docs/180-microsoft-integration/purview/media/adf_settings.png and b/docs/180-microsoft-integration/purview/media/adf_settings.png differ diff --git a/docs/180-microsoft-integration/purview/media/adf_settings2.png b/docs/180-microsoft-integration/purview/media/adf_settings2.png index 3faca80c4..7f4f78a79 100644 Binary files a/docs/180-microsoft-integration/purview/media/adf_settings2.png and b/docs/180-microsoft-integration/purview/media/adf_settings2.png differ diff --git a/docs/180-microsoft-integration/purview/media/asset-glossary-term-mapping.png b/docs/180-microsoft-integration/purview/media/asset-glossary-term-mapping.png new file mode 100644 index 000000000..8e7bd5963 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/asset-glossary-term-mapping.png differ diff --git a/docs/180-microsoft-integration/purview/media/assign-collection-admin.png b/docs/180-microsoft-integration/purview/media/assign-collection-admin.png new file mode 100644 index 000000000..2094c20e2 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/assign-collection-admin.png differ diff --git a/docs/180-microsoft-integration/purview/media/assign-glossary-term-to-dataproduct.png b/docs/180-microsoft-integration/purview/media/assign-glossary-term-to-dataproduct.png new file mode 100644 index 000000000..5c9ae7ca3 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/assign-glossary-term-to-dataproduct.png differ diff --git a/docs/180-microsoft-integration/purview/media/business-domain-roles.png b/docs/180-microsoft-integration/purview/media/business-domain-roles.png new file mode 100644 index 000000000..e32241837 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/business-domain-roles.png differ diff --git a/docs/180-microsoft-integration/purview/media/cluedin-entitytype-configuration.png b/docs/180-microsoft-integration/purview/media/cluedin-entitytype-configuration.png new file mode 100644 index 000000000..fd029ad84 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/cluedin-entitytype-configuration.png differ diff --git a/docs/180-microsoft-integration/purview/media/cluedin-stream.png b/docs/180-microsoft-integration/purview/media/cluedin-stream.png new file mode 100644 index 000000000..b939b6efa Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/cluedin-stream.png differ diff --git a/docs/180-microsoft-integration/purview/media/cluedin-vocab-from-glossary-terms.png b/docs/180-microsoft-integration/purview/media/cluedin-vocab-from-glossary-terms.png new file mode 100644 index 000000000..f85e59d02 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/cluedin-vocab-from-glossary-terms.png differ diff --git a/docs/180-microsoft-integration/purview/media/cluedin-vocabulary-configuration.png b/docs/180-microsoft-integration/purview/media/cluedin-vocabulary-configuration.png new file mode 100644 index 000000000..975c88e72 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/cluedin-vocabulary-configuration.png differ diff --git a/docs/180-microsoft-integration/purview/media/data-product-glossary-term-setup.png b/docs/180-microsoft-integration/purview/media/data-product-glossary-term-setup.png new file mode 100644 index 000000000..cbdb9bcfa Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/data-product-glossary-term-setup.png differ diff --git a/docs/180-microsoft-integration/purview/media/error-preparing-collection.png b/docs/180-microsoft-integration/purview/media/error-preparing-collection.png new file mode 100644 index 000000000..94c418558 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/error-preparing-collection.png differ diff --git a/docs/180-microsoft-integration/purview/media/purview-child-glossary-terms.png b/docs/180-microsoft-integration/purview/media/purview-child-glossary-terms.png new file mode 100644 index 000000000..817848a9b Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/purview-child-glossary-terms.png differ diff --git a/docs/180-microsoft-integration/purview/media/purview-dataproduct-dataaassets.png b/docs/180-microsoft-integration/purview/media/purview-dataproduct-dataaassets.png new file mode 100644 index 000000000..bb0a87c1f Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/purview-dataproduct-dataaassets.png differ diff --git a/docs/180-microsoft-integration/purview/media/purview-dataproduct.png b/docs/180-microsoft-integration/purview/media/purview-dataproduct.png new file mode 100644 index 000000000..3b37e5f2f Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/purview-dataproduct.png differ diff --git a/docs/180-microsoft-integration/purview/media/purview-glossaries-from-cluedin-vocab.png b/docs/180-microsoft-integration/purview/media/purview-glossaries-from-cluedin-vocab.png new file mode 100644 index 000000000..dad016ece Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/purview-glossaries-from-cluedin-vocab.png differ diff --git a/docs/180-microsoft-integration/purview/media/purview-glossaries-to-cluedin-glossaries.png b/docs/180-microsoft-integration/purview/media/purview-glossaries-to-cluedin-glossaries.png new file mode 100644 index 000000000..15fa28bc4 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/purview-glossaries-to-cluedin-glossaries.png differ diff --git a/docs/180-microsoft-integration/purview/media/purview-glossaries.png b/docs/180-microsoft-integration/purview/media/purview-glossaries.png new file mode 100644 index 000000000..ccf6be10b Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/purview-glossaries.png differ diff --git a/docs/180-microsoft-integration/purview/media/purview-glossary-term-setup.png b/docs/180-microsoft-integration/purview/media/purview-glossary-term-setup.png new file mode 100644 index 000000000..cc1da404d Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/purview-glossary-term-setup.png differ diff --git a/docs/180-microsoft-integration/purview/media/purview-resource-json.png b/docs/180-microsoft-integration/purview/media/purview-resource-json.png new file mode 100644 index 000000000..4580d9553 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/purview-resource-json.png differ diff --git a/docs/180-microsoft-integration/purview/media/role-assignments.png b/docs/180-microsoft-integration/purview/media/role-assignments.png new file mode 100644 index 000000000..964b233b6 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/role-assignments.png differ diff --git a/docs/180-microsoft-integration/purview/media/role-data-product-owner.png b/docs/180-microsoft-integration/purview/media/role-data-product-owner.png new file mode 100644 index 000000000..f06fa0edb Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/role-data-product-owner.png differ diff --git a/docs/180-microsoft-integration/purview/media/role-groups-and-app-permissions.png b/docs/180-microsoft-integration/purview/media/role-groups-and-app-permissions.png new file mode 100644 index 000000000..aabc066be Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/role-groups-and-app-permissions.png differ diff --git a/docs/180-microsoft-integration/purview/media/settings-auto-map-datasets.png b/docs/180-microsoft-integration/purview/media/settings-auto-map-datasets.png new file mode 100644 index 000000000..2d4b4c5b7 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/settings-auto-map-datasets.png differ diff --git a/docs/180-microsoft-integration/purview/media/settings-poll-datasources.png b/docs/180-microsoft-integration/purview/media/settings-poll-datasources.png new file mode 100644 index 000000000..25e889e46 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/settings-poll-datasources.png differ diff --git a/docs/180-microsoft-integration/purview/media/settings-sync-crawlers-and-enrichers.png b/docs/180-microsoft-integration/purview/media/settings-sync-crawlers-and-enrichers.png new file mode 100644 index 000000000..6409a014c Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/settings-sync-crawlers-and-enrichers.png differ diff --git a/docs/180-microsoft-integration/purview/media/settings-sync-dataproducts.png b/docs/180-microsoft-integration/purview/media/settings-sync-dataproducts.png new file mode 100644 index 000000000..afb95d790 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/settings-sync-dataproducts.png differ diff --git a/docs/180-microsoft-integration/purview/media/settings-sync-datasources.png b/docs/180-microsoft-integration/purview/media/settings-sync-datasources.png new file mode 100644 index 000000000..fa553d585 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/settings-sync-datasources.png differ diff --git a/docs/180-microsoft-integration/purview/media/settings-sync-purview-glossaries-to-cluedin-glossaries.png b/docs/180-microsoft-integration/purview/media/settings-sync-purview-glossaries-to-cluedin-glossaries.png new file mode 100644 index 000000000..8908c6748 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/settings-sync-purview-glossaries-to-cluedin-glossaries.png differ diff --git a/docs/180-microsoft-integration/purview/media/settings-sync-purview-glossaries-to-vocab.png b/docs/180-microsoft-integration/purview/media/settings-sync-purview-glossaries-to-vocab.png new file mode 100644 index 000000000..ac2da5272 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/settings-sync-purview-glossaries-to-vocab.png differ diff --git a/docs/180-microsoft-integration/purview/media/settings-sync-streams.png b/docs/180-microsoft-integration/purview/media/settings-sync-streams.png new file mode 100644 index 000000000..25b3771b0 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/settings-sync-streams.png differ diff --git a/docs/180-microsoft-integration/purview/media/settings-sync-vocab-to-purview-glossaries-root.png b/docs/180-microsoft-integration/purview/media/settings-sync-vocab-to-purview-glossaries-root.png new file mode 100644 index 000000000..316fc5bf8 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/settings-sync-vocab-to-purview-glossaries-root.png differ diff --git a/docs/180-microsoft-integration/purview/media/settings-sync-vocab-to-purview-glossaries.png b/docs/180-microsoft-integration/purview/media/settings-sync-vocab-to-purview-glossaries.png new file mode 100644 index 000000000..f9db04c3f Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/settings-sync-vocab-to-purview-glossaries.png differ diff --git a/docs/180-microsoft-integration/purview/media/sync-dataassets-to-cluedin-datasources.png b/docs/180-microsoft-integration/purview/media/sync-dataassets-to-cluedin-datasources.png new file mode 100644 index 000000000..9b1affddc Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/sync-dataassets-to-cluedin-datasources.png differ diff --git a/docs/180-microsoft-integration/purview/media/sync-dataproducts-to-cluedin-dataSets.png b/docs/180-microsoft-integration/purview/media/sync-dataproducts-to-cluedin-dataSets.png new file mode 100644 index 000000000..308ad9416 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/sync-dataproducts-to-cluedin-dataSets.png differ diff --git a/docs/180-microsoft-integration/purview/media/sync-datasources-assets.png b/docs/180-microsoft-integration/purview/media/sync-datasources-assets.png new file mode 100644 index 000000000..c877e678e Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/sync-datasources-assets.png differ diff --git a/docs/180-microsoft-integration/purview/media/sync-datasources-glossary-term.png b/docs/180-microsoft-integration/purview/media/sync-datasources-glossary-term.png new file mode 100644 index 000000000..80d453ba7 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/sync-datasources-glossary-term.png differ diff --git a/docs/180-microsoft-integration/purview/media/sync-datasources-to-cluedin.png b/docs/180-microsoft-integration/purview/media/sync-datasources-to-cluedin.png new file mode 100644 index 000000000..d76480a06 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/sync-datasources-to-cluedin.png differ diff --git a/docs/180-microsoft-integration/purview/media/sync-streams-asset-lineage.png b/docs/180-microsoft-integration/purview/media/sync-streams-asset-lineage.png new file mode 100644 index 000000000..6497beeeb Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/sync-streams-asset-lineage.png differ diff --git a/docs/180-microsoft-integration/purview/media/sync-streams-process-and-export-target.png b/docs/180-microsoft-integration/purview/media/sync-streams-process-and-export-target.png new file mode 100644 index 000000000..bc39ead19 Binary files /dev/null and b/docs/180-microsoft-integration/purview/media/sync-streams-process-and-export-target.png differ