Some issues with Purview's under-the-hood implementation.

Question

Some issues with Purview's under-the-hood implementation.

Mofei Zhuang 140 Microsoft Employee

What is the current write limit and the maximum concurrent write limit for a single Purview Account? Under what specific circumstances will an HTTP 408 error be triggered?
I would like to know about the underlying architecture of Purview. Is the Index layer currently implemented using Azure AI Search (formerly Azure Search)? And is Azure Cosmos DB used for the data storage layer?
Based on my testing, if an Entity contains a large number of relationships, querying it is still significantly slower than querying an entity without relationships, even when ignoreRelationship=true is specified. What is the technical reason behind this behavior?

Pilladi Padma Sai Manisha 10,190 Reputation points Microsoft External Staff Moderator

2026-06-17T04:12:10.1266667+00:00
Hi @Mofei Zhuang
Thanks for your questions.

Write limits and HTTP 408 errors

There isn't a publicly documented per-account write throughput limit or maximum concurrent write limit for a single Microsoft Purview account. The effective throughput depends on several factors, such as the API being used, request size, entity complexity (attributes, classifications, relationships), and the current service load.

An HTTP 408 (Request Timeout) generally indicates that the service could not complete processing within the allowed time window. This may occur when:

Processing very large payloads or batches

Entities contain a large number of relationships or complex metadata structures

Multiple concurrent write operations create contention

The service is under temporary load or throttling conditions

For sustained high-volume workloads, Microsoft recommends implementing retry logic with exponential backoff.

Purview internal architecture

Microsoft doesn't publicly disclose the underlying implementation details or guarantee which Azure services are used internally (for example, whether specific indexing or storage layers are backed by Azure AI Search or Azure Cosmos DB). Since these implementation details may change over time, they shouldn't be relied upon for application design or performance expectations.

Why entities with many relationships are slower to query

The behavior you're observing is expected.

Even when ignoreRelationship=true is specified, the service may still need to process relationship metadata internally during entity retrieval. Relationships are deeply integrated into Purview's graph-based metadata model and can impact:

Entity graph traversal and validation

Authorization and access checks

Internal joins and metadata aggregation

Response construction and filtering

As a result, entities with a large number of relationships can experience higher query latency compared to entities without relationships, even when relationships are excluded from the final response payload.

If you're encountering significant performance degradation at scale, it would be helpful to share:

Approximate number of relationships per entity

Total number of entities in the catalog

API being used (Atlas APIs, Data Map APIs, etc.)

Whether the issue occurs consistently or only during peak ingestion activity

For deeper insights into service behavior or undocumented limits, we recommend opening a support request, as those details are not publicly documented and may require investigation by the product engineering team.

1 answer

Your answer

Answer 1

For Submit Scenario

Here is the sample data structure, we are using custom entity， please see response_1781823816744.txt

and for QualifiedName field we use in this formart : pds://bench-rel-20260618050106/sub/0008ba92-f566-482b-8ea6-95b06465a802/appinsights/res-appinsights-0008ba92-0014

From the test result I found that :

we put 200 entities for a batch to submit , because when we add more than this it will occur an error like the URL is too long or some like that error info.

If the request can success, the avg cost time will be around 4000 ms. (without any relationship， just the independent entities)

So, my question is there any other way I can submit more for one time. Or is there any optimization I can do to accelerate the submit process.

We're using DataMapClient.GetEntityClient().BatchCreateOrUpdateAsync to batch Submit

For Catalog search

Here is the way we use catalog to search our target entities, I found that even I remove all the relationship the query perf is not so satisficed.

Can you help me to check if is the right way to use the catalog search ?

            `var prodSubResults = await _catalogService.SearchEntityByServiceIdAndAttributesAsync(`

new Dictionary<string, string>

{

{ "ServiceId", serviceId },

{ "Environment", "Prod" }

},

CustomEntityConstants.SUBSCRIPTION_CUSTOM_ENTITY_NAME,

FilterOperator.Eq);

public async Task<IEnumerable<SearchResultValue>> SearchEntityByServiceIdAndAttributesAsync(IReadOnlyDictionary<string, string> attrDic, string typeName, FilterOperator Oper)

{

var filters = new List<object>

{

new { typeName = typeName }

};

foreach (var kvp in attrDic)

{

filters.Add(new

{

attributeName = kvp.Key,

@operator = Oper.GetEnumMemberValue(),

attributeValue = kvp.Value

});

}

var resList = new List<SearchResultValue>();

string? continuationToken = null;

var config = new QueryConfig

{

Limit = 1000,

Filter = BinaryData.FromObjectAsJson(new

{

and = filters

}),

};

do

{

var response = await purviewClient.DataMapClient.GetDiscoveryClient().QueryAsync(config);

resList.AddRange(response.Value.Value ?? Enumerable.Empty<SearchResultValue>());

continuationToken = response.Value.ContinuationToken;

config.ContinuationToken = continuationToken;

} while (!string.IsNullOrEmpty(continuationToken));

return resList;

}

Share via

Some issues with Purview's under-the-hood implementation.

1 answer

Your answer