I have been conducting several tests with the "Data collector for Graph" extension, which have been successful overall, but I encountered an issue that I have not been able to resolve, which I will explain below.
After successfully synchronizing Contacts, Groups, and Users, .... ///////////////////////////////////////////////////////////////////////////////////////////// [2024-09-04 16:01:37][Info] Synchronization of data source 'Graph Teams Discovery - ' succeeded. [2024-09-04 16:02:01][Info] Synchronization of data source 'Graph lnkPersonToTeam Discovery - ' succeeded. [2024-09-04 16:02:06][Info] Synchronization of data source 'Graph User Discovery - ' succeeded. /////////////////////////////////////////////////////////////////////////////////////////////
I noticed that only the first 100 records were retrieved out of a total of approximately 3,200. I would like to mention that the pagination limit defined in my configuration files is set to more than 3,000: <max_chunk_size>10000</max_chunk_size>.
I was reviewing documentation online and came across the following link (https://learn.microsoft.com/en-us/answers/questions/2033138/how-to-extract-all-users-using-azure-data-factory), which states that there is a limitation in the Microsoft API regarding the number of records it returns.
/////////////////////////////////////////////////////////////////////////////////////////
The Graph API limits the number of records returned in a single request. Even if you specify $top=999, you might not get all the records in a single response. You need to handle pagination to retrieve all the records.
The API will return a @odata.nextLink property if there are more records to retrieve. You should use this link to request the next page of records.
Using ConsistencyLevel=eventual and $count=true helps in scenarios where you want to get the total number of records. However, this setting might still limit the number of records returned per page. ///////////////////////////////////////////////////////////////////////////////////////////
I also found the following lines in the file located at /combodo-graph-data-collector/data/GraphPerson.json
Dear Combodo Support,
I have been conducting several tests with the "Data collector for Graph" extension, which have been successful overall, but I encountered an issue that I have not been able to resolve, which I will explain below.
After successfully synchronizing Contacts, Groups, and Users, ....
/////////////////////////////////////////////////////////////////////////////////////////////
[2024-09-04 16:01:37] [Info] Synchronization of data source 'Graph Teams Discovery - ' succeeded.
[2024-09-04 16:02:01] [Info] Synchronization of data source 'Graph lnkPersonToTeam Discovery - ' succeeded.
[2024-09-04 16:02:06] [Info] Synchronization of data source 'Graph User Discovery - ' succeeded.
/////////////////////////////////////////////////////////////////////////////////////////////
I noticed that only the first 100 records were retrieved out of a total of approximately 3,200. I would like to mention that the pagination limit defined in my configuration files is set to more than 3,000: <max_chunk_size>10000</max_chunk_size>.
I was reviewing documentation online and came across the following link (https://learn.microsoft.com/en-us/answers/questions/2033138/how-to-extract-all-users-using-azure-data-factory), which states that there is a limitation in the Microsoft API regarding the number of records it returns.
/////////////////////////////////////////////////////////////////////////////////////////
The Graph API limits the number of records returned in a single request. Even if you specify $top=999, you might not get all the records in a single response. You need to handle pagination to retrieve all the records.
The API will return a @odata.nextLink property if there are more records to retrieve. You should use this link to request the next page of records.
Using ConsistencyLevel=eventual and $count=true helps in scenarios where you want to get the total number of records. However, this setting might still limit the number of records returned per page.
///////////////////////////////////////////////////////////////////////////////////////////
I also found the following lines in the file located at /combodo-graph-data-collector/data/GraphPerson.json
{"@odata.context":"https:\/\/graph.microsoft.com\/v1.0\/$metadata#users(manager())","@odata.nextLink":"https:\/\/graph.microsoft.com\/\/v1.0\/users?$expand=manager&$skiptoken=XXXXXX","value":{......
My specific question is: Is there any parameter that I can modify from the iTop extension that would allow me to overcome this limitation?
Thank you very much for your time.
Last edit: Cristian Ozan 2024-09-04