Paging

Response paging has been implented on most APIs that can return more than one item. When a response is paged, the header will include four items to help you navigate the pages of data:

  • PageNumber: The current page of data you are on
  • PageSize: The maximum number of items on each page
  • TotalNumberOfRecords: The actual count of items to be returned
  • TotalPages: The total number of pages based on the PageSize value

With these four pieces of information from the header you can modify your process to iterate through the pages of data to return all of the requested items. This is done by appending additional paremeters to the API request with each iteration.

Request Parameters for Paging

When a response is paged, the following parameters can be used navigate the requests for more items.

  • pageSize=X – this can be any integer between 1 and 1,000. Values greater than 1,000 will be automatically limited.

  • pageNumber=X – after the first request, you use this to increment the request to the next page and get more data. You can continue to increment this number until it is equal to the TotalPages value, at which case all data has been returned.

Example

When requesting all meters in a database that has 1,100 meters, the first response will have 250 items.

https://{host}/api/v3/meter

The response headers will have these four values:

    PageNumber: 1
    PageSize: 250
    TotalNumberOfRecords: 1100
    TotalPages: 5

This means that there are four more pages of data to request. To get the second page, the following request would be made:

https://{host}/api/v3/meter?pageNumber=2

This can be continued through all four additional pages. Once the PageNumber and the TotalPages are equal, the looping of requests is finished.

Using the PageSize request parameter can be advantageous in two scenarios, depending on your workflow and process.

  1. When many items will be returned, 30,000 meters for example, it would be beneficial to increase the request size to 1,000 items. The API will still respond in a timely fashion and your application will have to make 75% fewer requests.

  2. If there are only a few hundred items (less than 1,000) for something you would otherwise request muiltiple times through a process, then increase the PageSize to get everything at once, saving the overhead of many requests to get a small amount of data.

We encourage you to experiment with these values given your development needs, look at timings and response sizes, and consider using field reduction with your requests to get the right amount of data as quickly as possible.

Parallel Processing

With the ability to page the response also allows for parallel requests to be made. Once the first request is completed, your process will know how many total pages it needs to request. If your libraries and code allow, you could initiate multiple requests at once, each with the next page number (in series) to speed up the time to load data. In general, no more than 5 requests should be made at a time with large amounts of data. Still, the overall processing win can be substantial with these kinds of APIs.

Paging in the C# SDK

Due to the fact that paging data is included in the response headers, special consideration needs to be taken with the C# SDK. Paginated APIs must be called with their *WithHttpMessagesAsync counterparts. This is because the normal response does not include any header data.

The response of an API call with http messages is of type:
Microsoft.Rest.HttpOperationResponse<ResponseBodyType, HeaderType>:

  • Body
    • Response body object from version of the call without HTTP messages (ResponseBodyType).
  • Headers
    • Class Name: {ApiMethodName}Headers (HeaderType)
      • PageNumber
      • PageSize
      • TotalPages
      • TotalNumberOfRecords
  • Request
    • …
  • Response
    • …

For example, instead of calling GetAccountsAsync:

var accounts = await client.GetAccountsAsync(filter, pageNumber, pageSize);

You would instead call GetAccountsWithHttpMessagesAsync:

var accountResponse = await client.GetAccountsWithHttpMessagesAsync(filter, pageNumber, pageSize);

var accountData = accountResponse.Body; //This is the same object returned by GetAccountsAsync()
var pagingHeaders = accountResponse.Header;

// The paging header data is accessed through properties on the header object
var responsePageNumber = pagingHeaders.PageNumber;
var responsePageSize = pagingHeaders.PageSize;
var responseTotalPages = pagingHeaders.TotalPages;
var responseTotalNumberOfRecords = pagingHeaders.TotalNumberOfRecords; 

// ...