REST api querying large list

How often you have situation that our customers say that we will never ever have more than 100 items in our SharePoint list, then how quickly the list grows with 1000, 5000 items and finally it reach to a point that it exceed the list view threshold and our REST call don’t work anymore. I guess, it’s all most every time or?

Querying large list  “/_api/web/lists/getbytitle(‘<list title>’)/items”  that has more than 5000 items, you will get following exception “Request failed. The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator”.

In SSOM we have ContentIterator class but as for CSOM or JSOM the options are limited. I found solution at PnP but isn’t working for me. I have tried with Rowlimit and $filter and $top and nothing really worked. But finally I managed to find the right query.


The above query give you top 5 elements ordered by ID. Note that ID is OOTB index column available in each SharePoint list.


In result, you will find a next URL that contains paging option, using that you can query next set of items. So loop until you don’t have next URL.

		<link  rel="next" href="https://<yoursharepoint>/_api/web/lists/GetByTitle('largelist')/Items?%24skiptoken=Paged%3dTRUE%26p_ID%3d203&%24Select=ID&%24OrderBy=ID&%24top=5" />

In the sample code below I fetch 200 list items per page. I don’t recommend more than 200 items per page as it can affect the performance.

   function getDataFromUrl(endpoint) {
        return jQuery.ajax({
            url: endpoint,
            method: "GET",
            headers: {
                "Accept": "application/json; odata=verbose",
                "Content-Type": "application/json; odata=verbose"
   function getLargeList(nextUrl) {
        var dfd = new $.Deferred();
        if (nextUrl == undefined) {
        getDataFromUrl(nextUrl).done(function (listItems) {
            var items = listItems.d.results;
            var next = listItems.d.__next;

            $.when(getLargeList(next)).done(function (){
        return dfd.promise();
   function getListData {
        var documentLibName = 'LargeList';
        var listServiceUrl = _spPageContextInfo.webAbsoluteUrl + "/_api/web/lists/GetByTitle('" + documentLibName + "')/Items?$Select=ID&$OrderBy=ID&$top=200";

        $.when(getLargeList(listServiceUrl)).done(function () {
	// Process data

If you like to include additional fields in the $Select just make sure you have column index defined for those columns.

Posted in SharePoint 2013 | Tagged , | 1 Comment

Activating Document Lib Tree navigation using CSOM

Some of you looking for how to activate treeview metadata navigation using CSOM.

Featured image

At the moment i don’t see this in the api, maybe it comes in the next CUs. But i found following working around.

The metadata navigation settings are stored in the rootfolder of the document as property bag key “client_MOSS_MetadataNavigationSettings”.

Featured image

Prerequisites: In order to activate the navigation you need  include a metadata field in the document lib and “Meteadata navigation and Filtering” feature is activated in the site.

Featured image

In your CSOM code using the following function to activate the treeview.

private static void AddMetadataNavigationToDocLib(ClientContext ctx, List docLib)
	var rootWeb = ctx.Site.RootWeb;

	ctx.Load(rootWeb, p => p.AvailableFields);
	ctx.Load(docLib, p => p.RootFolder, p => p.RootFolder.Properties);

	if (rootWeb.AvailableFields.Any(p => p.InternalName.Equals(Common.Constants.TagsFieldName)))
		var field = rootWeb.Fields.GetByInternalNameOrTitle(Common.Constants.TagsFieldName);

		ctx.Load(field, p => p.Id, p => p.InternalName, p => p.TypeAsString);

		var sb = new StringBuilder();
		sb.Append("<MetadataNavigationSettings SchemaVersion='1' IsEnabled='True' AutoIndex='True'>");
		sb.Append("<NavigationHierarchies><FolderHierarchy HideFoldersNode='False' />");
		sb.AppendFormat("<MetadataField FieldID='{0}' FieldType='{1}' CachedName='{2}' CachedDisplayName='{3}' />", field.Id, field.TypeAsString, field.InternalName, field.InternalName);
		sb.Append("</NavigationHierarchies><KeyFilters /></MetadataNavigationSettings>");

		docLib.RootFolder.Properties["client_MOSS_MetadataNavigationSettings"] = sb.ToString();
Quote | Posted on by | Tagged , , , | Leave a comment

Programmatically remove app registration

Previous post you can see how to get app’s client ID. Here is the code to remove app registration from farm.

//load the principal manager
SPAppPrincipalManager manager = SPAppPrincipalManager.GetManager(targetWeb);
//Check if the app principal is registered in this farm using client ID
SPAppPrincipal appPrincipal = manager.LookupAppPrincipal(SPAppPrincipalIdentityProvider.External, SPAppPrincipalName.CreateFromAppPrincipalIdentifier(clientId));
//if the principal not found then add
if (appPrincipal != null)

//remove the principal from farm
Posted in Uncategorized | Leave a comment

How to programmatically install an provider hosted app?

There are lots of blog posts you would find in the internet to publish, register and install an app via user interface.  Like me, you would easily run into simple customer requirement to install an app programmatically then things get harder to find. MSDN documentation is very blank in this matter. My good friend Frank and I, spend lots of hours try to install an app programmatically, just before we gave up we found a solution. So let me describe.

We had a close look at the SharePoint, to see how he does it and here are the steps we found,

  1. Load the app package binary file and list item from “App for SharePoint” library in App catalog site collection.
  2. Read the App’s Client Id from the list item.
  3. Register an with App’s Client ID. (only if not already registered)
  4. Load and install the app on a target web.
  5. Trust an app programmatically.

Client Side: You can perform all the above steps except Step 5. Because of the security reason, SharePoint doesn’t provide API to trust an app programmatically. Without this step you may get the app installed successfully but you will not get access to app. When you click on the app it will throw “You are not authorized to perform this operation”  

Server Side: Yeah baby 🙂

Before you jump in to the code below, understand where to find App’s client ID.

Product ID / App ID:When you upload the app binary file to “App for SharePoint” library in app catalog you will see that, SharePoint automatically display the Product Id in the default view.

Client ID: Client ID is available in AppManifest.xml  inside the app binary file. To register an app you need client ID. The API to register an app tells you that “App Identifier” as input parameter, in the reality it expecting client ID. This means do we have to unpack the app binary file? the answer is No. The client ID is also available in in the hidden field in the “Apps for SharePoint” library. AppPermissionXML field of the list item contains XML string with client ID.

Here we go

//check if the app is not already installed in the target web
IList<SPAppInstance> existingInstances = targetWeb.GetAppInstancesByProductId(appProductId);

if (existingInstances.Count > 0) throw new Exception(“App is already installed in the target web”);

//Get the app list item from the “Apps for SharePoint” library in App catalog
SPListItem appCatalogItem = appCatalogItems.FirstOrDefault();
if(appCatalogItem==null) throw new Exception(“App is not found in the App catalog”);

//Get the permission XML
var appPermissionXml = appCatalogItem[“AppPermissionXML”].ToString();

//read the client Id from the Permission XML
var xmlDoc = new XmlDocument();
var root = xmlDoc.DocumentElement;
var clientIdNode = root.SelectSingleNode(“//@ClientId”);

//load the client id, title and stream
var clientId = clientIdNode.InnerText;
var appItemTitle = appCatalogItem.Title;
var appPackageStream = appCatalogItem.File.OpenBinaryStream();

//load the principal manager
SPAppPrincipalManager manager = SPAppPrincipalManager.GetManager(targetWeb);
//Check if the app principal is registered in this farm using client ID
SPAppPrincipal appPrincipal = manager.LookupAppPrincipal(SPAppPrincipalIdentityProvider.External, SPAppPrincipalName.CreateFromAppPrincipalIdentifier(clientId));
//if the principal not found then add
if (appPrincipal == null)
	//end points  can be blank
	List<string> endpoints = new List<string>();

	//app  key
	SecureString secureString = new SecureString();
	DateTime now = DateTime.Now;
	SPAppPrincipalCredential credential = SPAppPrincipalCredential.CreateFromSymmetricKey(secureString, now, now);

	//create external parameters
	SPExternalAppPrincipalCreationParameters sPExternalAppPrincipalCreationParameters = new SPExternalAppPrincipalCreationParameters(clientId, appItemTitle, endpoints, credential);
	sPExternalAppPrincipalCreationParameters.SkipExternalDirectoryRegistration = false;

	//register the principal
	appPrincipal = manager.CreateAppPrincipal(sPExternalAppPrincipalCreationParameters);
//load it install in the target web
SPAppInstance appInstance = targetWeb.LoadAndInstallApp(appPackageStream);
//Trust the app
SPAppPrincipalPermissionsManager sPAppPrincipalPermissionsManager = new SPAppPrincipalPermissionsManager(targetWeb);
sPAppPrincipalPermissionsManager.AddAppPrincipalToWeb(appPrincipal, SPAppPrincipalPermissionKind.Read);
Posted in SharePoint 2013 | Tagged , , , | 2 Comments

Get App Instance ID using JSOM

I had to search MSDN longer to find an answer. Here is the JavaScript to get App instance ID using JSOM.

APP instance ID

Posted in SharePoint 2013 | Tagged , | Leave a comment

Sync button for Document Library created with custom list definition

In my previous post I documented the limitation of SP2013 and SkyDrivePro related to document libraries that are created using custom list definition. I found one workaround for this.

In your list template element specify the Type=101 (DocumentLib) and DocumentTemplate=10000. You must also specify the FeatureId attribute. Id of the Feature that deploys the List template element.xml


Adjust your list instance also with TemplateType=101 and include the FeatureId Attribute.


There you go, now we have the SYNC button.


Note that if your Event Receiver is targeted to the custom list template id (eg ListTemplateId=10000), it won’t fire. So you should change the target to the specific list instance by using ListUrl attribute.


Posted in SharePoint 2013, Uncategorized | Tagged | 15 Comments

SP2013: Sync button doesn’t appear for Document Library created with custom list definition

One of the great function in SP2013 is  the SkyDrive Pro Sync. Using that one can Sync the document libraries in their local PC.

For all the document library created using OOTB document library template SharePoint shows nice sync button on the top of the page.
Sync button

This Sync button doesn’t appear for the document libs that are created using custom list Definition.

custom doc lib

<SharePoint:SPSyncPromotedActionButton runat=”server”/> in the master page is responsible for displaying sync button in the page. After long trail and hour spend in a small subject, i throw the SPSyncPromotedActionButton in .Net Reflector and saw that SPSyncPromotedActionButton control is made visible only if the List of types DocumentLibrary (101) and MySiteDocumentLibrary(700).

Image 1

image 2

perhaps one can use jquery to inject the html make the button available but this will not work as SkyDrive Pro Client also display the document libs that are using OOTB documentLibrary template.

skydrive pro doc lib selection

I can’t understand why this limited only to the OOTB document library Definition.

Posted in SharePoint 2013 | Tagged , , , | 3 Comments