Access Denied Error with App Only Access Token When Reading Profile Info

This is yet another rather strange error that I ran across and couldn’t find any info out on the interwebs about it so I though I would document it here.  Suppose you have a SharePoint App that needs to access some User Profile information.  You will probably use the PeopleManager class and ask for user profile properties using the PersonProperties class or one of the methods off of the PeopleManager class.  You write your code up using the standard TokenHelper semantics to get a user + app key to retrieve this information, i.e. something like var clientContext = spContext.CreateUserClientContextForSPHost().  In your AppManifest file you ask for (at a minimum) Read rights to User Profiles (Social).   Works great, okay, good start.

Now you determine that you need to retrieve that same information but use an App Only token.  So you use whatever method you want to get an App Only token.  You use the same code but now you get an Access Denied error message.  Why is that – App Only tokens are supposed to have the same or greater rights than user + app tokens.  Well…for right now…I don’t know why not.  NOTE:  I DO understand needing to be a tenant admin to install an app that requires access to User Profiles, but this is different; it happens after the app is installed.  But I do know how I fixed it.  I added Tenant…Read rights to my AppManifest file.  Now my App Only token is able to read properties from the User Profile in o365.  Just thought I would share this “not at all obvious” tip so that if you get stuck hopefully your favorite search engine will find this post.  Happy coding!

Updating Trust Between OnPrem Farms and ACS for Apps When Your SharePoint STS Token Signing Certificate Expires

For those of you who are “in the app way” with SharePoint 2013 (no, not a lot different from being “in the pregnant way”, as they say), you’ll reach that point sooner or later where the token signing certificate for your SharePoint STS expires if you are using low trust apps on premises.  Why is that?  Because in order to establish the trust with ACS that is necessary to use low trust apps on prem, you had to replace the token signing certificate that comes out of the box.  Most folks simply use a self-signed certificate, which when you get it from the IIS Manager snap-in is good for one year.  Even if you get it from a public root authority, they are typically good for a year or two, although there are exceptions.

All that being said, when it comes time to change your STS token signing cert then you will need to re-establish that trust between your on premises farm and ACS.  Having just beat my head against the wall for the better part of a day to get this to work I thought I’d share some tips that I used to get me through it.  Here are some things you should be looking at:

  1. Get the latest version of the PowerShell modules for MSOL.  For some reason this is always harder than it should be, so again, to save you some time, here’s where I went to get the latest versions:  http://msdn.microsoft.com/en-us/library/azure/jj151815.aspx .  This is the page that describes managing Azure Active Directory using PowerShell and it seems to consistently have the “right” links.  On this page you will find a link to the Microsoft Online Services Sign-On Assistant, which you must get and install first. Right below that you will find a link for the 32-bit and 64-bit version of the PowerShell module for Azure Active Directory.  I use this one because it has all the MSOL goodness we require.  One other thing to note – you must first uninstall your existing version, and it will tell you that you need to reboot after doing so.  Yeah, you need to do this.  You may or may not get prompted to reboot again after installing it.  The net of all this is that I tried to skate by and skip the reboots a couple times and then the Azure AD module just flat out wouldn’t install.  One reboot later everything installed just fine and using the new modules cleared the first set of roadblocks I had been hitting.
  2. Start with the scripts for establishing trust that we previously published here:  http://msdn.microsoft.com/en-us/library/dn155905.aspx.
  3. There are two modifications to the scripts you will need to make; without them you will get errors and stuff will still be broken.  Here are the changes:

    1. Find this line of in the main script:  [Guid[]] $ExistingKeyIds = Get-MsolServicePrincipal -AppPrincipalId $SP_APPPRINCIPALID | Get-MsolServicePrincipalCredential | % KeyId.  Comment it out (by putting a # sign in front of it) as well as the line that follows it:  Remove-MsolServicePrincipalCredential -AppPrincipalId $SP_APPPRINCIPALID -KeyIds $ExistingKeyIds.  At some point you may end up with key ID that is null, and that causes this part of the script to break.  I fixed it with some other PowerShell I wrote up, so that chunk of the script now looks like this:

#Remove existing connection to an Office 365 SharePoint site if the switch is specified.
    if ($RemoveExistingAADCredentials.IsPresent -and $RemoveExistingAADCredentials -eq $true) {
        #[Guid[]] $ExistingKeyIds = Get-MsolServicePrincipal -AppPrincipalId $SP_APPPRINCIPALID | Get-MsolServicePrincipalCredential | % KeyId
        #Remove-MsolServicePrincipalCredential -AppPrincipalId $SP_APPPRINCIPALID -KeyIds $ExistingKeyIds

    [string[]] $ExistingKeyIds = get-MsolServicePrincipalCredential -ReturnKeyValues $returnIds -AppPrincipalID $SP_APPPRINCIPALID | % KeyId

    foreach($id in $ExistingKeyIds)
    {
    If ($id)
    {
        Remove-MsolServicePrincipalCredential -AppPrincipalId $SP_APPPRINCIPALID -KeyIds $id
    }
    }
    }

b.  There is a second script that enumerates all of your web apps and adds each one as a ServicePrincipalName (SPN) to the AppPrincipal.  If you’ve already added those SPNs before though, then it causes all sorts of problems and you will end up getting 400 Bad Request errors anyways (as I described here:  http://blogs.technet.com/b/speschka/archive/2013/07/29/security-in-sharepoint-apps-part-3.aspx).  To fix it you just need to add one line of code to the PowerShell so it will remove the SPN before adding it.  So if the SPN already exists it will be removed before adding, and if it doesn’t exist it just gets added.  Again, the net is you end up with a working script and one SPN for each web app in your farm.  Here is the section of that script with the new line of code:

Write-Host “Adding SPN for” $NewSPN

#here’s the new line of code
$SPAppPrincipal.ServicePrincipalNames.Remove($NewSPN)


$SPAppPrincipal.ServicePrincipalNames.Add($NewSPN)
Set-MsolServicePrincipal -AppPrincipalId $SPAppPrincipal.AppPrincipalId -ServicePrincipalNames $SPAppPrincipal.ServicePrincipalNames

After doing ALL of these things, my low trust apps on premise are working again.  Hope your sailing goes a little smoother.


A Few Additional Notes

You may still experience some issues after using this code.  Here are a few suggestions:

  1. Try using the /_layouts/15/appregnew.aspx page to create a new app principal.  If things are messed up then you will see an error there, like invalid JWT token.
  2. If appregnew.aspx is not working look in the ULS logs.  You should be able to track down the invalid JWT token, and you may see a subsequent error along the lines of request aborted.  I call this part of the error out specifically because if you are just trying to deploy a new app from Visual Studio you will also see this error in there – request aborted, request was cancelled, something like that (sorry I don’t have the exact wording handy).
  3. If you are getting the invalid JWT tokens I recommend you do this:Also, I recommend when you run the script to create the trust you include the -SharePointWeb parameter.  That parameter should be the Url to a web app on your on prem farm.  If you don’t include it then you will get one randomly selected and I actually ran into a problem with that scenario in one of the farms where I was doing this fix up.
    1. Manually remove the SPServiceApplication “ACS”; use this PowerShell script to do it:  $p = Get-SPServiceApplicationProxy | ? DisplayName -EQ “ACS”, then Remove-SPServiceApplicationProxy $p
    2. Manually remove the SPServiceApplication “ACS-STS”; the PowerShell script is the same as the previous step, but use -EQ “ACS-STS”.
    3. Rerun the script to create the trust but DO NOT include any of the -Remove* parameters

  4. When you run the script to create the trust include the -SharePointWeb parameter.  This should be the Url to a web application on your local farm.  If you don’t include this parameter then you will have a web app randomly selected, and this scenario actually caused another issue in one of the farms where I was applying these fixes.

Create a List in the Host Web When Your SharePoint App is Installed…and Remove it from the Recent Stuff List

This is one of those questions that I’ve seen asked a lot but for some reason I’ve never really seen anyone post an answer to it.  I did a quick Bing search before writing this post but didn’t really see anything up there so I’m going to go out on a bit of a limb and guess that maybe there’s still some use for this kind of information.  In my particular scenario I am creating a SharePoint app and when it’s installed I want it to create a list in the host web, because I’ll be using it to store data for my application.  I’m using a provider hosted app so I’m not going to have an app web, but even if I DID have an app web, I’d rather have this list in the host web in case my app is ever uninstalled and then reinstalled later, my application data will still be there and I can pick right up where I left off.

So to get started on this little adventure, I created a new SharePoint application in Visual Studio 2013.  It began as just your basic out of the box application – I just wanted to get it up and working first to verify that all the plumbing is in place.  With that ready to go, the next thing I wanted to do is have something similar to a feature receiver that you have for full trust code that would fire when my app is installed.  Fortunately the cloud app model provides just such a thing.  I just needed to select my SharePoint App project in Visual Studio and in the properties windows there is a property called HandleAppInstalled; just double-click that to change it from it’s default value of false to true and when you do a new WCF endpoint is added to your provider hosted app.  It’s all fleshed out with a basic stub implementation, and then you can add your code in there.  That’s what we’ll begin with.

The stub implementation looks something like this:

public SPRemoteEventResult ProcessEvent(SPRemoteEventProperties properties)
{
     SPRemoteEventResult result = new SPRemoteEventResult();

     using (ClientContext ctx = TokenHelper.CreateAppEventClientContext(properties, false))
     {
          //blah blah blah
     }

     return result;
}

Now the challenge is that the CreateAppEventClientContext is geared towards creating a context for an app web, which we don’t want.  So instead we’re going to create a client context the way that Visual Studio 2012 did when you create a provider hosted app, in the default.aspx code behind.  That means we’re going to need the Url to the host web, a context token, and the host name of our application.  Here’s how we’re going to get all of that stuff.

To begin with, you can get the Url to the host web in the SPRemoteEventProperties parameter that is passed into our method.  The properties parameter has an object property called AppEventProperties that contains a URI property called HostWebFullUrl.  So we can get the first parameter we need – the host web Url – like this:  Uri hostUrl = properties.AppEventProperties.HostWebFullUrl.  The context token is actually very easy to get; it’s a property of the properties parameter, so we can get it like this:  properties.ContextToken.  The last thing we need – the host name of our application – is more difficult to get in a WCF.  That’s because you don’t have an HttpContext in a WCF, so you have to reach back into the ServiceContext to get the information you’re looking for.  Fortunately the ServiceContext contains a Host property that has a collection of URIs in the BaseAddresses property.  Here’s a chunk of code to pull out that information:

System.ServiceModel.OperationContext oc = System.ServiceModel.OperationContext.Current;

Uri localUrl = null;

//enumerate through the Host base addresses and look for the SSL connection
foreach (Uri u in oc.Host.BaseAddresses)
{
     if (u.Scheme.ToLower() == “https”)
     {
          localUrl = u;
          break;
     }
}

UPDATE:  The code above looking through the BaseAddresses works great when you’re using it with the IIS Express web site Visual Studio adds to your project when you create a new SharePoint App.  HOWEVER…it blows up when you move your code to the full blown version of IIS.  What I found in trying to get this working is that the you still get two BaseAddresses; the first one uses the HTTP scheme but IS the correct host name for your WCF service.  The second one uses the HTTPS scheme, but is the fully qualified domain name of the server on which the code is running.  As a result, the code above picks the second host name, but that is NOT the host name registered for the endpoint with SharePoint.  As a result, the code below to get a client context fails and everything blows up at that point.  NOTE:  This is because my IIS web name is not the same as my machine name, which would generally be the case in a production environment.  The code I’m using now that works with both IIS Express as well as IIS is just this:

if (oc.Host.BaseAddresses.Count > 0)
     localUrl = oc.Host.BaseAddresses[0];

I’m looking for the host that is using HTTPS, since that’s what our apps should always communicate over.  It is arguably unlikely that you would have two different host names for HTTP and HTTPS, but you never know so why tempt fate?  Now that I have the three parameters I need I can create my client context for the host web; note that I always FIRST check to make sure that localUrl is not null, which would happen if I had no SSL endpoints on my provider hosted app.  Getting my client context now looks like this:

//this is what was originally here
//using (ClientContext ctx = TokenHelper.CreateAppEventClientContext(properties, false))
using (ClientContext ctx = TokenHelper.GetClientContextWithContextToken(hostUrl.ToString(), properties.ContextToken, localUrl.Authority))

Awesome!  Now that I have a client context for my host web, as long as my user / application has sufficient rights I can create a new list in my host web.  I’m not really going to cover that in great detail because I think creating lists via CSOM is all over the interwebs.  But for completeness here’s a shortened version of my code to create a new list:

Web web = ctx.Web;

ListCreationInformation ci = new ListCreationInformation();
ci.Title = LIST_NAME;
ci.TemplateType = (int)ListTemplateType.GenericList;
ci.QuickLaunchOption = QuickLaunchOptions.Off;
                                
l = web.Lists.Add(ci);

//add a description and some fields in here  
//blah                              
                                
l.Hidden = true;
l.Update();

//this creates the list
ctx.ExecuteQuery();

Okay, all of that is great, I now have my list in my host web.  There is one important aspect to this scenario though, and that is that my list should be hidden from users.  Now you might think that I have that covered with my ListCreationInformation (ci.QuickLaunchOption) and the Hidden property of the list…but that is not enough.  Unfortunately SharePoint still throws it in the Recent bucket that shows up on the Quick Launch navigation.  In fact the Recent list is NOT a list, it’s a collection of NavigationNode items.  So if you REALLY want your list to be invisible, you need to remove it from over there as well.  Doing that is three step process (NOTE: I’m leaving out all the error handling and try…catch blocks for read-ability; you would of course want that in any code you write…but you knew that, I know…  🙂  ):


Step 1:  Get the quick launch navigation node collection

In this step you’re going to make a call back to get the root web of the site to get the quick launch navigation nodes.  This code is just continued from the create list code above, so it’s still in the “using” statement for my ClientContext (ctx) shown above:

//get the site and root web, where the navigation lives
Site s = ctx.Site;
Web rw = s.RootWeb;

//get the QuickLaunch navigation, which is where the Recent nav lives
ctx.Load(rw, x => x.Navigation, x => x.Navigation.QuickLaunch);
ctx.ExecuteQuery();

//now extract the Recent navigation node from the collection
var vNode = from NavigationNode nn in rw.Navigation.QuickLaunch
     where nn.Title == “Recent”
     select nn;

NavigationNode nNode = vNode.First<NavigationNode>();

Step 2:  Get the Child Nodes of the “Recent” Navigation Node

Okay, now that I have the “Recent” navigation node, I need to populate it’s child property, which is where I should find a node for the list I just created.  This gets simpler now, here is the code to retrieve that:

//now we need to get the child nodes of Recent, that’s where our list should be found
ctx.Load(nNode.Children);
ctx.ExecuteQuery();

Step 3:  Find The Node for the New List and Delete It

Now that I have the collection of items in the Recent navigation, I can find my item and delete it like you would any other item via CSOM.  Here’s the code:

var vcNode = from NavigationNode cn in nNode.Children
     where cn.Title == LIST_NAME
     select cn;

//now that we have the node representing our list, delete it
NavigationNode cNode = vcNode.First<NavigationNode>();
cNode.DeleteObject();

ctx.ExecuteQuery();

And there you have it.  You’ve now created a new list in the host web, made it hidden and not shown on the quick launch bar, and removed it from the Recent navigation list so it is REALLY out of sight.

 

Missing Context Token in Low Trust App with SharePoint 2013 – Part 2

As I noted in my first post on this topic – http://blogs.technet.com/b/speschka/archive/2013/05/28/missing-context-token-in-low-trust-app-with-sharepoint-2013.aspx – you may find yourself in a low trust app without a context token, which basically leaves you up the creek without a paddle.  I ran across another scenario where this was happening so thought I would add it to my list of app model troubleshooting tips I’m keeping here on Share-n-Dipity.  In this case, I had added a AppInstalled receiver to my application, and some code within it was failing. There weren’t any unhandled exceptions, but nonetheless when the page in my provider hosted app was loaded, it contained no context token.

The way I finally started narrowing this problem down was to remove the Xml for the AppInstalled receiver from the AppManifest.xml file for my application.  Specifically, I commented out this:

<InstalledEventEndpoint>~remoteAppUrl/Services/AppEventReceiver.svc</InstalledEventEndpoint>

With that commented out, my page began getting the context token again.  That of course then allowed me to redirect my troubleshooting efforts towards my event receiver.  Overall it fits with your debugging 101 concepts of always trying to reduce scope to the smallest reproducible scenario.  Just add this to the list of things to check off if you find yourself missing a context token with a low trust app.

 

Getting the Current User Identity in a Low Trust App in SharePoint 2013

Today’s post comes primarily because it’s a common question, but not because it’s a particularly brilliant answer; maybe just a bit unexpected.  If you followed the some of the differences between low trust and high trust apps in SharePoint, you will know that in a low trust app, SharePoint knows who the user is, versus a high trust app where the app tells SharePoint who the user is – see Security in SharePoint Apps Part 3 for more details (http://blogs.technet.com/b/speschka/archive/2013/07/29/security-in-sharepoint-apps-part-3.aspx).

The common misconception here though is that you can look at the context token that SharePoint sends over to determine who the user is that is making the request.  I explain more about the context token in Part 4 of the Security in SharePoint Apps series:  http://blogs.technet.com/b/speschka/archive/2013/07/30/security-in-sharepoint-apps-part-4.aspx.  If you look at the contents of the context token you’ll see that there really isn’t a single thing in there that uniquely identifies an individual.  The solution to this problem then, is that you need to make a CSOM call to actually get the user’s identity.  Fortunately this actually pretty easy; what I’ve got here is a slightly modified version of the basic code that Visual Studio generates for you when you create a new low trust SharePoint App project in a provider hosted web:

var contextToken = TokenHelper.GetContextTokenFromRequest(Page.Request);
var hostWeb = Page.Request[“SPHostUrl”];

using (var clientContext = TokenHelper.GetClientContextWithContextToken(hostWeb, contextToken, Request.Url.Authority))
{
 clientContext.Load(clientContext.Web, web => web.Title, user => user.CurrentUser);
 clientContext.ExecuteQuery();
 Microsoft.SharePoint.Client.User curUser = clientContext.Web.CurrentUser;
 Response.Write(“Current user is ” + curUser.Email);
}

I highlighted the parts there to illustrate the main difference with the code that you get out of the box.  As you can see, you can just use the CurrentUser of the context web in order to get information about which user is actually using your application.  As I said, not difficult, but maybe not the answer you were expecting.

 

Another Tip if Visual Studio Times Out Trying to Deploy Your App in SharePoint 2013

I’ve had this happen to me, seemingly randomly, a number of times before.  You open Visual Studio and create a new SharePoint App.  You go through the wizard and then hit F5 to run, deploy and debug your app.  Something happens along the way though…you watch the output window and it just keeps counting time while it’s trying to deploy your app.  A few seconds, becomes a few minutes, which becomes several minutes, and then finally it just quits and says sorry, there was a deployment error.  Now no more walking away, looking at the sky and shouting “SERENITY NOW” friends – instead, here’s a tip that will hopefully help you.

When I had this happen to me (again) today, I changed the SharePoint site collection to which I was deploying the app to one I had successfully deployed before.  Sure enough, it worked like a champ.  Hmm, that got me to wondering.  Rather than looking at the whole side loading apps feature (which you really should not try and use as a work-around), I went to the problematic site collection and just went into the appregnew.aspx page (i.e. https://yourSiteCollection/_layouts/15/appregnew.aspx).  I generated a new client ID in there and registered my app, then I took that client ID and pasted it into my appmanifest.xml and web.config file.  I tried deploying to that site collection again and – voila! – sure as I’m the master of my own domain, the app deployed successfully and started running.

I’ve heard of many other folks hitting this issue…this is nearly as frustrating as those who can take the reservation, but can’t hold the reservation, so hopefully this will buy you some peace and happiness.

How To Control App Token Lifetimes in SharePoint 2013

Today’s post is the first selection from the little twitter contest I announced on the Share-n-Dipity blog a few days ago:  http://blogs.technet.com/b/speschka/archive/2013/09/04/use-social-tools-to-tell-me-what-you-want-to-see-here-next.aspx.  Shariq wanted to know more about the lifetime for high trust app tokens, as he tweeted here:

So we’ll look at where you can control that, but we’ll also see that at the end of the day it really doesn’t matter.  The keys to controlling the token lifetime for apps in a high trust app is in TokenHelper.  If you look at the IssueToken method you’ll see that there are one or two JWT tokens created – one if you are using an app only token, and a second if you are using tokens for the app and the current user.  In both cases it creates a JsonWebSecurityToken, and the fourth parameter used to create that token defines how long the token is valid for.  For the app token, TokenHelper uses a constant called TokenLifetimeMinutes that is set to a value of 1,000,000 by default.  Definitely not a value you should need to worry about in any scenario, unless of course you wanted to limit it (which is really the more interesting scenario).  For the user token, it is hard-coded by default to use a lifetime of 10 minutes.  So if you drill into the IssueToken method you can find those values and change them to whatever you want.  So why doesn’t it really matter?

Well, if you dig further into the code, you see that TokenHelper includes a delegate that’s invoked in the GetClientContextWithAccessToken method.  It’s whole role in life is to add a bearer token to any request that comes from your app into SharePoint.  You can see it here:

clientContext.ExecutingWebRequest +=
                delegate(object oSender, WebRequestEventArgs webRequestEventArgs)
                {
                    webRequestEventArgs.WebRequestExecutor.RequestHeaders[“Authorization”] =
                        “Bearer ” + accessToken;
                };

If you set a breakpoint on that line of code and then step through a request from your app, you’ll see that none of the token “setup” functions – i.e. those that create the ClientContext – actually result in an HTTP call.  It’s only when you call the ExecuteQuery method on the ClientContext object that a request goes over the wire, and then the delegate fires and the token is added to the request.  So getting back to really the intent of Shariq’s original question, while you CAN cache the access token you get in a high trust app, there’s really not a good reason to do so (at least that I can think of so far), because it doesn’t generate a call to SharePoint anyways.  Just for proving this out I wrote a little sample app where I played with setting the token lifetime for the user to 5 minutes, but I cached the access token for 1 year.  Sure enough, after 5 minutes had passed all of my app calls started failing with 401 – unauthorized errors, which is exactly what you would expect to see.  I’ve attached a zip file to this posting with that sample application.

Now…where caching app tokens DOES help is when you are using low trust apps.  In that case you absolutely do save one or more roundtrips to ACS to get an access token if you are caching them.  If you have lots of users and/or lots of requests this can really add up.  I won’t bother going into the details of how to cache tokens for low trust apps here, because I’ve already posted about that in the series on app security.  For more details, see part 4 of that series at http://blogs.technet.com/b/speschka/archive/2013/07/30/security-in-sharepoint-apps-part-4.aspx.

Hopefully that explains the concepts in a little more detail and will be useful to you in your application security designs.  That was a great question Shariq, and thanks for responding to the twitter ‘test.  Feel free to continue to pile your requests in there.

You can download the attachment here: