Premature optimization is the root of all evil

August 25, 2016

Shibboleth: Integration issues

Previous: Shibboleth: controlling access to resources

In this section I discuss non-trivial issues that arose when I tried to integrate Shibboleth with my IdP. Since I procrastinated too much, I only remember three issues worth noting:

1. Shibboleth considers it an exception when it receives a SAML response with AuthnFailed error code. You get a nasty page that looks like a bug report. The normal workflow is to have user authenticated and deny or allow access based on user’s attributes. If users is known, but access is denied Shibboleth will return a civilized 401 Unauthorized.

2. Message “URL is malformed” may happen if IdP incorrectly handles the RelayState, e.g. base64-encodes it twice. Shibboleth expects RelayState to be in certain format, and when it finds that RelayState is broken, it will say “URL is malformed” without much explanation in the logs. If you see this message, ensure that RelayState that comes back in SAML response is exactly the same as RelayState that was sent in SAML request.

3. Hosts/DNS names are very important, because Shibboleth uses cookies to keep state, and cookies are attached to hosts, not IP addresses. If your secure site is, but you try to access it via http://localhost/secure, or, it will “almost” work, but Shibboleth cookie will be lost in the redirects, and Shibboleth will land you at

nUnit test cases with dates

nUnit has a great feature of running multiple similar test cases via data-driven approach:

[TestCase("", "")]
[TestCase("q", "q")]
[TestCase("xyz", "zyx")]
public void TestStringReverse(string s, string expectedResult)
    var result = Reverse(s);
    Assert.AreEqual(expectedResult, result);

However, that does not work with dates, since DateTime is not a primitive type and cannot be used in an attribute.

[TestCase(new DateTime(...), new DateTime(...), false)] // Does not work
public void TestIntervalIsGood(DateTime from, DateTime to, bool isGood)

The solution is to supply test data in runtime, using [ValueSource] attribute. This is more code, but it works.

        // warning: not tested
        public class Interval
            public DateTime From;
            public DateTime To;

        public class TestCase
            public int Id;
            public DateTime From;
            public DateTime To;
            public bool IsGood;

            // We need this override, so test cases are shown well in nUnit. Otherwise they all look like MyTestClass+TestCase
            // We could show actual dates, but it is kind of long, so I opted for a numeric ID
            public override string ToString()
                return "TestCase " + Id;

        private static TestCase T(int id, DateTime from, DateTime to, bool isGood)
            return new TestCase { Id = id, From = from, To = to, IsGood = isGood };

        private static readonly DateTime Before = ...;
        private static readonly DateTime Inside = ...;
        private static readonly DateTime After = ...;

        private static readonly TestCase[] TestCases = 
            T(1, Before, Before, false),
            T(2, Before, Inside, true),
            T(3, Before, After, true)

        public void TestIntervalIsGood([ValueSource(nameof(TestCases))] TestCase testCase)
            bool isGood = IsIntervalGood(testCase.From, testCase.To);
            Assert.AreEqual(testCase.IsGood, isGood);

August 10, 2016

Amusing Cryptography on Windows

Amusing fact #1: .NET framework does not have built-in class to load RSA private key from PKCS#1 (PEM) representation.
I borrowed this:

Amusing fact #2: creating an RSA key requires file system access. To the user profile. If you are running under an ASP.NET app pool user that has no profile, you get this:

System.Security.Cryptography.CryptographicException: The system cannot find the file specified.

   at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)
   at System.Security.Cryptography.Utils._CreateCSP(CspParameters param, Boolean randomKeyContainer, SafeProvHandle& hProv)
   at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle& safeProvHandle, SafeKeyHandle& safeKeyHandle)
   at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()
   at {my method}

To fix, one must either create the user profile, or use CspProviderFlags.UseMachineKeyStore when creating RSACryptoServiceProvider. If you choose the latter, make sure the user has write access to C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys. I believe %AllUsersProfile%\Microsoft\Crypto\RSA\MachineKeys should be used to accommodate customized or non-English systems.

If you use machine storage, but the user does not have write access to the above folder, you’ll get
System.Security.Cryptography.CryptographicException: Access denied.

I am not certain which exact calls raises the exception: first Microsoft function on the call stack is GetKeyPair(), but I am not calling it: the actual culprit must be inlined. I suspect it is either new RSACryptoServiceProvider(CspParameters) or RSACryptoServiceProvider.ImportParameters(RsaParameterTraits), I did not have time to investigate which one.

July 29, 2016

Shibboleth: controlling access to resources

Previous: Shibboleth installation and configuration.
Next: Shibboleth: Integration issues.

SAML attributes

Shibboleth allows or denies access to a directory based on the attributes of the user it receives from the Identity Provider. If Identity Provider cannot authenticate the user, Shibboleth will show an ugly looking exception page. If Identity Provider can authenticate the user, Shibboleth will make the access decision based on user’s attributes returned by the IdP. Attributes come back as part of the IdP authentication response and contain a name and a value:

<saml:Attribute Name="mycompany.securityClearance"><saml:AttributeValue>TopSecret</saml:AttributeValue></saml:Attribute>

Attribute names and values are not part of the SAML standard and are IdP specific, although there are some semi-standard attribute names on the Internet.

Mapping SAML attribute names to Shibboleth attribute IDs

In order for Shibboleth to consider an attribute, its name must be mapped to an internal id using attribute-map.xml file typically located at C:\opt\shibboleth-sp\etc\shibboleth\attribute-map.xml. Attributes that are not mapped cannot participate in the authorization decision. For each attribute, add the following line to the attribute-map.xml file:

<Attribute name="nameAsSeenOnTheWire" id="internalShibbolethId" />

Internal Shibboleth ID is arbitrary and does not have to match the name on the wire. E.g., you may have

<Attribute name="myCompany:securityClearance" id="clearance" />

Static access control

Once you have attribute IDs defined, you can control access to protected resources using <RequestMapper> section of shibboleth2.xml configuration file. The following snippet gives access only to the users with Secret or TopSecret clearance:

    <RequestMapper type="Native">
            <Host name="ikrivpc">
              <Path name="secure" authType="shibboleth" requireSession="true">
                    <Rule require="clearance">Secret</Rule>
                    <Rule require="clearance">TopSecret</Rule>

In theory you can have different rules for different paths, but I have not tried that. Full documentation for RequestMap rules can be found in Shibboleth wiki.

Users who are authenticated by not authorized to access the resource will get error 401 (Unauthorized).

Dynamic access control

Being an ISAPI extension, Shibboleth injects mapped attribute values as pseudo-headers into the HTTP request. Dynamic content such as .aspx files can access those values and make finer-grained authorization decisions based on that, or use the attributes in any other way the see fit. For attribute ID fooBar corresponding header name is SHIB_FOOBAR (capitalized). Note, that these are not real headers, you will not see them on the wire. They exist only between the IIS web server and the code of the .aspx page.

In ASP.NET one retrieves the attribute value using expression Request.ServerVariables["SHIB_FOOBAR"]. It then can be used for information purposes or to show/hide parts of the web side, etc.

<% if (Request.ServerVariables["SHIB_CLEARANCE"] == "TopSecret") { %>
    <!-- show top secret stuff here -->
<% } %>

Note that this comes on top of any static protection that you might have: users whose attributes don’t match the rules from <RequestMap> in shibboleth2.xml will have their access denied and the dynamic application code won’t have a chance to run.

What’s next
I will discuss interesting issues I ran into when integrating my IdP with Shibboleth.

July 28, 2016

Git: how to switch to UNIX line endings on existing repository

I cloned a UINX-oriented repository, and git promptly converted CR to CRLF in all text files, which caused some problems down the line. Here’s how to revert to the original line endings:

1. Make sure there is no unsaved work – commit or stash anything that’s pending.

2. At the root of the repository create a file called .gitattributes and put the following text in it:
* -crlf
This ensures that no files are subject to the line endings conversion.

3. Clear your index. This will not physically remove any files, but will update git internal data structures (don’t forget that final dot):
git rm --cached -r .

4. Rebuild your working directory from the repository using new rules:
git reset --hard

Voila. Enjoy your untouched line endings.

July 25, 2016

Json.NET and dictionaries

TL;DR Json.NET can serialize and deserialize dictionaries with simple keys like integers or strings. It can serialize, but not deserialize dictionaries with more complex keys.

Longer version: Json.NET uses ToString() to convert a dictionary key to JSON field identifier. A simple case works like this:

[read more…]

July 23, 2016

Shibboleth Installation and Configuration

Previous: SAML and shibboleth
Next: Shibbleth: controlling access to resources

Shibboleth is an open-source SAML implementation that is used for single sign-on. We are developing a SAML IdP, and I was testing it against Shibboleth. Here are some pieces of information that I want to keep for my records. I used Shibboleth with IIS 8.


I downloaded the latest package from the the download page. At the time of writing it was shibboleth-sp- The installer targets IIS 6, so I had to perform manual steps outlined in the IIS 7 Installer page.

The scripts on that page more or less work, but watch out for that extra new line before “/+[path”: you will need to remove it for the script to work. If you don’t, you will get this error

Handler "Shibboleth" has a bad module "ManagedPipelineHandler" in its module list

If this happens, make sure to repeat the manual steps for mapping .sso extension to Shibboleth filter.

After the installation is done, Shibboleth will protect the content of the http://yourhost/secure virtual directory.


My Shibboleth configuration file. Note: this is not the actual file we use, some company specific information was removed.

Shibboleth configuration is stored in the shibboleth2.xml file. Standard configuration process involves going through the file and replacing default values such as host names with our specific data. However, default configuration uses SAML discovery protocol, which we do not support, so I had to perform more significant modifications:

  1. Copy IdP’s SAML metadata to Shibboleth configuration directory (C:\opt\shibboleth-sp\etc\shibboleth).
  2. Add the following node under <ApplicationDefaults>:
    <MetadataProvider type="XML" file="YourIdpMetadata.xml"/>
  3. Remove <SSO> and <Logout> nodes under ApplicationDefaults/Sessions.
  4. Add <SessionInitiator> node in their place:
    <SessionInitiator type="SAML2" Location="/Login" isDefault="true" id="Intranet"
    relayState="cookie" entityID="https://your.idp/EntityId">

    I am not certain what is the role of the “location” attribute. It appears to be ignored: SAML requests are sent to the URL in the metadata.
  5. If you leave it at that, Shibboleth will come up with a very cryptic message when you try to access http://yourhost/secure:
    Unable to locate a SAML 2.0 ACS endpoint to use for response.
    “ACS” here stands for “Assertion Consumer Service”. To get rid of the error, add the following node under <SessionInitiator>:

    <md:AssertionConsumerService Location="/SAML2/POST" index="1" 
    Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" conf:ignoreNoPassive="true" />

    The documentation claims that an example is distributed with default shibboleth2.xml, but it is no longer the case. However, some examples can be found in the example-shibboleth2.xml file.

Restarting Shibboleth

Some configuration changes are picked up automatically, but for others you would have to restart Shibboleth. Run the following script as administrator

net stop shibd_default
net start shibd_default

Log level

Log levels are set in configuration file shibd.logger. This is a standard log4j configuration file, change “INFO” in the second line to “DEBUG” to get more detailed output. Don’t forget to restart Shibboleth after that.

What’s Next

I will discuss how to integrate Shibboleth SP with your IdP to control access to various resources.

July 22, 2016

Outlook glitch

A couple of hours ago my Outlook at work stopped working getting stuck on “trying to connect”. The Exchange Server appeared to be alive. Restarting and rebooting did not help.

I managed to unstick it by going into File -> Account Settings -> Account Settings (button) -> Double click on account -> More settings -> Security -> Always prompt for logon credentials (wow, what a way to structure the settings dialog!).

Next time the Outlook ran it asked me for user name and password, I entered them and voila – now it can connect. After that I went back to the same setting and unchecked the box.

The weird part is that it did not give me “invalid credentials” prompt, it would just hung.

July 16, 2016

SAML and Shibboleth

Next: Shibboleth installation and configuration.

What is SAML

I am currently dealing with a project that involves single sign-on (SSO) and SAML protocol. Single sign-on means you enter your credentials once, and then use your identity on multiple web sites. The web sites must talk between each other to verify your identity without exposing your user name and password. Two leading protocols for that purpose are OAuth and SAML.

An interesting twist is that we already had a quasi-SAML SSO implementation written long time ago by some consultants from a galaxy far, far away, but it fell quite short of actually implementing the standard. SAML is an XML-based standard coming from Oasis, like SOAP, WSDL, WS-Security, etc. Like most XML-based standards coming from OASIS, SAML is a verbose soup of XML tags governed by complex rules, which are easily misunderstood. Some of the errors in our old implementation came from simply not reading the standard carefully enough, but others arose from fundamental misunderstandings, which I will cover later.


[read more…]

June 16, 2016

REST: PUT request and calculated fields

If I have a REST server and do PUT on an resource, should the server return new resource state in the response, or should the response body be empty?

RFC-7231 is suspiciously silent about that. It has a lengthy discussion on what status code to return, but says virtually nothing about the response body. It is clear that empty response body is legal, since one of possible return codes is 204 No Content, but there is no recommendation about non-empty bodies.

One can make arguments for both empty and non-empty body. PUT is allegedly supposed to create or replace the resource in its entirety (although RFC does not explicitly say that), so the client should know what resultant state it seeks. Thus, sending back full fledged resource state will only waste the bandwidth.

From the other hand, the RFC states that the server is allowed to modify client input to make it conform to the internal rules, e.g. convert the input to a different format. Additionally, the resource may contain calculated fields that are hard or impossible to obtain on the client: modification time, number of hits, current orbital momentum, link to the XKCD cartoon the server deems the most relevant to the subject, etc. In most of those scenarios the client will have to issue an immediate GET request to fetch the data back from the server, so why not just serve it back right away and save a round-trip?

Presence of calculated fields raises other interesting subjects. E.g. if the client makes the same PUT request twice and gets different calculated fields back, does it mean the PUT implementation is not idempotent? If the client sends wrong values of the calculated fields in, should PUT request be rejected by the server? But I digress.

If you search the Internet, you will find equally convincing opinions (with lots of upvotes) that the body should be empty, because it’s THE RIGHT WAY, and that the body should not be empty, because it makes client’s life unnecessarily hard.

After giving it some thought, I think the server SHOULD return new resource state when practical. I initially wrote a server that returns an empty body, and immediately found myself doing GET in the client immediately following any successful PUT. This does not feel right. Of course, there are cases when the state is too large to return (think of PUTting a 100MB file), but I feel these are the exception rather than the norm.

PS. I suspect the RFC says nothing about the response body, because the committee members could not agree on what recommendation to PUT in 🙂