Just another WordPress site

December 21, 2015October 25, 2022

Cross-Forest Kerberos / SPNEGO issues

Recently I was confronted with an interesting Kerberos authentication issue, so I thought why not share it for prosperity’s sake.

First a bit of background on the architecture…

We were working on migrating a fairly large customer to our Online Workspace environment. The customer had separate account domains for development, testing, acceptance and production (where acceptance and production were part of the same forest, and the other two domains were separate forests) as well as separate resource domains (all single domain forests) for DTAP. The resource forests were all hosted and maintained at KPN’s Utility Hosting, the account domains were managed by the customer itself.

We setup 1-way Incoming Forest Trusts between our AD and the 4 resource forests, but unfortunately the customer’s AD admin insisted on External Trusts to the acceptance and production domains.

When the first users were migrated to our AD and the users started testing with their applications they noticed that Single Sign-On (SSO) wasn’t working for their SAP applications. When using the sapGUI it worked fine, but connecting to the web applications resulted in a login page.

The web application was configured to use Simple Protected Negotiation (SPNEGO) for SSO. After reproducing the issue while running a Wireshark trace I noticed the client was requesting a Kerberos TGS ticket for a different ServicePrincipalName (SPN) than we put in the address bar:

Doing a DNS lookup for the URL revealed they had configured it as a CNAME record instead of an A-record, so when the client did a DNS query for the FQDN in the url, it would request a TGS ticket for the real FQDN instead. No biggy, as long as there is a corresponding SPN configured on the account hosting the application.

As you can see in the packet trace above, the client (x.x.4.3) is contacting the local DC (x.x.1.2), receives a referral ticket and contacts the DC in the acceptance resource domain (x.y.1.13) where the server resides and gets a valid Kerberos ticket. So from a Kerberos point of view it all works fine, because we have a Forest Trust connecting to the resource domains, our local AD has the Name Suffix Routing information for these domains (stored in the Trusted Domain Object or TDO) and is able to issue the appropriate referral tickets…however SSO wasn’t working and the user was still prompted for credentials.

After talking to the SAP administrator I learned that the application was running under a service account in de production account domain. (the reason why they configured it this way was probably because in the past they had also configured the trusts between account and resource domains as External Trusts and couldn’t get Kerberos working between the account and resource domains). So the website FQDN was created in DNS as a CNAME pointing to the server in the acceptance resource domain, but it was running under a user account in the production account domain. (are you still following me?)

I first tried adding an entry in the local HOSTS file pointing the website address directly to the server IP (hence bypassing the CNAME to A-record translation), as a result the client was now trying to request a TGS ticket for the right SPN, but our KDC wasn’t able to issue a referral ticket because it had no Name Suffix routing information for the account domains because of the External Trusts not fully supporting Kerberos authentication. If you would like to know more about this, I would really recommend reading this (6 part!) article: https://jorgequestforknowledge.wordpress.com/2011/09/14/kerberos-authentication-over-an-external-trust-is-it-possible-part-6/

Packet trace with altered HOSTS file :

(local KDC has no routing information for the name suffix, so returns ‘PRINCIPAL_UNKNOWN’ error)

Now there is one more thing I should mention: Even if the trust had been a Forest Trust this probably wouldn’t have worked since the website address falls within the namespace of the acceptance account domain, while running as a user from the production account domain, on a server in the acceptance resource domain. (it keeps amazing me how difficult some admins make this shit)

After reading through Jorge’s blog I decided to try the Kerberos Forest Search Order (KFSO) GPO. Since I wasn’t sure how setting this GPO on the KDC would affect name suffix routing over the existing Forest Trusts, I decided to configure it on the clients instead. Unfortunately, this didn’t do jack nor shit.

So I went through the other Kerberos settings in the GPO and noticed this one: (Define host name-to-Kerberos Realm Mappings)

I couldn’t find much on this setting through Google, so I decided just to give it a try:

In our case we only needed to enable this for a hand full of servers so we added the list of individual servers to the specified realm, but if you need this to work for dozens of servers I would just enter the entire DNS suffix in there.

After configuring this GPO setting and doing a ‘gpupdate’ it worked! (well, at least after I removed the entry from the HOSTS file, because as it turned out they never configured the URL FQDN as a SPN, but only the server’s FQDN)

Voila, a valid Kerberos ticket:

And the packet trace shows that the client is sending the TGS request straight to the correct KDC:

Lessons learned:

Avoid using CNAME’s when configuring Kerberos
Avoid using External Trusts, if at all possible, always go for a Forest Trust
Keep shit simple! Publishing an application running on a server in Domain A, with a service account from Domain B and using the namespace from Domain C is, imho asking for trouble!
Enable Kerberos logging (in Windows 2012 R2 this is no longer enabled by reg_dword LogLevel=”1″, but instead in eventvwr under ‘Applications and Services Logs \ Microsoft \ Windows \ Security-kerberos’ right-click, enable)
Use wireshark (portable) and the builtin Kerberos filter.
Before reproducing or tracing always clear your existing Kerberos tickets with ‘klist purge’ command!

Cheers,

Enrico

October 12, 2015

Shovelhead 4-speed overhaul part #2

Now that we have the primary drive components out of the way we can proceed with the removal of the inner primary housing. But before we can remove the inner housing we need to get some stuff out of the way.

First I disconnected the wires from the solenoid and got reminded that I had forgotten to disconnect and remove the battery. (dumbass!) So always start by disconnecting the battery! (I did disconnect the spark plug wires though 🙂 )

After disconnecting and removing the battery (we need it out of the way anyway to get better clearance when we need to remove the starter motor later on) I removed the solenoid from the primary housing. To get the starter gear lever and the starter shaft out, we need to unscrew the pivot bolt, that holds the the starter gear lever, from the top of the housing. This bolt can be quite tight and there is almost no room because of the oil tank….so if you hadn’t guessed by now…..time to drain the oil tank and remove it from the frame!

After draining the oil from the tank, I disconnected the oil feed and return lines, removed the nuts from the mounting studs and lifted the oil tank out of the frame. (inspect the mounting stud rubbers and order new ones if needed!)

With the oil tank out of the way I could remove the pivot bolt and take out the gear lever and starter shaft assembly.

The front inner mounting bolts for the primary are secured with a safety wire, after removing the steel wire, I removed the upper and lower bolt (the center bolt doesn’t go into the crankcase), then removed the 2 rear bolts and the front 2 allen head bolts on the outside of the housing. (which were metric bolts in my case, so I first had to get a set of metric allen keys :S)

But wait…I’m forgetting something…. The starter motor is still bolted to the back of the primary 🙂

After disconnecting the wires from the starter motor, (label them! Especially if all your wires are the same shade of black, like mine :S) removed the chrome bracket from the back of the starter motor on the right side of the bike. Now it was time to unscrew the 2 studs/bolts that run through the starter motor and into the starter shaft housing on the back of the primary. (carefully lift it out as a unit, make sure you don’t pull the starter motor apart!)

Now I was finally able to get the primary out of the way and get on with removal of the gearbox 🙂

As you can see from the picture I try to make a habit out of screwing the bolts back where they came from. This can save you a lot of trouble with the reassembly!

More to come in Part #3!

October 5, 2015

Shovelhead 4-speed overhaul part #1

Until recently my 1982 Shovelhead didn’t leak a drop of oil. One of the main issues with these bikes (in regards to marking it’s territory) is a leaking primary, but mine was sealed great with the help of James Gaskets and some blue silicon. Another source for oil leakage is the oil seal on the mainshaft of the transmission. In my case this turned out to be the cause of my Shovel marking it’s territory. It started with a small drop every now and then but it got worse rather quick.

It can be difficult to pinpoint the exact source of the leakage, because the bottom of the engine, primary and tranny is often covered in lot’s of grease and dirt. So I started by cleaning the bottom of the engine and transmission with some brakecleaner, and the next morning I could clearly see the oil trail leading from behind the transmission sprocket to the bottom mounting bolts. Time to take it apart!

Before you can get the transmission out you need to get some stuff out of the way, the most obvious being the primary drive. After I drained the oil from the primary, I removed the primary cover and ran into the first challenge: the clutch.

A few years ago I replaced my clutch with a Barnett Scorpion clutch (no more slipping!), but to get the clutch hub of the mainshaft I needed two specific tools from Barnett, a Scorpion Clutch Lock Plate and a Scorpion Clutch Hub Puller.

First thing was to get the clutch hub nut loose (left hand thread! so turn clockwise to loosen!). Before you can unscrew this nut you’ll need to lock both the primary and the clutch. I used a piece of steel bar and rounded both ends using a file, stuck this between the crankshaft sprocket and the clutch shell to prevent the primary from spinning. I then used the clutch locking plate to lock the inner clutch hub to the clutch outer shell. (if you have some old clutch plates lying around you could weld 2 together to make your own locking plate)

The clutch hub nut can be very tight, so I heated it up first using a paint stripper and then, using a 1 1/8″ socket and a long steel rod over the wrench, I removed the nut. (I guess you could use an impact wrench too) Because the mainshaft is tapered, the clutch hub will be very tight, that’s why you need the puller. Be sure you screw the clutch nut back on the mainshaft for 5 or 6 turns and put a washer over the nut before you attach the puller. If you don’t, the puller center bolt will push against your clutch pushrod and you’ll brake shit. (trust me!)

Before you can remove the primary drive as a whole, you’ll need to remove the compensating sprocket from the crankshaft. This is normal right-hand thread and it takes a 1 1/2″ socket. I know some people use an impact wrench to remove the compensating sprocket, but I would personally recommend against that because there’s a risk of breaking loose the magnets on the rotor, and then you’ll have a whole new problem (especially if it goes unnoticed). Finally, loosen the primary chain tensioner and you can take out the complete primary drive.

1957 BSA B31 Restoration

For the last couple of months my brother and I have been working on his 1957 BSA B31 motorcycle. Approximately every Tuesday night we gather at his place to work on the bike. We started by removing the wheels to have new rims, spokes and tires fitted and are currently working on the engine. The cilinder has been bored to 400CC and we’re about to reassemble the engine. After that we still need to strip the frame and give it a new layer of paint, after that we’ll reassemble the bike and fit a new Amal MK1 Concentric carburetor to replace the Amal Monobloc.

You can watch the photo-blog here: https://plus.google.com/photos/105887025148501656794/albums/5908950495416198897

Update: 15-may-2014

We have been working on the bike steadily for the past few months (almost every wednesday evening). The frame was recently powdercoated along with some custom fenders, engine mounting brackets and some other small parts. The cylinder was bored to approximately 400CC’s and a Honda piston was fitted too match the bore. We’ve begun reassembling the engine and tried to bolt on the cylinder head yesterday when we discovered that the firering (the protruding ring on top of the cylinder base) was higher than the depth of the corresponding groove in the cylinder head (am I still making sense?).

Groove in cylinder head measures approximately 2.3mm

So either we have the head machined or the ring on the cylinder skimmed….

Update: 22-sept-2014

Well, as it turned out, we were wrong in thinking that the fire ring on top of the cylinder barrel should stick all the way into the groove in the head. That ring and the corresponding groove actually form the mating surface. So we put a bit of coppergrease on it and reassembled the engine.

BSA B31 engine reassembled with Mikuni carb and open airfilter

The engine is completely overhauled with new bearings and bored to 400CC’s. We also decided to invest in a more reliable oilpump made by ABSAF in Appingedam (Netherlands) and a new sump plate with magnetic drainplug by SRM Engineering.

As we were reassembling the engine, the paint work on the tanks, toolbox and front fork was also finished. Very awesome job!

BSA fueltank, oiltank and toolbox in matte black

Time to start rebuilding the bike!

We had some difficulties with the assembly of the front fork (new bushes and oil seals), and later more difficulties with the swingarm silent blocs. The silent blocs are 2 pair of bushings that fit into the swingarm with rubber in between the 2 bushes. The problem was that we forgot about them when the swingarm went away with the rest of the frame for powder coating. So when the frame and swingarm went into the oven to finish the powder coating, the rubbers vaporized, so we had to replace them. At first we thought the biggest problem was to get to outer bushes out of the swingarm, but we later realised that it was almost just as hard getting the new ones in 🙂 We finally managed to get them in using a hydraulic press and sawing a few millimeters off the ends (because we couldn’t get them in any further).

By now we have the engine and gearbox back in the frame, and both front fork and rear swingarm installed together with the wheels.

For more pictures and comments on our first project, check out: https://plus.google.com/photos/105887025148501656794/albums/5908950495416198897

November 12, 2013

Harley Davidson Shovelhead Barnet Clutch Installation

Haven’t been posting much lately, but I didn’t sit still all this time 🙂

If you would like to check my photo-log of the installation of a Barnett Scorpion clutch on my ’81 Harley FXS, please check here: https://plus.google.com/photos/105887025148501656794/albums/5932346697772510113

Barnett Scorpion Clutch on a Shovelhead — Barnett Scorpion Clutch on my Shovelhead