Adding Attributes to the FIM MV Schema Programmatically

Adding new attributes programmatically to the Metaverse is currently not supported by FIM in any of its forms, however, if you are willing to take the chance and make the changes needed directly to the database, this is possible.  This isn't necessary recommended and it is definitely not for the faint of heart. However, if you are determined to give it a try, here is what you have to do. First, make a backup of your database, then:

For single-valued non-reference attributes:

1. Add a column to the mms_metaverse table with the appropriate name and data type
2. Add a column to the mms_metaverse_lineagedate table with the appropriate name and a data type of datetime
3. Add a column to the mms_metaverse_lineageguid table with the appropriate name and a data type of uniqueidentifier
4. If you want the attribute to be indexed - add a new non-unique, non-clustered index to the mms_metaverse table with the name IX_mms_metaverse<attributeName>

Then, for all attributes (single or multi, reference or not) you will then need to:

1. Update the mv_schema_xml column of the mms_server_configuration
2. Close the client, restart the Sync Service and re-open the client

I have only done some preliminary testing, but so far it has worked okay. I have worked out a SQL Script to help you through the process.  Its called MVAttributeCreationScript.sql and you an download it from http://sdrv.ms/11oBRr8.

If you are uncomfortable attempting to update the schema in this fashion, there are a couple of other possibilities. However, these also carry their own risks:

1. Export the MV Schema, update the xml export file and then re-import it
2. Create a key stroke script to perform the updates through the UI as a user would

I will create some additional blog posts to expound on these two methods a little bit further, so be on the lookout!

The Last FIM Metaverse Extension You Will Ever Need

The third tool in the “The Last of” series is really another option for codeless provisioning (with out having to go to the FIM Service/Portal).  The Last FIM Metaverse Extension You Will Ever Need takes advantage of the configuration of a special MA to determine the initial flows needed to provision a new object.  This is done in two parts.  Here’s how it works. 

The Provisioning Management Agent
You will need to create an MA a special MA of type “Provisioning Management Agent (Insight)”.  This is an ECMA that inspects the configuration of the FIM Synch engine, allowing you to use the UI to define the provisioning rules for the other MAs in the environment.  You will not create run profiles for this MA, it will never be executed in that manner.

First, make sure you install the packaged Provisioning MA and the Insight.FIM.CodelessProvisioning.dll from http://fimmv.codeplex.com. Now, when configuring the MA, the first thing you will need to do (after giving it a name) will be to use the Connectivity tab to enable provisioning.  You can do this by MA and by MA object type.  One nice thing about this set-up, you can selectively turn on and off provisioning with out having to change any of the flow rules defined.  In this example, I have two MAs, the HR MA with one object type of person and a CRM MA with two types, person and group:

I have enabled provisioning for person objects in the HR MA and then clicked Next. Then, I clicked Next on the Configure Partitions and Hierarchies tab, we will be using the default values.

On the Object Types tab, you will see a list of Management Agents.  In essence, what this page is asking you is which MAs you want to define provisioning rules for.  At a minimum you will want to select the MAs that have at least item selected from the Connection tab. In this case I am going to define provisioning rules for the HR MA, which I have enabled for provisioning.  Additionally, I could define rules for the CRM MA if I plan on enabling it at some point in the future:

On the next tab, you will want to select the attributes that you are going to define initial flows for.  At a minimum you will need to check the attribute “Anchor”.  I have selected a few of the basic attributes I want to set during provisioning:

Next is the Anchors tab, leave the default values (Anchor attribute = Anchor) and click Next.

You should now be on the Configure Connector Filter tab, the provisioning code currently does not use this tab, but that could be a useful enhancement in the future.  Leave the filters blank and click Next.

Ditto with the Join and Project Rules, these won’t be used, leave them blank and click Next.

Now comes the interesting tab.  On the Attribute Flow section, define the flows that you want as your initial flows as Export flows for the MA and object type.  For example, if I want the ID, DisplayName, Email, FirstName and LastName set on a new HR MA person when the object is provisioned, I will define an Export flows for each of those:

Currently, only direct and constant flows are supported.  Although I am working on adding Advanced flows via the same mechanism as The Last FIM Management Agent Rules Extension You Will Ever Need in which the C# code that defines the advanced flow is placed directly in the Flow rule name.  Click Next when done defining the flow rules.

Click Next through the Deprovisioning and Extensions tab (you may need to provide a dll for the Rules extension name, you can simply use the Insight.FIM.CodelessProvisioning.dll).    That’s it!  You should now see your new Provisioning MA in the Sync client.  Don’t create more than one of these, currently only the first Provisioning MA found will be used.

Unfortunately, should you rename one of the MAs configured, you will need to update the schema of the Provisioning MA and then redefine the flow rules.

The Metaverse Extension
Okay, now all you need to do get this working is set the Metaverse Rules Extension to use Insight.FIM.CodelessProvisioning.dll and enable it:

Here’s what happens next.  When the provisioning code runs, it will go and look for a Provisioning MA.  It will then transverse the configuration of the MA to determine which objects to create and which flows to apply.  In this example, the provisioning code will see that I need to provision a new HR MA object of type person and set the ID, DisplayName, Email, FirstName and LastName on the object using the flow rules I defined.

Its an interesting concept and provides a well known UI to set up codeless provisioning without the overhead of the FIM Service. 

Let me know what you think!

The Last FIM Management Agent Rules Extension You Will Ever Need

For the next in the Last FIM series, I extended the concept introduced in The Last FIM Workflow You Will Ever Need to the Synchronization Engine.  In this case, this Management Agent Extension will take the Attribute Flow Name provided, compile it and run it like code.  So as an example, if you were creating an import flow to displayName and wanted to calculate it by concatenating  firstName + " " + lastName from the connector space, you could do the following as the Flow rule name: mventry[“displayName”].Value = csentry[“FirstName”].Value + " " + csentry[“LastName”].Value:

In order to make this work, download and install the CodePlex project from http://fimma.codeplex.com, then reference the Insight.FIM.CodelessSync.dll on the MA Extensions tab:

A few things to note, currently the code that you can place in the Flow rule name is limited to a single line, so only fairly simple calculations can be performed, but it may keep you from having to go to the Portal for "codeless" sync rules.  Only C# syntax is currently accepted, although a VB version could be written.  There may also be some value in writing a version that uses its own simplified syntax.  Only reference the .Value property of any mventry or csentry attribute.  The extension will inspect the data type to determine how to handle it from there.  Also, this can be nicely combined with an existing MA Extension code you might have, simply use it as your default call on your switch statement for import or export flow code:

public void MapAttributesForImport(string FlowRuleName, CSEntry csentry, MVEntry mventry)        {            switch (FlowRuleName)            {                case "existingFlowName":                    //existing code                    break;                default:                    runCommand(FlowRuleName, mventry, csentry, "mventry");                    break;            }                   }

The extension is fairly new, so may still need some tweaking, but I wanted to get it out to the community for your feedback and contributions.  Let me know what you think!

Also, be on the watch for the next in the Last FIM series, The Last FIM Metaverse Extension You Will Ever Need.

The Last FIM Workflow You Will Ever Need, Part 2

A year ago I introduced the idea of a FIM Workflow that would allow you to use the UI to write the code that you wanted executed at run time. Since then I have updated the workflow to allow you to also specify additional references and using/import statements. 

I also got approval to upload the project to CodePlex so that you can have access to it.  Check it out at http://fimwf.codeplex.com and let me know what you think!

Be on the lookout for the next item in Last FIM series,the Last FIM Management Agent Rules Extension You Will Ever Need.

FIM 2010 R2 Lab on Windows 7

I recently went about building out a FIM Lab on Windows 7 (64 bit).  Its completely unsupported, but in case it helps anyone, here is what I had to do.

With the basic FIM 2010 R2 installation instructions in mind, I began by installing IIS and SQL Server 2008 R2 on my Windows 7 machine.  Since this environment is just for my own personal lab use, I created all of the needed users and groups locally on the box.  Now comes the first challenge, getting SharePoint installed which requires a Server OS.  In order to install WSS, I downloaded and installed a neat little application called WSSOnVista from Bamboo Nation. This application uses API hooking to intercept the GetVersionEx call, fooling the install into believing its running on Windows Server. I used WSSOnVista to install WSS 3.0 with SP2.  Be sure to follow the instructions listed on the Bamboo Nation site, including using the WSS Advanced install option and selecting the Web Front End server type.

You will then need to download and install Orca.  You can use Orca to edit the FIM MSI packages, removing the server version checking requirements. You will need to copy the FIM install bits to some media that they can be edited (like a local drive).  Begin by right clicking the Synchronization Service.msi file and then selecting the “Edit with Orca” option.  You will then need to locate and remove the server OS version Launch Condition.  Right click the appropriate Row and click on the Cut Row(s) or Drop Row option:

Save your changes and launch the Synchronization Service.msi and install as you normally would.  You will need to perform a similar action on the Service and Portal.msi file, however, this time you are looking for the OSCheck under the CustomAction table.

Again, save the changes made to the msi file and then use it to install the FIM Service and Portal as you normally would.

I haven’t encountered any issues running on Windows 7 yet, but will update this article if any come up.

FIM PCNS And A Lack of Trust

If you read through the FIM documentation for setting up PCNS, you will find that either FIM and PCNS need to be in the same forest or be part of a forest trust. From http://technet.microsoft.com/en-us/library/cc720594(v=ws.10).aspx:

Forest trusts are only required if PCNS and ILM 2007 are located in different forests. If this is the case, a forest-level trust must be established. This is required for Kerberos mutual authentication for the ILM 2007 server to accept the request from a remote forest host.

This can become extremely limiting, especially if both forests do not have the proper forest and domain levels.  Theoretically, however, it’s possible to make Kerberos work over an External Trust (http://blogs.technet.com/b/activedirectoryua/archive/2010/08/04/conditions-for-kerberos-to-be-used-over-an-external-trust.aspx).  With an External Trust, we don’t have any of the same forest functional level restrictions, opening up PCNS as a viable option for more customers.  I would love to know if anyone has been able to get PCNS working in this configuration.  In the mean time, I will have to set up a lab and try it out.  I will report back on the results and let you know!

The Last FIM Workflow You Will Ever Need

Okay, probably not, but its still pretty cool.  I recently built a Workflow in FIM that will allow a user to define the code they want executed at run time.  The Workflow will compile the code and run it in a separate app domain when its initiated (http://www.west-wind.com/presentations/dynamicCode/DynamicCode.htm).  The UI will allow you to specify the input parameters and where the result of the code execution will go.  Here is a simple example, I am passing in the first and last name from the target of the request, I am also passing the the mail host that I have calculated in a previous step and saved to WorkflowData.  I am then using the code to calculate the user’s email address and save it to the email parameter on the target:

You can add and remove input parameters as needed during design time and then refer to them in the code using their specified index. I expect in future releases to be allow the user to specify additional namespaces and code references to import. It currently won’t do any syntax checking or formatting at design time, but otherwise its extremely versatile and flexible! 

UPDATE: Check out Part 2 of this article for the latest information and download instructions!

Sun One Boolean Attributes

Recently while working with the Sun One MA, I came across a problem caused by how FIM interprets boolean data coming from the directory.  In this case, the Sun One directory stores its boolean data as “YES” or “NO”, however when imported by FIM, this data always gets converted to False in the connector space.  (http://social.technet.microsoft.com/Forums/en-US/identitylifecyclemanager/thread/8ffc112e-d945-4916-83b3-78fbba716705)

Now we can’t use an advance import flow to correctly convert the data as it comes into the Metaverse because we have already lost data integrity.  The data that’s in the connector space is basically useless.  This only leaves us with a couple of options, we can write an XMA to correctly handle the data, but this can be fairly complicated.  We can also update the directory schema and use a string instead of a boolean, however, this could cause other downstream issues with other systems that might be consuming this data.  There is one other option, but its completely unsupported.  You could update the FIM database to make the system think this attribute is a string. 

Begin by running the following SQL Statement using the SQL Server Management Studio against your FIM database:

SELECT CAST(ma_schema_xml AS XML)   
  FROM FIMSynchronizationService.dbo.mms_management_agent
 WHERE ma_name = '<Sun One MA Name Here>'

In the results window you will get a single record that can be clicked on to open an xml document containing the schema for your Sun MA. Each attribute in the directory will appear using this syntax:

<attribute-type id="system_assigned_id" single-value="true/false">
     <name>attribute_name</name>
     <syntax>LDAPv3_syntax_oid</syntax>
</attribute-type>

If you do a quick search (Ctrl+F) for your attribute, you can manually update the LDAP Syntax OID from a boolean (1.3.6.1.4.1.1466.115.121.1.7) to a string (1.3.6.1.4.1.1466.115.121.1.15).  In my case, the new XML looked like:

<attribute-type id="Ah" single-value="true">
     <name>isManager</name>
     <syntax>1.3.6.1.4.1.1466.115.121.1.15</syntax>
</attribute-type>

I then used an update statement to write this information back to the database.  After performing a full import on this MA the connector space now reflected this data as a string of “YES” or “NO” just as it appears in the directory.  Now that the connector space had data I could work with, I wrote a quick advanced import flow rule to transform this value to a boolean:

case "isManager":

   //perform conversion for isManager from yes/no to boolean value
   if (csentry["isManager"].IsPresent)
   {
      if (csentry["isManager"].Value.Equals("yes", StringComparison.InvariantCultureIgnoreCase))
      {
         mventry["isManager"].BooleanValue = true;
      }
      else
      {
         mventry["isManager"].BooleanValue = false;
      }
   }
   break;

This approach does have its risks.  It will need to be re-done after performing a schema refresh in FIM and there is no guarantee it will keep working after an upgrade/patch.   The following SQL query could be used to script out this update, just replace the attributeName and SunOneMA variables and you should be good to go:

DECLARE @attributeName AS Varchar(255) = '<Attribute Name Here>'
DECLARE @SunOneMA AS Varchar(255) = '<Sun One MA Name Here>'
DECLARE @newSchemaOID AS Varchar(255) = '1.3.6.1.4.1.1466.115.121.1.15' --Directory String

-- get schema data
DECLARE @schema AS XML

SELECT @schema = CAST(ma_schema_xml as xml)
  FROM FIMSynchronizationService.dbo.mms_management_agent
 WHERE ma_name = @SunOneMA

-- update schema type
SET @schema.modify('
   declare default element namespace http://www.dsml.org/DSML;
   replace value of
      (dsml/directory-schema/attribute-type[name=sql:variable("@attributeName")]/syntax/text())[1]
   with
      sql:variable("@newSchemaOID")
   ')

-- save back to table
UPDATE FIMSynchronizationService.dbo.mms_management_agent
   SET ma_schema_xml = CONVERT(nvarchar(MAX), @schema)
 WHERE ma_name = @SunOneMA

FIM 2010 and Language Types

Recently, while working with James we had to support synchronization of identity data that used language types/codes.   Here’s a little background…

Directories that support RFC 3866 (in this case Sun One) allow you to specify the language of an attribute’s value.  Here’s a simple example, say your company has a presence in the States and in Germany and you want to be able to store the user’s title “Software products” in both English and German, you can do that like this:

title;lang-en: Software products
title;lang-de: Softwareprodukte

Now, FIM will support reading these attributes out of the directory and storing them into the CS, up to a point.  If you were to import this attribute, what you would get is a single multi-valued title attribute with both values, at which point you have lost all of the language information:

In order to preserve this information, we ended up writing our own XMA to bring in the attribute with its language type as an attribute in its own right.  So using the DirectoryServices (or DirectoryServices.Protocols, if you prefer) namespace we were able to access the directory and retrieve the full name of the attribute, we then wrote this out to our file so that FIM would see them as different items. (If you end up writing your XMA using the LDIF file format, like we did, you will have to actually transform the attribute name on the way out to the file to remove/replace the “;” or you will continue to have the issue above!)

We can now flow these attributes with the language information intact to other downstream systems.

XMA Creation Feature

So recently while creating some XMAs for FIM 2010 I noticed that while the initial creation page looked like this:

When I opened the XMA after it was created I got:

“One of these things is not like the others….”  Even though I didn’t select the checkbox for the “Run this management agent in a separate process” option, the FIM Sync Engine created it that way.  It will remember the setting from this point in, but if you forget, this can make attaching the debugger and stepping into your code a little more difficult.  (Just as any FYI, I usually do turn this feature on when I am running the XMA in a QA/Production environment to minimize the chance that I could bring done the main sync miiserver process).  This only appears to be an issue the XMA, I haven’t noticed this issue with any of the other MA types.

Exchange Room Resources

Recently while trying to provision Room Resource Mailboxes via FIM, or more specifically I was setting their Access Booking Policies via the Set-CalendarProcessing command in a PowerShell Workflow, I received the following error:

The values for ResourceMake, ResourceModel, and ResourceType must be included in the ResourceProperties collection.

You may also see this error if you try to open the object in the Exchange Management Console and try to save it. 

Upon further inspection, the issue appeared because the msExchResourceSearchProperties attribute didn’t contain the resource type, in this case “Room”, anywhere in the multi-valued list.  After manually adding this value to the existing list, this issue was resolved.  Since we are getting this list from user input in the portal, a more permanent solution will involve adding a Workflow to ensure that “Room” appears in this list when we add/update it in the portal.

TEC 2011

For those of you who were able to make it to my presentation at TEC 2011 (State side), I promised a blog entry going into some more technical detail on the Ensynch Accelerated SQL XMA (coming soon).  If you are interested in the slides I presented you can get them here.  Jeremy also had another suggestion that I will be trying out, so keep an eye out, I will let you know how it goes!

Custom Attributes in a Function Evaluator Workflow Activity (or lack there of)

Okay, so as some of you have found out if you bind a custom attribute to a custom resource in FIM, those attributes don’t show up in the Function Evaluator Workflow Activity drop-downs here:

 

or here:

There are two ways to go about fixing this, you can bind the custom property to a “known” resource type, like user.  Or you can skip the lookup and go ahead and reference them anyway using the CustomExpression option.  This will mean hand typing the attribute name into the Destination and Custom Expression fields, but it will keep your bindings cleaner:

File Based Management Agents In MIIS/ILM/FIM

I had a recent need to really compare the capabilities of each of the file based Management Agents in FIM.  Can you name all five? Don't worry, I won't leave you hanging, they are:

• Attribute-value pair text file
• Delimited text file
• Directory Services Markup Language (DSML) 2.0
• Fixed-width text file
• LDAP Data Interchange Format (LDIF)

Here are some of the things that they can and can't do (this is for you Joe) and just for kicks, I also added in the SQL MA. If you are using one of these file types in an Extensible Management Agent (XMA), the following still applies: 

	Multi-valued Attributes	Attribute Level Updates 1	Multi-valued Level Attribute Updates 2
Attribute-value pair	YES	NO	NO
Delimited	YES 3	NO	NO
DSML	YES	NO 4	NO
Fixed-width	YES 3	NO	NO
LDIF	YES	YES	ON IMPORT ONLY 5
SQL MA	YES	YES	NO 6

Okay, now for the caveats (can’t get away without some of those):

1. An Attribute Level Update implies that a delta import can contain only the attribute that has changed (along with the other required columns, like the type of change and the anchor)
   
   So, here’s what that might look like.  Suppose I have a user with the following attributes:
   

   	ID:	12345
   	Name:	Sarah
   	Status:	Active
   	Phone:	555-123-4567

                                        
   If Sarah’s phone number changes to 555-987-6543, I can simply tell FIM something like: 

   	ID:	12345
   	Type Of Change:	Update
   	Phone:	555-987-6543

   
   This has the advantage of giving FIM less work to do to determine what has changed on the records being imported and greatly speeds up delta imports. 
    
2. A Multi-valued Level Attribute Update supports adding and deleting specific values from a multi-valued attribute
    
   Let’s take another look at Sara’s record:
   

   	ID:	12345
   	Name:	Sarah
   	Status:	Active
   	Phone:	555-123-4567
   	Phone:	555-456-7890

                                         
   Now, Sarah has two Phone numbers, or a single attribute with multiple values. With multi-value level attribute update support, we can do things like add a new phone number to the list, delete a phone number from the list or update a phone number (in essence by doing an add of the new value and then a delete of the old one):

   	ID:	12345
   	Type Of Change:	Add
   	Phone:	555-987-6543

   
   Without this support, the source system would be required to do a “replace” action and provide FIM with all of the current values at the time of import which FIM will use to override all the values that it has for that attribute.  So if we start with Sarah’s record as listed just above and add the phone number 555-987-6543 and remove the phone number 555-123-4567, we would have to pass:
   

   	ID:	12345
   	Type Of Change:	Replace
   	Phone:	555-987-6543
   	Phone:	555-4567-7890

   
   As with attribute level updates, multi-valued level attribute update can greatly reduce the amount of work that FIM needs to accomplish.  To illustrate, just imagine applying this scenario to attributes like member on an AD group that can have thousands of values.
    
3. Using a multi-valued attribute in a delimited or fixed-width file requires the use of a header on the import file
   
   So for a comma delimited file this would look like: 
                                          ID, NAME, PHONE, PHONE, PHONE
                                          12345, Sarah, 555-123-4567, 555-987-6543, 555-456-7890
   
   This would import a record for Sarah with three attributes - ID, NAME and PHONE, the last of which will have three values. A fixed width file would work the same way.
    
4. While the DSML specifications themselves can actually handle attribute level updates using the addRequest, delRequest and modifyRequest operations, FIM only implements the ability to import a SearchResultEntry element which must contain all of the attributes on the object
   
   Just a side note for those that might be curious, you can actually place the addRequest, delRequest and modifyRequest nodes in the DSML file.  FIM will be able to parse the file and it wont cause any errors, however these elements are completely ignored and aren’t processed by FIM.  I also tried sending a DSML delta to FIM with just the attribute that changed and a change type of “modify”, and I suppose not surprisingly, the object in the connector space was updated so that it only had the one attribute I specified,  all the other attributes originally on the object were removed. Had any of these attributes been defined as required, this update would have failed.
    
5. While you can import an update to a specific value in a multi-valued attribute, if you were to export this same change to an LDIF file, it will come through as a replace operation containing all values now present on the attribute
    
6. While the SQL MA does not support updates to a specific value on a multi-valued attribute out of the box, I hear rumor that some customizations can be done to make this happen

Small Bug Found in MIIS/ILM/FIM Identity Manager UI….

Okay, so it’s so small its hardly worth mentioning.  However, if you happen to run into the error “no-start-file-open” when running an import/export step or see the following message when trying to browse to your files while configuring the import step, you may be a victim of this issue.

The problem occurs when you have given your MA a name that ends in one or more periods (“.”).  While this is considered legal by the Identity Manager UI (FYI - Identity Manager will not allow periods at the beginning of the MA Name), the periods will get stripped off of the MaData folder automatically by Windows.  In my test case I named my MA “ma ,, test – with __special ,, characters – ..”, a bit excessive I know, but hey, I was testing.  However, my MaData folder actually turned out like:

You can see why ILM could then have a problem finding the files specified in the run profile since its using the MA Name to determine the file path (i.e. its looking for it in D:\Program Files\Microsoft Identity Integration Server\MAData\ma ,, test – with __special ,, characters – ..\).  Further proof that this is the case can be found in the Event Viewer:

The fix?  Simply open the MA properties in Identity Manager, remove any periods from the end of the MA name and it should begin working as expected.

XPath in FIM Sets

Recently, while working with Brad on a FIM implementation, he told me about some issues he was having with the restrictions placed on the XPath used when building a new Set. Specifically, he wanted to be able to find people that were missing a specific attribute. This was a little tricky, but I finally came up with a solution for strings and another one for dates:

/Person[not(starts-with(<AttributeNameOfTypeString>, ''))]

/Person[not(<AttributeNameOfTypeDateTime> &gt; '1900-01-01T00:00:00Z')]

So, if you wanted to find everyone who was missing an AD Distinguished Name (ADDN) you would use the following filter to define the Set:

/Person[not(starts-with(ADDN, ''))]

And if you wanted all of the Person objects missing a Termination Date (EmployeeEndDate):

/Person[not(EmployeeEndDate &gt; '1900-01-01T00:00:00Z')]

These examples are specific for the Person resource object, but this same logic could be used for any resource type in the FIM Portal.

UPDATE:  In certain instances, the above XPath will actually return more items than you would expect.  Check out David’s blog post to find out why and what really works.