The Last FIM Metaverse Extension You Will Ever Need

The third tool in the “The Last of” series is really another option for codeless provisioning (with out having to go to the FIM Service/Portal).  The Last FIM Metaverse Extension You Will Ever Need takes advantage of the configuration of a special MA to determine the initial flows needed to provision a new object.  This is done in two parts.  Here’s how it works. 

The Provisioning Management Agent
You will need to create an MA a special MA of type “Provisioning Management Agent (Insight)”.  This is an ECMA that inspects the configuration of the FIM Synch engine, allowing you to use the UI to define the provisioning rules for the other MAs in the environment.  You will not create run profiles for this MA, it will never be executed in that manner.

First, make sure you install the packaged Provisioning MA and the Insight.FIM.CodelessProvisioning.dll from http://fimmv.codeplex.com. Now, when configuring the MA, the first thing you will need to do (after giving it a name) will be to use the Connectivity tab to enable provisioning.  You can do this by MA and by MA object type.  One nice thing about this set-up, you can selectively turn on and off provisioning with out having to change any of the flow rules defined.  In this example, I have two MAs, the HR MA with one object type of person and a CRM MA with two types, person and group:

I have enabled provisioning for person objects in the HR MA and then clicked Next. Then, I clicked Next on the Configure Partitions and Hierarchies tab, we will be using the default values.

On the Object Types tab, you will see a list of Management Agents.  In essence, what this page is asking you is which MAs you want to define provisioning rules for.  At a minimum you will want to select the MAs that have at least item selected from the Connection tab. In this case I am going to define provisioning rules for the HR MA, which I have enabled for provisioning.  Additionally, I could define rules for the CRM MA if I plan on enabling it at some point in the future:

On the next tab, you will want to select the attributes that you are going to define initial flows for.  At a minimum you will need to check the attribute “Anchor”.  I have selected a few of the basic attributes I want to set during provisioning:

Next is the Anchors tab, leave the default values (Anchor attribute = Anchor) and click Next.

You should now be on the Configure Connector Filter tab, the provisioning code currently does not use this tab, but that could be a useful enhancement in the future.  Leave the filters blank and click Next.

Ditto with the Join and Project Rules, these won’t be used, leave them blank and click Next.

Now comes the interesting tab.  On the Attribute Flow section, define the flows that you want as your initial flows as Export flows for the MA and object type.  For example, if I want the ID, DisplayName, Email, FirstName and LastName set on a new HR MA person when the object is provisioned, I will define an Export flows for each of those:

Currently, only direct and constant flows are supported.  Although I am working on adding Advanced flows via the same mechanism as The Last FIM Management Agent Rules Extension You Will Ever Need in which the C# code that defines the advanced flow is placed directly in the Flow rule name.  Click Next when done defining the flow rules.

Click Next through the Deprovisioning and Extensions tab (you may need to provide a dll for the Rules extension name, you can simply use the Insight.FIM.CodelessProvisioning.dll).    That’s it!  You should now see your new Provisioning MA in the Sync client.  Don’t create more than one of these, currently only the first Provisioning MA found will be used.

Unfortunately, should you rename one of the MAs configured, you will need to update the schema of the Provisioning MA and then redefine the flow rules.

The Metaverse Extension
Okay, now all you need to do get this working is set the Metaverse Rules Extension to use Insight.FIM.CodelessProvisioning.dll and enable it:

Here’s what happens next.  When the provisioning code runs, it will go and look for a Provisioning MA.  It will then transverse the configuration of the MA to determine which objects to create and which flows to apply.  In this example, the provisioning code will see that I need to provision a new HR MA object of type person and set the ID, DisplayName, Email, FirstName and LastName on the object using the flow rules I defined.

Its an interesting concept and provides a well known UI to set up codeless provisioning without the overhead of the FIM Service. 

Let me know what you think!

Sun One Boolean Attributes

Recently while working with the Sun One MA, I came across a problem caused by how FIM interprets boolean data coming from the directory.  In this case, the Sun One directory stores its boolean data as “YES” or “NO”, however when imported by FIM, this data always gets converted to False in the connector space.  (http://social.technet.microsoft.com/Forums/en-US/identitylifecyclemanager/thread/8ffc112e-d945-4916-83b3-78fbba716705)

Now we can’t use an advance import flow to correctly convert the data as it comes into the Metaverse because we have already lost data integrity.  The data that’s in the connector space is basically useless.  This only leaves us with a couple of options, we can write an XMA to correctly handle the data, but this can be fairly complicated.  We can also update the directory schema and use a string instead of a boolean, however, this could cause other downstream issues with other systems that might be consuming this data.  There is one other option, but its completely unsupported.  You could update the FIM database to make the system think this attribute is a string. 

Begin by running the following SQL Statement using the SQL Server Management Studio against your FIM database:

SELECT CAST(ma_schema_xml AS XML)   
  FROM FIMSynchronizationService.dbo.mms_management_agent
 WHERE ma_name = '<Sun One MA Name Here>'

In the results window you will get a single record that can be clicked on to open an xml document containing the schema for your Sun MA. Each attribute in the directory will appear using this syntax:

<attribute-type id="system_assigned_id" single-value="true/false">
     <name>attribute_name</name>
     <syntax>LDAPv3_syntax_oid</syntax>
</attribute-type>

If you do a quick search (Ctrl+F) for your attribute, you can manually update the LDAP Syntax OID from a boolean (1.3.6.1.4.1.1466.115.121.1.7) to a string (1.3.6.1.4.1.1466.115.121.1.15).  In my case, the new XML looked like:

<attribute-type id="Ah" single-value="true">
     <name>isManager</name>
     <syntax>1.3.6.1.4.1.1466.115.121.1.15</syntax>
</attribute-type>

I then used an update statement to write this information back to the database.  After performing a full import on this MA the connector space now reflected this data as a string of “YES” or “NO” just as it appears in the directory.  Now that the connector space had data I could work with, I wrote a quick advanced import flow rule to transform this value to a boolean:

case "isManager":

   //perform conversion for isManager from yes/no to boolean value
   if (csentry["isManager"].IsPresent)
   {
      if (csentry["isManager"].Value.Equals("yes", StringComparison.InvariantCultureIgnoreCase))
      {
         mventry["isManager"].BooleanValue = true;
      }
      else
      {
         mventry["isManager"].BooleanValue = false;
      }
   }
   break;

This approach does have its risks.  It will need to be re-done after performing a schema refresh in FIM and there is no guarantee it will keep working after an upgrade/patch.   The following SQL query could be used to script out this update, just replace the attributeName and SunOneMA variables and you should be good to go:

DECLARE @attributeName AS Varchar(255) = '<Attribute Name Here>'
DECLARE @SunOneMA AS Varchar(255) = '<Sun One MA Name Here>'
DECLARE @newSchemaOID AS Varchar(255) = '1.3.6.1.4.1.1466.115.121.1.15' --Directory String

-- get schema data
DECLARE @schema AS XML

SELECT @schema = CAST(ma_schema_xml as xml)
  FROM FIMSynchronizationService.dbo.mms_management_agent
 WHERE ma_name = @SunOneMA

-- update schema type
SET @schema.modify('
   declare default element namespace http://www.dsml.org/DSML;
   replace value of
      (dsml/directory-schema/attribute-type[name=sql:variable("@attributeName")]/syntax/text())[1]
   with
      sql:variable("@newSchemaOID")
   ')

-- save back to table
UPDATE FIMSynchronizationService.dbo.mms_management_agent
   SET ma_schema_xml = CONVERT(nvarchar(MAX), @schema)
 WHERE ma_name = @SunOneMA

File Based Management Agents In MIIS/ILM/FIM

I had a recent need to really compare the capabilities of each of the file based Management Agents in FIM.  Can you name all five? Don't worry, I won't leave you hanging, they are:

• Attribute-value pair text file
• Delimited text file
• Directory Services Markup Language (DSML) 2.0
• Fixed-width text file
• LDAP Data Interchange Format (LDIF)

Here are some of the things that they can and can't do (this is for you Joe) and just for kicks, I also added in the SQL MA. If you are using one of these file types in an Extensible Management Agent (XMA), the following still applies: 

	Multi-valued Attributes	Attribute Level Updates 1	Multi-valued Level Attribute Updates 2
Attribute-value pair	YES	NO	NO
Delimited	YES 3	NO	NO
DSML	YES	NO 4	NO
Fixed-width	YES 3	NO	NO
LDIF	YES	YES	ON IMPORT ONLY 5
SQL MA	YES	YES	NO 6

Okay, now for the caveats (can’t get away without some of those):

1. An Attribute Level Update implies that a delta import can contain only the attribute that has changed (along with the other required columns, like the type of change and the anchor)
   
   So, here’s what that might look like.  Suppose I have a user with the following attributes:
   

   	ID:	12345
   	Name:	Sarah
   	Status:	Active
   	Phone:	555-123-4567

                                        
   If Sarah’s phone number changes to 555-987-6543, I can simply tell FIM something like: 

   	ID:	12345
   	Type Of Change:	Update
   	Phone:	555-987-6543

   
   This has the advantage of giving FIM less work to do to determine what has changed on the records being imported and greatly speeds up delta imports. 
    
2. A Multi-valued Level Attribute Update supports adding and deleting specific values from a multi-valued attribute
    
   Let’s take another look at Sara’s record:
   

   	ID:	12345
   	Name:	Sarah
   	Status:	Active
   	Phone:	555-123-4567
   	Phone:	555-456-7890

                                         
   Now, Sarah has two Phone numbers, or a single attribute with multiple values. With multi-value level attribute update support, we can do things like add a new phone number to the list, delete a phone number from the list or update a phone number (in essence by doing an add of the new value and then a delete of the old one):

   	ID:	12345
   	Type Of Change:	Add
   	Phone:	555-987-6543

   
   Without this support, the source system would be required to do a “replace” action and provide FIM with all of the current values at the time of import which FIM will use to override all the values that it has for that attribute.  So if we start with Sarah’s record as listed just above and add the phone number 555-987-6543 and remove the phone number 555-123-4567, we would have to pass:
   

   	ID:	12345
   	Type Of Change:	Replace
   	Phone:	555-987-6543
   	Phone:	555-4567-7890

   
   As with attribute level updates, multi-valued level attribute update can greatly reduce the amount of work that FIM needs to accomplish.  To illustrate, just imagine applying this scenario to attributes like member on an AD group that can have thousands of values.
    
3. Using a multi-valued attribute in a delimited or fixed-width file requires the use of a header on the import file
   
   So for a comma delimited file this would look like: 
                                          ID, NAME, PHONE, PHONE, PHONE
                                          12345, Sarah, 555-123-4567, 555-987-6543, 555-456-7890
   
   This would import a record for Sarah with three attributes - ID, NAME and PHONE, the last of which will have three values. A fixed width file would work the same way.
    
4. While the DSML specifications themselves can actually handle attribute level updates using the addRequest, delRequest and modifyRequest operations, FIM only implements the ability to import a SearchResultEntry element which must contain all of the attributes on the object
   
   Just a side note for those that might be curious, you can actually place the addRequest, delRequest and modifyRequest nodes in the DSML file.  FIM will be able to parse the file and it wont cause any errors, however these elements are completely ignored and aren’t processed by FIM.  I also tried sending a DSML delta to FIM with just the attribute that changed and a change type of “modify”, and I suppose not surprisingly, the object in the connector space was updated so that it only had the one attribute I specified,  all the other attributes originally on the object were removed. Had any of these attributes been defined as required, this update would have failed.
    
5. While you can import an update to a specific value in a multi-valued attribute, if you were to export this same change to an LDIF file, it will come through as a replace operation containing all values now present on the attribute
    
6. While the SQL MA does not support updates to a specific value on a multi-valued attribute out of the box, I hear rumor that some customizations can be done to make this happen

Using Maexport To Import a Management Agent

Ever happen to see this little tidbit in the documentation for Identity Manager:

You can find it under the heading Import a Management Agent from a File, and well, don’t believe it for a second!  Its currently not possible to import a Management Agent into Identity Manager using a command line tool.  For now, you will have to continue to use the UI.

Hit or MIIS

While working out at a client site on their MIIS implementation, occasionally during a sync on one of their MAs, the Identity Manager Application would become almost completely unresponsive. I couldn't start or stop the running of an MA, execute a Preview, complete an MV Search, perform any manual Joins or Disconnections. It was all very frustrating, I would have to wait a few minutes until MIIS came back from the abyss to continue my work. The only other information that I had was from performing a SQL Profiler Trace. MIIS was definitely working on something, but what and why? <spoiler>The Answer: it has to do with the Joining activity that MIIS does while sync'ing an object.</spoiler> So, here's a little more background...

While configuring a Management Agent, you are given the ability to define rules that MIIS can use to determine if the object that its currently looking at should be linked to user data that it already has from other systems. This can be done by simply going to the Configure Join and Projection Rules in the Metaverse Designer and clicking on the New Join Rule button at the bottom:

In its simplest form, a Join Rule can be designed that directly correlates data in the source with data in MIIS. For example, if the First and Last Name are the same, join the identities together:

On the backend, MIIS is executing a query against the Metaverse table to get all of the object_ids of user's with data matching the join criteria:

select distinct [mms_metaverse].object_id from [mms_metaverse] with (holdlock) where (([<MVAttributeName>] =N'<CSAttributeValue>'))

So if we are sync'ing an individual named Joe Smith, we get a query like (and, yes, there really are both sets of parentheses around each expression):

select distinct [mms_metaverse].object_id from [mms_metaverse] with (holdlock) where (([givenName] =N'Joe')) and (([sn] =N'Smith))

It will then run the following series of queries to build a portfolio about every object_id returned:

exec mms_getmvsvall_xlock @mvobjid = '<object_id>';
exec mms_getmvmulti @mvobjid = '<object_id>';
exec mms_getmvrefasobjid @mvobjid = '<object_id>';

Once MIIS has all of this data, it can join with an existing identity for Joe Smith, if one is found. In this scenario, if more than one Joe Smith is found, MIIS will consider this a Non-match and move onto the next join rule. It will continue to execute the join rules one after the other, in turn, until either a join is made or it reaches the end of the list. If no join is made, MIIS will then move onto projection, if this option has been defined.

Now, In an actual deployment these join scenarios are usually a little more complicated. To resolve a more complicated search/match, we usually have to go out to our rules extension code. There are two places you can inject code during the join procedure. The first of these is done by using the Rules extension mapping type. This will allow you to add some additional logic around the attribute equivalency search. Here is what I mean... you are trying to join based on Social Security Number and the source system has SSNs stored as 123-456-789 but MIIS has it stored as 123456789, while these two are not technically equivalent if doing a direct evaluation, we can add code to ensure that evaluation is done "correctly" via code executed in the MapAttributesForJoin Method.

The second place we can inject code is to pick the proper join out of the X number of candidates found by the Join search. So continuing from our example above, if we have defined a join based on SSN, but our data tends to be pretty dirty and multiple people have the same SSN. We have ensured that our SSN matching is performed properly above in the MapAttributesForJoin by reformatting the data, but the match/search logic finds 5 people that could potentially match my source SSN of 123-456-789. We can add additional code in the ResolveJoinSearch method to determine, based on other data elements, say last name, that number 3 of the 5 people found is the right individual to join with.

So, here is a more real world example. What if we want to join people base on their birth date with a last name or previous last name match? In this case we will have to go out to code to determine whether a join candidate can be found. We would start by creating a join rule based on birth date:

We have decided to use a Mapping Type of Rules extension to allow us to go out to code to reformat the birth date to ensure that it matches the format in MIIS. In our rules extension, MIIS would be looking for an entry point of "cd.person#1:mccdDOB->mccdDOB" in the MapAttributesForJoinMethod:

switch (FlowRuleName)
{

     case "cd.person#1:mccdDOB->mccdDOB":

          if ( csentry["mccdDOB"].IsPresent )
          {

               //reformat the dob with style used in the MV
               values.Add( System.Convert.ToDateTime(csentry["mccdDOB"].StringValue).ToString("yyyy-MM-dd") );

          }

          break;

     default:
          throw new EntryPointNotImplementedException();

}

We have also selected the Use rules extension to resolve option. This will allow us to add code to check current and previous last names against those individuals found with the same birth date. This is done using the following code in the ResolveJoinSearch Method:

//set index of final join candidate to -1 to indicate no join found
imventry = -1;

//create a variable to keep our place in the loop
int curIndex = 0;

switch (joinCriteriaName)
{

     case "cd.student#1":

          //loop through each person to check current/prev last name
          foreach (MVEntry mventry in rgmventry)
          {

               if ( csentry["sn"].IsPresent && mventry["sn"].IsPresent)
               {

                    //try to join on last name
                    if ( csentry["sn"].StringValue == mventry["sn"].StringValue )
                    {

                         //its a match, return the index of this individual
                         imventry = curIndex;

                    }

               }
               else if ( csentry["prevSn"].IsPresent && mventry["sn"].IsPresent)
               {

                    //try to join on previous last name
                    if ( csentry["prevSn"].StringValue == mventry["sn"].StringValue )
                    {

                         //its a match, return the index of this individual
                         imventry = curIndex;

                    }

               }

               //we are moving to the next object, increment the counter
               curIndex++;

          }

          break;

     default:
          throw new EntryPointNotImplementedException();

}

//return index of individual found
return imventry;

So, now that we have all of the basics of joining down, we can get back to the problem. Did you catch it? When MIIS is performing the join search, it executes a set of stored procedures to get all of the data about the matching Metaverse objects using an exclusive lock, locking up our Metaverse table from any other reads or updates until the search is complete. This usually isn't a problem because this process happens so quickly you usually won't notice it. With that said, be very careful not to create inefficient join rules that return more than a few results. If more than a few records need to be returned and evaluated, not only will being to see a degradation in the UI response of Identity Manager, it will also take exponentially longer for the MA to complete.