Monday, October 20, 2008

Hit or MIIS

While working out at a client site on their MIIS implementation, occasionally during a sync on one of their MAs, the Identity Manager Application would become almost completely unresponsive. I couldn't start or stop the running of an MA, execute a Preview, complete an MV Search, perform any manual Joins or Disconnections. It was all very frustrating, I would have to wait a few minutes until MIIS came back from the abyss to continue my work. The only other information that I had was from performing a SQL Profiler Trace. MIIS was definitely working on something, but what and why? <spoiler>The Answer: it has to do with the Joining activity that MIIS does while sync'ing an object.</spoiler> So, here's a little more background...

While configuring a Management Agent, you are given the ability to define rules that MIIS can use to determine if the object that its currently looking at should be linked to user data that it already has from other systems. This can be done by simply going to the Configure Join and Projection Rules in the Metaverse Designer and clicking on the New Join Rule button at the bottom:

In its simplest form, a Join Rule can be designed that directly correlates data in the source with data in MIIS. For example, if the First and Last Name are the same, join the identities together:

On the backend, MIIS is executing a query against the Metaverse table to get all of the object_ids of user's with data matching the join criteria:
select distinct [mms_metaverse].object_id from [mms_metaverse] with (holdlock) where (([<MVAttributeName>] =N'<CSAttributeValue>'))
So if we are sync'ing an individual named Joe Smith, we get a query like (and, yes, there really are both sets of parentheses around each expression):
select distinct [mms_metaverse].object_id from [mms_metaverse] with (holdlock) where (([givenName] =N'Joe')) and (([sn] =N'Smith))
It will then run the following series of queries to build a portfolio about every object_id returned:
exec mms_getmvsvall_xlock @mvobjid = '<object_id>';
exec mms_getmvmulti @mvobjid = '<object_id>';
exec mms_getmvrefasobjid @mvobjid = '<object_id>';
Once MIIS has all of this data, it can join with an existing identity for Joe Smith, if one is found. In this scenario, if more than one Joe Smith is found, MIIS will consider this a Non-match and move onto the next join rule. It will continue to execute the join rules one after the other, in turn, until either a join is made or it reaches the end of the list. If no join is made, MIIS will then move onto projection, if this option has been defined.

Now, In an actual deployment these join scenarios are usually a little more complicated. To resolve a more complicated search/match, we usually have to go out to our rules extension code. There are two places you can inject code during the join procedure. The first of these is done by using the Rules extension mapping type. This will allow you to add some additional logic around the attribute equivalency search. Here is what I mean... you are trying to join based on Social Security Number and the source system has SSNs stored as 123-456-789 but MIIS has it stored as 123456789, while these two are not technically equivalent if doing a direct evaluation, we can add code to ensure that evaluation is done "correctly" via code executed in the MapAttributesForJoin Method.

The second place we can inject code is to pick the proper join out of the X number of candidates found by the Join search. So continuing from our example above, if we have defined a join based on SSN, but our data tends to be pretty dirty and multiple people have the same SSN. We have ensured that our SSN matching is performed properly above in the MapAttributesForJoin by reformatting the data, but the match/search logic finds 5 people that could potentially match my source SSN of 123-456-789. We can add additional code in the ResolveJoinSearch method to determine, based on other data elements, say last name, that number 3 of the 5 people found is the right individual to join with.

So, here is a more real world example. What if we want to join people base on their birth date with a last name or previous last name match? In this case we will have to go out to code to determine whether a join candidate can be found. We would start by creating a join rule based on birth date:

We have decided to use a Mapping Type of Rules extension to allow us to go out to code to reformat the birth date to ensure that it matches the format in MIIS. In our rules extension, MIIS would be looking for an entry point of "cd.person#1:mccdDOB->mccdDOB" in the MapAttributesForJoinMethod:
switch (FlowRuleName)
{

     case "cd.person#1:mccdDOB->mccdDOB":

          if ( csentry["mccdDOB"].IsPresent )
          {

               //reformat the dob with style used in the MV
               values.Add( System.Convert.ToDateTime(csentry["mccdDOB"].StringValue).ToString("yyyy-MM-dd") );

          }

          break;

     default:
          throw new EntryPointNotImplementedException();

}
We have also selected the Use rules extension to resolve option. This will allow us to add code to check current and previous last names against those individuals found with the same birth date. This is done using the following code in the ResolveJoinSearch Method:
//set index of final join candidate to -1 to indicate no join found
imventry = -1;

//create a variable to keep our place in the loop
int curIndex = 0;

switch (joinCriteriaName)
{

     case "cd.student#1":

          //loop through each person to check current/prev last name
          foreach (MVEntry mventry in rgmventry)
          {

               if ( csentry["sn"].IsPresent && mventry["sn"].IsPresent)
               {

                    //try to join on last name
                    if ( csentry["sn"].StringValue == mventry["sn"].StringValue )
                    {

                         //its a match, return the index of this individual
                         imventry = curIndex;

                    }

               }
               else if ( csentry["prevSn"].IsPresent && mventry["sn"].IsPresent)
               {

                    //try to join on previous last name
                    if ( csentry["prevSn"].StringValue == mventry["sn"].StringValue )
                    {

                         //its a match, return the index of this individual
                         imventry = curIndex;

                    }

               }

               //we are moving to the next object, increment the counter
               curIndex++;

          }

          break;

     default:
          throw new EntryPointNotImplementedException();

}

//return index of individual found
return imventry;
So, now that we have all of the basics of joining down, we can get back to the problem. Did you catch it? When MIIS is performing the join search, it executes a set of stored procedures to get all of the data about the matching Metaverse objects using an exclusive lock, locking up our Metaverse table from any other reads or updates until the search is complete. This usually isn't a problem because this process happens so quickly you usually won't notice it. With that said, be very careful not to create inefficient join rules that return more than a few results. If more than a few records need to be returned and evaluated, not only will being to see a degradation in the UI response of Identity Manager, it will also take exponentially longer for the MA to complete.

Thursday, October 16, 2008

Creating a Scrolling List Gadget

One of the most important distractions for me as I develop is my music, it helps to keep me focused. In an effort to share that, I added a widget to the right of my blog that shows a list of the music that I am "currently" listening to. Not finding a good alternative, I simply used a link list widget and inserted a couple of the songs from my current playlist. Well.... not being satisfied with the geek factor, what I really wanted was something more dynamic, a kind of scrolling list, a music reel or marquee, if you will. So I set out to find out how to make my own Blogger Widget....

I started with the Blogger help files:

However, I found these a little limiting. I expanded my search and found that, as many others have done, the easiest way to do this to create a new Google Gadget. And better yet, once our Gadget is ready, Google will even host it for us on their GGE (or for those of us looking for more features at Google Code).

A Gadget consists of a xml file with three basic elements:

  • ModulePrefs
  • UserPref
  • Content

And looks something like this...

<?xml version="1.0" encoding="UTF-8" ?>
<Module>
     <ModulePrefs title="" description="" height="" width="" thumbnail="" screenshot="" author_email="" author="" title_url="">
          <!-- all of the following ModulePref Elements are optional -->
          <Locale lang="" language_direction="" messages="" country="" />
          <Link rel="" href="" method="" />
          <Require feature="">
               <Param name="">
          </Require>
          <Optional feature="">
               <Param name="">
          </Optional>
          <Preload href="" authz="none(default),signed,oauth" />
          <Icon mode="" type="">
               <!--base64 encoded data needed when using mode attribute-->
               <!--can also use url to icon file if mode attribute is not used-->
          </Icon>
     </ModulePrefs>
     <UserPref name="userPref1" display_name="Example" datatype="string(default),bool,enum,hidden,list,location" urlparam=""
        required="true,false(default)" default_Value="" />
     <UserPref name="UserPref2" display_name="stringExample" />
     <UserPref name="userPref3" display_name="enumExample" datatype="enum" default_value="A">
          <EnumValue value="A" display_value="Alpha">
          <EnumValue value="B" display_value="Beta">
          <EnumValue value="G" display_value="Gamma">
     </UserPref >
     <Content type="html,url" href="" />
          <![CDATA[
               <!--All the html and javascript needed to get the job done -->
               <!--CDATA section only needed if the content type is html-->
          ]]>
     </Content>
     </Module>
A couple of things to keep in mind before you begin. Blogger will provide a user setting for Height and Title by default. List Items can only be Added and Removed, in other words no Edit functionality. To change an entry, it will have to be removed and recreated. Further, it will host your Gadget inside of an iframe, so you will lose any styles applied to the blog. I have tried to compensate for this in my design below. Also, a <div> tag will be added with a class of widget-content and a black border. It will also add scroll bars, if necessary.

I wanted consumer's of my Gadget to be able to set the following options:
  • Number of Items to Show - integer from 1 to 99,999
  • Time to wait before Scrolling (in seconds) - integer from 1 to 99,999
  • Content Font Style Definition
  • List of Items

Arriving at a simple UserPrefs section that looks like...

<UserPref name="myNumItems" display_name="Items to Show" />
<UserPref name="myTimer" display_name="Time to Scroll" />
<UserPref name="myStyle" display_name="Content Font Style" default_value="8pt Georgia" />
<UserPref name="myList" display_name="List of Items" datatype="list" />

You could add additional style items to give user's more display flexibility, things like color, line size, padding, etc. Finally, I took a stab at the CDATA portion of the Content element. After several rounds of modifications, saves, publishing and testing on my iGoogle page, I arrived at the following code:

<div id="content_div"> </div>

<script type="text/javascript">

// Get UserPrefs Options
var prefs = new _IG_Prefs(); //new gadgets.Prefs();

// Get Details of UserPrefs
var items = prefs.getArray("myList");
var style = prefs.getString("myStyle");
var numToShow = 99999;
var timer = 99999;

//validate entries
if (!isNaN(prefs.getString("myNumItems")))
{
     numToShow = new Number(prefs.getString("myNumItems"));
}

if (numToShow > 99999)
{
     numToShow = 99999
}

if (!isNaN(prefs.getString("myTimer")))
{
     timer = new Number(prefs.getString("myTimer"));
}

if (timer > 99999)
{
     timer = 99999;
}

// keep track of where we are in our loop
var curIndex = 0;

// load list once for initial display
displayList();

// reload list after delay
setInterval("displayList()", timer * 1000);

function displayList()
{

     // Build HTML to show the user
     var html = "<table id='content-table' cellPadding='1px' border='0'>";

     // Loop through and display items
     if (items.length > 0)
     {
          var loopUntil = numToShow;
          var itemToShow = new Number(curIndex);

          // if user has set the number to show greater than the number of items we
          // have in our list, only loop through the number in the list

          if (items.length < loopUntil)
          {
               loopUntil = items.length;
          }

          //loop through our items, keeping track of where we are and what needs to be shown
          for (var i = 0; i < loopUntil; i++)
          {
               //if we have moved past the end of the list start back at the beginning
               if (itemToShow >= items.length)
               {
                    itemToShow = 0;
               }

               html += "<tr><td>" + items[itemToShow] + "</td></tr>";
               itemToShow++;
          }

          // Start loop at the next one in the list
          curIndex++;

          if (curIndex >= items.length)
          {
               curIndex = 0;
          }
     }

     html += "</table>"

     document.getElementById("content_div").innerHTML = html;

     //load in our style element
     var cells=document.getElementsByTagName("td");
     for (var i=0; i < cells.length; i++)
     {
          cells[i].style.font = style;
     }
}

</script>

Don't be surprised if Google warns you about missing elements, like thumbnail and width, when you publish your Gadet. I was able to accept the warning and still publish successfully.

I then went to import this Gadget to my blog using the url that Google gave me during publishing, http://hosting.gmodules.com/ig/gadgets/file/102621166658384583871/scrolling-list.xml. Next, using the Edit HTML tab in the Layout section of my blog, I was able to remove the border around my Gadget and change the padding to the left of the Gadget by checking the Expand Widget Definitions and locating the <div> tag just before my Gadget. I also set the scrolling attribute on the iFrame to no. While changing the Font Style in user settings, I will sometimes get JavaScript errors, however, in my mind, the benefits here outweigh the issue.

You can see the fruits of my labor over to the right, titled "Currently Developing To...".....Wait for it.....

Let me know what you think and feel free to add the Gadget to your own site! Happy blogging!

Welcome!

I have been a developer for over 10 years, spending the last 6 of those as a consultant working for the world-class IT provider, Ensynch. While working for Ensynch, I have had the privilege of being exposed to an extensive portfolio of technologies. Its my objective to begin a blog about my adventures. And I have some very big shoes to fill here, following in the footsteps of those folks you see to the right. Getting into the blogging game a little late, my first (second?) post will be, not coincidentally, about creating this blog.

Based out of Tempe, Ensynch has both a professional services and a talent solutions arm. Ensynch has been awarded with such prestigious awards as Microsoft's West Region Partner of the Year and The Business Journal's #1 Phoenix Computer Consultant company. Ensynch provides expertise in several core areas, including, Identity Management, Portals and Collaboration, Systems Management, Infrastructure Optimization, Unified Communications and Application Development/Lifecycle Management.

So without further ado, here goes.....