Life of a Hungarian SharePoint Geek

I got tired of attaching debugger to w3wp.exe. And you?

pholpar — Fri, 12 Jun 2009 22:19:00 +0000

As a SharePoint developer, I spend most of my days developing and testing web parts, custom pages, fields and event receivers. Part of this activity is debugging, where one should attach the debugger to the w3wp.exe process instance that belongs to the application pool of the web site we developing on.
If you have done that yet, you know it is not an exciting task to select the correct process if you have dozens of instances. You can do that by trial and error, and checking if the red lights next to the breakpoints are turned on, or you should look for the next instance. It is one step better to run iisapps.vbs as described here, and use the process id it displays for the correct application pool, but it is still a cumbersome manual process, that a developer does not enjoy.
Developers enjoy development, and developing tools that help development. A good platform for those tools is the Visual Studio IDE and in that environment maybe the simplest way to create tools is writing macros.
To help my job I created a macro in Visual Studio 2005 that attaches the debugger to the w3wp.exe instance of the configured application pool. I hope it works for Visual Studio 2008 too, but I have not tested it with that yet.

Configuring the name of the application pool in Visual Studio
The configuration can be done using the project properties. These properties are stored in the csproj.user file of the project, so if you use version control and work in a team, the value can be different for teammates. I selected the Command line arguments text field on the Debug tab to store the name of the application pool we want to attach the debugger to.
There might be multiple projects in the solution, and each project may be multiple configuration (like debug or release), so I decided to use the debug configuration for the first startup project. It is important, because there may be multiple startup projects in a solution.

The macro
After this introduction, let’s see the code of the macro. We need the import the following namespaces:

Imports System
Imports EnvDTE
Imports EnvDTE80
Imports System.Management
Imports System.Text.RegularExpressions

The entry point for the macro is the AttachMacro sub. In that we first determine the projct whose configuration we will use. If there is only a single project in the solution we can go with that, if there are multiple projects then at least one of the must be set as startup project, but only the configuration of the first one will be used.
If we found the projects, we use the GetArgumentsForDebug function (see details later) to read the application pool name from the configuration.
Finally we call the AttachToAppPool function (see details later) to attach or debugger to the process of the application pool.

Public Sub AttachMacro()

Try
    ' get the startup project first
    Dim project As Project
    Dim solutionBuild As SolutionBuild = DTE.Solution.SolutionBuild
    Dim startUpProjs As Array = solutionBuild.StartupProjects
    Dim projName As String

    If startUpProjs.Length = 0 Then
        If DTE.Solution.Projects.Count = 1 Then
    project = DTE.Solution.Projects.Item(1)
        Else
            MsgBox("There is no startup project and solution contains multiple project!", MsgBoxStyle.Exclamation, "Alert")
            Exit Sub
        End If
    Else
        projName = solutionBuild.StartupProjects(0)
        project = GetProjectByName(projName)
        If project Is Nothing Then
            MsgBox(String.Format("Startup project '{0}' not found by name!", projName), MsgBoxStyle.Exclamation, "Alert")
            Exit Sub
        End If
    End If

Dim appPoolName As String = GetArgumentsForDebug(project)

    If String.IsNullOrEmpty(appPoolName) Then
        MsgBox(String.Format("Command line arguments property is not set for stratup project '{0}', debug mode!", projName), MsgBoxStyle.Exclamation, "Alert")
        Exit Sub
    End If

Dim processFound As Boolean = AttachToAppPool(appPoolName)

    If Not processFound Then
        MsgBox(String.Format("No worker process found for application pool called '{0}'!", appPoolName), MsgBoxStyle.Exclamation, "Alert")
        Exit Sub
    End If

Catch ex As System.Exception
MsgBox(ex.Message)
End Try

End Sub

The GetProjectByName function is a simple helper method to get the project object using the name of the startup project.

Private Function GetProjectByName(ByVal projectName As String) As Project

Dim result As Project = Nothing
For Each project As Project In DTE.Solution.Projects
    If project.UniqueName = projectName Then
        result = project
        Exit For
    End If
Next

GetProjectByName = result

End Function

In the GetArgumentsForDebug function we read the Command line arguments value that is stored in the StartArguments parameter from the debug config of the specifed project.

Private Function GetArgumentsForDebug(ByVal project As Project) As String

Dim configuration As EnvDTE.Configuration
GetArgumentsForDebug = String.Empty

For Each configuration In project.ConfigurationManager
    If configuration.ConfigurationName = "Debug" Then
        Dim startArgsObj As Object = configuration.Properties.Item("StartArguments")
        If Not startArgsObj Is Nothing Then
            GetArgumentsForDebug = CType(startArgsObj.Value, String)
        End If
        Exit Function
    End If
Next

End Function

An interesting part of the macro is the GetProcessIdByAppPoolName function. In this function we use WMI (IMPORTANT: don’t forget to reference the System.Management assembly!) to get the list of all w3wp.exe processes, and select the one that belongs to the specified application pool. The ID of the process is returned.
In the comparison we use the GetAppPoolNameFromCommandLine function (see later), that receives the CommandLine property of the process, that looks like this for a w3wp.exe process:
c:windowssystem32inetsrvw3wp.exe -a \.pipeiisipm5f3cda83-745a-423a-88b2-103a2f632200 -ap "MyAppPool"
You can see that the name of the application pool is at the end of the string.

Private Function GetProcessIdByAppPoolName(ByVal appPoolName As String) As Long

GetProcessIdByAppPoolName = -1
Dim scope As ManagementScope = New ManagementScope("\localhost
ootcimv2")

Dim searcher As ManagementObjectSearcher = New ManagementObjectSearcher("select * from Win32_Process where Name='w3wp.exe'")
searcher.Scope = scope

For Each process As ManagementObject In searcher.Get()
    Dim commandLine As String = process.GetPropertyValue("CommandLine")
    If GetAppPoolNameFromCommandLine(commandLine).ToUpper() = appPoolName.ToUpper() Then
        GetProcessIdByAppPoolName = process.GetPropertyValue("ProcessId")
        Exit For
    End If
Next

End Function

In the GetAppPoolNameFromCommandLine function we use a simple regular expression to get the name of the application pool from the CommandLine string.

Private Function GetAppPoolNameFromCommandLine(ByVal commandLine As String) As String

GetAppPoolNameFromCommandLine = String.Empty
Dim re As Regex = New Regex("-ap ""(.+)""", RegexOptions.IgnoreCase)
Dim matches As MatchCollection = re.Matches(commandLine)
If matches.Count = 1 Then
    If matches.Item(0).Groups.Count > 1 Then
        GetAppPoolNameFromCommandLine = matches.Item(0).Groups(1).Value
    End If
End If

End Function

In the AttachToAppPool function we attach the debugger to the process having the same process ID that we determined earlier.

Private Function AttachToAppPool(ByVal appPoolName As String) As Boolean

Dim dbg2 As EnvDTE80.Debugger2 = DTE.Debugger
Dim trans As EnvDTE80.Transport = dbg2.Transports.Item("Default")
Dim dbgeng(2) As EnvDTE80.Engine
dbgeng(0) = trans.Engines.Item("T-SQL")
dbgeng(1) = trans.Engines.Item("Managed")

AttachToAppPool = False
For Each process As EnvDTE80.Process2 In dbg2.LocalProcesses
    If process.ProcessID = GetProcessIdByAppPoolName(appPoolName) Then
        process.Attach()
        AttachToAppPool = True
        Exit For
    End If
Next

End Function

To make things even more comfortable it is the best to assign a keyboard shortcut to the macro using the Visual Studio Tools/Options… menu item as shown on the screenshot below:

Columns missing when using the Lists.GetListItems SharePoint webservice

pholpar — Wed, 10 Jun 2009 22:23:00 +0000

More people complained about that the result of the Lists.GetListItems method call does not contain the empty fields, so the resulting XML cannot be load into a DataSet:
http://social.msdn.microsoft.com/Forums/en-US/sharepointdevelopment/thread/95855246-b8f8-4dda-897a-c4480cc74044/#0a2d42ed-7365-413c-a12f-ccbf95025c12
http://www.tech-archive.net/Archive/SharePoint/microsoft.public.sharepoint.portalserver.development/2006-09/msg00056.html

A dirty workaround for that issue may be to adding the missing attributes to the XML from code before trying to load into the DataSet.

In the following code I illustrate this approach. In this code I first get the columns of the default view of the list using the List.GetListAndView method, then retrive the data using the Lists.GetListItems method, iterate through all the rows and columns, and if an attribute is missing for a column, add an empty attribute.

String listName = "YourListName";

listService.Credentials = new NetworkCredential("user", "password", "domain");

XmlNamespaceManager nsmgr;

// get info about the default view
// the 2nd parameter is null -> it is the default view
XmlNode listView = listService.GetListAndView(listName, null);
nsmgr = new XmlNamespaceManager(listView.OwnerDocument.NameTable);
nsmgr.AddNamespace("a", "http://schemas.microsoft.com/sharepoint/soap/");

List fieldNames = new List();
foreach (XmlNode field in listView.SelectNodes("a:List/a:Fields/a:Field", nsmgr))
{
    XmlAttribute attr = field.Attributes["Name"];
    // it should not be null, but we check it
    if (attr != null)
    {
        // we store all fields in a collection for later use
        fieldNames.Add(attr.Value);
    }
}

// get data from list
XmlNode items = listService.GetListItems(listName, null, null, null, null, null, null);

nsmgr = new XmlNamespaceManager(items.OwnerDocument.NameTable);
nsmgr.AddNamespace("z", "#RowsetSchema");
nsmgr.AddNamespace("rs", "urn:schemas-microsoft-com:rowset");

XmlNodeList itemNodeList = items.SelectNodes("rs:data/z:row", nsmgr);

foreach(XmlNode itemNode in itemNodeList)
{
    foreach (String fieldName in fieldNames)
    {
        String wsFieldName = "ows_" + fieldName;
        XmlAttribute attr = itemNode.Attributes[wsFieldName];
        // if the attribute is missing, we should add it
        if (attr == null)
        {
            attr = itemNode.OwnerDocument.CreateAttribute(wsFieldName);
            itemNode.Attributes.Append(attr);
        }
    }
}

DataSet listDataSet = new DataSet();

// read the result into a data set
XmlTextReader readerListDataSet = new XmlTextReader(items.OuterXml, XmlNodeType.Document, null);
listDataSet.ReadXml(readerListDataSet);

After this kind of preparation of XML it can be loaded into the DataSet.

You should know that this approach may not scale and perform well in case of a large amount of data, and I consider it a dirty workaround, but I don't know currently other solution for this request.

How to deploy a custom field with custom properties from a feature

pholpar — Tue, 12 Aug 2008 21:46:00 +0000

When MOSS 2007 was still in beta and features and custom fields were new areas to discover we created the classic regular expression field type too, just to play with and learn the new technology.

Our implementation SPFieldRegEx was inherited from the Text type and had three custom properties defined in the field type definition XML:

                255


              The value does not match the regular expression

The RegEx property stores the regular expression pattern, the MaxLen controls the maximal length of the field content and finally, the ErrMsg holds the validation message to be displayed when the input text does not match with the regular expression.

There is nothing interesting in that up to this point, but if you would like to deploy this custom field using a feature setting custom values to the properties you might encounter some difficulty.

Since I haven't found that documented neither in the WSS SDK nor on developer blogs in the past years, I decided to share my experience.If you create your feature definition for the field as you do normally with the built in field types, the result is the following XML:

But if you try to install the feature, you get the following error:

Feature definition with Id 6fd6ca04-3ac3-490f-b22f-4461a2253001 failed validation, file 'feature_definition2.xml', line 5, character 299:

The 'RegEx' attribute is not allowed.

If you remove the RegEx attribute, the same error message appears with MaxLen, if you remove that too, the ErrMsg causes problem.

So what to do to make this attributes allowed?The schema of the features is defined in the wss.xsd. Now the most important part for us is the FieldDefinition complexType that is responsible – what a surprise! – for describing the format of the field definitions in the features. Besides other things it contains the list of the allowed attributes.

…


…

…

…

…


…

…

…

…


…





…

The fragment above contains only the most widely used attributes for example.

One quick and dirty solution would be to include our custom attributes in the XSD schema but this probably wouldn't be a supported method. Fortunately in this case MS has left the back door open: if you check the last attribute in the schema, it is anyAttribute with namespace ##other, meaning that you can inject your own attributes in the XML files using your own namespace.

After a minor modification in the feature definition XML (highlighted below) the XML was passed the schema check and our custom field feature was installed successfully.

xmlns:grp="http://schemas.grepton.com/sharepoint/">
grp:RegEx="[0-9]" grp:MaxLen="20" grp:ErrMsg="Error!"/>

Negative item count in document libraries

pholpar — Fri, 14 Mar 2008 21:03:00 +0000

A few days ago we received a complaint from one of our customers. They found that after deleting all documents from it, a SharePoint document library that serves as the base of a custom application shows negative item count on the All Site Content page.

We checked where this value comes from and found it is the ItemCount property of the SPList object.

An additional side effect of this strange behavior is that the indexing fails because it cannot cast the negative value to an unsigned integer (UInt32).

We could reproduce the behavior using the following code. It assumes that the address of your site is your site is http://moss/site and the name of the document library is CopyTest. The code creates a folder and subfolders, copies it using the CopyTo() method of SPFolder class, and finally deletes the folders.

string siteUrl = "http://moss/site";

using (SPSite site = new SPSite(siteUrl))

{

using (_web = site.OpenWeb())

{

SPList list = _web.Lists["CopyTest"];

Console.WriteLine("Items.Count:{0}, ItemCount:{1}", list.Items.Count, list.ItemCount);

Console.WriteLine("Creating 1.0 folder … ");

SPFolder rootFolder = _web.GetFolder("/CopyTest");

SPFolder version1Folder = rootFolder.SubFolders.Add("/CopyTest/1.0");

version1Folder.Update();

list.Update();

Console.WriteLine("OK");

Console.WriteLine("Items.Count:{0}, ItemCount:{1}", list.Items.Count, list.ItemCount);

Console.WriteLine("Creating Test1 in 1.0 folder … ");

SPFolder testFolder = version1Folder.SubFolders.Add(version1Folder.Url + "/Test1");

testFolder.Update();

list.Update();

Console.WriteLine("OK");

Console.WriteLine("Items.Count: {0}", list.Items.Count);

Console.WriteLine("ItemCount: {0}", list.ItemCount);

Console.WriteLine("Creating Test1.1 in Test1 folder … ");

SPFolder test2Folder = version1Folder.SubFolders.Add(version1Folder.Url + "/Test1/Test1.1");

test2Folder.Update();

list.Update();

Console.WriteLine("OK");

Console.WriteLine("Items.Count: {0}", list.Items.Count);

Console.WriteLine("ItemCount: {0}", list.ItemCount);

Console.WriteLine("Creating Test2 in 1.0 folder … ");

SPFolder test3Folder = version1Folder.SubFolders.Add(version1Folder.Url + "/Test2");

test3Folder.Update();

list.Update();

Console.WriteLine("OK");

Console.WriteLine("Items.Count: {0}", list.Items.Count);

Console.WriteLine("ItemCount: {0}", list.ItemCount);

Console.WriteLine("Creating 2.0 folder in the root … ");

SPFolder version2Folder = rootFolder.SubFolders.Add("/CopyTest/2.0");

version2Folder.Update();

list.Update();

Console.WriteLine("OK");

Console.WriteLine("Items.Count:{0}, ItemCount:{1}", list.Items.Count, list.ItemCount);

Console.WriteLine("Copy 1.0 items to 2.0");

foreach (SPFolder subFolder in version1Folder.SubFolders)

{

Console.WriteLine("Copy {0} folder … ", subFolder.Name);

subFolder.CopyTo(version2Folder.Url + "/" + subFolder.Name);

SPFolder folder = _web.GetFolder(version2Folder.Url + "/" + subFolder.Name);

if(folder.Exists)

Console.WriteLine("Folder exists, url is: {0}", folder.ServerRelativeUrl);

folder.Update();

list.Update();

Console.WriteLine("OK");

Console.WriteLine("Items.Count:{0}, ItemCount:{1}", list.Items.Count, list.ItemCount);

}

Console.WriteLine("Deleting 1.0 folder … ");

SPFolder deleteFolder = _web.GetFolder("/CopyTest/1.0");

if (deleteFolder.Exists)

{

deleteFolder.Delete();

list.Update();

Console.WriteLine("OK");

}

Console.WriteLine("Deleting 2.0 folder … ");

deleteFolder = _web.GetFolder("/CopyTest/2.0");

if (deleteFolder.Exists)

{

deleteFolder.Delete();

list.Update();

Console.WriteLine("OK");

}

Console.WriteLine("SPList.ItemCount = {0}", list.ItemCount);

}

We escalated the issue to MS support and waiting for a resolution now.

I should note that there is an interesting blog post about the relation of SPList.ItemCount and SPList.Items.Count here. The main point is that if the number of items is changed (for example, by deleting or creating items) in the document libraries, then the value stored int he ItemCount does not changed until SPList.Update() is called. You should be aware of this behavior in your custom applications.

Using Month element in DateRangesOverlap can return items not in the specified month

pholpar — Mon, 03 Mar 2008 22:21:00 +0000

One can logically think that if you set the CalendarDate property of the SPQuery and apply Month as the Value in the DateRangesOverlap element then the query returns items from the month specified in the CalendarDate property.

For example, using the following code .

SPQuery query = new SPQuery();

query.CalendarDate = new DateTime(2008, 3, 1);

. and applying the following CAML query .

. one can expect that items having overlap with March 2008 will be only returned.

I've found that it is not exactly so, as elements from the end of February and beginning of April may be also returned. The reason for that I think is the simple fact that these items have overlap with the calendar range of the month view of March 2008 (e.g. 24/02/2008-05/04/2008), as the month calendar view includes the full weeks at the beginning and the end of the month, not only the days between 01/03/2008-31/03/2008.

Since DateRangesOverlap is (another) under-documented element of SharePoint and CAML I can't decide if it is a bug or a feature. Since it is used primarily in the calendar views it seems to be logical that it should return all items that need to be displayed in the month view. So it is probably by design, sad that this fact is not documented.

This fact caused a misbehavior (or let's call it bug) in one of our custom application, so we should return to our original complex CAML query assembled from Lt, Gt, Leq, Geq, composite And and Or elements.

Creating Keywords and Best Bets for MOSS Search programmatically

pholpar — Tue, 13 Nov 2007 21:42:00 +0000

You can administer keywords and best bets on the MOSS admin UI at Search keywords at Site Collection Administration.

Stefan Goáner gave a code snippet about How To: create Keywords and Best Bets for MOSS Search programmatically. A similar code can be found in the very useful SharePoint Server 2007 Presentations: Enterprise Search Deep Dives presentation series, Customizing and Extending Search in Office SharePoint Server 2007.

Based on my experiments there might be a little but important problem with this code. When creating the new Keyword instance, you should use DateTime.UtcNow instead of DateTime.Now to enable the Keyword immediately:

keywords.AllKeywords.Create("myKeyword", DateTime.UtcNow);

When you create the keyword through the UI, you can specify only the date part, no hours and minutes. In this case the keyword is created with a start date (StartDate property) 0:00 AM UTC for the specified date. For example, currently our local time is GMT+1, so the start date would be 23:00 PM for the previous day.

When I used the code samples “as is”, the keywords were not displayed in the results. After one hour, the repeated search already displayed the keyword matches. When I used the DateTime.UtcNow, the results were displayed immediately. Of course, if your configured time zone is west from the GMT time line, then DateTime.Now should work also, as it is refers to a time in the past if you interpret it as UTC time.

If you try to create a best bet on the UI that refers to an URL already used in an existing best bet you cannot save the new best bet. I found another interesting behaviour of creating keywords and best bets from code. The code that creates the best bets with a common URL but different titles and descriptions will run without errors:

Keyword keyword1= keywords.AllKeywords.Create("keyword1", DateTime.UtcNow);
BestBet bestBet1 = keyword1.BestBets.Create("BestBet1", "Description1", new Uri(http://www.company.com));
keyword1.Update();
Keyword keyword2= keywords.AllKeywords.Create("keyword2", DateTime.UtcNow);
BestBet bestBet1 = keyword2.BestBets.Create("BestBet2", "Description2", new Uri(http://www.company.com));
keyword2.Update();

In the example above the best bet for keyword2 will refer to the BestBet with title BestBet1 created for keyword1.

High Confidence Results in MOSS 2007

pholpar — Tue, 13 Nov 2007 16:57:00 +0000

At the MSDN forum I found an interesting question that asks what high confidence results are. After some investigation I think I have the answer for this question.

First of all, high confidence results are displayed by the Microsoft.Office.Server.Search.WebControls.HighConfidenceWebPart (Microsoft.Office.Server.Search assembly). On the standard search result page (results.aspx) there are two instances of this web part. One is Search High Confidence Results, the other is Search Best Bets. If you check the properties of the HighConfidenceWebPart either by modifying the shared web part, or by using Reflector, you can see that this web part can display keyword matches with best bets and the mysterious high confidence results. By default the Search High Confidence Results web part instance is configured to display the high confidence results and the Search Best Bets web part instance is configured to display keywords and best bets (as one can guess from their name).

The SDK contains information about that when the Enterprise Search returns the results there is a HighConfidenceResults table that is “The result set containing high-confidence results”. Well, it is not very descriptive, is it?

Let's see the formatting XSL for the HighConfidenceWebPart. You can check it in the XSL Editor in the Data View Properties section of the tool part. Fortunately, there is a template for the HighConfidenceResults, displayed below:

IMNRC('
')

You can see, that all of the properties are related to persons. I found that if you search for a person specifying the full name, and there is match, then it is displayed as a high confidence result. In the web part properties you can specify if the title, image, description and other properties should be displayed for a high confidence match. There is a ResultsPerTypeLimit property (“Maximum matches per High Confidence type “) that similar to the BestBetsLimit property (“Best Bets limit ” on the user interface). Based on my experience, the BestBetsLimit works as expected but the ResultsPerTypeLimit seems to have no effect on the displayed results. Checking the default formatting XSL shows that the BestBetsLimit property is used (BBLimit parameter), but the ResultsPerTypeLimit is not used, although it is declared as an XSL parameter with the same name.

You should include the following condition in the HighConfidenceResults template (see above) after checking the "$DisplayHC = 'True' and $IsFirstPage = 'True'" condition:

ResultsPerTypeLimit " >

Remark: The XSL parameter values are populated in the ModifyXsltArgumentList method of the HighConfidenceWebPart web part.

Lesson learned (the hard way): Never use underscore in your hostname

pholpar — Wed, 24 Oct 2007 19:56:00 +0000

Note – This is a repost of the original writing that was lost when the SharePointBlogs site crashed. Original article was published on 2007.05.11.

Last year, just after the release of MOSS 2007 we run into a very strange problem when we tried to configure FBA. After using this feature on B2 and B2TR more-or-less successfully on the RTM it was totally unusable. When the users entered their correct username and password the login screen appeared again without any sign of incorrect credentials or access denied message. If they provided incorrect credentials, it was handled by the system correctly (e.g. they received a warning message about the incorrect username or password).

Using SQL Profiler we created traces that showed that using the correct credentials the FBA was successful (at least on the database level). It means that the SP responsible for FBA returned success.

It was an interesting addition to the issue that when we checked the working with FireFox the FBA was working as it should be. Later there were issues with FireFox (it is another story) but users were able to log in at least.

When we used the Central Admin web site there we experienced another issue that we thought may be related to the login problem: after selecting the application to manage from the HTML list on the admin pages, the selection was lost and the first item got selected, although the setting we made on the pages were applied correctly to the selected web application.

We read all important blogs about using FBA on MOSS 2007, trying to remember what we made differently on the beta versions. We reinstalled the system several times, even the DC was reinstalled of the test domain (who knows?) without any success.

It was interesting that there were a red ball on the right side of the status bar of the browser alerting us about the browser "Could not find a privacy policy" for the login page. So we played with P3P, the browser privacy settings, but FBA was still not working.

We felt it was almost sure that the direct cause of the problem is the incorrect cookie-handling in the browser, but were not able to catch the root of the problem.

I tried to remember what could be different between the previous beta-based and the current RTM installation. Somehow my attention was focused for a moment on the name of the MOSS server machine and I immediately knew I had found it! While we gave the name MOSS for the server previously, my colleague who made the installation started to name it MOSS_DEV recently. I did not know why, but I was sure that the underscore will be the source of our pain.

After a short google I found the proof of my theory. This page on SAP site about Fully Qualified Domain Names (FQDN) tells that "The browser does not accept cookies if a host name contains the underscore character".

This MS support article says that "Cookies Are Not Saved If the Host Name Is Invalid".

Also I found a very similar incident in IBM's knowledge base: Form login does not work when the hostname contains an underscore. I feel this one very instructive. Unfortunately, I found it only after solving the problem myself. It could have saved a lot of time for us.

Main thing quoted, but I suggest you to read the entire article: "Internet standards do specify that hostnames should not contain underscores (RFC 1123 & RFC 932)".

Well, we already know it. Sad, that MS Windows Server 2003 is not aware of this when allowing users to set host name containing underscore without alerting them for the dangers. The browser made by the same company follows the RFC rules. Not very consistent.

Hunting the encoding problem in content deployment – part 3

pholpar — Wed, 24 Oct 2007 19:17:00 +0000

In the former parts of this series I wrote about the encoding problem we encountered using the standard MOSS 2007 content deployment. In this part I will shortly describe a workaround and the final solution.

The workaround we created was a "simple" command line tool that iterates through the sites and pages. The special characters were stored as a constant string array and we computed the corresponding “encoded” characters at runtime based on the rules we descried in the previous parts. We replaced all "encoded" characters into the original one in all off the fields that can contain text information. This process required considerable time, especially as we had pages that contain Russian text and it is full with special characters. We had to extend the array of special characters as users add more and more content with characters we did not include originally into the array. Also, it was not easy to synchronize the correction process with the content deployment.

We experimenting with other methods, like creating event receiver that replaces the special characters "on-the-fly" as content deployment pushes content to the publishing server. Unfortunately, it seems that event receivers do not fire as expected when content created through content deployment. Probably it has performance reasons.

Other "nice try" was handling the problem on the database level. We created a simple tool that corrected the content directly through the content database. We even planned to create some kind of INSTEAD OF TRIGGER that inserts the corrected content instead of the "encoded" one by calling managed code to restore the original content.

Fortunately we should not have to go live with this workaround as Microsoft released a hotfix for this issue and we receive that patch. After installing the hotfix we have no more problems with special characters.

Hunting the encoding problem in content deployment – part 2

pholpar — Wed, 24 Oct 2007 19:08:00 +0000

Note – This is a repost of the original writing that was lost when the SharePointBlogs site crashed. Original article was published on 2007.05.06.

In the previous part we saw that special characters in meta-information of publishing pages handled incorrectly by the export process (and by the content deployment that is based on the export process). This part will show you how we used Lutz Roeder's .NET Reflector and SQL Server Profiler to check the internal working of the export process and identify the potential source of the problem.

Most of the classes related to the export process are located in the Microsoft.SharePoint.Deployment namespace within the Microsoft.SharePoint.dll assembly so if not specified otherwise you should find classes I am writing about that place.

As we already know the export package is created by the Run() method of the SPExport class. In this method the exportation (serialization) of the objects is made by the call this.SerializeObjects(). Within this method the line serializer.Serialize(deployObject, writer.BaseStream) is responsible for serialization to the target file. The serializer object in this case is an ObjectSerializer:

ObjectSerializer serializer = new ObjectSerializer(deploymentContext);

In the ObjectSerializer(DeploymentStreamingContext deploymentContext) constructor you will find this line:

this.AddSerializationSurrogates(selector, this.m_context);

The AddSerializationSurrogates method is responsible for registering different type of serializers for the SharePoint object types that should be persisted during the export. The following line handles the serializer for the SPFile object type. Since we had problems with the SPFile Property value in the Manifest.xml, we will follow this track:

selector.AddSurrogate(typeof(SPFile), context, new FileSerializer());

As you see SPFile objects are persisted using the FileSerializer object that is a derived class of DeploymentSerializationSurrogate. DeploymentSerializationSurrogate implements the System.Runtime.Serialization.ISerializationSurrogate interface, so object data is provided to the serializer through the GetObjectData(object obj, SerializationInfo info, StreamingContext context) method.

If you check the definition of this method in the DeploymentSerializationSurrogate class, you can see that if the DataSet property of the obj object (after casting to ExportObject that is a subclass of DeploymentObject) is not null then the GetDataFromDataSet(object obj, SerializationInfo info, StreamingContext context) method is used during serialization. For the FileSerializer this is the case so we should check its GetDataFromDataSet method. Within this method the call HandleMetaInfo(objectManager, fileMetaData.ItemArray[ 8 ], info, settings) is responsible for persisting meta-information.

We investigated the SPS content database and created SQL trace during the export process and found that the file information can be read through the Docs view (that is based on table AllDocs). The meta-information is stored in an image data type column called MetaInfo within the view.

The stored procedure proc_DeplGetFileData is used for retrieving file information for deployment. The MetaInfo column is the 9^th field in the SELECT statement in this SP. You can see that in the GetDataFromDataSet method the HandleMetaInfo method is also called using the 9^th field (fileMetaData.ItemArray[ 8 ]) of each DataRow from the DataSet of the given ExportObject .

It means that within the HandleMetaInfo method the new MetaInfoHandler object instance is created using the value of the MetaInfo column. If you check the constructor of the MetaInfoHandler you can see that the constructor parameter is casted to a byte array when calling the Parse(byte[] propertyBytes) method.

Since the MetaInfo is binary data we suspected that the following lines may cause the problem:

property.TheString = new string(chArray, startIndex, (num2 – startIndex) + 1);

And later:

property.Value = new string(chArray, num3, index – num3);

These lines handle parts of MetaInfo as simple ASCII text converting each byte to a single character. This is incorrect in the case of special characters since these characters may consist of two (or more) bytes.

Instead of these, one should use the following lines:

UTF8Encoding utfEncoding = new UTF8Encoding();

property.TheString = utfEncoding.GetString(propertyBytes, startIndex, (num2 – startIndex) + 1);

And similarly:

UTF8Encoding utfEncoding = new UTF8Encoding();

property.Value = utfEncoding.GetString(propertyBytes, num3, index – num3);

We should note that the following line may also cause problem:

property.Name = new string(chArray, num3, index – num3);

Probably we had no issue with that because we don't use special characters in property names. In Hungary we learned the hard way in previous SharePoint versions that it's best to avoid accentuated letters in field, list, view, etc. names.

We tried to prove our theory using a single console application. For this application we copied the source of the entire MetaInfoHandler, MetaInfoProperty and SerializationInfoHelper classes from .NET Reflector into our project (since all of these classes are declared as internal we could not use them directly). Then we made a MetaInfoHandlerEx class that is equivalent with the MetaInfoHandler except the above mentioned code modifications.

In the main program we connected directly to the SPS content database and selected the MetaInfo for a given publishing page that contains special characters in its page content. We tried to use both MetaInfoHandler and MetaInfoHandlerEx classes to get the value of the property.

Our results show that the original version returns the incorrect value but our version handles the special characters correctly.

The code of this application is attached to this post. Don't forget to adjust the constant values (SQL server name, content database name, publishing page name whose content contains the special characters) in the code to match with your environment before making the test.

PS: We shared our founding more than a month ago with a local MS consultant we worked together on a project and whose responsibility was to help us to solve this kind of issues on the project. Unfortunately he told us that he can do nothing with this information.