Robert J Wheeler

Over the past year I have been developing a series of site collections to roll out across a global farm.  Each of the collections have several sub webs and use a set of pre-defined metadata values to boost search capabilities and to ensure that all content is marked with the same values by each user.  As a document management system, maintaining structure is as equally important as flexibility to meet the various differences between regional requirements.

When I started the project off there were a few requirements which I was to use.  It seemed logical at the time to develop a solution package for a site collection, add the features that make it up such as document libraries, views, default pages, etc.  Using a prototype site to get the general idea out and then reverse engineering to get the site definition appeared to be an easy way to get these out.  I knew that I had around 25 collections to build, each with their own set of libraries and views, and particular web parts so making it as easy as possible was the go.  What I would eventually find out is that although possible to reverse engineer a site, there were little problems hidden within the files that made up the solution, little things which would not work for generic use.  Before I knew it I had 3 large site collections being used and a lot of features!

The first big drawback was trying to change the way a document library worked. I needed to inherit values from the parent web and maintain individuality within each sub web, I needed to update views to limit the number displayed, and most importantly, I needed to replace the way users would create sub webs and provide them with a web part to do this.  Now I started to find that each change required would mean hours of work changing the XML and adding to custom lists and updating the id used.  Each library needed to be deployed as a feature which meant updating once deployed was time consuming.  A new way needed to be found to provision sites as the web manifest was not going to suffice any longer.  It was good to build the sub web structure, but that was about all.

Time to re-think the way this can be done.  I figured that there were a lot of common features used across the initial 3 collections and these would be better placed in a common solution where I could call on them when needed.  The second revision also meant that using custom lists were to be removed.  At the time they seemed useful but at the end of the day, having a lot of custom lists is not really a good thing.  To avoid this I created a copy of the standard document library, renamed the feature and then deployed this with the common solution.  I was left with only list instances as features, but still, this was too much going forward.  My site definitions were repeated as well, and deploying lots of features to create list instances was also wasteful and difficult to manage.  Time to re-think this yet another time.

The round of changes would be very drastic.  I would continue with the common deployment of features in one solution but would really trim back the number of features to deploy.  I also decided that it might be best for me to generate my own list provisioner where I could pass in the properties needed to create the list, add my event handlers when necessary, assign content types to the list and set some default values.  Of course I did not stop there.  I also created another feature to simply assign content types to any list I created with the ability to remove the default one which is Document, a feature to set some web properties (which I will explain later the reason behind this), my own security model (again, this will be explained later) and some other stuff to help round out the site definitions.

To help move this along I created some snippets which I could use to generate the site definitions, add the features, set the modules up and add web parts.  now I was ready to provision.  I started building site collections like crazy, where one collection would take around 2 months to build, deploy, debug, re-deploy, etc., it now took me only 3 hours.  But there were a lot of issues with the features and the more I created site definitions, the more the features failed and had to be debugged.  Even though there were several defects with the features, across several site collections the same error would appear (or very similar).  After a while they disapated and I was able to produse relatively bug free sites.

So, I was happy with my process, got around half-way through building when I had a new idea.  Why limit myself to the provisioning that comes with SharePoint?  There had to be another easier (well, as we will see not really, but definately more robust) way to generate these sites.

It was time to find a new way to provision.

I wanted to harness the features of SharePoint, leverage the work I did in replacing the list instances, custom lists, the overall number of solutions, really encapsulate the commonality of each collection but using only generic definitions and feature code.  This would be an enormous undertaking and one which if successful, would revolutionise the way we create large sites in SharePoint.  Alright, that might be stretching it a bit.

Occasionally the need arises to build dynamic CAML queries in order to get items from a SharePoint list.  This was the case recently while delivering a solution for a customer.  In my solution I used a combination of controls including ASP .NET and SharePoint controls.

In my solution I created a form which posted to the server the values which were to be used for the query.  For the solution I decided to use a string builder function and a string array to store the values, and then I looped through the results and used those which contained values to build the CAML query.

Read the article

Ever wonder how to access the placeholders in a Sharepoint masterpage and add new controls programmatically?  Let's assume you are adding a Label control with some text to display.  You could create the control and then call the master page from within your ASPX page.  When SharePoint renders the page you can programmatically add the control.  To catch the error if one is generated, use the InnerException.Message to see what it is since SharePoint will return a very generic message.

Label label = new Label();
label.Text = "Something to display";

try
{
    this.Master.FindControl("PlaceHolderMain").Controls.Add(label);
}
catch (Exception ex)
{
    Response.Write(ex.InnerException.Message);
}

Recently I developed a wizard using SharePoint lists.  The challange was to use only WSS 3.0, no compiled code, and it needed to be designed so that users could manage the questions themselves.  The extra limitations were that the solution had to be provided using only the basic templates (Surveys were no available) and it needed to be generic enough to cover a wide range of questions.

To see how this was accomplished, and in a very short time, read my article Create Wizard Functionality With SharePoint Lists

SharePoint 2007 integrates Active DIrectory users with the index it produces, and applies security trimming when it returns the values matching the search criteria. The fact that larger volumes for disc space and made more affordable means that giving users large areas to store objects is not a problem. Of course each user wants to secure their documents, and the groups they belong to may have different locations to store data, makes it very easy to misplace a document

When access is granted to the fileshare, all files under the branch of the tree generally inherit the parents permissions as well as the any new permissions on sub branches.  Now, SharePoint can index all of the files and this is good, but when the file structure is such that users should be denied access to particular folders is when it becomes an issue.

Say for instance the user decides to create 1 root folder, 3 sub folders and under each sub folder, 5 more sub folders.  This makes a total of 15 folders to index which is not a very hard task.  Howeer if there are 20 root folders and n number of sub folders and so on, then we run into the problem of having to break down the structure and make it more managable, while still performing efficient crawls.

Recently I have provided a solution which covers this topic.  For sake of explaination we will keep the structure as Division, Branch, Section, Unit, Member – where a member belongs to a unit, there are one or more units to to a section, and so on.  With this type of scenario there may be thousands of users (members) but all grouped together.

 Start by taking the unit and create a content source based on the folder name and contents.  This will allow for a quick

In my solution I worked down the folder access until I found those of a particular team

It takes a fair amount of effort on the part of the administrator but when it is running, the results are as you would expect – fast, reliable and efficient.

Adding file shares to search within MOSS is relatively easy.  The results of searching within the scope of the file share will display the links as 'file://…' and will display the first few lines of the content.

For this example, on the SharePoint Server, create a shared folder: i.e. C:Shared => \[server]Shared

Open the SharePoint Central Administration and navigate to the Shared Services where the index will be created.  Click on the Search Settings link under the Search section.

We want to add the shared drive as a content source but we do not want to add it more than once.  When satified that the source is unique and can be access for crawling, click the link next to Content Sources.  This will get us into the Manage Content Sources screen where we will add a new content source.  Click New Content Source from the menu to add the source.

Give the source a name: e.g. Shared Drive Content and then select File Shares from the options below.  Enter the start address for the content which should be the one that was created earlier: i.e. \[server]Shared.  Choose whether or not to crawl sub-folders or just the share itself.

When finished, click OK which will take you back to the `Content Sources' page.  Click on the content source created to show the context menu and select "Start Full Crawl".  This will begin the process of indexing the items on the shared drive.

Next we will add a scope which will use the content source created above. Back on the Search Settings page, scroll down to Scopes and click the link next to the item 'Scopes'.  You will find the content sources and the number of files created next to them.  Click 'New Scope' and enter a Title for this.  Click Okay to save.

Back on the View Scopes page you will see the scope that was just added with a link to 'Add Rules'.  Click this link to add the content source.  On the Add Scope Rules page select Content Source, and select the source created above.  Click OK to save.

Navigate back to the Search Settings page and scroll down to Scopes again.  Click 'Start Update Now' which will update the scopes that were added.

Navigate to the site that will use the new scope and click Site Settings -> Manage All Settings -> Search Scopes.  Create or select the group that will use the scope created.  You will see the list of scopes included in the group, and all scopes that are available to use, and the order of preference.  Select the scope created and click OK.At this point the scope is added and the drop-down list next to the search box will display the name.  Select this scope and then search for the items.

Notice that the results are returned like other search results with a hyperlink pointing the files.  This is all well and useful except what about security on the file share?  What about results that we do not want people without access rights to see?  This will be explained in the next part 'Using Folder Access Permissions with Shared Drive Indexes in MOSS'.

Recently I have started a project where I will need to index millions of documents which consumes over a terrabyte of data on a shared drive.  Whilst researching the limitations of Microsoft SharePoint Server for Search I found very little in terms of real world situations which have been discussed.

In the real world, users who have files to index and have used a file system, or shared drive to store them, would probably think that they way they file documents works for them.

Another point, what about security?  Uses in the past have relied on network administrators to provide them with a location, are members of security groups which may have access rights, may have document level security, and so on.  How does this relate to the enterprise search and how will this affect the outcome?

Recommendations are made offering two types of solutions: 1) scale out; 2) scale up.  Let's quickly review what this means.

Scale Out

If we are to provide a solution where the indexes used isolate specific folders or file types, and we do this across the enterprise we can push the work off to multiple servers and run indexing to collect