The purpose for this blog has always been to give back to the community that has propelled my career and helped me put food on the table for my wife and two boys.  SharePoint is ON FIRE and the admin/developer/blogosphere/searchipedia community is very strong and growing stronger by the day.  So I've always wanted my blog to contribute to that in some small part.

I decided to move my blog to a hosted location that I have 100% confidence in from a performance, reliability, and long term safety.  Plus it never hurts to have an easy url (like your last name) where people can go to see what you're up to.  No I haven't gotten down with Twitter or Facebook yet for fear that my inner geek will get addicted to being "plugged in" and I'll never get anything done!

So… without further delay…  please permanently redirect your browsers to my new and (hopefully) final blog resting place:

www.houberg.net     or, if you just want the RSS feed:  http://www.houberg.net/Rss.aspx

I have gone ahead and copied all of my posts up to the new site… just to make sure they live on.

For all of you out there in the ether who have ever posted a comment on this blog or added this blog feed to your feed reader, I thank you and I appologize for the move.  But now is better than later.  I'm planning some really cool stuff in the near future and I when the time is right to post on it I wanted it to live at a permanent residence without further change.

Thanks for the space SharePoint Blogs.  Please continue providing this excellent resource to the community!

Lesson #1: Authentication and Localhost Loopback Check:

I do a lot of projects related to migrating content into SharePoint.  I learned long ago that migration throughput is much higher if my migration server is a WFE and I upload content to "localhost".  Basically, I add an entry in my host file that points the Portal host header to 127.0.0.1.  The performance benefit comes from the fact that certain API and web service calls are much faster since data traffic doesn't leave the box until it's headed for the SQL Server (via SharePoint).

Well, the last couple of projects keep hanging me up because I can't seem to authenticate to the "localhost" host header site.  That stupid auth box keeps popping up!  And then it hits me… Once again I've forgotten about the magic "DisableLoopbackCheck" registry value.  A while back, one of the MS updates (or service packs, I forget which), Microsoft introduced a security feature that is meant to prevent reflection attacks by blocking authentication when using 127.0.0.1 with a host name that doesn't match the FQDN of the machine.  The fix for this is to implement one of two possible registry tweaks described here:  http://support.microsoft.com/kb/896861  I'll be honest, I usually just go with the sledgehammer approach and use the DisableLoopbackCheck key.  Most of the SharePoint servers I deal with are internal and are in no danger of a reflection attack.

Lesson #2: MOSS Search Error

I run into some crazy stuff sometimes related to search.  Since we tend to put hundreds of thousands or even up to tens of millions of documents into SharePoint regularly, we occasionally find a rough edge or two.  Also, KnowledgeLake software uses the search API exposed by MOSS search so that also adds a dimension of interest.  Today I came across a wierd result where our search interface was surfacing a strange error.  The error shows up in the UI as something like "specified network name is no longer available".  In the event log we find something like "Query machine [servername] has been taken out of rotation due to this error: The specified network name is no longer available"

My first thought was that I had a corrupt index because certain search criteria would yield normal results while others cause the error.  So after a hard core "reset all crawled content" and full crawl didn't fix the problem.  I researched further.  It turns out that this is an issue on Windows Server 2008 machines with something called the "TransactNamedPipe" function call.  If the function is used to request more than 8,000 bytes of data, the error occurs.  So a quick trip to the hotfix store and PROBLEM SOLVED!  If you come across this error, have a go at the hotfix found in this KB article:  http://support.microsoft.com/kb/946487/en-us

So that's it for the lessons learned on this gig.  As for Bermuda… Well, what do you expect!  It was beautiful.  I was fortunate to have the weekend to myself.  So I buzzed my little scooter all over the country (island).  I checked out the crystal caves and visited several awesome beaches, the best of which is found in Horseshoe Bay.  This truely is a postcard beach.  If you're a photo buff, check out some of the shots I snapped here:  http://www.flickr.com/photos/37780535@N05/sets/72157617260859413/  Living smack dab in the middle of the US, it's a privilidge when I'm able to get out and admire this amazing earth that God created.  Bermuda is amazing.. a bit expensive!.. but amazing.  It is basically untouched by industrial polution so the water is impossibly clear!

Finally, I have to give a shout out to Mark and Ben at Mad Hatters.  Mark is the head waiter, Ben is the Executive Chef, and the two of them co-own an awesome restaurant tucked back in Hamilton Bermuda.  It's a bit hard to find!  If you can locate the Robin Hood pub, then you're just about across the street!  Anyway, if you ever make it to Bermuda, I HIGHLY recommend it.  I went twice and each time in addition to the (exceptional) fare that I ordered, Ben also whipped up a little extra something special.  The first trip yielded an unexpected soft shell crab.  The second trip offered an extra "salmon sampler" plate.  Both were absolutely delicious.  Ben is very gifted!  Also, I was blown away by a black olive pesto that topped some veal I had ordered.  I hate black olives.  I hate almonds.  I hate walnuts.  I can only handle so much garlic.  But Ben managed to chop it all up very finely and saute it in some olive oil.  It was absolutely brilliant.  All in all, I had some great food in Bermuda, including the local favorite "wahoo" fish steak and fish chowder.  But Mad Hatters was definately the "hidden gem" of the trip.

Anyway, I had a good time in Bermuda.  Greg and Murray were excellent hosts and with Sharon's help on the business rules side, we knocked it out of the park.  Thanks guys! 

It looks like there are some interesting potential projects on tap for the summer… Including a very large project that involves an MSFAST solution.  Looking forward to that one!  If that one lands on my plate, I'm hoping to have a bunch of cool stuff to post about MSFAST.

Cheers

How was it?  Absolutely incredible… and the most difficult training I've ever attended in my entire life!  It was 20 straight days of mental gymnastics.  I don't want to come off sounding negative though.  It was a very rewarding experience.  Microsoft is clearly aiming for a very "pure" Master certification here.  If you aren't extremely experienced with virtually all aspects of SharePoint, then there is a good chance you won't be successful.  I had a few weak spots and I had to work my tail off (many late night study sessions) to compensate.  But I think that was the case with a lot of the attendees.  It's hard to have broad AND deep experience in every aspect of IT Pro and development that SharePoint touches.  So each one of us found areas that could be improved.

So just to be clear before I get a bunch of specific responses to this post, there isn't much that I can say legally.  But I do want to convey just a little flavor of the experience in general.

Let's say you've got your 4 MCTS exams covered, you've got several years of field experience with SharePoint, and you've talked your company into putting up the $$,$$$ to cover the training expenses.  Congratulations, you're ahead of the game!  But wait… You've got to make it past the interview.  You have to be prepared to prove to the MCM training team that you have what it takes to be successful and trust me, they can figure out pretty quickly if you're all talk and no walk!  If you pass this interview, then you'll make it into a rotation. (Congrats to those of you heading to the R3 rotation in June!)

So from there, I can't say a whole lot about the training specifically, but I can talk about the experience as a whole.

First of all, you're in the fire from day 1 and that fire has an interesting way of forging you together with your classmates.  One of the things I enjoyed most about the training was the people in the class.  We had a very culturally diverse group with folks from the UK, India, Austria, Germany, Portugal, Netherlands, Chicago, California, and of course St. Louis, MO (me).  But it doesn't matter where you came from.  Everyone enters the pressure cooker at the same time and the best way to make it through is to connect as a team.  Group study was key.  These people are some of the most intelligent and experienced SharePoint minds in the world and everyone has strengths and weaknesses in different areas. 

Also, I was fortunate that every single one of my classmates were just plain "good people".  So I want to mention them.  For the sake of privacy, I'll just abbreviate – here's a shout out guys – you know who you are!  We had:

• Ali, a very experienced MCS consultant and all around great guy
• Ram, a fun dude and high quality MCS consultant out of Chicago and fooseball master – at least in our class
• Dn, an extremely experienced Microsoft field engineer who blew my mind with the crazy problems he's solved. Dn was awesome.
• Gln, the instructor from Germany who asked great questions and had a cool British accent
• Tj, a soft-spoken but confident consultant from India
• Ptr, another quiet but extremely friendly (and intelligent) guy.  You don't realize just how smart he is until you talk to him a bit.
• Brt, another MCS consultant with mad skills and amazing study habits
• GP, MCS consultant from India, nice guy, fun to be around, asked some great questions
• Rmn from Austria (another cool accent), MCS lifer (15 years!), dev guru, could probably code circles around me
• Chs, a very well spoken independent consultant and all around Information Architecture philosophizer
• Sp, an incredibly knowledgeable independent consultant who really added a lot to the class… particularly after the instructors were gone for the day
• Mir, an extremely intelligent person from the Netherlands.  Very pleasant contributor who's trying to become the "second" in the Netherlands.
• Mau, an X- product team member who had some really cool nuts-and-bolts background info on WSS, and owner of the class mascot "Asta"
• Nkl, a wickedly smart SharePoint guru from India.  He had some awesome tips to provide from lessons learned
• Kl, an independent consultant and hosting guru from Chi town. Gotta love this guy.. he a fan of Da Bears!
• Me, I brought a bit of infrastructure and SQL to the table.

I don't want to forget our Program Manager and Class "MC".  I don't know if he wants his name called out, but I wanted to publically say that I really appreciated all his hard work and effort in this program.  He did a fantastic job of managing the entire process and the candidates.  This training really has it's highs and lows and "J" was there to help guide us through.  Thanks man.

Let's be honest, SharePoint is a HUGE product and nobody is a Master of all the deep dark corners of SharePoint when they walk in the door.  But as you can see from the list above, the whole group together IS functions as a Master of SharePoint.  When experiences are shared everybody learns.  Then you mix in the excellent instructors that Microsoft has commissioned, the top minds in their respective fields.  I don't want to call them out because they may or may not be the instructors for future courses, I don't know.  (OK, I have to say that Paul Randal – SQL Grand Master – was straight up amazing.  I really geek out on SQL stuff.)  But it is clear that Microsoft is serious about the true Master nature of this program. So what you end up with is not just training.  Rather it's a true LEARNING experience.  If you invest in the corporate learning experience and you put in the work, you can't help but come out much stronger than when you entered.  On top of that, you'll feel more confident in your skills and, if you pass, you'll feel validated in your knowledge.

The days are long.  Hours upon hours of lecture, but oddly enough, they fly by because this stuff is so relevant.  Before you know it, the "day" is over.  Heh, but then the study time begins.  There are labs to complete, material to review, and tests to study for late into the night.  Then you have to find a way to shut your brain off long enough to catch some sleep before the whole process starts all over the next day!  Usually this isn't a problem because you're wiped out mentally every day.  But you'll end up dreaming about this stuff too!

I don't know what I can say about the last day other than it was the hardest day of my professional career!  I'll just leave it at that.

Was it worth it?  Yup!  You know it's interesting.  This is by far the most expensive training course I've ever attended, but the resources that Microsoft brought to bear on this project were nothing short of incredible.  I can't image how much money has been spent to bring together this comprehensive set of presenters, courseware, and hardware resources.  I figure that MS is probably just breaking even!  At least in the early rotations.  I think they are mostly interested in getting competent SharePoint Masters out into solution space to ensure that SharePoint is successful.  I don't think this program is going to make them a bunch of money. 

Is it expensive?  Yes.  But it is THREE weeks of training.  If you do the math compared to some of the week long training courses out there then you'll see they are fair.  There is no way anyone else but Microsoft could put together this much brain-trust in one place.  This isn't your standard training class where you've heard most of the marketecture before.  This is REAL knowledge.  Hard core nuts-n-bolts information that will lay a strong foundation and enable you to tackle ANY SharePoint engagement no matter how complex.  That type of value will benefit ANY organization or independent consultant who works in the SharePoint space.  I was VERY fortunate that KnowledgeLake sees the value in having an MCM on staff.

I believe that the Microsoft Certified Master (MCM) SharePoint cert will quickly be recognized as the pinnacle of technical achievement in the SharePoint community.  A Master is someone you can trust for accurate information.  If for some reason they can't help you directly, then they have the connections to get it done.  Probably someone from the rotation they attended has the information.  Even if that isn't the case, then there are other lines of communication that can be tapped by a Master to get answers to the hard questions.

I still don't know if I'm a Master yet or not.  But even if I didn't pass all the tests, I can assure you that I WILL be a Master eventually, even if I find out that I have to re-take some stuff.  On that topic… If you know anyone who attends the training and fails one or more tests, please don't think less of them.  The Master tests are designed to make sure you are unquestionably competent in ALL aspects of SharePoint.  They are indescribably difficult.  The sheer volume of knowledge that has to be in place to even make it into the class puts each candidate on a higher plane already.  Even someone who washes out and doesn't achieve certification is probably a better SharePoint resource than most.  There is a very small minority of those who make it TO the class that pass everything the first time through.  There is NO SHAME in a retake!

Now that I've been through the gauntlet I am more aware of technical weaknesses.  The cool thing is that, armed with that information, there is a clear path to improvement.  There is no way to convey the difficulty of the program in words.  You have to experience it.  My biggest recommendation to potential candidates is to STUDY THE PRE-READING materials.  That is your best preparation tool.  Once again KL was very helpful in this regard and even gave me plenty of extra to pre-study before I even attended.  I was surprised to find that almost everything that was taught was available "somewhere" out in the ether.  But if you are experienced… and you are prepared… Don't fear the program!  Mind your training!  Save you it can! (gratuitous Star Wars reference).

Bottom line:  The MCM training just put me on a knowledge rocket ship.  The G-forces were intense.  It was like one of those roller coasters that scares you to death and thrills you at the same time.  I've never enjoyed a knowledge expansion exercise as much as this.  If you've got the skills, you've got the nerve, and you've got the $$$, then GO FOR IT!

I now know that if I haven't passed everything, I am capable.  So I won't give up.

To anyone else who is accepted into this training, GOOD LUCK!

To my classmates in R2, thanks for everything guys.  It was an honor. 

I've been holding off on this post for a few days now because I couldn't determine if this information was "in the wild" yet.  But it turns out that it's been reported on by at least one other blog post, so if this was protected info, I'm not the guy who released it!

That said… here's the deal.  Microsoft is doing a Wal-Mart rollback on ESP (FAST Search) for SharePoint!  If you have an EA with Microsoft that includes MOSS, then you can "add on" ESP for SharePoint for $25K per server.  The MOSS CALs cover the client access to said server.  So why is that cool?  Let us count the ways:

1. Unlike MOSS Search, FAST Search is scalable far beyond the 50 million item recommended limit for a SharePoint index server.  In fact, depending on the configuration, MSFAST can handle 50 – 75 million indexed items PER SERVER.
2. Note the PER SERVER.  With FAST, we can scale OUT instead of just UP.  Need to crawl more items…  No problem, just bolt on another FAST server.  Need to crawl items faster.  No problem, just bold on another FAST server.
3. Anybody want to guess how big a SharePoint farm scale now that MSFAST is doing the crawling?  Well, I can see architectures of 1/2 BILLION documents OR MAYBE MORE being possible with binary externalization being an additional enabler.
4. It just got WAY cheaper.  FAST used to be sold based on the features needed in the solution and it was well into the 6 and sometimes 7 figure range for cost.  Heh.  Nope.  Now it's sold PER SERVER for $25K as long as it's used to to index a SharePoint farm.
5. It's available TODAY!  While there is a licensing path to FAST for O14, you don't have to wait!  If you're index server is suffering and you've got an MS EA… what are you waiting for?

This is a game changer folks.  I'm currently architecting what could potentially be the largest SharePoint implementation in the world and FAST is a cornerstone technology for this solution.  We're talking 100s of millions of documents.  30 terabytes of metadata and binary content.

After I grind through SharePoint Master training in a few weeks here, I'll soon be heading off to MSFAST training.  So KnowledgeLake will soon be well position in the SharePoint enterprise architecture space.  That's right.  KnowledgeLake is more than just leaders in the ECM industry.  We have assembled a tiger team of seasoned SharePoint consultants as well.  Sort of like that Midas car commercial… If your looking for anything related to SharePoint deployment… "we do that".

The competition should be scared.  there are no more bullets in the gun. They shot across the bow with "MOSS Search has scalability limitations" and "you can't store blobs in a SQL database".  Both issues have now been address by either MIcrosoft server stack or 3rd party products (externalization).  And very soon, even externalization will be core stack.

So stay tuned as KnowledgeLake works to prove that the boundaries of what SharePoint can do for the enterprise have indeed been redefined!  As we push into a new level of scalable solutions, I'll publish experiences here on this blog.

I hit a big roadblock when I wanted to use a picture library as a source for an animated slideshow.  I can't upload custom webparts to the site because I don't have the rights to deploy solutions or features or to manipulate the web.config.  So I figured there has to be some way using RSS or something.  My first instinct was to start trolling for a free Flash based RSS reader type library.  No joy.  Most of them either cost money or they wanted to do a one time conversion to output a SWF for a Facebook page or something. Lame.

Moving on… It hit's me!  Dude, you're a Microsoft developer so take your butt to "Live Search" and type in the word "Silverlight".  Way cool!  Turns out there's an open source (uhhh free) on Codeplex that a few XSL script kitties have munged into a SharePoint page.  SWEET!

So after following the instructions on an excellent blog post called "Using Silverlight SlideShow and a SharePoint Picture Library" from the excellent SharePoint Evolved blog, I was rockin!  In fact, I was able to tweak the procedure a bit and squeeze TWO different slideshows point to TWO different picture libraries on the same page.  In my case, I needed one of the slideshows for some rotating "information" slides and the other slideshow for your standard "picture stack" slide show.  It looks great and it's dynamic.

So then I got a little creative.  I added an extra column to each picture library called OrderIndex.  Then I tweaked the CAML a bit to order the data by the OrderIndex.  Now I'm able to add and remove images at will AND re-order them however I see fit!  And that way cool Silverlight slideshow just picks up the changes and rolls with it. 

Rock on Silverlight!

I've been blessed to be part of this white hot technology called SharePoint for the last 5 years.  It all started in late 2003 early 2004 when a SharePoint MVP named Darrin Bishop took me under his wing and began to show me how SharePoint would one day RULE THE WORLD of corportate intranets and collaboration portals.  He was dead right.  I really enjoyed the platform.  There was a lot more sizzle to it than the Biztalk work I had been doing and I wanted to be more than just another .NET developer.

Turns out that Darrin was writing a book called The Rational Guide to Building SharePoint Web Parts and I was unknowingly the guinee pig for some of the concepts that he would later publish.  Anyway, Darrin helped provide me with a good SharePoint foundation, back when there weren't many people doing SharePoint.  From there, I executed several engagements on my own on the WSS V2 platform.

Eventually, I was led to make a change.  A long term engagement (over 6 months) with a client that involved a daily 2.5 hour 1 way commute got to be a bit much and there was no end in sight.  So I poked my head out onto Monster.com and found this company called KnowledgeLake who figured out that the SharePoint platform makes for a real nice "out of the box" storage repository for Document Management systems.  They needed Professional Services consultants with SharePoint experience.  Not only did I have experience architecting solutions, but I could also code against the object model.  I sent my resume to them and they called me within 45 minutes!  A great relationship was born.

Well, fast forward to 2009.  I know have 5 years of SharePoint architecture experience and I've passed all 4 SharePoint MCTS tests.  Low and behold, I've met the pre-requisites for the Microsoft Certified Master training program!  But, holy cow this is some expensive training!  Who could expect their employer to just randomly spring for an $18,000+ training program with no guarantee of success?  I didn't think they would go for it but KL never ceases to surprise me.  One of the VPs of the company heard about the Master program through the MS Partner Advisory Council.  He actually contacted ME! 

So I applied and was accepted into the program!  I am very flattered, honored, excited, and scared to death all at the same time right now.  I have 6 weeks to brush up on the areas of SharePoint that I have less experience in.  My career has been focused around large scale architecture and deployment, search, taxonomy design, and general IT infrastructure planning.  So I'll be spending the next 6 weeks broadening my skills with branding/theming, web content management, BDC, Forms Server, and Report Server/BI integration.

I've already poured through all of the background info I can find on the net.  Including the blog entries on the topic from Todd Baginski, Spencer Harbar, and Andrew Connell.  So based on what I've read so far, I'm pretty sure this is going to be the hardest thing I've ever done!

I will blog more about the program but only to the extent that I'm allowed.  I haven't figured out yet what "allowed" is yet, so there won't be much right away.

Hopefully though, I'll be sitting down, back in my office, on April 5th, blogging about the experience in general.  Pass or fail.  I plan to write it down here.  Oh yeah.  I don't plan on failing.

Anyway, this concludes the narritive on my SharePoint career!  As I move through new study material, I'm sure I'll have some additional interesting technical comments.

Cheers.

I'm not sure if I ever really gained any "followers" of this blog last year.  I figure I posted some decent reference material but probably not frequently enough for anyone to subscribe to the feed.  So what the heck have I been doing since my last post in June of 2008?

Well, the short answer is that I've been building a house!  The contract was placed for the sale of our old house in June of 2008.  Since that time, it's been a tornado of events that lead me to today where I type this from my brand new office (I can still smell the new paint!) in the loft of our new home!  We sold the old place, moved into an apartment (didn't want 2 house payments!), started a construction loan, dug a big hole, watched hurricane Ike fill it with water, pumped it out, poured the basement, and then watched my general orchestrate a symphony of subcontractors as they built us a beautiful home!

Anyway, so that explains the haiatus!  In the mean time, I had to pay the bills, so I have to once again sing the praises of KnowledgeLake.  They graciously helped me by giving me content migrations to work on for a group of customers.  The benefit to that was that I was working alone, usually from home, where my schedule could be extremely flexible.  When they dug the hole and struck water, I could go out to decide how deep the basement would be (and give up my walk-out).  When one of the concrete walls in the basement came out a bit too far, I could help decide where the new stair placement would be and how it would affect my other walls.  When the master bath shower ended up sticking out into the middle of the room, I was there to change the plans so that it didn't. 

Anyway, to make a long story short, KnowledgeLake totally rocks.  There have been seasons where I worked my tail off for them to dream up some creative solution or to pull a customer engagement out of the fire.  The last 6 – 8 months was a season that had me working just 8 hours a day, no more, no less and often in a very hit and miss schedule.  They let this be a season that I could focus more on my personal life than my job.  Thanks KL.  That type of employer loyalty is hard to find and I don't take it for granted.  It just makes me want to work all the harder this year!

So there it is.  I've been off doing migrations and building a house.  Not much interesting technically.  But it's a new season now.  And in my next post, I'll be talking about what will be driving more blog posts in the near future!

The culmination of the efforts of several incredible individuals has officially materialized.  ANNOUNCING:

Using Microsoftr Office SharePointr Server to implement a large-scale content storage scenario with rapid search availability

This document is a scalability whitepaper that has been reviewd by Microsoft and posted on TechNet. This white paper presents the key decision points, architecture design and definition, test criteria and results from a scalability and performance test conducted at the Microsoft Partner Solutions Center (MPSC) in Redmond, WA.

Don't be discouraged by the 97 pages of content in this document!  It's an easier read than it appears.  Many of the pages are graphs and charts that document the results of the test runs.

I just wanted to thank Andy Hopkins for breathing life into this effort and finding the funding that got it done!  If he hadn't brought in KnowledgeLake, I'd still be just messing around with conversion and migration into SharePoint!

And I can't forget my good friend Paul Learning.  This guys is amazing folks.  His architecture guidance and his ability to keep us "on track" with Microsoft guidelines were excellent.  Paul is a taxonomy wiz kid and a dedicated professional.  I can't even count how many late nights he worked to bring this effort to life.  Thanks Paul.  As each of us navigate our careers, I expect to stay in touch with you all allong the way.

I also want to thank the SharePoint architects and product team members that consulted with us on this effort.  You guys were willing to share your knowledge and I'm a better architect for it.  I'm not sure if any of you read this blog but in case you do, I just wanted you to know that I appreciate that you added your knowledge to the mix.  My hope is that you will continue to see me evangelizing SharePoint Scalability and Search technologies for many years to come.

Finally, (can you hear the Emmy exit music?) I wanted to thank KnowledgeLake.  These guys continue to bless me with the opportunities that allow me to further my career and the success of KnowledgeLake.  I don't take this privilege lightly.  Ron Cameron, Bob Bueltmann, Gregg Smith, Chris Caplinger, and Tom Brownell.  I can't imagine working for a better 5 guys.  The servant leadership culture that you all live and breathe is truely appreciated.

So, the whitepaper is here… forever added to the ocean of information in TechNet.  My hope is that it helps the community and furthers the image of SharePoint as a highly scalable collaborative and content management platform!

Download (.docx): http://go.microsoft.com/fwlink/?LinkId=120881

TechNet topic: http://go.microsoft.com/fwlink/?LinkId=120901

Download (.doc): http://go.microsoft.com/fwlink/?LinkID=120890

Download (.pdf): http://go.microsoft.com/fwlink/?LinkID=120891

Microsoft's new hypervisor technology "Hyper-V" is set to storm onto the scene.  I saw some excellent demos this week.  So good in fact, that I'll probably have Windows Server 2008 with Hyper-V running on my laptop and/or my desktop before the week is out.  The big thing I picked up here is that the hypervisor layer has been pushed below the kernal layer so we get a very "close to the metal" connection with system RAM and disk I/O interfaces (SCSI, SATA, HBA, etc).  So the old addage of losing 15-20% performance from virtualization probably goes out the window.  It's probably closer to 3 – 5% now.

I also enjoyed talking with the enterprise search product manager from Microsoft.  I look forward to talking with him again regarding how FAST is going to complete the enterprise search story for SharePoint.  With any luck, KnowledgeLake will be in on the ground floor of this new frontier.

Lastly, Hao Zhai, a friend of mine and the Commercial Product Manager for KnowledgeLake attended a session on "Unstructured Data Storage".  I totally missed this session so I'm going to plagerize the email he sent me.  This single email signifies the beginning of Microsoft "getting it right" regarding large scale document repository architecture.  Check this out from Hao:

Went to a session today on “Unstructured Data Storage”. The presenter confirmed that Microsoft will provide a RBS (Remote Blob Storage) API for SQL 2008 that allows binary data to be stored on “remote” devices. NetApp, EMC and IBM have all officially signed up to write providers for this API, Hitachi and Fujitsu were also mentioned as “in the works”. This is in addition to the FileStream feature where blobs are stored on locally attached NTFS volumes. In both cases, SQL Server will manage the backup/link consistency between DB and blob store.

The SharePoint team has committed to leverage FileStream and RBS in O14. To me, this means SharePoint will be able to truly compete head-to-head with Documentum/OnBase/FileNet in terms of storage scalability.

This will replace the current “External Blob API” in SharePoint v3 SP1. The current API will continue to exist for backward compatibility for those who built solutions around it while waiting for O14. However, with the SharePoint team committing to implement a new/better solution OOTB leveraging SQL Server 2008, any investment in the current API would have very short-lived relevancy.

WOW!  Amazing stuff!  Guess what guys, without blobs in the SQL database, we will be able to take SharePoint into the BILLIONS of documents!  Once remote blob storage bursts onto the scene for SharePoint even simple taxonomies will be able to scale to INSANE numbers of documents!  I CAN'T WAIT!!!

Lastly, I want to throw a shout out to the Novell booth in the partner expo center.  Thanks to them, I scored a $1000 travel voucher by winning the Tiger Woods long drive challenge in their booth!  Also, I'm not sure if Rob, Khai, or Mac read this blog or not but I had fun with those guys!  We all did a few round trips into the long drive line trying to one-up each others drive lengths!  It was all friendly competition and I had a BLAST!

So there it is.  With any luck we'll all soon be maximizing our massive multi-core servers using Hyper-V to run large SharePoint farms that do blob externalization all while being crawled using FAST search!  So uhh, just a heads up to the Documentum guy who tried to sell his wares during my Birds of a Feather scalability session…. Might want to freshen up that resume. 

This timer job is associated with each web application in the farm.  It is automatically created and configured to run weekly when the web application is created.  I mentioned that it might be a good idea to run some sort of statistics refresh after a large migration (into SharePoint) or similar operation.  Well, the interesting thing is that I traced this timer job definition all the way down to a stored procedure that exists in every content database.  The stored procedure is called proc_UpdateStatistics. 

So if it were my farm, I might be inclined to try to modify the schedule for this timer job but this may not be possible (haven't had time to check).  The other possibility is to just execute the stored procedure manually but that might not be supported by Microsoft.  Interestingly, according to this KB article, Microsoft specifically says that it's OK to update statistics, so maybe it is supported.

Anyway, I just thought all of that was interesting.  On a related note, I want to clarify, that this timer job runs the statistics update on CONTENT DATABASES.  So it is highly likely that you will need to create an additional maintenance plan specifically for the SSP search database.  If the search database is large and/or supports millions of documents, it's a good idea to keep it cleaned up regularly. 

Finally, a bit of a warning.  No matter how you manage your indexes/statistics refresh, you don't want to be updating statistics any time a crawl is running.  Particularly if the crawl has a lot to do (full crawl), if you're trying to update statistics at the same time, you could end up grinding the SQL Server into oblivion.