EMC and Microsoft at the SharePoint conference!

Time:  Yesterday, 8.30 Pacific
Where:  SharePoint Conference 2011 in Anaheim, California
Venue: Conference kick-off keynote

On stage: A DJ and a high performance SharePoint farm in two racks.

Backbone: EMC VNX 5700 Unified Storage Array with NEC high density Servers.

So what?: how about a 14TB Content Database with FAST search and 100 millions documents, thats what!

Delighted to say that EMC was center stage at the SharePoint conference and a main focus of the SharePoint main demo.
The demo which lasted 10 minutes showed how a large-scale sharepoint farm with a extremely large content database of 14TB, with FAST Search Server, running at full load suffering a SQL Server node outage.
The environment, using SQL Server Denali CTP3 and SQL’s AlwaysOn technology was key in failing over the SQL Content databases in seconds.

EMC was key in maintaining performance around the clock with this solution, and the VNX Array was not stressed at all even though very large IOPS/second were generated.

SharePoint 2010 Sp1 bring much larger content database supported sizes, up to 4TB for general use
and UNLIMITED for Documents and Records Center in specific circumstances (- 5% content access or 1% content modified per month/avg)

More information
================
www.emc.com/sharepoint

whitepaper – Managing Multi-Terabyte Environments – http://go.microsoft.com/fwlink/?LinkId=223599

SP1 Announcement
http://blogs.technet.com/b/office_sustained_engineering/archive/2011/05/11/announcing-service-pack-1-for-office-2010-and-sharepoint-2010.aspx
Joel’s quick blog entry for the new limits –  http://www.sharepointjoel.com/Lists/Posts/Post.aspx?ID=457
Technet article on the details –  http://technet.microsoft.com/en-us/library/cc262787.aspx

SharePoint Conference Season

Hi all,

While May, June and August are the era for the big platform events such as EMC World, TechED, and VMWorld…
October is the season for my two major application events.

I am happy to announce that EMC are proud Gold Sponsors at:-

• SharePoint Conference USA                    Anaheim, California – Oct 3-6
• SQL PASS Summit                                    Seattle, WA – Oct 11-14

EMC @ the SharePoint Conference

• Large booth where key experts from the EMC Business Units will be able to describe to you how to make your life easier with SharePoint
• Demonstrations, mini-lectures, and Q&As
• Free give-aways.  Yes, again, like TechEd, we will have free t-shirts and on the final day many, many cash spot-prizes for wearing your EMC T-shirt

      Two Sessions

• Proven Best Practices for Virtualized SharePoint and FAST Search  – Tuesday, Oct 4 – 3:15pm – 4:30pm

Speaker(s):  James Baldwin, Eyal Sharon  (James & Eyal show)
Level: 200
Understand technical best practices to design and deploy a virtualized SharePoint that leverages FAST Search. Understand how design a flexible and robust architecture that supports your advanced collaboration requirements. Understand how to architect a solution that addresses IT challenges for data growth, application availability and simplified management that also enables your users to find and leverage the right business information to make better decisions.

• Searching Video with FAST Search for SharePoint 2010

Speakers:-  Matt Roberts, Nate Treloar
Level 300
Demonstrate how to integrate external video metadata generation services with native SharePoint Search capabilities

Dont forget Europe!

The European SharePoint Conference is taking place in Berlin, Germany   – October 17-20.

I will be there presenting the following session:-

Optimize, Store and Protect SharePoint 2010 Server…Best Practices     Wednesday 15:00 – Session W21

Learn about the critical best practices and considerations for optimizing and growing SharePoint farms, storing user data efficiently and securely, while backing up TB’s a data in minutes. RBS (Remote Blob Store) and Virtualization, are just two of the many techniques discussed in this session. Realize the considerations for providing fast, automated disaster recovery for the entire SharePoint environment through SAN-based technology.

EMC @ SQL PASS Summit

We will have something kinda special at the SQL PASS Summit.  Can’t say more.

But what I can say…

• Large booth area in the Pavillion, with SQL Experts from EMC including two heros from our team, Tony Wu and Bruce Ye, travelling all the way from Shanghai.
• Demos, booths, best practices and most importantly application-led conversations around;
• SQL Server scalability – Infrastructure
• Optimized Data Protection
• High availability to where? Same SAN? Same site? next door? next state? next country?  – All of the above <—
• Something Flashy
• Proven Solutions around high-speed SQL deployments, one of which is in build right now with Michael and David in our Cork labs.

Hope to see you there.

James.

Content Database Storage Provisioning for SharePoint – where’s my SQL DBA?!?!

So, you are a SharePoint Admin….

You want to create a new content Database …
It will be very large….you want it on seperate storage…

What do you need to do to achieve this?
Well, from Central Admin/Stsadm/PowerShell….

…you cannot specify which storage SQL will create the content database on (it will use the SQL instance defauts for Data and Log files)….
…you cannot specify the inital size of the content database…
…you need to find your SQL DBA….
…and your storage admin….who needs to arrange a time to create and unmask the storage….

…ever try building a house and installing a kitchen?  You need the kitchen guy, the plumber and electrician….

Well, not any more.    Our microsoft platform-focused development teams in EMC have listened and are working diligently on creating integrated, lightweight tools to help…

Announcing….. the EMC Storage Integrator Tool ….as part of a bigger Microsoft management release from EMC – life just got easier for us…

EMC Enhances Management of Virtualized Microsoft Applications

I cant say much more at this point, but I will have answers, details and demos hopefully next week….

Its been a while….

Hello folks,

It’s been a while since we last spoke but I return after spending the latter half on an exciting project as EMC’s architect of the EMC Data Computing Appliance, working closely with Greenplum’s platform team in understanding what the best balanced Data Warehousing solution should consist of in terms of compute, storage and network.

http://www.emc.com/about/news/press/2010/20101013-01.htm

http://www.emc.com/campaign/global/greenplumdca/index.htm

My design did well and beat Exadata v2 in the mainstream metrics with massive scale-out ability.  22GByte/sec read bandwidth without tweaking was great to achieve.

It was an interesting exercise, working with vendors such as Brocade engineering in getting the best from all hardware.  I burn my summer on it, but I absolutely loved it.  Linux, an MPP database architecture and DW-style database architecture was a wild difference to what I do day-to-day in Microsoft land, but it wasnt hard once one has the interest to do it – and boy, was I interested in being involved in such a project, leading a supreme engineering team out of Cork to pull off a major project in a minor number of months. 

But, I’m back and am working with our Proven Solutions teams in creating real (not suggested servings) solutions for real customers on Microsoft platforms, being SQL and SharePoint centric.  VNX came out, FASTVP on Symmetrix came out, Proven Solutions teams working their socks off in generating serious whitepapers, solving serious issues – those Whitepapers will be stored in a new page found at the top of the frame, called Documents and Whitepapers.  Feel free to browse.  I will update as interesting whitepapers and solutions and demos appear…

EMC World 2011 and Tech-Ed USA 2011 are coming….back-to-back….Vegas then Atlanta – we’ll be there……

See you there….

Free webcast – SharePoint 2007 & 2010 Storage & Virtualization Best Practices

Hi all,

Eyal and I recently delivered a webcast in association with TechTarget.com, sponsored by EMC.

We talk about Microsoft SharePoint 2007 & 2010 storage and virtualization best practices.

SharePoint is tricky, with so many moving parts, volumes, and storage requirements.  We try to dispel the confusion and set out best practices based on testing in our Proven Solutions Labs in EMC.

This has been converted and is available on-demand and is available at:

http://www.bitpipe.com/detail/RES/1278422421_403.html

Feel free to take a look, hope it helps.

James.

P.S. If you are ever looking for some REALLY good deep-dive SQL Training…

Check out Paul Randal and Kimberely Tripp @ SQLskills.com – they’re brilliant. They are in Dublin this week in conjunction with Prodata

We’ll be at Microsoft Tech-Ed 2010

Hello all,

Just incase you find yourself at Microsoft Tech-Ed 2010 New Orleans next week….we’ll be there.

Eyal Sharon, Brian Henderson, and myself will be there amoungst other crack Microsoft folks, showcasing how EMC technology definately enhances the experience of a Microsoft solution.

* Hyper-V, small, medium, and seriously enterprise will be shown
* Automated Disaster recovery
* EMC integration and management
* Move your SQL database off 60 FC 15k Drives to just 5 EFD Flash drives and get better performance!….

…just tiny bits of whats on offer, so…

If yer at it…please call by the EMC booth. 

Thanks – James.

EMC World 2010 – Boston

Hi all,

If you find yourself at EMC World, why not drop into the Solutions Pavillon where we are showcases SharePoint and SQL solutions.

I’m also presenting the following sessions

Tuesday 08:00         SharePoint Storage Best Practices

Wednesday 08:00   Birds of a Feather – Expert Panel – SharePoint, SQL, Oracle and SAP

Thursday 13:00   SharePoint Storage Best Practices (repeat)

I’ll drop the slide here into here once we are done.

Cheers!

J

It’s nearly here… SharePoint 2010

In May this year,

just after the SQL Server 2008 R2 release (which will be made at the SQL Pass Europe conference on April 21st)

Comes the big guns, SharePoint 2010 and Office 14, I mean the Office 2010 system.

Right, smack in the middle of EMC World 2010 in Boston, Mass.

We will be there with architectural guidance, performance and scalability and how the new built-in Search capability weights up against the old built-in search capability in SharePoint 2007.   Come see us in the EMC Proven Solutions Pavillon.

Our next steps after that? Scale up and bring FAST search server for the Enterprise into play.

One of the bigger challenges for us in the lab is to devel into testing more functions and features available in the new SA model versus the old SSP, but we’ll get them.  More to come on that as we progress.

 The Technet documentation build-up is going fast and furously right now, a recent update was the SharePoint 2010 hardware and software requirements.

We’ll keep in touch with our progress.

James.

The iSCSI performance issue…

Just an update to the iSCSI performance issue.

Microsoft has released the KB article we worked on last week

http://support.microsoft.com/kb/2020559

Due to the significant change required, the design change request has been submitted and will not be in Windows 7, but is currently being considered for Windows 8 platform.

Best Practice for Hyper-V with iSCSI

Hello folks.

In EMC Proven Solution testing for a given use case, I have come across a serious issue in relation to iSCSI responses, which inheritently causes slow storage response times, very slow cluster polling and enumeration.

Test config
Windows 2008 R2 with Hyper-V.   6-node Hyper-V cluster.  65 Disks.  2x iSCSI NIC per node.

What I saw was a slowness in the cluster in bringing a VM’s Virtual Machine Configuration cluster resource online which had a large amount of disks configured.  They would time out as they passed their default pending and deadlock time-outs.

Firstly, when you online, refresh (or failover) a VM configuration, Hyper-V performs a sanity check to ensure the underlying components of the VM (network, storage, etc) are available.  This means scanning all the disks.
So, say I had a VM with 25 disks (in my case), the VM config took over 10 minutes to online!

Why?  Well working with Microsoft OEM Support, they asked me to try to tighten the TCPAckFrequency to 1(millisecond).  I say, OK I’ll try it.
This brought the online time from 10 minutes to 19 seconds!  Result…or maybe…

I needed to fully understand the issue, so out came WireShark in order run some Ethernet traces…

Let me explain what the actual issue is…

The problem is basically that iSCSI is a victim of Nagle’s Algorithm.  Optimization of TCP networks in terms of minimizing congestion due to TCP overhead.  iSCSI is essentially stung by send colalescing.

Windows 2003 onwards, the default TCP acknowledge time is 200ms.
This means that if a TCP segment (1462 bytes) is not full, it may need to wait up to 200ms before the data actually sent from the Windows host.

This is a problem when using iSCSI for two reasons
1) You need the fastest response time possible from storage for your application
2) iSCSI payloads can be very small, esp SCSI control OpCode (CDBs).

Now, the iSCSI CDB (control/query) OpCode commands involved in enumerating the disks during the online action have a tiny payload (10 bytes).

From looking at the ethernet trace, the cluster disk driver is performing SCSI(10) read commands (LUN read inquiry, read capacity, etc).  It does this sequentially, at least twice for each disk involved in the virtual machine.

With the default TCPAck time of 200ms, for each SCSI Read command issues by the cluster node, the payload of the command is 10bytes (on wire is 66bytes).  The SCSI Read command does not fill a TCP segment and so while the TCP payload is sent to the storage controller and the controller responds, the node waits to send the ACK until a segment on that NIC is filled or hits the max ACK time of 200ms.

So, let’s say I am the cluster disk driver and want to read LUN metadata as part of onlining a cluster disk…
…I send the command via the iSCSI Initiator, Storage Controller responds…wait….wait….wait…TcpAckFrequency trigger fires after a max of 200ms….only then is my ACK sent to storage….and the TCP transmission completes and the data is returned to the Cluster disk driver process.

This means for each SCSI command we attempt to send to storage (target and target LUN), we will typically end up waiting for the 200ms timer!  This is regardless of how busy SCSI data I/O is because this algorithm has Winsocket-based granularity.

In a Windows iSCSI cluster, this issue really comes to light because the cluster operates (controls/validates) on resources in the cluster in a sequential manner.
So, for say 65 LUNs, each LUN takes a significantly longer time to validate/control because the duration elongation due to the 200ms ACK timer happens multiple times per LUN.

Storage response times should be in the sub-10ms range, not 200ms 🙂

This is also why I did not see this issue in Fibre Channel environments.

So, while setting the TCP ACK frequency to hardcode to 1ms per-NIC helps, it may have adverse performance implications in terms of wire and Storage Port congestion.

But…for iSCSI Networks, this should not really be of concern because by best practice, you should have isolated iSCSI networks with minimal hops between host and storage.

 

The real fix I believe is by using the TCP_NODELAY option in SockOPTNS for the iSCSI Initator.  This by-passes the Nagle Algorithm completely for that process, so you dont need to even wait for that 1ms (seems a short time, but it is still a trigger time) – the SCSI command will fire immediately.

I have a design change request in for Microsoft to consider this as the way forward. Probably wont see it until Windows 8, but hey!

How to set the TCPAckFrequency on your iSCSI NICs

Regedit – backup your registry – before proceeding 🙂

Subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\<Interface GUID>
Entry: TcpAckFrequency
Value Type: REG_DWORD, number
Valid Range: 0-255
Set to 1.

Do this only for your iSCSI NICs, unless directed otherwise.

Hope this helps

James.