Windows Server Forum / Exchange Server / Design / January 2006
Hardware Recommendations
|
|
Thread rating:  |
Ryan Burrus - 10 Jan 2006 21:22 GMT I am trying to find some information as to the recommended hardware configuration for Exchange Server 2003. We only have about 150 users and all access it using Outlook 2003. I have read a few things from Microsoft that says the Exchange data files and the page file should be on separate drives. Also that the Exchange data files and the Windows system files should be on separate drives. Also that the Exchange transaction logs should be on a separate drive than the Exchange data files. This would equate to 4 drives which doesn't sound right to me. Does anyone know what the recommended scenario would be for an organization our size? Any advice would be much appreciated. Thanks!!
Jonathan Norris - 12 Jan 2006 16:21 GMT It is still best practice to seperate the logs, DB, and OS. Actually it would equate to more drives than 4 if you use RAID which is recommended.
Smaller environments sometimes don't always follow best practices since it does require a larger server and more hardware. I have seen people use a single RAID 5 for everything. However its not considered Best Practice, but it is still supported by Microsoft.
You will be taking a larger risk with DR because if you loose the Single RAID container then you loose everything.
If you do decide to go with less disks I highly recommend you do backups to tape for sure and do it at a daily minimum.
Since I am a consultant I always go with Best Practices and don't recommend anything less.
 Signature Jonathan No Warrenties Implied, Did you do a FULL backup today??????
> I am trying to find some information as to the recommended hardware > configuration for Exchange Server 2003. We only have about 150 users and all [quoted text clipped - 6 lines] > scenario would be for an organization our size? Any advice would be much > appreciated. Thanks!! Jonathan Norris - 12 Jan 2006 16:31 GMT Actually if i were to use two Disks for the OS (Raid 1) and two Disks for Transaction Logs (Raid 1) and three disks in RAID 5 for the Database .STM and .EDB Files it would be 7 Disks.
Also keep in mind by having everything on the same spindles you will have an I/O bottleneck.
OS/Page File is Read and Write intensive Transaction Logs are write intensive during normal operation and during recovery its read intensive. DB File .stm and .edb are both Read and Write Intensive.
 Signature Jonathan No Warrenties Implied, Did you do a FULL backup today??????
> I am trying to find some information as to the recommended hardware > configuration for Exchange Server 2003. We only have about 150 users and all [quoted text clipped - 6 lines] > scenario would be for an organization our size? Any advice would be much > appreciated. Thanks!! John Fullbright - 12 Jan 2006 19:02 GMT I wouldn't do that, "three disks in RAID 5 for the Database .STM and .EDB Files it would be 7 Disks.", that is.
Let's assume a 3:1 read/write ratio for tha deatabases. We'll also assume 10K RPM SCSI disks which are capable of ~85 20ms IOPS.
Where P is the performance of a single spindle and N' is the number of data spindles in the set:
RAID 5 Read performance = P*N' = 85*2 = 170 RAID 5 Write performance =P*N'/4 = 85*2/4 = 42.5
And we apply the read/write raitio which is required when read and write performance are asymmetrical. With a 3:1 ratio we are 75% read and 25% write:
Performance = 170*.75 + 42.5*.25 =127.5+10.625 =138.125 IOPS
If you were to buy bigger drives and go with just a mirror:
RAID 1 Read performance = P*N = 170 RAID 1 Write performance = P*N/2 = 85
Again we apply the read/write ratio:
170*.75 + 85*.25 = 127.5 + 21.3125 = 148.8125 IOPS
My recommendation - Three mirrors for a total of 6 spindles. This is about 8% better overall performance on the database array with 1 fewer spindles. The write performance is a 100% improvement.
Do the math.
John
> Actually if i were to use two Disks for the OS (Raid 1) and two Disks for > Transaction Logs (Raid 1) and three disks in RAID 5 for the Database .STM [quoted text clipped - 25 lines] >> scenario would be for an organization our size? Any advice would be much >> appreciated. Thanks!! Jonathan Norris - 12 Jan 2006 20:08 GMT I never use Raid 5, it was just an example. Typically I use RAID 10 for this reason. Then again I always use SAN. I can't remember using less at a client site.
The only reason I would recommend a RAID 5 is for more disk Redundancy.
Yes I am familiar with doing IOP calculations. Raid 5 with 3 disks would work fine for both Capacity and Performance with these assumptions
150 Mailboxes 200 MB Mailbox (which is large) 10 K RPM Disk 66% Read Weight .50 IOP Per User (Typically .25 for the average user) Microsoft uses 2.0 80 % User Concurrency.
You could use a Raid 1, but plan on having downtime if you loose a drive.
 Signature Jonathan No Warrenties Implied, Did you do a FULL backup today??????
> I wouldn't do that, "three disks in RAID 5 for the Database .STM and > .EDB Files it would be 7 Disks.", that is. [quoted text clipped - 60 lines] > >> scenario would be for an organization our size? Any advice would be much > >> appreciated. Thanks!! Ryan Burrus - 12 Jan 2006 20:12 GMT We currently have a server with 3 hard drives (RAID 5) and 1GB of RAM. We are beginning to notice some delays with people trying to access their mailboxes (usually when opening large attachtments).
> I never use Raid 5, it was just an example. Typically I use RAID 10 for this > reason. Then again I always use SAN. I can't remember using less at a [quoted text clipped - 78 lines] > > >> scenario would be for an organization our size? Any advice would be much > > >> appreciated. Thanks!! Jonathan Norris - 12 Jan 2006 21:02 GMT Do you have seperate Drives for the Transaction Logs and OS?
I would expect READ / Write Delays if you had them on the same Spindles/drives.
This is why it is not Best practice.
 Signature Jonathan No Warrenties Implied, Did you do a FULL backup today??????
> We currently have a server with 3 hard drives (RAID 5) and 1GB of RAM. We > are beginning to notice some delays with people trying to access their [quoted text clipped - 82 lines] > > > >> scenario would be for an organization our size? Any advice would be much > > > >> appreciated. Thanks!! John Fullbright - 12 Jan 2006 21:08 GMT Open perfmon and collect the physical disk - sec/transfer counter for your DB & Log drives. I bet they're not within the recommendations of the MS Whitepaper "Optimizing Storage Performance for Exchange 2003".
Slow disk is the number one cause of preceived performance problems in Exchange.
> We currently have a server with 3 hard drives (RAID 5) and 1GB of RAM. We > are beginning to notice some delays with people trying to access their [quoted text clipped - 98 lines] >> > >> much >> > >> appreciated. Thanks!! Asher_N - 12 Jan 2006 20:10 GMT Another one of my pet peeves. Similar to the old, 'the arrow will never reach the target' problem. While your number are sound, any decent RAID controller has on-board cache. writes are cached and delayed until read are satisfied.
So while the theorical problem points to fsaster writes with mirror, if you use the same drives and same RAID controler to do R1 or R5, you should get similar results. Also, 150 users is not much. I run 100 users on a single 5 drive partitioned as 2 logical drives (OS, Stores and logs). The performance is more than adequate.
> I wouldn't do that, "three disks in RAID 5 for the Database .STM and > .EDB Files it would be 7 Disks.", that is. [quoted text clipped - 60 lines] >>> recommended scenario would be for an organization our size? Any >>> advice would be much appreciated. Thanks!! John Fullbright - 12 Jan 2006 21:13 GMT Cache only goes so far. Beyond a base amount, adding more cache is a case of diminishing returns. If you do rely on cache, it needs to power protected to prevent corremption in the case of power failure. The main advantage of cache in RAID 5 is to remove 1 disk access from the 4 required for each write. This effectively changes the formula for write performance from P*N'/4 to P*N'/3. Still worse than a mirror.
> Another one of my pet peeves. Similar to the old, 'the arrow will never > reach the target' problem. While your number are sound, any decent RAID [quoted text clipped - 71 lines] >>>> recommended scenario would be for an organization our size? Any >>>> advice would be much appreciated. Thanks!! Jonathan Norris - 12 Jan 2006 21:20 GMT I am using a calculator developed by Exchange Rangers and have used it without any issues (both theoretically and in the real world) It uses the same calculations as John Fullbright had illustrated with more considerations.
I ran the information you provided, if you can provide your average mailbox size, User Concurrency, Disk Size, Disk RPMs, I will rerun it and give you the projections.
Interesting enough it doesn't even provide recommendations for RAID1 on the DB spindles.
Agreed Raid 1 does provide more performance, but I would expect to be given my pink slip if I recommended it as a customer solution for a Database due to the lack of redundancy and high risk we would assume.
Unless your users have a really high IOP Profile or really huge mailboxes I wouldn't expect RAID 5 to be a bottleneck (unless you have your OS and Transaction Logs sitting on the same Disks/spindles). Which isn't Best Practice or Recommended by anyone I know (With all due respect). You may also consider running Microsoft Best Practice Analyser to see if it gives you any recommendations. Perhaps you have write cache enabled on the drives?
I would also recommend you run performance monitor to see what the bottleneck is before you spend money.
Here is a link you may want to check out for Troubleshooting Performance with Exchange.
http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/e2k3perf.mspx
 Signature Jonathan No Warrenties Implied, Did you do a FULL backup today??????
> Another one of my pet peeves. Similar to the old, 'the arrow will never > reach the target' problem. While your number are sound, any decent RAID [quoted text clipped - 71 lines] > >>> recommended scenario would be for an organization our size? Any > >>> advice would be much appreciated. Thanks!! John Fullbright - 12 Jan 2006 21:47 GMT I don't understand what you're talking about when you say " but I would expect to be given my pink slip if I recommended it as a customer solution for a Database due to the lack of redundancy and high risk we would assume."
RAID 1 is 100% redundant, it's mirroring..
The Microsoft Whitepaper "Disk Subsystem Performance Analysis for Windows" gives a set of criteria where it is appropriate to use RAID 5. http://download.microsoft.com/download/e/b/a/eba1050f-a31d-436b-9281-92cdfeae4b4 5/subsys_perf.doc#_Toc72126973 Quoting from the paper:
"RAID-5 might be the right choice for a write-heavy workload if the following is true:
· The workload consists of large requests in comparison to the array's stripe unit size (that is, they constitute several complete stripes of data on average), or the controller has a write-back cache, allowing writes to be delayed and coalesced into full stripes, and
· The workload is largely sequential (in terms of LBNs), and
· The array controller can detect and optimize for full stripe writes, and
· The number of spindles is reasonable, as described in "Rules of Thumb" later in this paper, and
· Cost is a serious issue"
Exchange databases are a random workload that uses a 4K (small requests) IO size. A three drive RAID 5 set uses more spindes at higher cost and provides lower performance than a mirror. I don't see where any of the criteria are met for a RAID 5 recommendation.
The same paper discusses the characteristics of RAID 1/0+1/10:
" · Performance: For write requests, both parts of a mirror must be updated, reducing the available write throughput by >50% for 2-way mirrors, >66% for 3-way mirrors, and so on. On the other hand, read performance improves as the number of mirrors in each set increases. · Reliability: For an M-way mirrored set, all M disks must be lost before data is lost. In the common case of a simple 2-way mirror, MTBF is now increased because it is the longer of two semi-independent disk lifetimes. In most cases, the mirror can be restored quickly (for example, by replacing the disk by hand or using a hot spare), further reducing the window of vulnerability. For non-trivial mirrored striped arrays, most multi-disk failures are survivable.
· Availability: If connectivity is lost to one disk, or even M-1 disks of every M-mirrored set in a striped array, work can still proceed, albeit at a different level of throughput; failure of a mirror spindle decreases read performance and increases write performance for that set. When disconnected disks become reachable again, they can be synchronized back to the corresponding active disks, thereby restoring the original array characteristics."
Both a RAID 5 set with 3 disks and a Mirror can withstand the failure of a single drive. I'm not seeing a redundancy problem that would result in a pink slip here. I do see a potential perfrormance problem with RAID 5 that could however result in that pink slip.
>I am using a calculator developed by Exchange Rangers and have used it > without any issues (both theoretically and in the real world) It uses the [quoted text clipped - 111 lines] >> >>> recommended scenario would be for an organization our size? Any >> >>> advice would be much appreciated. Thanks!! Jonathan Norris - 13 Jan 2006 17:07 GMT I got this from http://www.microsoft.com/technet/prodtechnol/exchange/guides/StoragePerformance/ 1c471676-2312-4ffe-adf2-15a9cfd529c4.mspx
The RAID solution you use should be based on the cost and performance trade-offs that are appropriate for your company. As a result, in many cases, more than one type of RAID solution may be recommended for a particular data storage requirement. General recommendations are as follows:
• Use Raid-1+0 for the transaction log file volumes, database volumes, and SMTP queues • Use Raid-1 for the transaction log file volumes, SMTP queues, and MTA queues.
• In general, Raid-5 does not provide the best trade-off between reliability/availability and performance. • Raid-0 is never recommended.
The only concern I have with running RAID 1 is if you do loose a disk you will have to down the server to rebuild it unless the RAID controller supports hot swap, then again if you loose the disk in the Transaction Logs or OS your just as dead in the water.
Interesting enough they don't come out and say RAID-1 is suitable for the DBs in any part of the article.
Perhaps MS should write another article for the SMB segment. Since Raid 0+1 isn't a cost effective solution for 150 users its open for debate.
I am not disagreeing with you that RAID 1 provides more performance than RAID 5.
However running the numbers... RAID 5 will support the number of users and the IOPS required to support this many users. But RAID 1 would use less disks and provide more performance.
So your solution would be better in this case. I spend so little time in the SMB segment that I never implement this scenerio. I stand corrected:)
Here is another article for the person who posted the question,
http://www.microsoft.com/technet/prodtechnol/exchange/guides/E2k3HighAvGuide/c18 04c14-c099-4207-b6b9-de5bda972b76.mspx
 Signature Jonathan No Warrenties Implied, Did you do a FULL backup today??????
> I don't understand what you're talking about when you say " but I would > expect to be given my pink slip if I recommended it as a customer solution [quoted text clipped - 175 lines] > >> >>> recommended scenario would be for an organization our size? Any > >> >>> advice would be much appreciated. Thanks!! Al Mulnick - 19 Jan 2006 02:38 GMT "Interesting enough they don't come out and say RAID-1 is suitable for the DBs in any part of the article.
Perhaps MS should write another article for the SMB segment. Since Raid 0+1 isn't a cost effective solution for 150 users its open for debate.
I am not disagreeing with you that RAID 1 provides more performance than RAID 5."
Jonathan, I see that you came to the conclusion that RAID 1 was pretty good in this case. Since you're on the learning track, can I suggest that you rethink the RAID concepts as you've come to know them? Also, rethink the way that controllers deal with RAID. Cache is fine but keep in mind that Exchange is cache-unfriendly.
Also, when you read those documents that folks write, keep in mind they write them for the general audience. Although I've personally seen some of the documents for the SMB segment (and they're pretty good if I say so myself), don't expect Microsoft to publish papers that say that RAID 1 is recommended for Exchange. Will it work? Yep. Is it as good as RAID 5? For some types of operations, yes. For others, no. If you have read heavy applications, you'll get better performance from a RAID 5 than you will from RAID 1. Write performance would be less. Both would have the same type of fault tolerance. I throw this out there hopefully to be helpful. Besides, you wouldn't have to count on tools from Rangers that way ;)
Johnathan (?), I'm surprised to see in your calculation a number of 2 for the number of spindles in the data set. "RAID 5 Read performance = P*N' = 85*2 = 170" Can you expand on that? I see two, but it's a raid5 set meaning you have three possible targets. Are you saying that only two are valid for both read and writes? Traditionally, I would have expected 3 for the write and 2 for the read to account for the parity penalty. Curious. I realize you're the guy to know, but I'm interested in knowing why that number is used vs. another.
Cheers, Al
>I got this from > http://www.microsoft.com/technet/prodtechnol/exchange/guides/StoragePerformance/ 1c471676-2312-4ffe-adf2-15a9cfd529c4.mspx [quoted text clipped - 260 lines] >> >> >>> recommended scenario would be for an organization our size? Any >> >> >>> advice would be much appreciated. Thanks!! John Fullbright - 19 Jan 2006 19:54 GMT Al Mulnick. Wow, another blast from the past. Still hangn' with Jim at Bubba's Bait & Tackle?
Anyway:
P = Performance of a single spindle in IOPS N = Number of spindles in the RAID set. N' = Number of data spindles in the RAID set (N - parity spindles)
RAID 5 performance calculations:
Write performance = P*N'/4 Read performance = P*N'
(depending on the caching architecture implemented by the controller and the amount of write cache, in some cases write performance can be expressed P*N'/3)
RAID 1/10/0+1 performance caluclations:
Write performance = P*N/2 Read performance = P*N
RAID 4/RAID DP (as implemented by Network Appliance) performance calculations
Write performance = P*N' Read performance = P*N'
For a 3 drive RAID 5 set, N' would be 2. The set size would be three and the other spindle is parity.
> "Interesting enough they don't come out and say RAID-1 is suitable for the > DBs in any part of the article. [quoted text clipped - 309 lines] >>> >> >>> recommended scenario would be for an organization our size? Any >>> >> >>> advice would be much appreciated. Thanks!! Al Mulnick - 22 Jan 2006 00:29 GMT Ah. That'd be it. I think it was a mistake on my part to use a different caching/controller thought. Must be using the wrong devices :)
Al
> Al Mulnick. Wow, another blast from the past. Still hangn' with Jim at > Bubba's Bait & Tackle? [quoted text clipped - 352 lines] >>>> >> >>> recommended scenario would be for an organization our size? Any >>>> >> >>> advice would be much appreciated. Thanks!! John Fullbright - 22 Jan 2006 07:06 GMT Some controllers try to coalesce writes, and advertise less of a penalty. This is where you have to be careful. For sequential writes, coalescing writes works really well. For random workloads, it usually amounts to so much FUD. Think about it. Envision your 100G exchange database as a 1000 page novel. Coalescing writes is like trying to write an enire sentence instead of a word at a time. If you are sequentially writing the next chapter, writing a sentence at a time works well. A random workload writes one word on line seven of page one, and the next on line three of page four hundred and eightyseven. The next is, well random; you get the idea. There is no real opportunity to coalesce writes so the effectiveness of this feature for random write workloads is very low. Unless, that is, you never overwrite. If you mark the word on page one as dirty, then update a pointer in a virtualized structure that represents a sentence to point to the next available open spot, and you do this for every write, then you essentially make a random workload look like a sequential workload to the disk controller. There is only one storage vendor that does this.
> Ah. That'd be it. I think it was a mistake on my part to use a different > caching/controller thought. Must be using the wrong devices :) [quoted text clipped - 368 lines] >>>>> >> >>> Any >>>>> >> >>> advice would be much appreciated. Thanks!! Al Mulnick - 22 Jan 2006 16:43 GMT Right. I was thinking of RAID in terms of using available spindles at any given time. The more I think about it, RAID 5 would always have the penalty because it would always have to update the parity as part of the cycle. But I was looking at it from the point of view that a controller doesn't have to only write to the two and then leave the one dedicated to parity. Nor would it want to in case that's the one that's gone. However, I was thinking of the controllers that keep track of the least busy disk and write to it vs. thrashing the others. I've seen some strange configurations ;)
Still, I appreciate the explanation. Thanks John.
> Some controllers try to coalesce writes, and advertise less of a penalty. > This is where you have to be careful. For sequential writes, coalescing [quoted text clipped - 393 lines] >>>>>> >> >>> Any >>>>>> >> >>> advice would be much appreciated. Thanks!! John Fullbright - 22 Jan 2006 17:22 GMT " However, I was thinking of the controllers that keep track of the least busy disk and write to it vs. thrashing the others."
Again, this stratgy could work for sequential writes, but won't work for a random write workload unless you never overwrite.
If you're looking for a disk subsystem that is capable of beating the RAID write penalty, then the features you want to see are:
1. Never overwrite, always write to new space. 2. Coalesce writes across a stripe 3. Calculate parity once for a stripe in RAM.
You'll see a few vendors doing #2 or #3, but only one vendor does all three.
> Right. I was thinking of RAID in terms of using available spindles at any > given time. The more I think about it, RAID 5 would always have the [quoted text clipped - 408 lines] >>>>>>> >> >>> Any >>>>>>> >> >>> advice would be much appreciated. Thanks!! John Fullbright - 22 Jan 2006 23:59 GMT I'll be in your neck of the woods in early February. Give me a call or drop me a line.
John
> Right. I was thinking of RAID in terms of using available spindles at any > given time. The more I think about it, RAID 5 would always have the [quoted text clipped - 408 lines] >>>>>>> >> >>> Any >>>>>>> >> >>> advice would be much appreciated. Thanks!! Al Mulnick - 30 Jan 2006 01:54 GMT I'm guessing you don't check your home email very often? :)
> I'll be in your neck of the woods in early February. Give me a call or > drop me a line. [quoted text clipped - 420 lines] >>>>>>>> >> >>> Any >>>>>>>> >> >>> advice would be much appreciated. Thanks!! John Fullbright - 30 Jan 2006 03:13 GMT Hmmm. I actually check once every week or so. You can ping me at the comcursed address if you are having difficulty.
> I'm guessing you don't check your home email very often? :) > [quoted text clipped - 427 lines] >>>>>>>>> >> >>> Any >>>>>>>>> >> >>> advice would be much appreciated. Thanks!! Al Mulnick - 31 Jan 2006 03:12 GMT I have 3x now. I'm guessing there's something else. How about the other direction? You can send to this one once you remove the extras.
> Hmmm. I actually check once every week or so. You can ping me at the > comcursed address if you are having difficulty. [quoted text clipped - 437 lines] >>>>>>>>>> >> >>> size? Any >>>>>>>>>> >> >>> advice would be much appreciated. Thanks!! John Fullbright - 22 Jan 2006 22:04 GMT "I am using a calculator developed by Exchange Rangers and have used it without any issues (both theoretically and in the real world) It uses the same calculations as John Fullbright had illustrated with more considerations."
The "Exchange Rangers", all 68 of them collaborating to produce one calculator. The mental image of Adrian, Scott, Lee, Ace, Elias, etc. all in one room is enough to make you go blind.
Wasn't that Greg D.'s *other little band of merry men, besides the Blackbelts?
Makes you wonder; what came first, the chicken or the egg?
>I am using a calculator developed by Exchange Rangers and have used it > without any issues (both theoretically and in the real world) It uses the [quoted text clipped - 111 lines] >> >>> recommended scenario would be for an organization our size? Any >> >>> advice would be much appreciated. Thanks!!
|
|
|