The Ultimate Folding Farm Guide

[SPD010273]

Ultimate Folding Farm guide

This is meant as a guide to those looking to setup a cheap, yet effective folding farm that will provide maximum returns while not breaking the bank.

This guide is tailored to SMP folding, but many aspects still apply to regular FAH client setups.

Determining what Hardware to get

The first step to building a farm is to determine what combination of hardware will give you the best bang for your hard earned buck. It really helps to assemble a table like the following:



Essentially you use this to determine which processor offers the most “Mhz for the buck”.
Keep in mind, however, that Allendale processors, though cheap, have either 1MB to 2MB shared L2 Cache compared to their E6xxx C2D brothers with 4MB shared L2. This comes out to make a difference PPD wise. This should be considered when purchasing a farm. If the funds are available, pay the extra $20-40 for more cache.

Next, search for a motherboard and some RAM that's reliable. The motherboard should have onboard LAN and video, and provide any other functions you need (OCing, second LAN port). As far as RAM goes, a general rule of thumb is to get 512MB per core. WU's aren't that big, and RAM is pretty cheap now-a-days. Make sure you have at least 2 identical sticks to take advantage of the bandwidth provided by dual-channel. 256MB/core is acceptable only if your node has a hard drive. Remember, diskless nodes need to have enough RAM to store the OS, F@H client, cores, WU, and WU data locally.

Now that you’ve determined the best combination of CPU/motherboard/memory for your budget, you need to determine what type of folding farm you’ll have:

Diskless – each node does not have a hard drive, but instead downloads the OS from a host and loads it into memory along with the F@H client

Pros- Lowest power consumption, Lower price, Localization of client files makes it easier to update FAH versions.
Cons- Difficult to setup, one computer must act as the host.

CDROM only – each node has no hard drive, but 1 CD-ROM drive. Each node needs to be loaded with Knopptix or Overclockix, which have built in folding support.

Pros- Lower power consumption, no host computer needed.
Cons- Slightly difficult and tedious to set up. If power is lost, you must reload each OS.

Hard drive – Easiest setup, all you need is a hard drive for each node, and 1 CD-ROM to load up the OS.

Pros– Very easy to set up.
Cons– Higher relative power consumption (+25W/node), additional heat, additional noise, additional point of failure.

Location, location, location!

The second step is to determine how much power the farm will consume, and likewise, how much heat it will produce.

If your farm is going to draw more than 15A @ 120VAC (Most PSU's are rated to draw a maximum of 8-12 A @ 120VAC, though they tend to draw less depending on load and wattage), you need to find a wall receptacle in your house/dorm/apartment/whatever that can handle the power draw. Most standard wall receptacles are rated at 15 Amps, and exceeding this is very dangerous as it can lead to an electrical fire. DO NOT think you can daisy-chain surge protectors and get away with it, this is dangerous and WILL overload a standard wall outlet.

Since your folders are *hopefully* going to run 24/7 for team 11314, you need to find a way to get rid of all that darn heat that we overclockers despise so much. The complexity of your cooling system is up to you, you can do anything from increasing the distance between boards (to allow the CPU heatsink more room for airflow), to a large water cooling loop.

Housing 101

The next step would to choose a standard motherboard size (mATX/ATX/whatever), and design a housing for it. The most common design would be a rack. Racks offer the highest density, but put all your folders in one place, concentrating the demand for power in one spot, so unless you can’t find a suitable power source, your next choice are several small racks.

Regular ATX/mATX cases work, but take up additional space, increase costs. They are optional, depending on budget constraints.

Instead of constructing a rack, you can use anything from a bookshelf to steel shelving to make a place to put all of your folders.

Attached to this post are some F@H Farms to give you an idea of different rack layouts

Setup Time

Ok, now you have your power, housing, hardware, and cooling taken care of, time to get your nodes set up and working. Depending on what hardware setup you chose (diskless/CDROM/HDD), you’ll have to set it up accordingly. I’ll write a simple guide here, but link to several in-depth guides that can walk you through more complicated aspects of setup.

Diskless:
Here's the low-down. Since most motherboards come with the ability to PXE boot (named LAN boot ROM under BIOS), that's amost a non-factor. You need one computer to act as a server. This computer will be your DHCP server and FTP server (this means if you have a router, turn off its DHCP function). PXE booting works by assigning an IP address do a node, then the node downloads a boot image through TFTP, then boots that image that will download additional files (and a folding client). The difficulty comes in setting up the server. Most guides to diskless clusters are based around Linux, but there are a few Windows flavors (keep in mind this is for the server, the nodes boot a tiny linux kernel). I'm currently looking for more links for this section.

Helpful Diskless Folding site:
http://reilly.homeip.net/folding/

CDROM:

Each node is equipped with a CD-ROM drive. Each node is booted with an Overclockix CD, which loads the OS and client into CD. Overclockix can also be installed to a hard drive or run over a network.
Overclockix
http://overclockix.octeams.com/

Overclockix – Folding farm setup guide:
http://overclockix.octeams.com/How_t...lding_Farm.htm

Hard Drive:

Setup is pretty easy. Download the latest version of Ubuntu linux 64-bit for your SMP setup:
http://www.ubuntu.com/getubuntu/download

Burn this image to a disk and pop it in the CD drive. Setup is pretty straight forward, only a few options to fool with.

NOTE: Ubuntu does not play well with RAID arrays 'right out of the box', and you’ll have to configure either fakeraid or dmraid with your setup.

Next, obtain the Linux SMP client from here:
http://folding.stanford.edu/release/FAH_SMP_Linux.tgz
Detailed SMP install guide for 64-bit Linux:
http://forums.extremeoverclocking.co...d.php?t=245511

Suggested Setups:

Penny Pincher:

$150 shipped for some decent dual core power.

CPU: AMD Athlon64 X2 “Windsor” 3600+ - $61
http://www.newegg.com/Product/Produc...82E16819103760

Mobo: MSI AM2 K9N6SGM-V - $50
http://www.newegg.com/Product/Produc...82E16813130068

RAM: A-DATA 1GB (2x512MB) DDR2-800 - $46
http://www.newegg.com/Product/Produc...82E16820211064

PSU: Logisys 480W (peak), ~330-350W continuous - $16
http://www.newegg.com/Product/Produc...82E16817170012

Budget:
This hardware is capable of 3.33Ghz, and for $250 shipped, how can you say no?!

CPU: Intel C2D “Allendale” E4400 - $126
http://www.newegg.com/Product/Produc...82E16819116037

Mobo: ASRock ConRoe1333-D667 - $53
http://www.newegg.com/Product/Produc...82E16813157113
Good board, at 1333FSB, PCIe and SATA start to go out of spec, but this thing can OC pretty well for its price.

RAM: Corsair ValueSelect DDR2-667 1GB (2x512) - $40
http://www.newegg.com/Product/Produc...82E16820145568

PSU: Logisys 480W (peak), ~330-350W continuous - $16
http://www.newegg.com/Product/Produc...82E16817170012
Cheap, but well rated. Seems to go above and beyond the call of duty.

Powerhouse:
Very capable hardware, but still manages to be relatively cheap.
$500 shipped per node.

CPU: Intel C2Q “Kentsfield” Q6600 - $290
http://www.newegg.com/Product/Produc...82E16819115017

Mobo: Gigabyte GA-P31-DS3L - $83
http://www.newegg.com/Product/Produc...82E16813128062

RAM: A-DATA 1GB (2x512MB) DDR2-800 - $46
http://www.newegg.com/Product/Produc...82E16820211064

PSU: Coolmax ESP12V 500W - $60
http://www.newegg.com/Product/Produc...82E16817159040
Very good PSU, very good reviews.

Tying it all together - Additional Hardware

Now that you have everything in the right place, it's time to lay the infrastructure and obtain additional hardware that will increase the farm's uptime, life, and reliability.

The following hardware is needed to run the farm:

X+2 port switch – You need a Ethernet hub/switch/router to give each node an internet connection. This applies to CDROM and Hard drive farms. Diskless farms have a host acting as DHCP and obtain WU’s through that server. You don’t need to be overly sophisticated with your network. For small farms, you could buy additional network cards for a host computer and share the host’s internet connection with the nodes. For larger farms, a regular, reliable 10/100 switch will suffice.

Heavy duty surge protector– Regular extension cords won’t work here, you need a surge protector to distribute power and protect your equipment from damage. Again, be careful not to overload wall receptacles. Make sure this has a heavy gauge cord.

The following hardware is recommended for the farm:

Router – You can manually configure a router to allow traffic through certain ports, protecting your nodes from interfering network traffic. These can also be used to hide the farm from the rest of the LAN.

UPS – This is an upgrade from your run-of-the-mill surge protector. This can protect the nodes from brownouts and power fluctuations (see note). It also cleans up the AC line. Again, nothing fancy, just pick one out that can stand up to the load and is reliable. a UPS is highly recommended as brown-outs can cause corruption. Anything from a huge UPS that can run your farm for 30min-1hr, to a simple line conditioner with a short back up time will do. This is meant to give you time to properly shut down FAH cores so you don't have to restart from a fresh WU (we all know how finicky the SMP F@H core is).

NOTE: Low end UPS's contain lead acid batteries which have a short lifespan and have the capability to damage other equipment when it fails. Lower end UPS's aren't as adept at protecting against brownouts either, so it's highly suggested that if you are going to purchase a UPS, get a name brand UPS and save yourself some headaches later down the road.

Monitoring software – Highly recommended, use this to monitor temps and monitor progress. Nothing sucks worse than taking a production hit and finding out some of your nodes are overheating/locked up/or running into other problems.

To Overclock, or not to Overclock...

As many here would say, fold/overclock it until it melts. That is a good way to look at your nodes, but remember, these computers aren't your personal gaming rigs, these crunch numbers for F@H. Accuracy is of the up most importance. There's no point if you're folding data that Stanford can't use because it isn't accurate. If you are going to overclock your nodes, make sure you do the following after you've determined your max OC:

-Burn in each node 24-36 hrs with Orthos (dual core)
-Burn in RAM with memtest86/memtest86+ for 12-36hrs directly after CPU burn in.
-Run a full system benchmark or looping burn in program, such as heavyload, that stresses CPU, RAM, and other components simultaneously for 12-24hrs.
-Unless your FSB is set to a stock like speed (200Mhz, 266Mhz, 333Mhz), drop it 5Mhz to ensure stability (Do this if FSB speed is 206-265Mhz, 272-332Mhz, or 339Mhz+)

Weigh carefully the pro's and con's, considering the extra power draw, extra heat, and increased wear on components.

Keeping an Eye on Your Rigs

No matter what setup you're running, you're probably going to want to monitor your node's progress, as well as maintain each system, update software, and other administrative tasks. Luckily for you, there's tons of programs in every flavor of OS to monitor both the FAH client and allow for remote desktop.

Below is a list, by OS, Remote Desktop clients and FAH client links:

Linux - 32bit
Cspace (Remote Desktop), scroll down for source
http://cspace.in/
FAHMON (source)
http://fahmon.silent-blade.org/uploa....2.src.tar.bz2

Linux - 64bit
Still looking...

Windows - 32bit
Cspace (Remote Desktop)
http://cspace.in/downloads/CSpaceSetup126.exe
FAHMON
http://fahmon.silent-blade.org/uploa...FahMon_222.zip

Windows - 64bit
Still looking...

Additional Monitoring programs are linked in the Difinitive FAH thread...
http://forums.extremeoverclocking.co...d.php?t=262145

Links:

Diskless FAH Farm setup guide:
http://www.extremeoverclocking.com/a...ss_Farm_1.html

Generalized FAH farm how-to:
http://www.talix.com/Folding/Folding_Farm.html

Compute Cluster setup – good ideas for networking and setup:
http://www.mini-itx.com/projects/cluster/?p

PSU calculator – Get a general idea of the wattage you’ll need:
http://www.extreme.outervision.com/p...ulatorlite.jsp

FAH Info - Estimate the average PPD/Ghz for a given core
http://www.fahinfo.org/index.php?news=true

Please feel free to PM me with any suggestions or helpful links.

[901-Memphis]

Great stuff, i say sticky.

[frozndevl]

Great content, but Id say reformat your table of of hardware to narrow it down. It makes it hard to read, just have some of the column headings 2 rows high... much easier on the eyes.

[NM_JJ]

Great thread. For the table, MHz/$ is a good easy statistic but I would caution that the CPUs with 4MB of shared L2 cache will put out more ppd on average with SMP than those with 2MB of shared L2 cache at the same clock speed. Maybe an average ppd/$ taken from fahinfo? I know this would take more work. A quick glance at the list looks like the q6600 may be the best bang for the buck. Plus it may OC higher. If I had $2K for a farm I would definitely go 4 q6600s versus the other choices. Should have lower overall power too.

[SPD010273]

Thanks for the input guys, I'm currently looking for public domain pics of folding farms to give people various ideas of how to set nodes up.

I was a little iffy on using FAHinfo's PPD figures because all I can find is listings for Core's only, no listing by speed, but I linked it until I can figure how to get it represented on that graph. Thanks NM JJ for the link.

[HotRod]

Originally Posted by SPD010273

Thanks for the input guys, I'm currently looking for public domain pics of folding farms to give people various ideas of how to set nodes up.

Here are a few of my old farm pics.

http://forums.extremeoverclocking.co...1&d=1106500647
http://forums.extremeoverclocking.co...2&d=1106500647

[SPD010273]

Perfect, just what I was looking for

[MrObvious]

SPD: That one image is taking up too much width. Can you crop it down?

[SPD010273]

Ok, updated some of the text.
Gonna redo the table to include PPD/Ghz in the next couple days.
MrObvious, you were referring to the main table, right?

[901-Memphis]

PPD on the chart would be huge!

Can't wait.

And make sure you fix the resolution of the image so its easy to view please!

I think you already addressed this though.

Loving it so far

[HotRod]

Lets keep this thread clean and only post if you need to have anything changed or added.

[Taichan]

Umm i'd just like to add some UPS can't protect against brownouts, and i've actually had a few cheap UPS die from brown outs but they are good to protect against power outages

[SPD010273]

thanks for pointing that out. I've been reading up on UPS's and generally your only option is name brand with the ability to hold up the the load of your farm.

Another note is to stay away from UPS's that use lead acid batteries (generally these are off-brand ups's with lower power ratings), and the batteries tend to last 6months to 5 years, and when they die, you'll know it. (boom/smoke/big suprise on the next power outage/brownout).

I'd highly suggest to get some way to protect against short term outages or brownouts, especially if you are going to run SMP cores, as they don't support checkpointing, and and disruption can lead to the current WU being dumped and a new one getting started.

I'm currently at a home comp with a whacky keyboard, I'll reword the guide and add some good UPS links when i get back to my appt.

Until then this is just a heads up, and FYI i'll have pics up of the small farm I'm starting

EDIT: Okay I did a ton of tinkering over the last semester with PXE booting and diskless node and I'll put more detail into the appropriate sections as well as generally update the sticky.

[runlinux]

I have a diskless node up and running right now, and its soo easy to setup and monitor.

following the setup on this link : http://reilly.homeip.net/folding/ i was able to get my linux machine acting as a DHCP and TFTP server. because its folding 2x linux smp anyways, i figured i would set it up. i turned off DHCP on my home router as to not screw things up here at home. wireless still works as the wireless router still acts as the AP, so that's all good. by using this guide, i added one q6600 diskless into the fold, and im gunna have to install ubuntu on a hard drive as my other boards NIC isnt supported - darn nvidia chipset, lol.

all in all, its really easy to get a diskless farm going using that guide. he hit the nail on the head! i havent been able to get the windows DHCP and TFTP serving working, but i did not tinker too much with it. for most of us, i can see the windows setup witll working. it still boots the diskless clients into the linux fah client, but its just a lot easier to use assuming it works as simple as he states it.

[SPD010273]

Good to hear you had some success. I was able to get a windows based TFTP server up and computers could boot from it. I'm planning on updating this thread with newer info and an updated chart after I finish finals next week.

There's some good folding hardware that's poped up on the radar recently, namely Intel's E1200 DC Celeron for $50 at newegg. These are overclockable to around 3.4 Ghz, but I'm still looking for some MFLOPS figures for this processor so it'll be easier to gauge it's folding performance. I also have a growing farm in my dorm room XD. Currently at about 10 computers, but I'm having major problems completing WU's on them.

Anyways, update coming soon, sorry mods.

[Leganfuh]

This is an old photo, 4 more Vostro's have been added.

(1) Dell XPS 720 Quad 2.4, GeForce 8800GTX 1-SMP 1-GPU

(20) Dell Vostro 400's Quad 2.4, 14 with GeForce 8800GT 40-SMP 14-GPU

(1) Dell Dimension 8300 P4 3.2 1-FAH504 Console (Retired)

(1) Dell Optiplex GX270 P4 3.2 1-FAH504 Console (Retired)

Mike
Leganfuh aka: The Commander
Team XCPUS

[Mike Johnson]

This is just what I neeed as putting together a farm seemed daunting but you've definately inspired me to go on.

One question though. Do the motherboards need onboard VGA in order to set up before connecting to a server?

[SPD010273]

onboard is desired because it will use less power. If you have a board w/o onboard you might want to look into ebaying a low end pci graphics card because in my experience most boards will not boot without a display adapter connected. good luck with your build!

[gamer11200]

Just a heads up about the original post. The suggested setups should be updated. The prices for the components have changed (Q6600 listed at $290 there, but is now $190)

[vistageek92]

The CPU table was a help....i have just got a q6600...about to start OC'ing!