Having made a spreadsheet of a lot of the used servers/parts on the market and taking into account power draw, there is a thin selection of servers where it is actually cost effective. (I think I made a post about this in the past, but I did not post my spreadsheet).
The first major problem: parts. Some of the cpu's may only be $15; however, to find a motherboard with features you would like, could end up costing more than if you just built a new system.
The next problem is that you are competing against people looking for replacement parts for their servers. If something hits the market fairly cheap, it will not stay on it for long.
Final problem, heat, power consumption, and noise. Tech has come a long way, and some of those nicely priced cpu's will add a lot to your electric bill over the course of running. They were meant to be used in machine rooms. Depending on the case, they are going to be super loud. 1-2u fans have to pack a lot of rpm, and they sound horrible.
The Pro's, if you are diligent and do your homework, you can end up with quite a steal! I put together a few used servers for a buddy of mine, and we got really lucky with what we purchased. The performance to dollar ratio was staggering, and we fell into a nice generation of intel cpu's that were pretty efficient. You have to know what you are doing though!!!! Do your homework if you plan on going this route. Excel will be you friend.
---------------------------------------------------------------------------------------------------------
If you are just looking to build a computer in a server chassis and rack mount them, this is fairly common and easy. The racks may be the most expensive part of the build; however, you can just buy stand-alone rails and make your own cabinet. Keep in mind, there are some fairly nice expandable ones by startech on the market--something to consider. Ebay has a nice selection of open u-cabinets that are not too pricey. I try to stay away from 1-2u chassis for the previously mentioned reasons. 3u and up, you can use Noctua radiators without a fuss (measure your chassis to be extra sure: they are not all the same--no matter what the dimensions say on the box). I would also switch out the fans for noctua ones if in the budget. I found some of the stock fans were not too horrible, but I usually budget for quieter fans when I plan a build anyway. There are some decent consumer-ish server chassis on the market now that accommodate a wide range of motherboard layouts. It all depends on what you want and how much space you have.
Cheers!