[Discuss] I need a recommendation for a Linux-compatible high-end PC builder

[Alan W. Irwin]

On 2018-01-11 21:20-0800 pw wrote:

> Well, as far as OpenMP + MPICH clusters go, I manage one used for parallel 
> modelling at work.
>
> It's a 2 server cluster with 48 cores in one node and 64 in the second.
> Supermicro 2U, similar to the ones shown on this link:
>
> http://www.supermicro.com/products/nfo/Xeon_Phi.cfm
>
> University of Texas has a really big system called 'Stampede' based on these.
>
> I recommend slurm and sview for monitoring and job management although,
> to date, we haven't needed to partition for simultaneous jobs. So most
> jobs are just run on the command line using  mpiexec in a shell script.
>
> University of Texas is using slurm as far as I know.
>
> The network was originally gigabit but to really make shared disk space
> and network passing move infiniband is the way to go between nodes.
>
> We just upgraded the interconnect between the two nodes
> to 40GBe because the gigabit network actually made one model slow down.
>
> Personally, I would have preferred 56GBe because the connection can then
> run at the full PCI bus rate of the two nodes without having to bond any
> connections. NFS doesn't do well if one of the links in a bond goes
> down. But the 40Gbe was affordable. There are also 3 more PCIe slots in
> each server, so virtual nodes could be used by partitioning the systems.
> The infiniband switch we are using has 18 ports, so lots of room.
>
> Anyway, long story short, wanting to run with large data
> blocks over the network gigabit network wasn't fast enough for our
> purposes.

Hi Peter:

Thanks for all that useful clustering information.

I have seen quite good-looking scaling results for small numbers of
host computers organized into a cluster (12 computers were ~10 times
faster than one) reported at phoronix for 100 base T (not even 1000
base T) for a number of different tests.  Those good results
completely surprised Michael Larabel (the Phoronix principal tester,
editor, etc.) since he was fully aware of the bad connectivity
bandwidth for his cluster, and how everyone always cautions about
that.  Nevertheless, he got those near-linear results. Do you think
that near-linearity would hold if you are dealing with say 5 Banana Pi
M3's with 8 cores each?  In other words, because the connection speed
within the M3 is so high compared to the 1000 base T internet between
the M3's, would you predict a near-linear increase in benchmark
numbers as each of the 5 Banana Pi M3's was added in to create a
40-core cluster?

I typically run my build and test task using the
following shell command:

make -j<NNN> test_everything

where NNN is the number of parallel tasks that can be done, and where
I have configured (with CMake) the test_everything target to build
~1000 independent tests and those test prerequisites with GNU make
keeping track of all dependencies (test prerequisites) and building
them in the needed order.  And because the dependencies are not severe
(most builds and tests independent of each other so can safely be run in
parallel), I would expect on such a system that the effective speed of
the command would increase linearly with NNN until it saturated near
NNN = the number of cores on this hypothetical single computer system.

Now, what exact syntax would I use to run that same command on the
above 40-core cluster?  From what you said above, would it be

mpiexec make -j<NNN> test_everything

And to paraphrase the prior question, would I expect the speed to be
proportional to NNN until it suddenly saturated near NNN = 40?

Excuse basic cluster questions like this, but I have always
systematically ignored the cluster literature before because I never
thought it would be relevant to me....  But it suddenly becomes
relevant in a revolutionary way when a 40-core cluster (with each core
having good although not outstanding quality and outstanding
connectivity within each group of 8 and good 1000 base T connectivity
between each group of 8) is within reach for ~$500.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________