eng.kulanov.org.ua

Cluster Computing Software

Googling not so long ago I’ve found some interesting (facts) information about clusters software distributions. Of course it’s not a secret, you can just look at wikipedia… Additional documentation can be found at http://www.clustergate.ru/CG_Clustering/:

• MOSIX – Scalable Cluster Computing for Linux. MOSIX is a management system that allows a Linux cluster or a Grid of clusters to perform like a single computer with multiple processors. It is particularly suitable to run intensive computing and applications with moderate amounts of I/O.
  In a MOSIX based system there is no need to modify or to link applications with any library, or even to assign processes to different nodes – it is all done automatically, just fork and forget, like in an SMP.
• OpenSCE OpenSCE (Open Scalable Cluster Environment) is an Open-Source project aims to create a scalable and extensible cluster environment. The aim of the project is to allow user to

  Allows anyone to deploying a cluster quickly and effectively

  Reduce complexity in maintaining a Beowulf cluster

  Provides a user friendly environment that increase users productivity

• Perceus/warewulf cluster portal
• Bootable Cluster CD The BCCD was created to facilitate instruction of parallel computing aspects and paradigms. Part of the difficulty instructors face is lack of dedicated resources to explore distributed computing aspects lack of time to preconfigure and test the supporting environment. The BCCD image addresses this problem by providing a non-destructive overlay way to run a full-fledged parallel computing environment on just about any workstation-class system…We’re happy to say that this now includes the MAC too!
• C3 Cluster Command and Control ORNL is developing user interfaces for assisting in the system management of PC clusters. C3 is a command line interface that may also be called within programs. C2G provides a Python/TK based GUI, that among other features, may invoke the C3 tools. The Cluster Command and Control (C3) tool suite was developed for use in operating the HighTORC Linux cluster at Oak Ridge National Laboratory. This suite implements a number of command line based tools that have been shown to increase system manager scalability by reducing time and effort to operate and manage the cluster.
• ClustermatiC! Redesigning the Cluster Architecture. A project of the Cluster research Lab in the Advanced Computing Laboratory at Los Alamos National Laboratory
• High Performance Computing in Engineering — center of High Performance Computing technology (all in Russian). St.Petersburg (Russia) Politechnical State University
• LTSP.org Linux Terminal Server Project (the portal remark: I think it is like cluster of terminals or thin clients). LTSP is an add-on package for Linux that allows you to connect lots of low-powered thin client terminals to a Linux server. Applications typically run on the server and accept input and display their output on the thin client display. LTSP is available as a set of packages that can be installed on any Linux system. It is also available as a part of complete distributions, such as K12Ltsp and SkoleLinux.
• Robert G. Brown’s personal cluster pages. This page is devoted to beowulfery and cluster computing. There in an online book on cluster engineering, talks and tutorials, and software packages. You can either click directly to a resource page on the navigation bar above or browse the list below, where each link is accompanied by a short abstract.
• Cluster Computing Info Centre the site has been developed by Rajkumar Buyya (editor of the book High Performance Cluster Computing, Volume 1, Prentice Hall, 1999, 881 pages, ISBN: 0130137847 and Volume 2, Prentice Hall, 1999, 664 pages, SBN: 0130137855)
• SystemImager is software that automates Linux installs, software distribution, and production deployment. SystemImager makes it easy to do automated installs (clones), software distribution, content or data distribution, configuration changes, and operating system updates to your network of Linux machines. You can even update from one Linux release version to another.
  It can also be used to ensure safe production deployments. By saving your current production image before updating to your new production image, you have a highly reliable contingency mechanism. If the new production enviroment is found to be flawed, simply roll-back to the last production image with a simple update command.
  Some typical environments include: Internet server farms, database server farms, high performance clusters, computer labs, and corporate desktop environments.
• ROCKS Cluster distribution. Since May 2000, the Rocks group has been addressing the difficulties of deploying manageable clusters. We have been driven by one goal: make clusters easy. By easy we mean easy to deploy, manage, upgrade and scale. We are driven by this goal to help deliver the computational power of clusters to a wide range of scientific users. It is clear that making stable and manageable parallel computing platforms available to a wide range of scientists will aid immensely in improving the state of the art in parallel tools.
• The Linux Virtual Server (LVS) is a highly scalable and highly available server built on a cluster of real servers, with the load balancer running on the Linux operating system. The architecture of the server cluster is fully transparent to end users, and the users interact as if it were a single high-performance virtual server.
• OpenSSI (Single System Image) Clusters for Linux project is a comprehensive clustering solution offering a full, highly available SSI environment for Linux. Goals for OpenSSI Clusters include availability, scalability and manageability, built from standard servers. Technology pieces include: membership, single root and single init, cluster filesystems and DLM, single process space and process migration, load leveling, single and shared IPC space, device space and networking space, and single management space.
• The Open Cluster Group is an informal group of people dedicated to making cluster-computing practical for high performance computing. Our core working principles are:

  We believe the adoption of clusters for mainstream high performance computing is limited by the lack of well -accepted software stacks that are robust and easy to use by the general user community

  We fully embrace the Open Source model of software distribution. Anything contributed to the group must be freely distributable – preferably as source code under a well-known open source license

  We seek common solutions for cluster computing solution stacks – fragmentation of available software stacks doesn’t help anyone

  On the pages you can find software for project OSCAR.

• Linux Clustering with CSM and GPFS (HTML) The cluster installation and administration is based on Cluster Systems Management (CSM) for Linux, which dramatically simplifies administration of a Linux cluster by providing management from a single point-of-control.
  This redbook also presents the General Parallel File System (GPFS) for Linux Clusters. GPFS provides a cluster-wide file system that allows users shared access to files spanning multiple disk drives. GPFS is based on a shared disk model, providing lower overhead access to disks not directly attached to the application nodes, and using a distributed protocol to provide data coherence for access from any node.
• Linux Clustering with CSM and GPFS (in PDF)
• Linux HPC Cluster Installtion IBM Red Book (in HTML) This redbook will guide system architects and systems engineers through a basic understanding of cluster technology, terms, and Linux High-Performance Computing (HPC) clusters. We discuss some of the design guidelines used when architecting a solution and on how to install and configure a working Linux HPC cluster.
• Linux HPC Cluster Installtion IBM Red Book (in PDF)
• Building an HPC Cluster with Linux, pSeries, Myrinet, and Various Open Source Software (PDF); IBM RedPaper
• Cluster Development Kit (Portland Group) Linux clusters now are easily affordable by any group interested in serious scientific computing. Getting hardware installed and a network up and running are just the first steps in building a cluster. The PGI CDK. Cluster Development Kit╝ contains all the software development and cluster management tools you need to get a Linux cluster running production applications.
• Center for HPC Cluster Resource Management and Scheduling – source for Torque, Maui and other cluster software.
• Beowulf.org — is a collection of resources for the expanding universe of users and designers of Beowulf class cluster computers. These enterprise systems are built on commodity hardware deploying Linux OS and open source software
• Cluster on Demand Run your application on our high performance computing clusters. We provide cost-effective computing services and solutions accessible over the internet for small and mid-sized organizations worldwide. Login securely and launch your application on a powerful cluster today.
• LCFG is a system for automatically installing and managing the configuration of large numbers of Unix systems. It is particularly suitable for sites with very diverse and rapidly changing configurations
• SourceForge.net Clustering — the list of cluster developments on the site SourceForge.net
• quattor is a system administration toolkit providing a powerful, portable and modular toolsuite for the automated installation, configuration and management of clusters and farms running UNIX derivates like Linux and Solaris. quattor was started in the scope of the EDG project (2001-2003). Development and maintenance is coordinated by CERN (IT department) in collaboration with other partner institutes (in particular BARC, IN2P3/LAL, INFN/CNAF, NIKHEF and UAM Madrid) in the scope of ELFms.
• cfengine , or the configuration engine is an agent/software robot and a high level policy language for building expert systems to administrate and configure large computer networks. Cfengine uses the idea of classes and a primitive intelligence to define and automate the configuration and maintenance of system state, for small to huge configurations. Cfengine is designed to be a part of a computer immune system, and can be thought of as a gaming agent.
• HA Linux Project The High Availability Linux Project. The basic goal of the High Availability Linux project is to:Provide a high availability (clustering) solution for Linux which promotes reliability, availability, and serviceability (RAS) through a community development effort.
• Hadoop is a software platform that lets one easily write and run applications that process vast amounts of data.Hadoop implements MapReduce, using the Hadoop Distributed File System (HDFS) (see figure below.) MapReduce divides applications into many small blocks of work. HDFS creates multiple replicas of data blocks for reliability, placing them on compute nodes around the cluster. MapReduce can then process the data where it is located.

Andrey Y Shevel.

Of course, it’s not GRID software and some information became old, here we should also point some words about SGE (Sun Grid Engeen), Condor, etc … The next step is to make some analysis about Cluster soft with Heterogeneity support: both Linux and Windows working nodes.