Linux Clustering

What is Linux Clustering

Linux clustering is the ability to make more than one Linux system act in coordination toward some common function. The goal could be to make it so that a system failure won't knock an important service offline, or maybe to make a system that is able to handle problems that are bigger than any individual computer's physical resources.

There are many free and commercial options for the software to enable these systems and to manage them. This page will try to explain the different types of clustering and offer pointers to resources that will help you in your journey to set up your cluster.

What to add to the page next?
How I started with Linux Clustering
Types of clusters
High Availability Software
Load Balancing Software
High Performance Computing Cluster Software
Software: File Systems
Software: Installation
Software: Management/Administration
Software: Monitoring
Software: Queueing and Scheduling
Miscellaneous Useful Links
Linux Clustering Books
General High Performance Computing Books
High Availability Books
Distributed/Parallel Computing Books
Parallel Programming Books
What do you think?

What to add to the page next?

There are a few topics we'd like to add information about to the page. What topic would you find most interesting?

Loading poll. Please Wait...

Share this poll
- Stumbleupon
- Facebook
- MySpace
- Twitter
- Digg
- Delicious
- Email

How I started with Linux Clustering

I was introduced to Linux by a good friend in college. We spent many a night breaking the system and then fixing it, or sometimes having to reinstall to figure out what we did.

At my first job out of college, I was setting up a print server and was curious how I could make it so that if the computer broke, everyone would still be able to print. This first intro, to what Red Hat referred to as their Piranha package at the time (today that functionality has been rolled into the Red Hat Cluster Suite), was my first contact with any kind of clustering in Linux. Since then, I've worked with everything from HA web servers to Top500 systems to nationally distributed grids.

Types of clusters

When I think of clustering, I tend to classify clusters in three types, high availability, load balancing, and high performance computing.

High Availability: No one wants any services they offer to be unreachable, especially since that downtime will almost undoubtably cost them money. High availability clusters offers the solution to this need for a lack of downtime.
Load Balancing: Load balancing is almost a special subset of the high availability. The idea is to create a set up where you have multiple systems all serving the same service at the same time because the demand for the service is more than a single server can handle.
High Performance Computing: Also known as Beowulf clusters, the idea here is to hook a bunch of regular computers up to act in a coordinated way so that they can do calculations or jobs that are bigger than any single computer. In this way, you can build a supercomputer using Commodity Off The Shelf (COTS) hardware. These systems do everything from calculating the origins of the universe to modeling weather to rendering movie special effects.

High Availability Software

High availability is a must for any business these days. You lose money if you have a failure and you suffer downtime. Here we'll list Linux HA options.

Corosync Cluster Engine: The Corosync Cluster Engine is an OSI Certified implementation of a complete cluster engine. It is a widely distributed de-facto standard cluster infrastructure for application high availability.
LifeKeeper: SteelEye's LifeKeeper for Linux is a software application that ensures the continuous availability of applications by maintaining system uptime. LifeKeeper maintains the high availability of Linux cluster systems by monitoring system and application health, maintaining client connectivity and providing uninterrupted data access regardless of where clients reside - on the corporate Internet, intranet or extranet.
Linux-HA Project: The Linux-HA project maintains a set of building blocks for high availability cluster systems, including a cluster messaging layer, a huge number of resource agents for a variety of applications, and a plumbing library and error reporting toolkit.

Linux-HA is perhaps the best known of all the Linux HA software, home to the Heartbeat Daemon.
OpenAIS: The OpenAIS Standards Based Cluster Framework is an OSI Certified implementation of the Service Availability Forum Application Interface Specification (AIS). The Application Interface Specification is a software API and policies which are used to develop applications that maintain service during faults. Restarting and failover of applications is also provided for those deploying applications which may not be modified. The OpenAIS software is built to operate on the Corosync Cluster Engine which allows any third party to implement plugin cluster services using the infrastructure provided.
Red Hat Cluster Suite: Red Hat Cluster Suite provides two distinct types of cluster:
* Application/Service Failover - Create n-node server clusters for failover of key applications and services
* IP Load Balancing - Load balance incoming IP network requests across a farm of servers
With Red Hat Cluster Suite, applications can be deployed in high availability configurations so that they are always operational-bringing "scale-out" capabilities to Enterprise Linux deployments.
Veritas Cluster Server: VeritasTM Cluster Server from Symantec is a high availability solution for reducing both planned and unplanned downtime. By monitoring the status of applications and automatically moving them to another server in the event of a fault, Cluster Server can dramatically increase the availability of an application or database.

Load Balancing Software

Load balancing of certain services, such as mail or web, is a must for any organization that needs to service a large client base.

Keepalived: Keepalived is a userspace daemon for LVS cluster nodes healthchecks and LVS directors failover. Keepalived implements a framework that gives the daemon the ability of checking a LVS server pool states. When one of the server of the LVS server pool is down, keepalived informs the Linux kernel to remove this server entry from the LVS topology.
Linux Virtual Server: The Linux Virtual Server is a highly scalable and highly available server built on a cluster of real servers, with the load balancer running on the Linux operating system. The architecture of the server cluster is fully transparent to end users, and the users interact as if it were a single high-performance virtual server. The Linux Virtual Server as an advanced load balancing solution can be used to build highly scalable and highly available network services, such as scalable web, cache, mail, ftp, media and VoIP services.
Red Hat Cluster Suite: Red Hat Cluster Suite provides two distinct types of cluster:
* Application/Service Failover - Create n-node server clusters for failover of key applications and services
* IP Load Balancing - Load balance incoming IP network requests across a farm of servers
With Red Hat Cluster Suite, applications can be deployed in high availability configurations so that they are always operational-bringing "scale-out" capabilities to Enterprise Linux deployments.

High Performance Computing Cluster Software

Computational/High Performance Linux clusters started back in 1994 when Donald Becker and Thomas Sterling built a cluster for NASA. This cluster was made up of 16 DX4 processors connected by 10 Mbit Ethernet, and they named it Beowulf.

There are a variety of software packages to build and manage these kinds of systems.

ISLE Cluster Manager: SGI's ISLE Cluster Manager provides the necessary power and flexibility to monitor essential system metrics from a single point of control. ISLE Cluster Manager reduces the time and resources spent administering the system by improving software maintenance procedures and automating repetitive tasks. The comprehensive features of ISLE Cluster Manager help lower total cost of system ownership, increase productivity, and provide a better return on your investment.
MOSIX: MOSIX is an on-line management system targeted for high performancer computing on Linux clusters, multi-clusters and Clouds. It supports both interactive processes and batch jobs. MOSIX can be viewed as a multi-cluster operating system that incorporates automatic resource discovery and dynamic workload distribution, commonly found on single computers with multiple processors.
OSCAR: OSCAR allows users, regardless of their experience level with a *nix environment, to install a Beowulf type high performance computing cluster. It also contains everything needed to administer and program this type of HPC cluster.
Platform Cluster Manager: Platform Cluster Manager makes clusters simple with a complete solution that is easy to deploy, manage and use. Managed through a web interface, even users with limited Linux administration experience can deploy and manage sophisticated HPC clusters with state of the art software tools.
Platform HPC Workgroup Manager: Platform HPC Workgroup Manager includes familiar easy-to-use cluster management tools such as Platform Cluster Manager, Platform LSF Workgroup Edition, Platform MPI, Platform LSF web interface, and Platform ISF Adaptive Cluster.
Rocks: Rocks is an open-source Linux cluster distribution that enables end users to easily build computational clusters, grid endpoints and visualization tiled-display walls.
Scyld Clusterware: Scyld ClusterWare is an HPC cluster management solution. It is designed to make the deployment and management of a Linux cluster as easy as the deployment and management of a single system.
xCAT: xCAT offers complete and ideal management for HPC clusters, RenderFarms, Grids, WebFarms, Online Gaming Infrastructure, Clouds, Datacenters, and whatever tomorrow's buzzwords may be. It is agile, extendable, and based on years of system administration best practices and experience.

Software: File Systems

Cluster and High Performance File Systems

File systems designed to run in a clustered or high performance environment.

OpenAFS: AFS is a distributed filesystem product, pioneered at Carnegie Mellon University. It offers a client-server architecture for federated file sharing and replicated read-only content distribution, providing location independence, scalability, security, and transparent migration capabilities. AFS is available for a broad range of heterogeneous systems including UNIX, Linux, MacOS X, and Microsoft Windows. IBM branched the source of the AFS product, and made a copy of the source available for community development and maintenance. They called the release OpenAFS.
DRBD: DRBD refers to block devices designed as a building block to form high availability (HA) clusters. This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network based raid-1.
GFS: Red Hat GFS is commonly used in clusters of enterprise applications to provide high speed access to a consistent file system image across the server nodes. This allows the cluster nodes to simultaneously read and write to a single shared filesystem.
GPFS: General Parallel File System (GPFS) is a high performance shared-disk file system that can provide data access from nodes in a cluster environment. Parallel and serial applications can readily access shared files using standard UNIX® file system interfaces, and the same file can be accessed concurrently from multiple nodes. GPFS is designed to provide high availability through logging and replication, and can be configured for failover from both disk and server malfunctions.
Lustre: Lustre is designed to enable I/O performance and scaling beyond the limits of traditional storage technology. Often used in High Performance Computing environments, Lustre is also applicable to any enterprise storage environment where very high I/O bandwidth is required. Lustre can scale to tens of thousands of nodes and petabytes of storage with groundbreaking I/O and metadata throughput.
PVFS: PVFS is an open-source, scalable parallel file system targeted at production parallel computation environments. It is designed specifically to scale to very large numbers of clients and servers. The architecture is very modular, allowing for easy inclusion of new hardware support and new algorithms, which makes PVFS a good research testbed as well.

Software: Installation

Installation software

Things that will help you install/provision your cluster easier

FAI - Fully Automatic Installation: FAI is a non-interactive system to install, customize and manage Linux systems and software configurations on computers as well as virtual machines and chroot environments, from small networks to large-scale infrastructures and clusters. FAI a tool for mass unattended Linux installation. You can take one or more virgin PCs, turn on the power, and after a few minutes, the systems are installed, and completely configured to your exact needs, without any interaction necessary.
Perceus: Perceus is the next generation cluster and enterprise tool kit for the deployment, provisioning, and management
of groups of servers. Employing the power of the Perceus OS and framework, the user can quickly purpose a
machine out of the box.
SystemImager: SystemImager is software which automates Linux installs, software distribution, and production deployment. SystemImager makes it easy to do automated installs (clones), software distribution, content or data distribution, configuration changes, and operating system updates to your network of Linux machines. You can even update from one Linux release version to another!

Software: Management/Administration

Administration and Management Software

Stuff to help you manage your cluster.

C3: The Cluster Command and Control (C3) tool suite is a command line interface that may also be called within programs. This suite implements a number of command line based tools that have been shown to increase system manager scalability by reducing time and effort to operate and manage the cluster.
Cfengine: Cfengine is a distributed agent framework for performing policy-based network and system administration that is used on Unix-like and Windows systems.
Webmin/Cloudmin: Webmin is a web-based interface for system administration for Unix. Using any modern web browser, you can setup user accounts, Apache, DNS, file sharing and much more. Cloudmin provides a web interface for management of multiple systems

Software: Monitoring

Monitoring Software

Stuff that will let you keep an eye on your cluster to make sure it's running well.

Big Brother: Big Brother Professional Edition (BBPE) is an easy-to-implement, affordable, web-based solution for IT infrastructure monitoring and diagnostics. Get real-time monitoring for any server (Windows, UNIX, Linux) or device, on any network, from a web browser.
Ganglia: Ganglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. It is based on a hierarchical design targeted at federations of clusters. It leverages widely used technologies such as XML for data representation, XDR for compact, portable data transport, and RRDtool for data storage and visualization.
Nagios: Nagios is a powerful tool that provides instant awareness of an organization's IT infrastructure and allows the user to detect and repair problems and mitigate future issues before they affect end-users or customers.
Performance Co-Pilot: Performance Co-Pilot (PCP) provides a framework and services to support system-level performance monitoring and management. It presents a unifying abstraction for all of the performance data in a system, and many tools for interrogating, retrieving and processing that data.

Software: Queueing and Scheduling

Queueing and Scheduler Software

For high performance computing clusters, you need a way to coordinate jobs running on your system. This will help you do that.

Condor: Condor is a specialized workload management system for compute-intensive jobs. Like other full-featured batch systems, Condor provides a job queueing mechanism, scheduling policy, priority scheme, resource monitoring, and resource management.
Lava: Platform Lava is an open source entry-level workload scheduler designed to meet a wide range of workload scheduling needs for clusters up to 512-nodes.
LSF: Platform LSF, the industry-leading workload management solution for high performance computing (HPC) environments, schedules batch and interactive workload for compute- and data-intensive applications in cluster and grid environments.
Maui Cluster Scheduler: The Maui Cluster Scheduler is an open source job scheduler for clusters and supercomputers. It is an optimized, configurable tool capable of supporting an array of scheduling policies, dynamic priorities, extensive reservations, and fairshare capabilities.
Moab Cluster Suite: Moab Cluster Suite is an intelligent management middleware that provides simple Web-based job management, graphical cluster administration and management reporting tools. Using Moab, a user can dynamically change HPC resource pools on the fly to match specific workload needs.
PBS Works: Formerly known as PBS Pro, PBS Works makes it possible to easily create intelligent policies to manage distributed, mixed-vendor computing assets, including applications, as a single, unified system. Based on a policy-driven architecture, PBS Works continually optimizes how your technical HPC resources are used, ensuring that all resources are used effectively and efficiently.
Sun Grid Engine: Sun Grid Engine is a distributed resource management (DRM) system that manages the distribution of users' workload to available compute resources. Sun Grid Engine monitors the current state of all resources in the cluster and is able to assign these jobs to the best suited resources.

There is also a community portal here.
TORQUE Resource Manager: TORQUE is an open source resource manager providing control over batch jobs and distributed compute nodes. It is a community effort based on the original *PBS project and has incorporated significant advances in the areas of scalability, fault tolerance, and feature extensions

Miscellaneous Useful Links

Pages/sites that you'll probably find helpful.

Cluster Builder: ClusterBuilder.org assists cluster administrators, technical evaluators and purchase evaluators to research popular information about components they may need in their current or future cluster, grid or utility hosting environments.
Cluster Monkey: A portal with news, articles, and features about Linux clustering and high performance computing.
freshmeat: freshmeat maintains the Web's largest index of Unix and cross-platform software, themes and related "eye-candy", and Palm OS software. Thousands of applications, which are preferably released under an open source license, are meticulously cataloged in the freshmeat database, and links to new applications are added daily.
Furbeowulf Cluster Computing: The name says it all.
HPC Community: The HPCCommunity.org effort is a technical community for the High Performance Computing (HPC) community. HPCCommunity.org is sponsored by Platform Computing and is dedicated to supporting both Platform Computing and non-Platform technologies.
HPC Wire: HPCwire is a news and information site covering the entire ecosystem of High Productivity Computing (HPC), target at an audience interested in computationally- and data-intensive computing, including infrastructure topics such as software, middleware, hardware, networking, storage, tools and applications.
IEEE Technical Committee On Scalable Computing: The IEEE Technical Committee on Scalable Computing (TCSC) addresses theoretical and experimental aspects of designing, developing, and evaluating scalable network computing systems, especially clusters and grids, and their applications. Specific topics of interest include cluster and grid interconnection networks, middleware, single-system image, resource and scheduling management policies, distributed programming environments, principles of scalable and reliable software engineering, and high-performance and high-availability computing applications. The TCSC sponsors workshops and conferences on these and related topics.
Linux Clusters Institute: LCI is the premier international forum to share information on management, administration, and scientific computing techniques for high performance clustered computing. Founded in 1998, it includes some of the world's foremost specialists in building and deploying clustered high-performance computing systems.
LinuxHPC.org: LinuxHPC.org is a website for system administrators, developers, and technical managers, offering recent industry news, events, mailing lists and links, etc. related to high performance technical computing and clustering with Linux.
Top500: Top500 is the home of the twice annual ranking of the top 500 supercomputers in the world.