Global file systems

Problem deferral?

Improved CPU performance, a faster network thanks to InfiniBand, more cores per processor but bottlenecks still cannot be avoided because of one key factor:
storage space
The following facts are a reality:

• Calculation results of multiple TB are not uncommon
• File servers are overflowing
• Above a certain number of nodes, scaling under NFS no longer improves performance
• IP over InfiniBand is not an alternative to native access

The storage space should be available in one or as few as possible file systems to prevent inconvenient copying.
In addition, it should be available on every cluster node and use existing high speed networks for transferring data.

Separating two terms

True cluster file systems
When it comes to file systems, the two standard concepts need to be mentioned separately.
True cluster file systems require the use of shared storage (SAN).
Simultaneous write access at block device level is essential.
Important well-known cluster file systems such as GFS2 from Red Hat or OCFS2 from Oracle are now part of the Linux kernel.

• Intelligent and complex locking services

Cluster file systems use an intelligent locking system - the distribution locking manager (DLM) - to ensure data and media data consistency with simultaneous write access.
Fibre channel, iSCSI or InfiniBand are used to connect to the SAN.

Distributed file systems
Distributed file systems function in a totally different way.
They involve logically combining storage pools located on several different computers. The resulting logical volumes can then be provided to the other client computers.

• Distributed file systems are founded on a multi-component client/server model.

Clients attempting to access storage areas in a distributed file system first establish a connection to a metadata server which then redirects them to the actual server containing the data, to which the target storage address is locally connected (redirection).

Complex solutions such as global file systems are nothing out of the ordinary - our experts are happy to advise you.

Lustre? Yes!

Lustre in three sentences
The Lustre file system was developed for use in high performance computer (HPC) clusters.
Like NFS or SMB/CIFS, Lustre is a network file system.
Unlike these, however, Lustre is a cluster file system, i.e. the file system can be distributed across several servers.

• Components of Lustre cluster
• The metadata server (MDS) that co-ordinates the locking mechanisms for file access.
• The actual data servers, known as the object storage servers (OSS).
• The management server, which allows configuration and management of the Lustre cluster.

• Outstanding and flexible
• The individual object storage servers can be connected to a variety of storage systems:
  From internal hard disks, to DAS systems, through to fibre-channel SANs, anything can be used for Lustre volumes.
• The machines involved in a Lustre file system communicate using the LNET protocol specially developed for Lustre, which supports TCP/IP and high-speed interfaces such as InfiniBand or Myrinet and benefits from the high bandwidth and low latency.

Effective use

As a distributed file system, Lustre allows a group of computers (normally HPC nodes) to simultaneously access a shared file system.
In addition to high performance and excellent scalability, the primary benefit is the fact that distributed storage allows job flow optimisation.

• Specific example:

Since the beginning of 2008, a transtec Lustre system has been in operation in the computer centre at the University of Karlsruhe.
With a gross capacity of 750 TB and based on InfiniBand, the file system is provided by several Linux storage servers, which make up an HA cluster. The storage area is used by several HPC clusters and achieves an extremely high data throughput.

Recommendation:

Without redundant hardware equipment, Lustre should not currently be used for permanent data storage.
While it is not particularly well suited for home directories, it is widely used in high-speed scratch areas such as cluster calculations, as a glance at the Top 500 shows.

Our promise

We are there for you

• From conception and project design
• to installation, configuration
• and integration into your IT environment
• through to training your users

Trust our years of experience in this business and let us help you optimise your structures.
Our experts will be happy to advise you.