File System Guide | Basic Ceph Administration

The Ceph File System (CephFS) is a file system compatible with POSIX standards that are built on top of Ceph’s distributed object store, called RADOS (Reliable Autonomic Distributed Object Storage). CephFS provides file access to a Ceph cluster and uses the POSIX semantics wherever possible. For example, in contrast to many other common network file systems like NFS, CephFS maintains strong cache coherency across clients. The goal is for processes using the file system to behave the same when they are on different hosts as when they are on the same host. However, in some cases, CephFS diverges from the strict POSIX semantics.

The Ceph File System has two primary components:

  1. Clients
    The CephFS clients perform I/O operations on behalf of applications using CephFS, such as, ceph-fuse for FUSE clients and kcephfs for kernel clients. CephFS clients send metadata requests to an active Metadata Server. In return, the CephFS client learns of the file metadata, and can begin safely caching both metadata and file data.
  2. Metadata Servers (MDS)
    The MDS does the following:
  • Provides metadata to CephFS clients.
  • Manages metadata related to files stored on the Ceph File System.
  • Coordinates access to the shared Ceph cluster.
  • Caches hot metadata to reduce requests to the backing metadata pool store.
  • Manages the CephFS clients’ caches to maintain cache coherence.
  • Replicates hot metadata between active MDS.
  • Coalesces metadata mutations to a compact journal with regular flushes to the backing metadata pool.
  • CephFS requires at least one Metadata Server daemon (ceph-mds) to run.

The diagram below shows the component layers of the Ceph File System.

The bottom layer represents the underlying core storage cluster components:
- Ceph OSDs (ceph-osd) where the Ceph File System data and metadata are stored.
- Ceph Metadata Servers (ceph-mds) that manages Ceph File System metadata.
- Ceph Monitors (ceph-mon) that manages the master copy of the cluster map.

The Ceph Storage protocol layer represents the Ceph native librados library for interacting with the core storage cluster. The CephFS library layer includes the CephFS libcephfs library that works on top of librados and represents the Ceph File System. The top layer represents two types of Ceph clients that can access the Ceph File Systems.

The diagram below shows more details on how the Ceph File System components interact with each other.

Lab Environment:


  • A running Ceph cluster.
  • Installation of the Ceph Metadata Server daemons (ceph-mds).

Basic Administration:

1. Deploy Metadata Servers

Each CephFS file system requires at least one MDS. The cluster operator will generally use their automated deployment tool to launch required MDS servers as needed. In this case, used ceph orch apply mds and orchestrate with placement option

Configure ceph keyring related MDS

2. Create Ceph File System

A Ceph file system requires at least two RADOS pools, one for data and one for metadata.

Once the pools are created, you may enable the file system using the ceph fs new command. Once a file system has been created, your MDS(s) will be able to enter an active state and other standby. Notice the available size.

3. Mount CephFS on a Ceph client

Access Ceph-client host then install required package

Copy related files for authentication purposes between Ceph cluster and client

Mount the file system and verify that the file system is mounted and showing the correct size

Try to write files on /mnt

on Ceph Dashboard => File Systems will look like:

4. Removing CephFS and Pools

To delete CephFS or Pools, follow these steps. First, unmount /mnton the Client host.

Back to Ceph cluster node, stop MDS daemon use ceph orch command then delete cephfs

To delete pools execute the following, replace POOL_NAME with the name of the pool:

If you get an error when deleting pools, please follow this guide. Finally pools can be delete like this

See other content

References :

#CEPH #Storage #CentOS8 #SDS



Cloud Consultant | RHCSA | RHCE in Red Hat OpenStack | Google Cloud ACE | AWS SAA | LinkedIn:

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store