Ceph Storage Cluster deployment on Rocky Linux 9/CentOS 9

  • Post author:
  • Reading time:32 mins read
  • Post category:Ceph
  • Post comments:0 Comments

Hello there, it’s been a while.

Didn’t really have a topic to write about, but today I have something a bit more fancy… at least for me.

In this post, I want to go through a simple deployment of a Ceph Cluster using 3 VMs. I will go through the steps to bootstrap the cluster using “cephadm” adding a couple of nodes and creating our first CephFS filesystem.

The last step will be to mount the filesystem on a client and create a few files.

So, what is Ceph you might ask? Ceph is an open-source software-defined storage platform. This allows you to span your storage system over multiple servers, which makes scaling out your storage (somewhat) easy. It provides object, block and file storage options, giving you a huge amount of flexibility.

Now, I am writing all this, but this will be the first time deploying Ceph myself. 🙂

All of this, you can also get from the official documentation. I’m just making it a bit more compact.

Let’s begin.

Ceph Installation/Deployment

For this I am assuming that you have 3 hosts deployed with at least one 10GB disk attached. You can add more disks or use physical servers, I just don’t have the resources. The steps will be the same. Also, I will be using “Rocky Linux 9 Generic Cloud” VMs, so no firewall in my case.

Here is my setup.

HostnameNetworkdisk
ceph-0110.10.0.31/243x 10GB
ceph-0210.10.0.32/243x 10GB
ceph-0310.10.0.33/243x 10GB
The host names I will be using. You have to use a disk size of a minimum of 5GB, else ceph won’t use those.

Installation of the required packages

First things first. We have to install the required packages. Do this on every host.

This will install additional packages like podman and lvm.

ceph-01

# Install the repository
ceph-01 :: ~ » sudo dnf install centos-release-ceph-reef
# Install the cephadm package
ceph-01 :: ~ » sudo dnf install cephadm
# Install podman
ceph-01 :: ~ » sudo dnf install podman
# Not required but recommended are the ceph tools (optional)
ceph-01 :: ~ » sudo dnf install ceph-common

ceph-02

# Install the repository
ceph-02 :: ~ » sudo dnf install centos-release-ceph-reef
# Install the cephadm package
ceph-02 :: ~ » sudo dnf install cephadm
# Install podman
ceph-02 :: ~ » sudo dnf install podman
# Not required but recommended are the ceph tools (optional)
ceph-02 :: ~ » sudo dnf install ceph-common

ceph-03

# Install the repository
ceph-03 :: ~ » sudo dnf install centos-release-ceph-reef
# Install the cephadm package
ceph-03 :: ~ » sudo dnf install cephadm
# Install podman
ceph-03 :: ~ » sudo dnf install podman
# Not required but recommended are the ceph tools (optional)
ceph-03 :: ~ » sudo dnf install ceph-common

Setting up the firewall

Next, we need to set up the firewall. Either disable it or better yet, open the following ports on the servers.

ceph-01 :: ~ » sudo firewall-cmd --add-port={8443,3000,9095,9093,9094,9100,9283}/tcp

Bootstrap the ceph cluster

Now we can bootstrap our cluster. We only need to do this on one system. Enter the IP of the host you are trying to bootstrap.

This will show a bunch of things, the interesting ones are the “Ceph Dashboard” credentials. This is the integrated web interface for ceph. It basically allows you to manage your ceph cluster using a GUI interface, but we will be using the CLI for this.

ceph-01

# Bootstrap ceph cluster
ceph-01 :: ~ » sudo cephadm bootstrap --mon-ip 10.10.0.31
...
Ceph Dashboard is now available at:

             URL: https://ceph-01:8443/
            User: admin
        Password: 6t6bp4ueg6
...

Add additional Hosts

Next, we can add the other two hosts to the cluster.

But first we have to add the SSH public keys to the two additional hosts. How you do this, is up to you, I will just copy / paste the key into the root “authorized_keys” file. If you already have SSH configured, you could use the “ssh-copy-id” command.

The public key is located at /etc/ceph/ceph.pub

Here the method using ssh-copy-id

# Switch to root
ceph-01 :: ~ » sudo -s
# Copy the public key to remote host
ceph-01 :: root » ssh-copy-id -f -i /etc/ceph/ceph.pub root@*<new-host>*

You could also just manually add it. For this, copy the key.

# Output and copy the key
ceph-01 :: ~ » cat /etc/ceph/ceph.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKU8XD6xkuqKE8suXP3AhXppGuUVHrFZDR5LQsDbnX9WBfGeqrU/1sstqif3nTcbfnXXGZXbH8HYR9lefIJnDsoJRUgwzInZCm2IvGFa+ZKCJV3ADnf4bFZDQn7x4svBGQylMQlMPZup2MLLLbD4yHXCTil1Ah+X3irdMvPZN4rGz1rrj1BNehQ2ihDwD0yY3+qLzzVra6lNkdWW0= ceph-267b2084-e0ac-11ee-9381-52540074873a

And just paste it into the root authorized_key file.

# Switch to root
ceph-02 :: ~ » sudo -s
# Paste the key into the autorized_keys file
ceph-02 :: root » vim ~/.ssh/authorized_keys

Once that’s done, we can add the nodes to the cluster. This will deploy and configure ceph on the other hosts automatically.

# Add the nodes to the cluster
ceph-01 :: ~ » ceph orch host add ceph-02 10.10.0.32
ceph-01 :: ~ » ceph orch host add ceph-03 10.10.0.33

With the command “ceph status”, we can check the status of the cluster. The green marked line shows our nodes. We also have an error seen in the red line. This is because we didn’t add any OSDs yet. This is basically our storage. There will be one OSD per physical disk.

ceph-01 :: ~ » ceph status
  cluster:
    id:     f581d60e-e2f8-11ee-8f4b-525400c55f38
    health: HEALTH_WARN
            OSD count 0 < osd_pool_default_size 3
 
  services:
    mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 (age 20s)
    mgr: ceph-01.bhovln(active, since 10m), standbys: ceph-02.tzstuj
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     

Add OSD to cluster

With the following command, we can add all unused and available storage.

ceph-01 :: ~ » ceph orch apply osd --all-available-devices

If you want to add a single disk, use this command.

ceph-01 :: ~ » ceph orch daemon add osd <hostname>:/dev/vdb

If you want to check what happens first, execute the command with the “–dry-run” option. You have to execute this twice (you will get a message telling you to try again in a bit.

ceph-01 :: ~ » ceph orch apply osd --all-available-devices --dry-run
+---------+-----------------------+---------+----------+----+-----+
|SERVICE  |NAME                   |HOST     |DATA      |DB  |WAL  |
+---------+-----------------------+---------+----------+----+-----+
|osd      |all-available-devices  |ceph-01  |/dev/vdb  |-   |-    |
|osd      |all-available-devices  |ceph-01  |/dev/vdc  |-   |-    |
|osd      |all-available-devices  |ceph-01  |/dev/vdd  |-   |-    |
|osd      |all-available-devices  |ceph-02  |/dev/vdb  |-   |-    |
|osd      |all-available-devices  |ceph-02  |/dev/vdc  |-   |-    |
|osd      |all-available-devices  |ceph-02  |/dev/vdd  |-   |-    |
|osd      |all-available-devices  |ceph-03  |/dev/vdb  |-   |-    |
|osd      |all-available-devices  |ceph-03  |/dev/vdc  |-   |-    |
|osd      |all-available-devices  |ceph-03  |/dev/vdd  |-   |-    |
+---------+-----------------------+---------+----------+----+-----+

Once added, you can check the disks with this command.

ceph-01 :: ~ » ceph orch device ls
HOST     PATH      TYPE  DEVICE ID   SIZE  AVAILABLE  REFRESHED  REJECT REASONS  
ceph-01  /dev/vdb  hdd              10.0G  Yes        16m ago                    
ceph-01  /dev/vdc  hdd              10.0G  Yes        16m ago                    
ceph-01  /dev/vdd  hdd              10.0G  Yes        16m ago                    
ceph-02  /dev/vdb  hdd              10.0G  Yes        16m ago                    
ceph-02  /dev/vdc  hdd              10.0G  Yes        16m ago                    
ceph-02  /dev/vdd  hdd              10.0G  Yes        16m ago                    
ceph-03  /dev/vdb  hdd              10.0G  Yes        15m ago                    
ceph-03  /dev/vdc  hdd              10.0G  Yes        15m ago                    
ceph-03  /dev/vdd  hdd              10.0G  Yes        15m ago                    

Great. Let’s check the health again.

ceph-01 :: ~ » ceph status
  cluster:
    id:     f581d60e-e2f8-11ee-8f4b-525400c55f38
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 (age 17m)
    mgr: ceph-01.bhovln(active, since 27m), standbys: ceph-02.tzstuj
    osd: 9 osds: 9 up (since 14s), 9 in (since 2m)
 
  data:
    pools:   1 pools, 1 pgs
    objects: 2 objects, 449 KiB
    usage:   640 MiB used, 89 GiB / 90 GiB avail
    pgs:     1 active+clean
 
  io:
    recovery: 43 KiB/s, 0 objects/s

Ok. That’s it. We have a running cluster. It’s not doing much right now though, let’s change that.

Ceph Configuration

Create a CephFS

Alright. Since we want to actually access the storage somehow, we need a system that allows us to connect. For this, we will first create a Ceph filesystem.

This command will also create two pools. One for the metadata and one for actual files.

The default for the replicas is 3, meaning that we get 2 copies for every file we create. It’s a bit more complex than that, but I think that explains it simply enough. We could lose 2 nodes without data loss, but the cluster would be in a degraded state.

ceph-01 :: ~ » ceph fs volume create CEPH-FS-01

Let’s check if it has been created.

ceph-01 :: ~ » ceph fs ls
name: CEPH-FS-01, metadata pool: cephfs.CEPH-FS-01.meta, data pools: [cephfs.CEPH-FS-01.data ]

Change replication size

Like I said earlier, the default for the replications is 3. If you want to change that, use the following command.

ceph-01 :: ~ » ceph osd pool set cephfs.CEPH-FS-01.meta size 2

We can also change the minimum before the cluster stops IO completely.

fedora :: ~ » ceph osd pool set cephfs.CEPH-FS-01.meta min_size 1

Keep in mind, that this is not recommended.

Mount the CephFS

Generate minimal config

Ok. Now let’s see if we can mount the filesystem. I will use a “client” Fedora 39 system for this. But first, we need to create a minimal configuration for the client. Execute this on the ceph cluster host and copy the output.

ceph-01 :: ~ » ceph config generate-minimal-conf
# minimal ceph.conf for f581d60e-e2f8-11ee-8f4b-525400c55f38
[global]
        fsid = f581d60e-e2f8-11ee-8f4b-525400c55f38
        mon_host = [v2:10.10.0.31:3300/0,v1:10.10.0.31:6789/0] [v2:10.10.0.32:3300/0,v1:10.10.0.32:6789/0] [v2:10.10.0.33:3300/0,v1:10.10.0.33:6789/0]

Next, we switch to the client. Install the “ceph-common” package.

fedora :: ~ » sudo dnf install ceph-common

Now we need to paste the minimal config to /etc/ceph/ceph.conf.

fedora :: ~ » sudo vim /etc/ceph/ceph.conf
# minimal ceph.conf for f581d60e-e2f8-11ee-8f4b-525400c55f38
[global]
        fsid = f581d60e-e2f8-11ee-8f4b-525400c55f38
        mon_host = [v2:10.10.0.31:3300/0,v1:10.10.0.31:6789/0] [v2:10.10.0.32:3300/0,v1:10.10.0.32:6789/0] [v2:10.10.0.33:3300/0,v1:10.10.0.33:6789/0]

Create user

Ok. Back to the server. We need to create a user. Use the following command.

# Syntax
ceph-01 :: ~ » ceph fs authorize <filesystem-name> client.<username> / rw
# Example
ceph-01 :: ~ » ceph fs authorize CEPH-FS-01 client.bob / rw
[client.bob]
        key = AQACoPRl/ihvNBAAqBkigFRlb7A6zs8XlaT2hg==

Again, copy the output and enter it into the /etc/ceph/ceph.client.bob.keyring.

fedora :: ~ » sudo vim /etc/ceph/ceph.client.bob.keyring
[client.bob]
        key = AQACoPRl/ihvNBAAqBkigFRlb7A6zs8XlaT2hg==

Change the permission.

fedora :: ~ » sudo chmod 644 /etc/ceph/ceph.conf
fedora :: ~ » sudo chmod 600 /etc/ceph/ceph.client.bob.keyring

Mounting the Ceph Filesystem

Alright. One more step and we can mount the filesystem.

We need the fsid from the ceph cluster. For this execute the following command.

ceph-01 :: ~ » ceph fsid
f581d60e-e2f8-11ee-8f4b-525400c55f38

Now we can mount the filesystem on the client. Create a folder for the mount point and execute the mount command with the following parameters.

## Create a folder for the mountpoint
fedora :: ~ » sudo mkdir /mnt/CEPH
## Mount the filesystem
# Syntax
fedora :: ~ » sudo mount -t ceph <username>@<fsid>.<filesystem-name>=/ <mountpoint>
# Example
fedora :: ~ » sudo mount -t ceph user@f581d60e-e2f8-11ee-8f4b-525400c55f38.CEPH-FS-01=/ /mnt/CEPH/

Fantastic. It should be mounted. Let’s check.

fedora :: ~ » mount
...
user@f581d60e-e2f8-11ee-8f4b-525400c55f38.CEPH-FS-01=/ on /mnt/CEPH type ceph (rw,relatime,seclabel,name=user,secret=<hidden>,ms_mode=prefer-crc,acl,mon_addr=10.10.0.31:3300/10.10.0.32:3300/10.10.0.33:3300)

Actually. Since we only have one fsid entry in the ceph.conf file on the client, we could also just type this without the fsid.

fedora :: ~ » sudo mount -t ceph user@.CEPH-FS-01=/ /mnt/CEPH/

Testing the filesystem

Ok. I want to check the used space on the OSDs.

ceph-01 :: ~ » ceph osd status
 0  ceph-03  48.6M  9.94G      0        0       0        0   exists,up  
 1  ceph-02  44.6M  9.95G      0        0       0        0   exists,up  
 2  ceph-01  27.0M  9.96G      0        0       0        0   exists,up  
 3  ceph-03  27.4M  9.96G      0        0       0        0   exists,up  
 4  ceph-02  48.6M  9.94G      0        0       0        0   exists,up  
 5  ceph-01  49.1M  9.94G      0        0       0        0   exists,up  
 6  ceph-03  27.0M  9.96G      0        0       0        0   exists,up  
 7  ceph-02  27.4M  9.96G      0        0       0        0   exists,up  
 8  ceph-01  27.0M  9.96G      0        0       0        0   exists,up

Around 30MB. Alright, let’s create and copy a few files onto the CephFS from the client.

fedora :: ~ » dd if=/dev/zero of=/mnt/CEPH/testfile bs=100M count=1

Again. Let’s check the OSDs.

ceph-01 :: ~ » ceph osd status
ID  HOST      USED  AVAIL  WR OPS  WR DATA  RD OPS  RD DATA  STATE      
 0  ceph-03  92.6M  9.90G      0        0       0        0   exists,up  
 1  ceph-02  84.6M  9.91G      0        0       0        0   exists,up  
 2  ceph-01  92.6M  9.90G      0        0       0        0   exists,up  
 3  ceph-03  59.4M  9.93G      0        0       0        0   exists,up  
 4  ceph-02  72.6M  9.92G      0        0       0        0   exists,up  
 5  ceph-01  73.1M  9.92G      0        0       0        0   exists,up  
 6  ceph-03  51.0M  9.94G      0        0       0        0   exists,up  
 7  ceph-02  63.4M  9.93G      0        0       0        0   exists,up  
 8  ceph-01  55.0M  9.94G      0        0       0        0   exists,up 

It looks like, it distributed it nicely.

Adding additional managers

Ok. One more thing before I want to take a quick peek at the dashboard. We currently only have 2 managers in our 3 node cluster, and I would like to change that. I want to define a minimum of 3 managers in my cluster. For this, we will use the following command.

ceph-01 :: ~ » ceph orch apply mgr 3
Scheduled mgr update...

Once that’s done, doing its thing, we should see the mgr count go up. Having one active and two standbys.

ceph-01 :: ~ » ceph status
  cluster:
    id:     f581d60e-e2f8-11ee-8f4b-525400c55f38
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 (age 75m)
    mgr: ceph-01.bhovln(active, since 86m), standbys: ceph-02.tzstuj, ceph-03.rfjafn
    mds: 1/1 daemons up, 1 standby
    osd: 9 osds: 9 up (since 58m), 9 in (since 61m)
 
  data:
    volumes: 1/1 healthy
    pools:   3 pools, 273 pgs
    objects: 50 objects, 100 MiB
    usage:   649 MiB used, 89 GiB / 90 GiB avail
    pgs:     273 active+clean

Great.

Ceph Dashboard

Now I want to take a quick look at the Ceph Dashboard. Remember the output with the URL from the beginning. Paste that into a browser (https://10.10.0.31:8443) and you will get to the login screen. Log in with the auto-generated credentials, next it will ask you to change your password.

Once we are in, we can see all the options we have on the left. With the “Expand Cluster” button, we could add additional servers to our cluster.

Click on “Dashboard” and we get an overview of the current health of our cluster. The “Inventory”, used capacity, Cluster Utilization and so on.

Select the “Hosts”.

I have expanded the first host, so we can take a look at the information provided. We can see the list of physical disks in the server and the assigned osd, the daemons that are running on that host and so on. Here we could also add additional hosts to our cluster.

One last tab, the “File Systems”. Here we can see the pools that the filesystem belongs to, we can create subvolumes, check the connected clients and browse the directories.

The Dashboard looks overall very powerful.

Alright. That is it for today. I will keep poking at ceph for a while, looks fun but my tiny NUC seems a bit overwhelmed by ceph and keeps freezing up. But I haven’t lost any data yet. 🙂

Till next time.

Leave a Reply