.. _cephfs_add_remote_mds:

.. note::
   It is highly recommended to use :doc:`/cephadm/index` or another Ceph
   orchestrator for setting up the ceph cluster. Use this approach only if you
   are setting up the ceph cluster manually. If one still intends to use the
   manual way for deploying MDS daemons, :doc:`/cephadm/services/mds/` can
   also be used.

============================
 Deploying Metadata Servers
============================

Each CephFS file system requires at least one MDS. The cluster operator will
generally use their automated deployment tool to launch required MDS servers as
needed.  Rook and ansible (via the ceph-ansible playbooks) are recommended
tools for doing this. For clarity, we also show the systemd commands here which
may be run by the deployment technology if executed on bare-metal.

See `MDS Config Reference`_ for details on configuring metadata servers.


Provisioning Hardware for an MDS
================================

The present version of the MDS is single-threaded and CPU-bound for most
activities, including responding to client requests. An MDS under the most
aggressive client loads uses about 2 to 3 CPU cores. This is due to the other
miscellaneous upkeep threads working in tandem.

Even so, it is recommended that an MDS server be well provisioned with an
advanced CPU with sufficient cores. Development is on-going to make better use
of available CPU cores in the MDS; it is expected in future versions of Ceph
that the MDS server will improve performance by taking advantage of more cores.

The other dimension to MDS performance is the available RAM for caching. The
MDS necessarily manages a distributed and cooperative metadata cache among all
clients and other active MDSs. Therefore it is essential to provide the MDS
with sufficient RAM to enable faster metadata access and mutation. The default
MDS cache size (see also :doc:`/cephfs/cache-configuration`) is 4GB. It is
recommended to provision at least 8GB of RAM for the MDS to support this cache
size.

Generally, an MDS serving a large cluster of clients (1000 or more) will use at
least 64GB of cache. An MDS with a larger cache is not well explored in the
largest known community clusters; there may be diminishing returns where
management of such a large cache negatively impacts performance in surprising
ways. It would be best to do analysis with expected workloads to determine if
provisioning more RAM is worthwhile.

In a bare-metal cluster, the best practice is to over-provision hardware for
the MDS server. Even if a single MDS daemon is unable to fully utilize the
hardware, it may be desirable later on to start more active MDS daemons on the
same node to fully utilize the available cores and memory. Additionally, it may
become clear with workloads on the cluster that performance improves with
multiple active MDS on the same node rather than a single overloaded MDS.

Finally, be aware that CephFS is a highly-available file system by supporting
standby MDS (see also :ref:`mds-standby`) for rapid failover. To get a real
benefit from deploying standbys, it is usually necessary to distribute MDS
daemons across at least two nodes in the cluster. Otherwise, a hardware failure
on a single node may result in the file system becoming unavailable.

Co-locating the MDS with other Ceph daemons (hyperconverged) is an effective
and recommended way to accomplish this so long as all daemons are configured to
use available hardware within certain limits.  For the MDS, this generally
means limiting its cache size.


.. _manual-mds:

Adding an MDS
=============


In the below instructions, ``{id}`` is an arbitrary name, such as the hostname of the machine.

#. Create the mds data directory.

   .. code:: bash

       mkdir -p /var/lib/ceph/mds/{cluster-name}-{id}

#. Create and import the keyring.

   .. code:: bash

       ceph auth get-or-create mds.{id} mon "allow profile mds" mgr "allow profile mds" osd "allow rw tag cephfs *=*" mds "allow" > /var/lib/ceph/mds/{cluster-name}-{id}/keyring

#. Start the daemon the manual way.

   .. code:: bash

       ceph-mds --cluster {cluster-name} -i {id} -m {mon-hostname}:{mon-port} [-f]

#. Alternatively, start the daemon using a service manager.

   .. code:: bash

       sudo systemctl start ceph-mds@${id}

#. If starting the daemon fails with this error:

   ::

       mds.-1.0 ERROR: failed to authenticate: (22) Invalid argument

   Then make sure you do not have a keyring set in ceph.conf in the global
   section; move it to the client section; or add a keyring setting specific to
   this mds daemon. And verify that you see the same key in the mds data
   directory and ``ceph auth get mds.{id}`` output.

#. Optionally, configure the file system the MDS should join (:ref:`mds-join-fs`):

   ::

       $ ceph config set mds.${id} mds_join_fs ${fs}

#. Now you are ready to :ref:`create-fs`.


Removing an MDS
===============

If you have a metadata server in your cluster that you'd like to remove, you may use
the following method.

#. (Optionally:) Create a new replacement Metadata Server. If there are no
   replacement MDS to take over once the MDS is removed, the file system will
   become unavailable to clients.  If that is not desirable, consider adding a
   metadata server before tearing down the metadata server you would like to
   take offline.

#. Stop the MDS to be removed. ::

	$ sudo systemctl stop ceph-mds@${id}

   The MDS will automatically notify the Ceph monitors that it is going down.
   This enables the monitors to perform instantaneous failover to an available
   standby, if one exists. It is unnecessary to use administrative commands to
   effect this failover, e.g. through the use of ``ceph mds fail mds.${id}``.

#. Remove the ``/var/lib/ceph/mds/ceph-${id}`` directory on the MDS. ::

	$ sudo rm -rf /var/lib/ceph/mds/ceph-${id}


.. note:: When an active MDS either has health warning MDS_TRIM or
   MDS_CACHE_OVERSIZED, confirmation flag (--yes-i-really-mean-it)
   needs to be passed, else the command will fail. It is not recommended to
   restart an MDS which has these warnings since slow recovery at restart may
   lead to more problems.

.. _MDS Config Reference: ../mds-config-ref
