ceph(nautilus)集群安装
环境
CentOS Linux release 7.7.1908 (Core)
192.168.206.100 admin(admin-node)
192.168.206.101 node0(osd.0、mon、mds、mgr)
192.168.206.102 node1(osd.1、mon)
192.168.206.103 node2(osd.2、mon)
步骤一: Host修改(所有节点执行)
192.168.206.100 admin
192.168.206.101 node0
192.168.206.102 node1
192.168.206.103 node2
步骤二: 更新软件源(所有节点执行)
sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
cat>/etc/yum.repos.d/ceph.repo<<EOF
[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirrors.163.com/ceph/rpm-nautilus/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=http://mirrors.163.com/ceph/keys/release.asc
priority=1
EOF
yum -y update
步骤三: 安装ceph-deploy命令(admin管理节点执行)
yum -y install ceph-deploy
步骤四: 安装NTP并同步,关闭防火墙(所有节点执行)
sudo yum install ntp ntpdate ntp-doc
sudo ntpdate 0.cn.pool.ntp.org
#关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
#SELINUX 设置
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
步骤五：创建普通用户cephd(ceph-deploy 必须以普通用户登录 Ceph 节点,所有节点执行)
# 创建 ceph 特定用户
sudo useradd -d /home/cephd -m cephd
sudo passwd cephd
# 添加 sudo 权限
echo "cephd ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephd
sudo chmod 0440 /etc/sudoers.d/cephd
步骤六: 免密登录
#cephd用户登录执行ssh-keygen,提示输入密码时，直接回车，因为它需要免密码登录到各个节点
#将公钥复制到 node0、node1、node2 节点
ssh-copy-id cephd@node0
ssh-copy-id cephd@node1
ssh-copy-id cephd@node2
#修改 ceph-deploy 管理节点上的 ~/.ssh/config,新增如下内容
Host node0
Hostname node0
User cephd
Host node1
Hostname node1
User ceph
Host node2
Hostname node2
User cephd
#修改权限
sudo chmod 600 ~/.ssh/config
步骤六: 集群创建
# 创建执行目录
$ mkdir ~/ceph-cluster && cd ~/ceph-cluster
# 添加mon节点
$ ceph-deploy new node0
#安装ceph
ceph-deploy install admin node0 node1 node2
#初始化monitor且收集密钥
ceph-deploy mon create-initial
#执行上面命令查看ceph-cluster目录,有如下文件
ceph.client.admin.keyring
ceph.bootstrap-mgr.keyring
ceph.bootstrap-osd.keyring
ceph.bootstrap-mds.keyring
ceph.bootstrap-rgw.keyring
ceph.bootstrap-rbd.keyring
ceph.bootstrap-rbd-mirror.keyring
#将配置拷贝到其他节点
ceph-deploy admin node0 node1 node2
#创建mgr,从Luminous开始mgr需要单独部署
ceph-deploy mgr create node0
#查看节点驱动
ceph-deploy disk list node0
#创建OSD(osd是存储数据的地方)
ceph-deploy osd create --data /dev/sdb node0
ceph-deploy osd create --data /dev/sdb node1
ceph-deploy osd create --data /dev/sdb node2
# 检查集群健康状况
ssh node0 sudo ceph -s
步骤七: 扩展集群
# admin节点ceph.conf中增加如下配置
public_network = 192.168.206.0/24
# 同步最新配置到其他节点
ceph-deploy admin node0 node1 node2
#增加monitor节点
ceph-deploy mon add node1
ceph-deploy mon add node2
步骤八: ceph文件系统cephfs
#创建元数据服务器
ceph-deploy mds create node0
#激活元数据服务器
ceph osd pool create cephfs_data 32
ceph osd pool create cephfs_meta 32
ceph fs new mycephfs cephfs_meta cephfs_data
查看最终集群状态
[cephd@admin ceph-cluster]$ ssh node0 sudo ceph -s
cluster:
id: 338c8a7c-231e-4add-850d-4e5f7ec7bf13
health: HEALTH_OK
services:
mon: 3 daemons, quorum node0,node1,node2 (age 81m)
mgr: node0(active, since 116m)
mds: mycephfs:1 {0=node0=up:active}
osd: 3 osds: 3 up (since 110m), 3 in (since 110m)
data:
pools: 2 pools, 64 pgs
objects: 22 objects, 2.2 KiB
usage: 3.0 GiB used, 24 GiB / 27 GiB avail
pgs: 64 active+clean
挂载文件系统
#查看admin的key
sudo ceph auth get client.admin
#创建本地目录
mkdir -p /mnt/mycephfs2
#挂载
sudo mount -t ceph 192.168.206.101:6789,192.168.206.102:6789,192.168.206.103:6789:/ /mnt/mycephfs2 -o name=admin,secret=AQChMeddoT5pLhAAsBj3u9xshOkvXcchlmBW/w==
#查看挂载
[root@node0 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 484M 0 484M 0% /dev
tmpfs 496M 0 496M 0% /dev/shm
tmpfs 496M 6.8M 489M 2% /run
tmpfs 496M 0 496M 0% /sys/fs/cgroup
/dev/mapper/centos-root 17G 1.9G 16G 12% /
/dev/sda1 1014M 182M 833M 18% /boot
tmpfs 496M 24K 496M 1% /var/lib/ceph/osd/ceph-0
192.168.206.101:6789,192.168.206.102:6789,192.168.206.103:6789:/ 7.6G 0 7.6G 0% /mnt/mycephfs2
tmpfs 100M 0 100M 0% /run/user/0
#验证
在同样挂载CephFS服务器上的其他挂载点,创建文件或目录,其他挂载点可以看到。
错误解决
RuntimeError: NoSectionError: No section: 'ceph'
解决办法:
# yum remove ceph-release
把这个东西卸了，应该是这个的版本不兼容 亲测有效。
File "/usr/lib/python2.7/site-packages/ceph_deploy/cli.py
解决办法:
yum install python-setuptools
RuntimeError: Failed to execute command: ceph --version
解决办法:
因为超时,在各个节点都执行 yum -y install ceph
参考:
国内ceph源：http://xiaoquqi.github.io/blog/2016/06/19/deploy-ceph-using-china-mirror/
pg、pgp、osd、object的关系 https://www.cnblogs.com/luohaixian/p/8087591.html
pg数调整 https://www.cnblogs.com/dengchj/p/10003534.html
tangz
fa98a6337b
文档更新