ceph 16.2.15(Pacific)编译

技术分享 1年前 (2025-02-10) 0 999+

以下流程在ubuntu22.04 和 openEuler20.03 都实际操作过

获取ceph源码

从https://docs.ceph.com/en/latest/releases/找到pacific版本对应的tag为16.2.15

git clone https://github.com/ceph/ceph.git  # 查看所有tag git ls-remote --tags origin |grep 16.2. 618f440892089921c3e944a991122ddc44e60516        refs/tags/v16.2.15^{}  # 根据git log的commit找到pacifc的release tag为v16.2.15 commit 618f440892089921c3e944a991122ddc44e60516 (HEAD -> ceph-pacific-learning, tag: v16.2.15, tag: ls, origin/ceph-pacific-learning) Author: Ceph Release Team <ceph-maintainers@ceph.io> Date:   Mon Feb 26 19:21:09 2024 +0000      16.2.15          Signed-off-by: Ceph Release Team <ceph-maintainers@ceph.io> # 或者 git show v16.2.15，可以确认即可   # 切换到pacific的release tag git checkout v16.2.15  # 这个tag是一个快照，不可修改，所以创建我们自己的分支 git checkout -b ceph-pacific-learning

编译

拉取submodule网络问题

执行命令git submodule update --init --recursive --progress更新子模块

如果子模块拉取失败：

方法一：这个地址会记录最新国内可用的github镜像地址，https://fcp7.com/github-mirror-daily-updates.html，修改ceph顶层目录下的.gitmodule，例如对于[submodule "src/erasure-code/jerasure/gf-complete"]，将url = https://github.com/ceph/gf-complete.git部分替换为镜像的地址如下：

[submodule "ceph-object-corpus"] 	path = ceph-object-corpus 	url = https://gitclone.com/ceph/ceph-object-corpus.git [submodule "src/civetweb"] 	path = src/civetweb 	url = https://gitclone.com/ceph/civetweb [submodule "src/erasure-code/jerasure/jerasure"] 	path = src/erasure-code/jerasure/jerasure 	url = https://gitclone.com/ceph/jerasure.git 	branch = v2-ceph [submodule "src/erasure-code/jerasure/gf-complete"] 	path = src/erasure-code/jerasure/gf-complete 	url = https://gitclone.com/ceph/gf-complete.git 	branch = v3-ceph [submodule "src/rocksdb"] 	path = src/rocksdb 	url = https://gitclone.com/ceph/rocksdb 	ignore = dirty [submodule "ceph-erasure-code-corpus"] 	path = ceph-erasure-code-corpus 	url = https://gitclone.com/ceph/ceph-erasure-code-corpus.git

执行git submodule sync --recursive更新git拉取的地址，再次执行git submodule update --init --recursive --progress

方法二：如果方法一中的镜像源没有地址，根据拉取失败的提示fatal: clone of 'https://github.com/spdk/dpdk.git' into submodule path '/data/ceph-learning/src/spdk/dpdk' failed，手动多拉取几次git clone https://github.com/spdk/dpdk.git /data/ceph-learning/src/spdk/dpdk --progress，或者将https换成git执行git clone git://github.com/spdk/dpdk.git /data/ceph-learning/src/spdk/dpdk --progress
方法三：那种方式上网，解决99%的编译问题

安装依赖

./install-deps.sh

如果是openEuler系统

index 7e137d687f1..b22d45f409d 100755 --- a/install-deps.sh +++ b/install-deps.sh @@ -410,7 +410,7 @@ else         $SUDO env DEBIAN_FRONTEND=noninteractive apt-get -y remove ceph-build-deps         if [ "$control" != "debian/control" ] ; then rm $control; fi          ;; -    centos|fedora|rhel|ol|virtuozzo) +    centos|fedora|rhel|ol|virtuozzo|openEuler) # 修改这里          builddepcmd="dnf -y builddep --allowerasing"          echo "Using dnf to install dependencies"          case "$ID" in

do_cmake.sh

cmake -LH可以看到cmake可用参数和描述信息，例如

// ceph-mgr is enabled WITH_MGR:BOOL=ON  // Build the mgr/dashboard frontend using `npm` WITH_MGR_DASHBOARD_FRONTEND:BOOL=ON  // build OCF-compliant cluster resource agent WITH_OCF:BOOL=OFF

dashboard依赖node、ng的一堆包，系统版本不同依赖问题也不同，编译会很麻烦，如果用不到它可以关掉：

./do_cmake.sh -DWITH_MGR_DASHBOARD_FRONTEND=OFF

如果需要dashboard，直接执行./do_cmake.sh后，到build目录下，按照下面的编译dashboard配置后，再回来执行make

编译

执行./do_cmake.sh后会在ceph源码顶层目录下生成一个build目录

cd build make -j16

Tips：make过程中也会下载一些软件包，没有那种方式上网可能会导致超时失败，失败后可以继续make -j16增量编译

vstart启动

要使用vstart启动ceph，还需要编译相关环境，在build目录下

make vstart

MON=1 OSD=2 MGR=1 RGW=1 NFS=1 ../src/vstart.sh -d -n -x

根据输出提示在shell中EXPORT

export PYTHONPATH=/data/ceph-learning/src/pybind:/data/ceph-learning/build/lib/cython_modules/lib.3:/data/ceph-learning/src/python-common:$PYTHONPATH export LD_LIBRARY_PATH=/data/ceph-learning/build/lib:$LD_LIBRARY_PATH export PATH=/data/ceph-learning/build/bin:$PATH alias cephfs-shell=/data/ceph-learning/src/tools/cephfs/shell/cephfs-shell CEPH_DEV=1

root@yielde:/data/ceph-learning/build# ceph -s 2025-02-09T18:23:21.560+0800 ffff73eae120 -1 WARNING: all dangerous and experimental features are enabled.  2025-02-09T18:23:21.568+0800 ffff73eae120 -1 WARNING: all dangerous and experimental features are enabled.    cluster:     id:     22b8a5ef-b731-43d6-9e5a-e3c3e77cad68     health: HEALTH_OK    services:     mon: 1 daemons, quorum a (age 3m)     mgr: x(active, since 2m)     mds: 1/1 daemons up, 2 standby     osd: 2 osds: 2 up (since 3m), 2 in (since 3m)     rgw: 1 daemon active (1 hosts, 1 zones)    data:     volumes: 1/1 healthy     pools:   8 pools, 321 pgs     objects: 229 objects, 20 KiB     usage:   2.0 GiB used, 200 GiB / 202 GiB avail     pgs:     321 active+clean    io:     client:   85 B/s rd, 0 op/s rd, 0 op/s wr

问题

无法访问dashboard，ceph -s输出Module 'dashboard' has failed: No module named 'routes'，缺python包pip3 install routes即可
启动cephfs可能会失败缺包，手动安装

Requirements ------------  The following packages are required to enable CephFS and RGW exports with nfs-ganesha:  -  ``nfs-ganesha``, ``nfs-ganesha-ceph``, ``nfs-ganesha-rados-grace`` and    ``nfs-ganesha-rados-urls`` packages (version 3.3 and above)  Ganesha Configuration Hierarchy

编译dashboard

如果使用-DWITH_MGR_DASHBOARD_FRONTEND=OFF关了dashboard的话可以忽略这个

安装nodejs

方法一下载编译好的源码包

wget https://nodejs.org/dist/v16.14.2/node-v16.14.2-linux-x64.tar.xz xz -d node-v16.14.2-linux-x64.tar.xz tar -xvf node-v16.14.2-linux-x64.tar   mkdir /usr/local/soft mv node-v16.14.2-linux-x64 /usr/local/soft  ln -s /usr/local/soft/node-v16.14.2-linux-x64/bin/node /usr/local/bin/node ln -s /usr/local/soft/node-v16.14.2-linux-x64/bin/npm /usr/local/bin/npm  npm config set registry https://registry.npmmirror.com npm install -g @angular/cli@16.2.0 ln -s /usr/local/soft/node-v16.14.2-linux-x64/lib/node_modules/@angular/cli/bin/ng.js /usr/local/bin/ng

方法二 nvm安装node（推荐）

# 复制脚本到主机 https://github.com/nvm-sh/nvm/blob/master/install.sh  bash install.sh # 重新打开终端 nvm ls-remote --lts  nvm install v16.14.2 npm -g install @angular/cli@16.2.10

编译nodeenv

旧版本的ceph编译dashboard会有nodejs和angular/cli版本冲突的问题

# /data/ceph-learning换成自己的ceph源码顶层目录 cd /data/ceph-learning/src/pybind/mgr/dashboard && vim CMakeLists.txt  # 修改 COMMAND ${mgr-dashboard-nodeenv-dir}/bin/nodeenv --verbose ${node_mirror_opt} -p --node=12.18.2 # 为 COMMAND ${mgr-dashboard-nodeenv-dir}/bin/nodeenv --verbose ${node_mirror_opt} -p --node=16.14.2  cd /data/ceph-learning/build  root@yielde:/data/ceph-learning/build# make mgr-dashboard-nodeenv  Installing collected packages: nodeenv Successfully installed nodeenv-1.9.1  * Install prebuilt node (16.14.2) ..... done.  * Appending data to /data/ceph-learning/build/src/pybind/mgr/dashboard/node-env/bin/activate  * Appending data to /data/ceph-learning/build/src/pybind/mgr/dashboard/node-env/bin/activate.fish Built target mgr-dashboard-nodeenv

其他

boost下载慢

生成build后，可以先下载好 https://download.ceph.com/qa/boost_1_73_0.tar.bz2，放到/data/ceph-learning/build/boost/src目录下

root@yielde:/data/ceph-learning/build/boost/src# ls -lh total 105M drwxr-xr-x 2 root root 4.0K Feb  9 14:38 Boost -rw-r--r-- 1 root root 105M Feb  9 15:27 boost_1_73_0.tar.bz2 drwxr-xr-x 2 root root 4.0K Feb  9 14:55 Boost-stamp