swift分布式存储多节点部署
1、机器 192.168.1.211Proxy Node 192.168.1.212Storage Node 192.168.1.213Storage Node 192.168.1.214Storage Node 系统为SLES11sp1 2、配置软件源 因为公司服务器无法连外网,所以配置局域网源和本地源来搭建环境 上传ISO镜像文件到各台机器 SLES-11-SP4-DVD-x86_64-GM-DVD1.iso 每台机器挂载镜像,配置本地源 # mkdir /mnt/cdrom # mount -o loop SLES-11-SP4-DVD-x86_64-GM-DVD1.iso /mnt/cdrom # mount /home/SLES-11-SP4-DVD-x86_64-GM-DVD1.iso on /mnt/cdrom type iso9660 (ro) # vi /etc/zypp/repos.d/Local-iso.repo [local-iso] name=local iso repo enabled=1 autorefresh=1 baseurl=file:/mnt/cdrom/ type=yast2 gpgcheck=0 http局域网源 /etc/zypp/repos.d # cat Icehouse.repo [Icehouse] name=Icehouse enabled=1 autorefresh=1 baseurl=http://192.168.1.206:8080/download.opensuse.org/repositories/Cloud/OpenStack/Icehouse/SLE_11_SP3/ type=rpm-md 3、创建swift用户并配置权限 创建组 # groupadd swift 创建用户 # useradd -g swift -d /home/swift -s /bin/bash -m swift 为swift赋予sudo # vi /etc/sudoers 在root ALL=(ALL) ALL一行下添加 swift ALL=(ALL) NOPASSWD:ALL # passwd swift 5、基础配置 注意关闭SElinux和防火墙 分别在四台机器上安装依赖 # su - swift sudo zypper install curlgcc memcached rsync sqlite3 xfsprogs Git-core libffi-dev python-setuptools sudo zypper install python-coverage python-dev python-nose python-simplejson python-xattr python-eventlet python-greenlet python-pastedeploy python-netifaces python-pip python-dnspython python-mock python-swiftclient openstack-swift 注: python-swiftclient是客户端工具,可以在服务器上安装使服务器只充当客户端。 每个节点上配置 ~> sudo chown -R swift:swift /etc/swift ~> /etc/swift> cat swift.conf [swift-hash] #random unique strings that can neverchange(DO NOT LOSE) swift_hash_path_prefix = 'od -t x8 -N 8 -An < /dev/random' swift_hash_path_suffix = 'od -t x8 -N 8 -An < /dev/random' 6、安装配置proxy sudo zypper install openstack-swift-proxy memcached 启动memcached /usr/sbin # # ./memcached -d -m 10 -u swift -l 192.168.1.211 11211 -c 256 -P /tmp/memcached/pid 注: memcached为什么这样启动,而不是在配置文件里写入是因为我发现写入配置文件后启动服务并没有生效。 /usr/sbin # service memcached status Checking for service memcached running /usr/sbin # netstat -an | grep 11211 tcp 0 0 192.168.1.211:11211 0.0.0.0:* LISTEN udp 0 0 192.168.1.211:11211 0.0.0.0:* 查看配置后的文件 /etc/swift> grep -v "^#" /etc/swift/proxy-server.conf | grep -v "^$" [DEFAULT] user = swift bind_port = 8090 workers = 8 [pipeline:main] pipeline= healthcheck proxy-logging cache tempauth proxy-logging proxy-server [app:proxy-server] use = egg:swift#proxy allow_account_management = ture account_autocreate = true [filter:tempauth] use = egg:swift#tempauth user_system_root= testpass .admin http://192.168.1.211:8090/v1/AUTH_system user_admin_admin = admin .admin .reseller_admin user_test_tester = testing .admin user_test2_tester2 = testing2 .admin user_test_tester3 = testing3 [filter:healthcheck] use = egg:swift#healthcheck [filter:cache] use = egg:swift#memcache memcache_servers = 192.168.1.211:11211 [filter:ratelimit] use = egg:swift#ratelimit [filter:domain_remap] use = egg:swift#domain_remap [filter:catch_errors] use = egg:swift#catch_errors [filter:cname_lookup] use = egg:swift#cname_lookup [filter:staticweb] use = egg:swift#staticweb [filter:tempurl] use = egg:swift#tempurl [filter:formpost] use = egg:swift#formpost [filter:name_check] use = egg:swift#name_check [filter:list-endpoints] use = egg:swift#list_endpoints [filter:proxy-logging] use = egg:swift#proxy_logging [filter:bulk] use = egg:swift#bulk [filter:container-quotas] use = egg:swift#container_quotas [filter:slo] use = egg:swift#slo [filter:dlo] use = egg:swift#dlo [filter:account-quotas] use = egg:swift#account_quotas [filter:gatekeeper] use = egg:swift#gatekeeper [filter:container_sync] use = egg:swift#container_sync 创建account、container以及object rings > cd/etc/swift /etc/swift> sudo swift-ring-builder account.builder create 18 3 1 /etc/swift> sudo swift-ring-builder container.builder create 18 3 1 /etc/swift> sudo swift-ring-builder object.builder create 18 3 1 注: 18代表2的18次幂,这个数字取决与你希望一个ring中会有多少个partition,3代表object的副本数,1代表至少一个小时后才能被移动。 让ring记录每个storage存储设备 /etc/swift> export ZONE=1 /etc/swift> export STORAGE_LOCAL_NET_IP=192.168.1.212 /etc/swift> export WEIGHT=100 /etc/swift> export DEVICE=sdb1 /etc/swift> sudo swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT WARNING: No region specified for z1-192.168.1.212:6002/sdb1. Defaulting to region 1. Device d4r1z1-192.168.1.212:6002R192.168.1.212:6002/sdb1_"" with 100.0 weight got id 4 /etc/swift> sudo swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT WARNING: No region specified for z1-192.168.1.212:6001/sdb1. Defaulting to region 1. Device d0r1z1-192.168.1.212:6001R192.168.1.212:6001/sdb1_"" with 100.0 weight got id 0 /etc/swift> sudo swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT WARNING: No region specified for z1-192.168.1.212:6000/sdb1. Defaulting to region 1. Device d0r1z1-192.168.1.212:6000R192.168.1.212:6000/sdb1_"" with 100.0 weight got id 0 /etc/swift> export ZONE=2 /etc/swift> export STORAGE_LOCAL_NET_IP=192.168.1.213 /etc/swift> sudo swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT WARNING: No region specified for z2-192.168.1.213:6002/sdb1. Defaulting to region 1. Device d1r1z2-192.168.1.213:6002R192.168.1.213:6002/sdb1_"" with 100.0 weight got id 1 /etc/swift> sudo swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT WARNING: No region specified for z2-192.168.1.213:6001/sdb1. Defaulting to region 1. Device d1r1z2-192.168.1.213:6001R192.168.1.213:6001/sdb1_"" with 100.0 weight got id 1 /etc/swift> sudo swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT WARNING: No region specified for z2-192.168.1.213:6000/sdb1. Defaulting to region 1. Device d1r1z2-192.168.1.213:6000R192.168.1.213:6000/sdb1_"" with 100.0 weight got id 1 /etc/swift> export ZONE=3 /etc/swift> export STORAGE_LOCAL_NET_IP=192.168.1.214 /etc/swift> sudo swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT WARNING: No region specified for z3-192.168.1.214:6002/sdb1. Defaulting to region 1. Device d12r1z3-192.168.1.214:6002R192.168.1.214:6002/sdb1_"" with 100.0 weight got id 12 /etc/swift> sudo swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT WARNING: No region specified for z3-192.168.1.214:6001/sdb1. Defaulting to region 1. Device d2r1z3-192.168.1.214:6001R192.168.1.214:6001/sdb1_"" with 100.0 weight got id 2 /etc/swift> sudo swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT WARNING: No region specified for z3-192.168.1.214:6000/sdb1. Defaulting to region 1. Device d2r1z3-192.168.1.214:6000R192.168.1.214:6000/sdb1_"" with 100.0 weight got id 2 平衡ring /etc/swift> sudo swift-ring-builder account.builder rebalance Reassigned 262144 (100.00%) partitions. Balance is now 0.00. /etc/swift> sudo swift-ring-builder container.builder rebalance Reassigned 262144 (100.00%) partitions. Balance is now 0.00. /etc/swift> sudo swift-ring-builder object.builder rebalance Reassigned 262144 (100.00%) partitions. Balance is now 0.00. 如果操作失误也可以移除 /etc/swift> sudo swift-ring-builder account.builder remove z1-192.168.1.212:6020 d0r1z1-192.168.1.212:6020R192.168.1.212:6020/sdb1_"" marked for removal and will be removed next rebalance. 不过我发现最后一个无法删除,只能在新建另外一个,然后删除想要删除的 删除后需要rebalance /etc/swift> sudo swift-ring-builder account.builder rebalance Reassigned 262144 (100.00%) partitions. Balance is now 0.00. 当然也可以加入设备名 /etc/swift> sudo swift-ring-builder account.builder remove z1-192.168.1.212:6020/sdb1 d0r1z1-192.168.1.212:6020R192.168.1.212:6020/sdb1_"" marked for removal and will be removed next rebalance. 拷贝ring到其他所有节点 所有节点 sudo chown -R swift:swift /etc/swift scp swift/*.ring.gz swift@192.168.1.212:/etc/swift scp swift/*.ring.gz swift@192.168.1.213:/etc/swift scp swift/*.ring.gz swift@192.168.1.214:/etc/swift 开启proxy /etc/swift> sudo swift-init proxy start 需要注意的是对于account.builder 、container.builder以及object.builder的命名一定要严格按照小写,并且名字不能修改,不然会出现类似下面的错误 sudo swift-init proxy start Starting proxy-server...(/etc/swift/proxy-server.conf) WARNING: SSL should only be enabled for testing purposes. Use external SSL termination for a production deployment. Traceback (most recent call last): File "/usr/bin/swift-proxy-server",line 23,in <module> sys.exit(run_wsgi(conf_file,'proxy-server',default_port=8080,**options)) ...... IOError: [Errno 2] No such file or directory: '/etc/swift/container.ring.gz' 因为/etc/swift/目录下是Container.ring.gz 7、安装配置storage节点(三个节点安装配置) sudo zypper install openstack-swift-account openstack-swift-container openstack-swift-object python-xml sudo mkdir -p /srv/node/sdb1 sudo chown swift:swift /srv/node/sdb1 设备配置xfs卷 机器为底层lvm逻辑卷格式 # lvcreate -n swiftlv -L 10G vg0 # mkfs.xfs /dev/mapper/vg0-swiftlv 把挂载信息写入配置文件 # vi /etc/fstab /dev/mapper/vg0-swiftlv /srv/node/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0 重新加载配置文件 # mount -a 查看 # mount /dev/mapper/vg0-swiftlv on /srv/node/sdb1 type xfs (rw,noatime,logbufs=8) 按实际情况修改rsync配置文件 /etc> sudo vi rsyncd.conf uid = swift gid = swift log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid address = 192.168.1.212 [account] max connections = 2 path = /srv/node/ read only = False lock file = /var/lock/account.lock [container] max connections = 2 path = /srv/node/ read only = False lock file = /var/lock/container.lock [object] max connections = 2 path = /srv/node/ read only = False lock file = /var/lock/object.lock 其他两台注意: address = 192.168.1.213 address = 192.168.1.214 /etc> sudo vi default/rsync RSYNC_ENABLE=true # service rsyncd start /etc>rsync rsync://pub@192.168.1.212 account container object /etc> rsync rsync://pub@192.168.1.213 account container object /etc> rsync rsync://pub@192.168.1.214 account container object 修改account、container以及object配置文件 ~> grep -v "^#" /etc/swift/account-server.conf |grep -v "^$" [DEFAULT] bind_ip = 192.168.1.212 bind_port = 6002 swift_dir = /etc/swift devices = /srv/node workers = 2 [pipeline:main] pipeline = account-server [app:account-server] use = egg:swift#account [filter:healthcheck] use = egg:swift#healthcheck [filter:recon] use = egg:swift#recon [account-replicator] [account-auditor] [account-reaper] ~> grep -v "^#" /etc/swift/container-server.conf |grep -v "^$" [DEFAULT] bind_port = 6001 workers = 2 [pipeline:main] pipeline = container-server [app:container-server] use = egg:swift#container [filter:healthcheck] use = egg:swift#healthcheck [filter:recon] use = egg:swift#recon [container-replicator] [container-updater] [container-auditor] [container-sync] ~> grep -v "^#" /etc/swift/object-server.conf |grep -v "^$" [DEFAULT] bind_port = 6000 workers = 2 [pipeline:main] pipeline = object-server [app:object-server] use = egg:swift#object [filter:healthcheck] use = egg:swift#healthcheck [filter:recon] use = egg:swift#recon [object-replicator] [object-updater] [object-auditor] 拷贝配置文件到其他两台机器,记得其他两台IP做修改 启动存储服务 etc/swift> sudo swift-init all start Starting container-updater...(/etc/swift/container-server.conf) Starting account-auditor...(/etc/swift/account-server.conf) Starting object-replicator...(/etc/swift/object-server.conf) Unable to locate config for proxy-server Starting container-replicator...(/etc/swift/container-server.conf) Starting object-auditor...(/etc/swift/object-server.conf) Starting object-expirer...(/etc/swift/object-expirer.conf) Starting container-auditor...(/etc/swift/container-server.conf) Starting container-server...(/etc/swift/container-server.conf) Starting account-server...(/etc/swift/account-server.conf) Starting account-reaper...(/etc/swift/account-server.conf) Starting container-sync...(/etc/swift/container-server.conf) Starting account-replicator...(/etc/swift/account-server.conf) Starting object-updater...(/etc/swift/object-server.conf) Starting object-server...(/etc/swift/object-server.conf) 可以使用curl来和存储进行交互 curl -k -v -H'X-Storage-User: system:root' -H 'X-Storage-Pass: testpass' https://192.168.1.211:8080/auth/v1.0 curl -k -v -H'X-Storage-User: system:root' -H 'X-Storage-Pass: testpass' http://192.168.1.211:8080/auth/v1.0 如果curl出错: swift@x-shcs-creditcard-v01:~> curl -k -v -H'X-Storage-User: system:root' -H 'X-Storage-Pass: testpass' https://192.168.1.211:8080/auth/v1.0 * Protocol "https" not supported or disabled in libcurl * Closing connection -1 curl: (1) Protocol "https" not supported or disabled in libcurl 则需要编译curl # tar jxf curl-7.19.0.tar.bz2 # cd curl-7.19.0/ # ./configure --prefix=/usr/local/curl --with-ssl=/usr/local/openssl curl version: 7.19.0 Host setup: x86_64-unknown-linux-gnu Install prefix: /usr/local Compiler: gcc SSL support: enabled (OpenSSL) SSH support: no (--with-libssh2) zlib support: enabled krb4 support: no (--with-krb4*) GSSAPI support: no (--with-gssapi) SPNEGO support: no (--with-spnego) c-ares support: no (--enable-ares) ipv6 support: enabled IDN support: no (--with-libidn) Build libcurl: Shared=yes,Static=yes Built-in manual: no (--enable-manual) Verbose errors: enabled (--disable-verbose) SSPI support: no (--enable-sspi) ca cert bundle: no ca cert path: /etc/ssl/certs/ LDAP support: no (--enable-ldap / --with-ldap-lib / --with-lber-lib) LDAPS support: no (--enable-ldaps) # make && make install /usr/local/bin # mv curl curl.bak /usr/bin # mv curl curl.bak # ln -s /usr/local/curl/bin/curl /usr/bin/curl 检测swift是否正常工作,出现像下面的输出就说明正常 /etc/swift> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass stat Account: AUTH_system Containers: 0 Objects: 0 Bytes: 0 Content-Type: text/plain; charset=utf-8 X-Timestamp: 1490690568.46981 X-Trans-Id: tx59301de1e9244b70a2065-0058da2208 X-Put-Timestamp: 1490690568.46981 新建一级子目录 swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass post container1 忘存储目录上传文件 swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container1 /etc/swift/*.ring.gz 查看存储里的目录和文件 swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list 下载存储目录里的文件到本地 swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass download container1 (编辑:李大同) 【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容! |