linux – 完整的数据备份到Amazon S3

我使用的堆栈的相关部分是Node.js,MongoDB和Elasticsearch.

到目前为止,备份是通过转储整个MongoDB数据库,保存ES配置以及复制应用程序目录中的所有其他文件(日志,代码本身等)来完成的.如果是本月的第一天,所有用户文件也会被复制,否则只会添加自本月第一天起更改的文件.然后将所有这些压缩到一个文件中并上传到Amazon S3.

数据大小已达到此进程占用过多磁盘空间的程度,并且无法一次性将文件上载到S3.

这个大小的应用程序的下一个级别是什么(8 GB用户文件,125,000个用户,3,000个其他文档,所有可在ES中搜索)？

我理解基于意见的问题在服务器故障上是不行的.我不是在征求意见,而是正常的,具有成本效益的解决方案适用于这种规模的应用.

更新：这些是我试图使用Duplicity的脚本和配置的相关部分.我正在使用Node来管理备份,因为它适合我现有的日志记录解决方案,已经安排在低活动时间内与其他所有内容保持一致,并且可以在操作系统之间移植.

节点脚本,日志记录当然需要改进：

// Walks a directory recursively and returns a flat list of files
function walkDir() {};

// Node based rm -rf
function rmrf() {};

exec("mongodump --out dump",{ cwd: process.cwd() },function(err,dta) {
    if (err) return log("Error backing up: couldn't dump MongoDB!");

    exec("sudo duply ats backup",function(err) {
        if (err) log("Error running Duplicity");
        else rmrf("dump");

        log("Exiting.");

        process.exit();
    });
});

Duplicity配置：

GPG_PW='GPG password'

TARGET='s3://s3-us-east-1.amazonaws.com/bucket'

TARGET_USER='Known working AWS credentials'
TARGET_PASS='AWS secret key'

SOURCE='/var/appdir'

MAX_AGE=6M

DUPL_PARAMS="$DUPL_PARAMS --exclude "/var/appdir/elasticsearch/data/**" "

我尝试过-s3-use-new-style,使用s3 http：//,然后设置S3_USE_SIGV4但没有运气.

这是我从Duplicity获得的日志：

Start duply v1.5.10,time is 2015-07-05 09:30:13.
Using profile '/root/.duply/ats'.
Using installed duplicity version 0.6.23,python 2.7.6,gpg 1.4.16 (Home: ~/.gnu                                                                     pg),awk 'GNU Awk 4.0.1',bash '4.3.11(1)-release (x86_64-pc-linux-gnu)'.
Signing disabled. Not GPG_KEY entries in config.
Test - Encryption with passphrase (OK)
Test - Decryption with passphrase (OK)
Test - Compare (OK)
Cleanup - Delete '/tmp/duply.25562.1436103014_*'(OK)

--- Start running command PRE at 09:30:14.155 ---
Skipping n/a script '/root/.duply/ats/pre'.
--- Finished state OK at 09:30:14.183 - Runtime 00:00:00.027 ---

--- Start running command BKP at 09:30:14.208 ---
Reading globbing filelist /root/.duply/ats/exclude
BackendException: No connection to backend
09:31:27.427 Task 'BKP' failed with exit code '23'.
--- Finished state FAILED 'code 23' at 09:31:27.427 - Runtime 00:01:13.218 ---

--- Start running command POST at 09:31:27.465 ---
Skipping n/a script '/root/.duply/ats/post'.
--- Finished state OK at 09:31:27.491 - Runtime 00:00:00.026 ---