UU Blog

Python写的备份小工具

项目地址: https://github.com/onecer/Pyrsync

readme: https://uublog.com/article/20170630/pyrsync-readme/

===========================================================================

打算用rsync同步,刚刚在写shell的时候,尝试了下,项目多了,目录多了。shell代码就眼花缭乱。

所以,这样封装了一下,备份的项目用yaml描述。100行代码,可以更方便的去管理备份源。

项目代码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
#!/usr/bin/env python
# coding:utf-8

# Author: phan

import os
import yaml
import commands
import time
import codecs

# 配置区域
# 日志路径
LOGFILE = '/var/log/pyrsync.log'
# 是否开启日志
ENABLE_LOG = True
# 啰嗦模式
VERBOSE_MODE = True

# 自定义常量
BASEDIR = os.path.dirname(__file__)

# 获取目录文件列表
def get_file_lists(dirpath):
filelists = []
for filename in os.listdir(dirpath):
filelists.append(os.path.join(dirpath,filename))
return filelists

# 获取当前时间
def get_time():
return time.strftime('%Y-%m-%d %H:%M:%S ',time.localtime(time.time()))

# 记录日志
def logger(line):
if ENABLE_LOG:
if os.path.isfile(LOGFILE):
with codecs.open(LOGFILE,'a','utf-8') as f:
f.write("{time}{logline}\n".format(time=get_time(), logline=line))
else:
with codecs.open(LOGFILE,'w','utf-8') as f:
f.write("{time}{logline}\n".format(time=get_time(), logline=line))

# 记录命令日志
def logger_commands(status,result):
logger('result:{status}'.format(status=status))
if VERBOSE_MODE:
logger(result)

# 执行命令
def runcommand(commandline):
return commands.getstatusoutput(commandline)

# 部署场景
def deploy_sences(isLocal,shellfile,actor=''):
commandline = os.path.join(BASEDIR, 'script', shellfile)
if isLocal:
(status,result) = runcommand(commandline)
logger_commands(status,result)
else:
upload_script = 'scp {command} {name}@{ip}:/tmp/pyrsync_{scriptName}'.format(
command=commandline,
name=actor['name'],
ip=actor['ip'],
scriptName=os.path.basename(commandline)
)
(status,result) = runcommand(upload_script)
logger_commands(status, result)
commandline = 'ssh {name}@{ip} "chmod u+x /tmp/pyrsync_{command} && /tmp/pyrsync_{command} && rm -f /tmp/pyrsync_{command}"'.format(
name=actor['name'],
ip=actor['ip'],
command=os.path.basename(commandline),
)
(status,result) = runcommand(commandline)
logger_commands(status, result)

# 同步文件
def sync_files(actor,sence):
rsyncCommand = 'rsync {action} --exclude-from="{exclude}" {name}@{ip}:{path} {savepath}'.format(
action=actor['action'],
exclude=os.path.join(BASEDIR,'exclude',sence['exclude']),
name=actor['name'],
ip=actor['ip'],
port=actor['port'],
path=sence['item'],
savepath=sence['savepath']
)
logger('Command:{command}'.format(command=rsyncCommand))
(status,result) = runcommand(rsyncCommand)
logger_commands(status,result)

# 执行剧本
def play_playbooks(filelists):
for filelist in filelists:
if os.path.isfile(filelist) and os.path.splitext(filelist)[-1]=='.yaml':
with open(filelist,'r') as f:
playbook = yaml.load(f)
sences = playbook['playbook']['sences']
for sence in sences:
if playbook['execute']['pre-local']:
deploy_sences(True,playbook['execute']['pre-local'])
if playbook['execute']['pre-remote']:
deploy_sences(False, playbook['execute']['pre-remote'],playbook['actor'])
sync_files(playbook['actor'],sence)
if playbook['execute']['post-remote']:
deploy_sences(False,playbook['execute']['post-remote'],playbook['actor'])
if playbook['execute']['post-local']:
deploy_sences(True,playbook['execute']['post-local'])

if __name__=='__main__':
play_playbooks(get_file_lists(os.path.join(BASEDIR,'playbook')))

配置格式如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
actor:
name: root
ip: 192.168.168.19
port: 22
action: -avz

execute:
pre-local:
pre-remote: test.sh
post-remote:
post-local:

playbook:
sences:
- 1:
item: /data/web1/uploads
savepath: /backup/web1
exclude: cvs.txt
- 2:
item: /data/web2/uploads
savepath: /backup/web2
exclude: cvs.txt
- 3:
item: /data/web3/uploads
savepath: /backup/web3
exclude: cvs.txt
- 4:
item: /data/sql
savepath: /backup/sql
exclude: cvs.txt

安装

Pyrsync

调用rsync同步文件,从yaml文件读取配置。简化rsync使用,和方便管理备份项更改。未来将提供web界面添加yaml配置。

安装依赖

sudo pip install -r requirements.txt

sudo yum install -y rsync

使用说明

  • 新建项目yaml配置文件 cp playbook/default.yaml.template playbook/default.yaml

  • 配置好项目使用哪个账户复制,ip、备份目录之类的。

  • 要提前做好要备份的主机公钥认证,好让rsync不用输入密码。

  • 执行 ./pyrsync.py

给作者打一针鸡血