跨平臺數(shù)據(jù)同步工具rclone簡介

1. 下載&安裝

從下面的地址下載對應(yīng)的二進制版本即可，之后安裝在系統(tǒng)的/usr/local/bin目錄下,或者其他用戶指定的路徑即可。
二進制//rclone.org/downloads/
源碼//github.com/rclone/rclone

2. rclone簡介

rclone是一個golang寫的開源遠程數(shù)據(jù)同步工具，后端支持文件系統(tǒng)以及多種類型的對象存儲產(chǎn)品，是一個跨平臺的數(shù)據(jù)同步工具，相比于s3cmd性能更加強大，同時還支持sync數(shù)據(jù)同步等功能實現(xiàn)差異數(shù)據(jù)的跨存儲后臺同步。并且還提供mount操作，可以直接將遠端對象存儲掛載到本地文件系統(tǒng)進行訪問。官方的Features介紹如下

MD5/SHA-1 hashes checked at all times for file integrity
Timestamps preserved on files
Partial syncs supported on a whole file basis
Copy mode to just copy new/changed files
Sync (one way) mode to make a directory identical
Check mode to check for file hash equality
Can sync to and from network, e.g. two different cloud accounts
Optional large file chunking (Chunker)
Optional transparent compression (Compress)
Optional encryption (Crypt)
Optional FUSE mount (rclone mount)
Multi-threaded downloads to local disk
Can serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna

3. 初始配置與檢查

查看軟件版本

╭─user@Mac~
╰─$rcloneversion1?
rclonev1.58.0
-os/version:darwin10.15.7(64bit)
-os/kernel:19.6.0(x86_64)
-os/type:darwin
-os/arch:amd64
-go/version:go1.17.8
-go/linking:dynamic
-go/tags:cmount

通過命令rclone config file獲取默認的配置文件路徑，之后在對應(yīng)的配置文件中填寫對應(yīng)的配置

╭─user@Mac~
╰─$rcloneconfigfile
Configurationfileisstoredat:
/Users/user/.config/rclone/rclone.conf
╭─user@Mac~
╰─$vi/Users/user/.config/rclone/rclone.conf
╭─user@Mac~
╰─$cat/Users/user/.config/rclone/rclone.conf
[s3v2]
type=s3
provider=Ceph
access_key_id=xxx
secret_access_key=sss
endpoint=https://s3.demo.com
location_constraint=
force_path_style=false

╭─user@Mac~/SourceCode/rclone?master?
╰─$rcloneconfigshow
[s3v2]
type=s3
provider=Ceph
access_key_id=xxx
secret_access_key=sss
endpoint=http://s3.demo.com
location_constraint=
force_path_style=false

╭─user@Mac~/SourceCode/rclone?master?
╰─$rcloneconfigdump
{
"s3v2":{
"access_key_id":"xxx",
"endpoint":"http://s3.demo.com",
"force_path_style":"false",
"location_constraint":"",
"provider":"Ceph",
"secret_access_key":"sss",
"type":"s3"
}
}%

通過rclone listremotes查看已經(jīng)配置好的遠端服務(wù)列表

╭─user@Mac~
╰─$rclonelistremotes
s3v2:

通過rclone lsd查看bucket列表或者是遠端的目錄列表

╭─user@Mac~
╰─$rclonelsds3v2:
-12022-01-1011:11:52-1demo-storage
╭─user@Mac~
╰─$rclonelsds3v2:demo-storage
02022-04-1110:42:45-1demo-storage
02022-04-1110:42:45-1wallpapers-zelda

通過rclone backend features獲取遠端服務(wù)支持的功能特性列表

╭─user@Mac~/SourceCode/rclone?master*?
╰─$rclonebackendfeaturess3v2:
{
"Name":"s3v2",
"Root":"",
"String":"S3root",
"Precision":1,
"Hashes":[
"md5"
],
"Features":{
"About":false,
"BucketBased":true,
"BucketBasedRootOK":true,
"CanHaveEmptyDirectories":false,
"CaseInsensitive":false,
"ChangeNotify":false,
"CleanUp":true,
"Command":true,
"Copy":true,
"DirCacheFlush":false,
"DirMove":false,
"Disconnect":false,
"DuplicateFiles":false,
"GetTier":true,
"IsLocal":false,
"ListR":true,
"MergeDirs":false,
"Move":false,
"OpenWriterAt":false,
"PublicLink":true,
"Purge":false,
"PutStream":true,
"PutUnchecked":false,
"ReadMimeType":true,
"ServerSideAcrossConfigs":false,
"SetTier":true,
"SetWrapper":false,
"Shutdown":false,
"SlowHash":false,
"SlowModTime":true,
"UnWrap":false,
"UserInfo":false,
"WrapFs":false,
"WriteMimeType":true
}
}

4. 數(shù)據(jù)上傳下載與遍歷

以上傳本地bin目錄到遠端S3為例

╭─user@Mac~/SourceCode/rclone?master?
╰─$lsbin
bisect-go-rclone.shcheck-merged.godecrypt_names.pymake_changelog.pynfpm.yamltest-repeat-vfs.shtest_proxy.pyupdate-authors.py
bisect-rclone.shconfig.pyget-github-release.gomake_manual.pynot-in-stable.gotest-repeat.shtidy-betaupload-github
build-xgo-cgofuse.shcross-compile.gomake_backend_docs.pymake_rc_docs.shtest-all-commits-compile.shtest_independence.gotravis.rclone.confwin-build.bat


╭─user@Mac~/SourceCode/rclone?master?
╰─$rclonecopybins3v2:demo-storage/bin-vv
2022/04/1314:50:25DEBUG:rclone:Version"v1.58.0"startingwithparameters["rclone""copy""bin""s3v2:demo-storage/bin""-vv"]
2022/04/1314:50:25DEBUG:Creatingbackendwithremote"bin"
2022/04/1314:50:25DEBUG:Usingconfigfilefrom"/Users/user/.config/rclone/rclone.conf"
2022/04/1314:50:25DEBUG:fscache:renamingcacheitem"bin"tobecanonical"/Users/user/SourceCode/rclone/bin"
2022/04/1314:50:25DEBUG:Creatingbackendwithremote"s3v2:demo-storage/bin"
2022/04/1314:50:25DEBUG:S3bucketdemo-storagepathbin:Waitingforcheckstofinish
2022/04/1314:50:25DEBUG:S3bucketdemo-storagepathbin:Waitingfortransferstofinish
2022/04/1314:50:25DEBUG:bisect-rclone.sh:md5=5476e61f31e1e2d697ce06a0f0100829OK
2022/04/1314:50:25INFO:bisect-rclone.sh:Copied(new)
...
2022/04/1314:50:26DEBUG:upload-github:md5=a26b267a5c784152a6378457b217c3b9OK
2022/04/1314:50:26INFO:upload-github:Copied(new)
2022/04/1314:50:26INFO:
Transferred:62.573KiB/62.573KiB,100%,0B/s,ETA-
Transferred:25/25,100%
Elapsedtime:0.8s

2022/04/1314:50:26DEBUG:13goroutinesactive
╭─user@Mac~/SourceCode/rclone?master?
╰─$rclonelsds3v2:demo-storage
02022-04-1314:50:32-1bin
02022-04-1314:50:32-1demo
02022-04-1314:50:32-1demo-storage
02022-04-1314:50:32-1wallpapers-zelda

通過lsd目錄查看指定路徑下的目錄列表

╭─user@Mac~/SourceCode/rclone?master?
╰─$rclonelsds3v2:demo-storage
02022-04-1315:02:41-1bin
02022-04-1315:02:41-1demo
02022-04-1315:02:41-1demo-storage
02022-04-1315:02:41-1wallpapers-zelda

通過ls命令查看指定目錄下的文件列表

╭─user@Mac~/SourceCode/rclone?master?
╰─$rclonelss3v2:demo-storage/bin
253.ignore-emails
382bisect-go-rclone.sh
902bisect-rclone.sh
...
370travis.rclone.conf
1224update-authors.py
1034upload-github
364win-build.bat

通過指定max-depth參數(shù)來設(shè)定ls命令所需遍歷的目錄層級深度

╭─user@Mac~/SourceCode/rclone?master?
╰─$rclonelss3v2:demo-storage--max-depth1?
382bisect-go-rclone.sh
902bisect-rclone.sh
142build-xgo-cgofuse.sh
...
370travis.rclone.conf
1224update-authors.py
1034upload-github
364win-build.bat

同時支持以json方式輸出，并輸出對應(yīng)文件的md5值信息。

╭─user@Mac~/SourceCode/rclone?master?
╰─$rclonelsjsons3v2:demo-storage/bin--hash1?
[
{"Path":".ignore-emails","Name":".ignore-emails","Size":253,"MimeType":"application/octet-stream","ModTime":"2021-12-16T1033.005446571+08:00","IsDir":false,"Hashes":{"md5":"8155885487321df7790cf63a9db7ba46"},"Tier":"STANDARD"},
{"Path":"upload-github","Name":"upload-github","Size":1034,"MimeType":"application/octet-stream","ModTime":"2021-07-08T1834.487717787+08:00","IsDir":false,"Hashes":{"md5":"a26b267a5c784152a6378457b217c3b9"},"Tier":"STANDARD"},
{"Path":"win-build.bat","Name":"win-build.bat","Size":364,"MimeType":"application/x-msdownload","ModTime":"2021-07-08T1834.487814575+08:00","IsDir":false,"Hashes":{"md5":"bf695fba05eb1ec8972eac29a59b21b4"},"Tier":"STANDARD"}
]

同時支持按文件files-only或者目錄dirs-only進行內(nèi)容篩選

╭─user@Mac~/SourceCode/rclone?master?
╰─$rclonelsjsons3v2:demo-storage--dirs-only1?
[
{"Path":"bin","Name":"bin","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T1532.917088000+08:00","IsDir":true},
{"Path":"demo","Name":"demo","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T1532.917140000+08:00","IsDir":true},
{"Path":"demo-storage","Name":"demo-storage","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T1532.917145000+08:00","IsDir":true},
{"Path":"wallpapers-zelda","Name":"wallpapers-zelda","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T1532.917149000+08:00","IsDir":true}
]

數(shù)據(jù)下載

╭─user@Mac/tmp/demo
╰─$ls
╭─user@Mac/tmp/demo
╰─$pwd
/tmp/demo
╭─user@Mac/tmp/demo
╰─$rclonecopys3v2:demo-storage/binbin
╭─user@Mac/tmp/demo
╰─$lsbin
bisect-go-rclone.shcheck-merged.godecrypt_names.pymake_changelog.pynfpm.yamltest-repeat-vfs.shtest_proxy.pyupdate-authors.py
bisect-rclone.shconfig.pyget-github-release.gomake_manual.pynot-in-stable.gotest-repeat.shtidy-betaupload-github
build-xgo-cgofuse.shcross-compile.gomake_backend_docs.pymake_rc_docs.shtest-all-commits-compile.shtest_independence.gotravis.rclone.confwin-build.bat

5. 使用sync數(shù)據(jù)同步

將遠端數(shù)據(jù)同步到本地

╭─user@Mac/tmp/demo
╰─$rclonesyncs3v2:demo-storage/binbin1?
╭─user@Mac/tmp/demo
╰─$ls
bin
╭─user@Mac/tmp/demo
╰─$lsbin
bisect-go-rclone.shcheck-merged.godecrypt_names.pymake_changelog.pynfpm.yamltest-repeat-vfs.shtest_proxy.pyupdate-authors.py
bisect-rclone.shconfig.pyget-github-release.gomake_manual.pynot-in-stable.gotest-repeat.shtidy-betaupload-github
build-xgo-cgofuse.shcross-compile.gomake_backend_docs.pymake_rc_docs.shtest-all-commits-compile.shtest_independence.gotravis.rclone.confwin-build.bat

模擬新建一個文件，同步到遠端

╭─user@Mac/tmp/demo
╰─$echonewfile1>bin/newfile1
╭─user@Mac/tmp/demo
╰─$rclonesyncbins3v2:demo-storage/bin
╭─user@Mac/tmp/demo
╰─$rclonelss3v2:demo-storage/bin
253.ignore-emails
382bisect-go-rclone.sh
...
370travis.rclone.conf
1224update-authors.py
1034upload-github
364win-build.bat

模擬修改一個文件，同步到遠端，可以看到只同步了差異部分

╭─user@Mac/tmp/demo
╰─$echonew123>bin/newfile1
╭─user@Mac/tmp/demo
╰─$rclonesyncbins3v2:demo-storage/bin-vv
2022/04/1315:39:01DEBUG:rclone:Version"v1.58.0"startingwithparameters["rclone""sync""bin""s3v2:demo-storage/bin""-vv"]
2022/04/1315:39:01DEBUG:Creatingbackendwithremote"bin"
2022/04/1315:39:01DEBUG:Usingconfigfilefrom"/Users/user/.config/rclone/rclone.conf"
2022/04/1315:39:01DEBUG:fscache:renamingcacheitem"bin"tobecanonical"/tmp/demo/bin"
2022/04/1315:39:01DEBUG:Creatingbackendwithremote"s3v2:demo-storage/bin"
2022/04/1315:39:01DEBUG:S3bucketdemo-storagepathbin:Waitingforcheckstofinish
...
2022/04/1315:39:02DEBUG:upload-github:Unchangedskipping
2022/04/1315:39:02DEBUG:win-build.bat:Sizeandmodificationtimethesame(differby0s,withintolerance1ns)
2022/04/1315:39:02DEBUG:win-build.bat:Unchangedskipping
2022/04/1315:39:02DEBUG:S3bucketdemo-storagepathbin:Waitingfortransferstofinish
2022/04/1315:39:02DEBUG:newfile1:md5=f9f9a6b3363a401f566becf83a5fad1bOK
2022/04/1315:39:02INFO:newfile1:Copied(replacedexisting)
2022/04/1315:39:02DEBUG:Waitingfordeletionstofinish
2022/04/1315:39:02INFO:
Transferred:7B/7B,100%,0B/s,ETA-
Checks:26/26,100%
Transferred:1/1,100%
Elapsedtime:0.3s

2022/04/1315:39:02DEBUG:25goroutinesactive

6. 掛載S3目錄到本地文件系統(tǒng)

rclone支持按指定目錄作為根目錄進行掛載,比如指定bucket名稱為demo-storage下的bin目錄作為根目錄

╭─user@Mac/tmp
╰─$rclonemounts3v2:demo-storage/bin~/mnt?

之后就可以在本地文件系統(tǒng)上訪問對應(yīng)的目錄數(shù)據(jù)了

╭─user@Mac~
╰─$cd~/mnt
╭─user@Mac~/mnt
╰─$ls
bisect-go-rclone.shcross-compile.gomake_manual.pytest-all-commits-compile.shtidy-beta
bisect-rclone.shdecrypt_names.pymake_rc_docs.shtest-repeat-vfs.shtravis.rclone.conf
build-xgo-cgofuse.shget-github-release.gonewfile1test-repeat.shupdate-authors.py
check-merged.gomake_backend_docs.pynfpm.yamltest_independence.goupload-github
config.pymake_changelog.pynot-in-stable.gotest_proxy.pywin-build.bat

7. 其他操作

通過link命令生成文件的訪問URL鏈接，支持通過expire設(shè)置URL有效時長。

╭─user@Mac~/mnt
╰─$rclonelinks3v2:demo-storage/bin/win-build.bat--expire1h4?
http://demo-storage.s3.demo.com/bin/win-build.bat?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ZK4XIPVNW9PMOADEZMSR%2F20220413%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220413T080748Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b4d60414a5d13662ac7026821e0aa7ffa10f4654142b87ea8629d2ba98b05e8c

通過size命令，獲取文件或者目錄的大小

─user@Mac~/mnt
╰─$rclonesizes3v2:demo-storage/bin
Totalobjects:26(26)
Totalsize:62.580KiB(64082Byte)

通過delete命令，刪除對應(yīng)的文件

╭─user@Mac~/mnt
╰─$rclonedeletes3v2:demo-storage/bin/win-build.bat

使用rmdir只能刪除空目錄，如果目錄殘留文件，需要使用purge命令刪除數(shù)據(jù)，之后再執(zhí)行rmdir才能徹底刪除目錄數(shù)據(jù)。

╭─user@Mac~/mnt
╰─$rclonermdirs3v2:demo-storage/bin-v
2022/04/131625INFO:S3bucketdemo-storagepathbin:Removingdirectory
╭─user@Mac~/mnt
╰─$rclonelss3v2:demo-storage/bin
253.ignore-emails
...
1034upload-github
╭─user@Mac~/mnt
╰─$rclonepurges3v2:demo-storage/bin-v
2022/04/1316:12:55INFO:build-xgo-cgofuse.sh:Deleted
...
2022/04/1316:12:55INFO:update-authors.py:Deleted
2022/04/1316:12:55INFO:S3bucketdemo-storagepathbin:Removingdirectory
╭─user@Mac~/mnt
╰─$rclonelss3v2:demo-storage/bin
╭─user@Mac~/mnt
╰─$rclonermdirs3v2:demo-storage/bin
╭─user@Mac~/mnt
╰─$rclonelsds3v2:demo-storage
02022-04-1316:13:16-1demo
02022-04-1316:13:16-1demo-storage
02022-04-1316:13:16-1wallpapers-zelda