0
  • 聊天消息
  • 系统消息
  • 评论与回复
登录后你可以
  • 下载海量资料
  • 学习在线课程
  • 观看技术视频
  • 写文章/发帖/加入社区
会员中心
创作中心

完善资料让更多小伙伴认识你,还能领取20积分哦,立即完善>

3天内不再提示

跨平台数据同步工具rclone简介

Ceph对象存储方案 来源:Ceph对象存储方案 作者:秦牧羊 2022-04-14 08:04 次阅读

1. 下载&安装

从下面的地址下载对应的二进制版本即可,之后安装在系统的/usr/local/bin目录下,或者其他用户指定的路径即可。
二进制//rclone.org/downloads/
源码//github.com/rclone/rclone

2. rclone简介

rclone是一个golang写的开源远程数据同步工具,后端支持文件系统以及多种类型的对象存储产品,是一个跨平台的数据同步工具,相比于s3cmd性能更加强大,同时还支持sync数据同步等功能实现差异数据的跨存储后台同步。并且还提供mount操作,可以直接将远端对象存储挂载到本地文件系统进行访问。官方的Features介绍如下

MD5/SHA-1 hashes checked at all times for file integrity
Timestamps preserved on files
Partial syncs supported on a whole file basis
Copy mode to just copy new/changed files
Sync (one way) mode to make a directory identical
Check mode to check for file hash equality
Can sync to and from network, e.g. two different cloud accounts
Optional large file chunking (Chunker)
Optional transparent compression (Compress)
Optional encryption (Crypt)
Optional FUSE mount (rclone mount)
Multi-threaded downloads to local disk
Can serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna

3. 初始配置与检查

查看软件版本

╭─user@Mac~
╰─$rcloneversion1↵
rclonev1.58.0
-os/version:darwin10.15.7(64bit)
-os/kernel:19.6.0(x86_64)
-os/type:darwin
-os/arch:amd64
-go/version:go1.17.8
-go/linking:dynamic
-go/tags:cmount

通过命令rclone config file获取默认的配置文件路径,之后在对应的配置文件中填写对应的配置

╭─user@Mac~
╰─$rcloneconfigfile
Configurationfileisstoredat:
/Users/user/.config/rclone/rclone.conf
╭─user@Mac~
╰─$vi/Users/user/.config/rclone/rclone.conf
╭─user@Mac~
╰─$cat/Users/user/.config/rclone/rclone.conf
[s3v2]
type=s3
provider=Ceph
access_key_id=xxx
secret_access_key=sss
endpoint=https://s3.demo.com
location_constraint=
force_path_style=false
╭─user@Mac~/SourceCode/rclone‹master›
╰─$rcloneconfigshow
[s3v2]
type=s3
provider=Ceph
access_key_id=xxx
secret_access_key=sss
endpoint=http://s3.demo.com
location_constraint=
force_path_style=false

╭─user@Mac~/SourceCode/rclone‹master›
╰─$rcloneconfigdump
{
"s3v2":{
"access_key_id":"xxx",
"endpoint":"http://s3.demo.com",
"force_path_style":"false",
"location_constraint":"",
"provider":"Ceph",
"secret_access_key":"sss",
"type":"s3"
}
}%

通过rclone listremotes查看已经配置好的远端服务列表

╭─user@Mac~
╰─$rclonelistremotes
s3v2:

通过rclone lsd查看bucket列表或者是远端的目录列表

╭─user@Mac~
╰─$rclonelsds3v2:
-12022-01-1011:11:52-1demo-storage
╭─user@Mac~
╰─$rclonelsds3v2:demo-storage
02022-04-1110:42:45-1demo-storage
02022-04-1110:42:45-1wallpapers-zelda

通过rclone backend features获取远端服务支持的功能特性列表

╭─user@Mac~/SourceCode/rclone‹master*›
╰─$rclonebackendfeaturess3v2:
{
"Name":"s3v2",
"Root":"",
"String":"S3root",
"Precision":1,
"Hashes":[
"md5"
],
"Features":{
"About":false,
"BucketBased":true,
"BucketBasedRootOK":true,
"CanHaveEmptyDirectories":false,
"CaseInsensitive":false,
"ChangeNotify":false,
"CleanUp":true,
"Command":true,
"Copy":true,
"DirCacheFlush":false,
"DirMove":false,
"Disconnect":false,
"DuplicateFiles":false,
"GetTier":true,
"IsLocal":false,
"ListR":true,
"MergeDirs":false,
"Move":false,
"OpenWriterAt":false,
"PublicLink":true,
"Purge":false,
"PutStream":true,
"PutUnchecked":false,
"ReadMimeType":true,
"ServerSideAcrossConfigs":false,
"SetTier":true,
"SetWrapper":false,
"Shutdown":false,
"SlowHash":false,
"SlowModTime":true,
"UnWrap":false,
"UserInfo":false,
"WrapFs":false,
"WriteMimeType":true
}
}

4. 数据上传下载与遍历

以上传本地bin目录到远端S3为例

╭─user@Mac~/SourceCode/rclone‹master›
╰─$lsbin
bisect-go-rclone.shcheck-merged.godecrypt_names.pymake_changelog.pynfpm.yamltest-repeat-vfs.shtest_proxy.pyupdate-authors.py
bisect-rclone.shconfig.pyget-github-release.gomake_manual.pynot-in-stable.gotest-repeat.shtidy-betaupload-github
build-xgo-cgofuse.shcross-compile.gomake_backend_docs.pymake_rc_docs.shtest-all-commits-compile.shtest_independence.gotravis.rclone.confwin-build.bat


╭─user@Mac~/SourceCode/rclone‹master›
╰─$rclonecopybins3v2:demo-storage/bin-vv
2022/04/1314:50:25DEBUG:rclone:Version"v1.58.0"startingwithparameters["rclone""copy""bin""s3v2:demo-storage/bin""-vv"]
2022/04/1314:50:25DEBUG:Creatingbackendwithremote"bin"
2022/04/1314:50:25DEBUG:Usingconfigfilefrom"/Users/user/.config/rclone/rclone.conf"
2022/04/1314:50:25DEBUG:fscache:renamingcacheitem"bin"tobecanonical"/Users/user/SourceCode/rclone/bin"
2022/04/1314:50:25DEBUG:Creatingbackendwithremote"s3v2:demo-storage/bin"
2022/04/1314:50:25DEBUG:S3bucketdemo-storagepathbin:Waitingforcheckstofinish
2022/04/1314:50:25DEBUG:S3bucketdemo-storagepathbin:Waitingfortransferstofinish
2022/04/1314:50:25DEBUG:bisect-rclone.sh:md5=5476e61f31e1e2d697ce06a0f0100829OK
2022/04/1314:50:25INFO:bisect-rclone.sh:Copied(new)
...
2022/04/1314:50:26DEBUG:upload-github:md5=a26b267a5c784152a6378457b217c3b9OK
2022/04/1314:50:26INFO:upload-github:Copied(new)
2022/04/1314:50:26INFO:
Transferred:62.573KiB/62.573KiB,100%,0B/s,ETA-
Transferred:25/25,100%
Elapsedtime:0.8s

2022/04/1314:50:26DEBUG:13goroutinesactive
╭─user@Mac~/SourceCode/rclone‹master›
╰─$rclonelsds3v2:demo-storage
02022-04-1314:50:32-1bin
02022-04-1314:50:32-1demo
02022-04-1314:50:32-1demo-storage
02022-04-1314:50:32-1wallpapers-zelda

通过lsd目录查看指定路径下的目录列表

╭─user@Mac~/SourceCode/rclone‹master›
╰─$rclonelsds3v2:demo-storage
02022-04-1315:02:41-1bin
02022-04-1315:02:41-1demo
02022-04-1315:02:41-1demo-storage
02022-04-1315:02:41-1wallpapers-zelda

通过ls命令查看指定目录下的文件列表

╭─user@Mac~/SourceCode/rclone‹master›
╰─$rclonelss3v2:demo-storage/bin
253.ignore-emails
382bisect-go-rclone.sh
902bisect-rclone.sh
...
370travis.rclone.conf
1224update-authors.py
1034upload-github
364win-build.bat

通过指定max-depth参数来设定ls命令所需遍历的目录层级深度

╭─user@Mac~/SourceCode/rclone‹master›
╰─$rclonelss3v2:demo-storage--max-depth1382bisect-go-rclone.sh
902bisect-rclone.sh
142build-xgo-cgofuse.sh
...
370travis.rclone.conf
1224update-authors.py
1034upload-github
364win-build.bat

同时支持以json方式输出,并输出对应文件的md5值信息

╭─user@Mac~/SourceCode/rclone‹master›
╰─$rclonelsjsons3v2:demo-storage/bin--hash1↵
[
{"Path":".ignore-emails","Name":".ignore-emails","Size":253,"MimeType":"application/octet-stream","ModTime":"2021-12-16T1033.005446571+08:00","IsDir":false,"Hashes":{"md5":"8155885487321df7790cf63a9db7ba46"},"Tier":"STANDARD"},
{"Path":"upload-github","Name":"upload-github","Size":1034,"MimeType":"application/octet-stream","ModTime":"2021-07-08T1834.487717787+08:00","IsDir":false,"Hashes":{"md5":"a26b267a5c784152a6378457b217c3b9"},"Tier":"STANDARD"},
{"Path":"win-build.bat","Name":"win-build.bat","Size":364,"MimeType":"application/x-msdownload","ModTime":"2021-07-08T1834.487814575+08:00","IsDir":false,"Hashes":{"md5":"bf695fba05eb1ec8972eac29a59b21b4"},"Tier":"STANDARD"}
]

同时支持按文件files-only或者目录dirs-only进行内容筛选

╭─user@Mac~/SourceCode/rclone‹master›
╰─$rclonelsjsons3v2:demo-storage--dirs-only1↵
[
{"Path":"bin","Name":"bin","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T1532.917088000+08:00","IsDir":true},
{"Path":"demo","Name":"demo","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T1532.917140000+08:00","IsDir":true},
{"Path":"demo-storage","Name":"demo-storage","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T1532.917145000+08:00","IsDir":true},
{"Path":"wallpapers-zelda","Name":"wallpapers-zelda","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T1532.917149000+08:00","IsDir":true}
]

数据下载

╭─user@Mac/tmp/demo
╰─$ls
╭─user@Mac/tmp/demo
╰─$pwd
/tmp/demo
╭─user@Mac/tmp/demo
╰─$rclonecopys3v2:demo-storage/binbin
╭─user@Mac/tmp/demo
╰─$lsbin
bisect-go-rclone.shcheck-merged.godecrypt_names.pymake_changelog.pynfpm.yamltest-repeat-vfs.shtest_proxy.pyupdate-authors.py
bisect-rclone.shconfig.pyget-github-release.gomake_manual.pynot-in-stable.gotest-repeat.shtidy-betaupload-github
build-xgo-cgofuse.shcross-compile.gomake_backend_docs.pymake_rc_docs.shtest-all-commits-compile.shtest_independence.gotravis.rclone.confwin-build.bat

5. 使用sync数据同步

将远端数据同步到本地

╭─user@Mac/tmp/demo
╰─$rclonesyncs3v2:demo-storage/binbin1↵
╭─user@Mac/tmp/demo
╰─$ls
bin
╭─user@Mac/tmp/demo
╰─$lsbin
bisect-go-rclone.shcheck-merged.godecrypt_names.pymake_changelog.pynfpm.yamltest-repeat-vfs.shtest_proxy.pyupdate-authors.py
bisect-rclone.shconfig.pyget-github-release.gomake_manual.pynot-in-stable.gotest-repeat.shtidy-betaupload-github
build-xgo-cgofuse.shcross-compile.gomake_backend_docs.pymake_rc_docs.shtest-all-commits-compile.shtest_independence.gotravis.rclone.confwin-build.bat

模拟新建一个文件,同步到远端

╭─user@Mac/tmp/demo
╰─$echonewfile1>bin/newfile1
╭─user@Mac/tmp/demo
╰─$rclonesyncbins3v2:demo-storage/bin
╭─user@Mac/tmp/demo
╰─$rclonelss3v2:demo-storage/bin
253.ignore-emails
382bisect-go-rclone.sh
...
370travis.rclone.conf
1224update-authors.py
1034upload-github
364win-build.bat

模拟修改一个文件,同步到远端,可以看到只同步了差异部分

╭─user@Mac/tmp/demo
╰─$echonew123>bin/newfile1
╭─user@Mac/tmp/demo
╰─$rclonesyncbins3v2:demo-storage/bin-vv
2022/04/1315:39:01DEBUG:rclone:Version"v1.58.0"startingwithparameters["rclone""sync""bin""s3v2:demo-storage/bin""-vv"]
2022/04/1315:39:01DEBUG:Creatingbackendwithremote"bin"
2022/04/1315:39:01DEBUG:Usingconfigfilefrom"/Users/user/.config/rclone/rclone.conf"
2022/04/1315:39:01DEBUG:fscache:renamingcacheitem"bin"tobecanonical"/tmp/demo/bin"
2022/04/1315:39:01DEBUG:Creatingbackendwithremote"s3v2:demo-storage/bin"
2022/04/1315:39:01DEBUG:S3bucketdemo-storagepathbin:Waitingforcheckstofinish
...
2022/04/1315:39:02DEBUG:upload-github:Unchangedskipping
2022/04/1315:39:02DEBUG:win-build.bat:Sizeandmodificationtimethesame(differby0s,withintolerance1ns)
2022/04/1315:39:02DEBUG:win-build.bat:Unchangedskipping
2022/04/1315:39:02DEBUG:S3bucketdemo-storagepathbin:Waitingfortransferstofinish
2022/04/1315:39:02DEBUG:newfile1:md5=f9f9a6b3363a401f566becf83a5fad1bOK
2022/04/1315:39:02INFO:newfile1:Copied(replacedexisting)
2022/04/1315:39:02DEBUG:Waitingfordeletionstofinish
2022/04/1315:39:02INFO:
Transferred:7B/7B,100%,0B/s,ETA-
Checks:26/26,100%
Transferred:1/1,100%
Elapsedtime:0.3s

2022/04/1315:39:02DEBUG:25goroutinesactive

6. 挂载S3目录到本地文件系统

rclone支持按指定目录作为根目录进行挂载,比如指定bucket名称为demo-storage下的bin目录作为根目录

╭─user@Mac/tmp
╰─$rclonemounts3v2:demo-storage/bin~/mnt↵

之后就可以在本地文件系统上访问对应的目录数据了

╭─user@Mac~
╰─$cd~/mnt
╭─user@Mac~/mnt
╰─$ls
bisect-go-rclone.shcross-compile.gomake_manual.pytest-all-commits-compile.shtidy-beta
bisect-rclone.shdecrypt_names.pymake_rc_docs.shtest-repeat-vfs.shtravis.rclone.conf
build-xgo-cgofuse.shget-github-release.gonewfile1test-repeat.shupdate-authors.py
check-merged.gomake_backend_docs.pynfpm.yamltest_independence.goupload-github
config.pymake_changelog.pynot-in-stable.gotest_proxy.pywin-build.bat

7. 其他操作

通过link命令生成文件的访问URL链接,支持通过expire设置URL有效时长。

╭─user@Mac~/mnt
╰─$rclonelinks3v2:demo-storage/bin/win-build.bat--expire1h4↵
http://demo-storage.s3.demo.com/bin/win-build.bat?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ZK4XIPVNW9PMOADEZMSR%2F20220413%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220413T080748Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b4d60414a5d13662ac7026821e0aa7ffa10f4654142b87ea8629d2ba98b05e8c

通过size命令,获取文件或者目录的大小

user@Mac~/mnt
╰─$rclonesizes3v2:demo-storage/bin
Totalobjects:26(26)
Totalsize:62.580KiB(64082Byte)

通过delete命令,删除对应的文件

╭─user@Mac~/mnt
╰─$rclonedeletes3v2:demo-storage/bin/win-build.bat

使用rmdir只能删除空目录,如果目录残留文件,需要使用purge命令删除数据,之后再执行rmdir才能彻底删除目录数据。

╭─user@Mac~/mnt
╰─$rclonermdirs3v2:demo-storage/bin-v
2022/04/131625INFO:S3bucketdemo-storagepathbin:Removingdirectory
╭─user@Mac~/mnt
╰─$rclonelss3v2:demo-storage/bin
253.ignore-emails
...
1034upload-github
╭─user@Mac~/mnt
╰─$rclonepurges3v2:demo-storage/bin-v
2022/04/1316:12:55INFO:build-xgo-cgofuse.sh:Deleted
...
2022/04/1316:12:55INFO:update-authors.py:Deleted
2022/04/1316:12:55INFO:S3bucketdemo-storagepathbin:Removingdirectory
╭─user@Mac~/mnt
╰─$rclonelss3v2:demo-storage/bin
╭─user@Mac~/mnt
╰─$rclonermdirs3v2:demo-storage/bin
╭─user@Mac~/mnt
╰─$rclonelsds3v2:demo-storage
02022-04-1316:13:16-1demo
02022-04-1316:13:16-1demo-storage
02022-04-1316:13:16-1wallpapers-zelda


原文标题:对象存储神器-rclone光速入门

文章出处:【微信公众号:Ceph对象存储方案】欢迎添加关注!文章转载请注明出处。

审核编辑:汤梓红


声明:本文内容及配图由入驻作者撰写或者入驻合作网站授权转载。文章观点仅代表作者本人,不代表电子发烧友网立场。文章及其配图仅供工程师学习之用,如有内容侵权或者其他违规问题,请联系本站处理。 举报投诉
  • 二进制
    +关注

    关注

    2

    文章

    793

    浏览量

    41593
  • 对象存储
    +关注

    关注

    0

    文章

    67

    浏览量

    11960
  • 数据同步
    +关注

    关注

    0

    文章

    17

    浏览量

    8153

原文标题:对象存储神器-rclone光速入门

文章出处:【微信号:cephbook,微信公众号:Ceph对象存储方案】欢迎添加关注!文章转载请注明出处。

收藏 人收藏

    评论

    相关推荐

    JAVA语言为什么能平台

    执行的代码。因此对于Java编程者来说,不需要考虑硬件平台是什么。所以Java可以平台。因为它有虚拟机(JVM),JAVA程序不是直接在电脑上运行的,是在虚拟机上进行的,每个系统平台
    发表于 11-27 17:06

    如何提高嵌入式平台的移植性与兼容性

    (四个字节)。这样写的目的,是为了提高平台的移植性与兼容性。不同平台数据类型定义都不尽相同,一套代码要想.
    发表于 11-08 06:25

    平台嵌入式数据库EffiProz介绍

    EffiProz:面向.NET程序员的平台嵌入式数据库 EffiProz:面向.NET程序员的平台嵌入式
    发表于 12-21 06:00

    FPGA时钟域处理简介

    (10)FPGA时钟域处理1.1 目录1)目录2)FPGA简介3)Verilog HDL简介4)FPGA时钟域处理5)结语1.2 FPGA简介
    发表于 02-23 07:47

    怎样通过MQTT实现ESP8266与ONENET平台数据交互呢

    本文将介绍通过MQTT协议连接ONENET,实现ESP8266与ONENET平台数据交互。
    发表于 02-24 07:50

    基于CANopen协议的车载平台数据通讯系统

    在分析CANopen 协议结构和特点的基础上,着重给出了用CANopen 协议开发车载平台数据通讯系统的方法,其中包括CAN 控制器的硬件接口设计和DLL 模块、对象字典、应用程序等软件部分
    发表于 07-07 14:59 25次下载

    面向MapGIS和ArcGIS的平台数据转换研究

    面向MapGIS和ArcGIS的平台数据转换研究_文缔尧
    发表于 01-07 21:28 2次下载

    平台构建工具xmake简介与安装使用说明

    大家好,我是ZhengN。本次给大家分享一个优秀的平台构建工具
    的头像 发表于 07-19 15:52 5792次阅读

    Rsync远程数据同步工具简介

    Rsync(remote synchronize)是一个远程数据同步工具,可通过LAN/WAN快速同步多台主机间的文件。Rsync使用所谓的“Rsync算法”来使本地和远 程两个主机之
    的头像 发表于 09-16 09:13 1058次阅读

    Rsync远程数据同步工具简介

    Rsync(remote synchronize)是一个远程数据同步工具,可通过LAN/WAN快速同步多台主机间的文件。Rsync使用所谓的“Rsync算法”来使本地和远 程两个主机之
    的头像 发表于 10-12 09:12 891次阅读

    一款​平台指纹识别工具原理解析

    一款平台社区网页指纹识别工具,类似glass、eholo,不同的是该工具调用 nuclei ,可以同时进行漏洞验证。
    发表于 07-28 15:56 1584次阅读
    一款​<b class='flag-5'>跨</b><b class='flag-5'>平台</b>指纹识别<b class='flag-5'>工具</b>原理解析

     节能监管平台数据的分析及相关产品选型

    ​  节能监管平台数据的分析及相关产品选型 摘 要 :目前全国大部分省市都已建立节能监管平台,已积累了大量建筑的能耗数据,这些数据未能在建筑节能中得以有效应用。从能耗
    的头像 发表于 01-16 13:44 579次阅读
     节能监管<b class='flag-5'>平台数据</b>的分析及相关产品选型

    一个嵌入式数据可视化工具——Serial Studio

    Serial Studio从名称就可以看的出来,它是一款串口(Serial)可视化(Studio)工具:主要用于嵌入式开发者的平台数据可视化工具(应用软件,或上位机软件)。
    的头像 发表于 03-04 10:12 5766次阅读

    RA4E1 AI 套件参考平台数据手册

    电子发烧友网站提供《RA4E1 AI 套件参考平台数据手册.rar》资料免费下载
    发表于 05-14 17:47 0次下载
    RA4E1 AI 套件参考<b class='flag-5'>平台数据</b>手册

    RA6M3 AI 套件参考平台数据手册

    电子发烧友网站提供《RA6M3 AI 套件参考平台数据手册.rar》资料免费下载
    发表于 05-14 18:12 0次下载
    RA6M3 AI 套件参考<b class='flag-5'>平台数据</b>手册