Topics:
DVR
nvrec
Mplayer
Links
Misc
Commands
Humor
New user
uploaded files
|
|
(links)-> (Parent)->rsync over sftp (mosso/rackspace) fuse mount |
submited by Russell Fri 01 Aug 08
Edited Fri 20 Jul 12 |
I'm Migrating webserver content to mosso (http://www.mosso.com), which I have to say has been 98% happy
experience for me but there are just a few bumps in the process that I have encountered.
Clearly the biggest problem for me has been the lack of ssh (scp) support. In the past I
dealt with my own servers where I had root ( or at least shell ) access and could copy
and change files at will.
I work with a local copy of the website where I make changes, and when I am happy with
them I copy the changes to the server. When I had my own server or ssh access, I would
use rsync to copy the local files to the remote server. Mosso doesn't support the ssh
connection the rsync needs, but they do support sftp. Here are my notes on how to get
rsync to work over this sftp using a fuse-sshfs mounted dir.
first, if you don't have it, install fuse-sshfs. Its in lots of distros, but perhaps not
on the default install. I'm using fedora.
mount remote mosso drive locally:
sshfs -o reconnect,cache_timeout=1000 <username>@ftp.<hostname>:<domain>/web/content <path-to-mount>
This will prompt you for your password and then mount the remote drive in
<path-to-mount>. ( this local directory must exist and be empty)
-o reconnect tells it to reestablish the connection (as needed) if it times-out or is
otherwise lost. Currently this will prompt for a password each time it needs to
reconnect. Searching for workaround. (update: see below)
-o cache_timeout tells it to use a much longer cache time locally, this speeds things up
a lot, without this fuse will reacquire directorys multiple times. If your the only one
changing files, then this speedup is safe.
synchronize local and remote files.
rsync -rtpvP --inplace <path-to-local-files> <path-to-mount>
note: Neither path has a server:<path> format, rsync behaves like these are local files
-vP options are verbose and progress, remove these to quiet up, but I like to see that
it's doing something.
-rtp is instead of the -a I would normally use. does subdirs (-r) preservers timestamps
(-t) and permitions (-p)
--inplace tells it to overwrite the old file with the new. The default is to transfer
to a temp file then rename it. This seemed to cause a problem over sftp (I think rename
where a file would replace another doesn't work over sftp ). Just be aware that a large
file will be invalid on the server during the upload. (or if the upload in broken)
add -l if you want to transfer symlinks as symlinks.
These options do not attempt to change the group or owner of the file (-go or -a) if
it tried to do this over a sftp connection, it would get an error. This *REALLY*
slows things down because it will attempt to do this on dirs and files that are already
transfered, each time you rsync.
Since Mosso is a hosting cloud from time to time you actually connect to a different server. (using the same host name) Each time you encounter a new one, you will be asked to accept a ssh key into your known hosts.
-------------------
Pitfalls:
THIS IS NOT FAST. if you have a high file count , or high directory count, (like I
do ) this will slowly crawl the whole site looking for changed files. To me it was more
important that I know the server has all the correct files, than making sure the process
was as quick as possible. If you want speed, manually copy files wit filezilla or other
client.
This won't transfer symlinks ( it sees them as non-regular files) . I solved that by
removing my symlinks, ( I only had a few)
If you have any webside content that is generated this will overwrite that with local
versions. use the --exclude <path> option of rync if needed.
a common rsync pitfall for me is including or not including the trailing slash in the local path when I should or shouldn't
rsync local-path/ remote-path/
will copy the *CONTENTS* of local-path to remote path if you mean to transfer the local path as a single directory, leave off the trailing slash.
This can also be used to create a backup copy, just reverse the order of remote and
local in the rsync command. I have also done this with wget, and found it simpler:
wget -N -r --level=0 -m ftp://<username>:<password>@ftp.<hostname>
this downloads the remote files to the current directory, skipping files with the same
timestamp as the local copy and recursively does the whole thing. I think it's faster
than doing a rsync over sftp, but rsync lets you create backup versions of changed files
(filename~) if you want that, use rsync with the --backup option
---------------------
update Jan/2012
Some time ago, rackspace must have updated the sftp server so it now looks for and will use an authorized_keys file. You can use this method to auto login and auto-reconnect to the rackspace server. (without being prompted for a password when a reconnect is tried after a timeout) This is similar but not the same to setting up authorized_keys for a ssh login. You can't use ssh-copy-id because you don't have a ssh access top a shell.
the procedure:
on your client:
ssh-keygen
I use the defaults, read the man page to select other keytypes and sizes. This creates .ssh/id_rsa.pub (among other things) you then copy or append that file onto rackspaces's sever to:
.ssh/authorized_keys
you need to create the .ssh directory under the root login directory for the sftp account you are using. it should be permitions : rwx------ (chmod 700 .shh ). The authorized_keys file should have the same permitions.
Standard disclamers: Do this only from a secure computer. Never share the private key. If you suspect a security failure, delete the authorized_keys file and make new keys. Using this method to login will not ever require the connecting server to supply a password ( after all, the shared keys establish this as a trusted connection) so changing the password on the sftp account won't deny access via this method.
Replys:
Rsync over sshfs is inefficient (Anonymous)
rsync over SSH: -S program (Marco)
KeepAlive (Mike)
in conclusion? (hozza)
|
Add comment or question...:
|