Sunday, November 2, 2008

TimeMachine via AFP (netatalk) to a DroboShare

This rainy weekend, I turned the hardcore geek on so now I get to brag about it a little.

On my MacBook Pro, I used an Ubuntu vmware image to cross compile netatalk , dbus , avahi , openssl , expat , libdaemon and berkeley db to the embedded Marvell-Linux (arm-gnueabi) running on the DroboShare. Anyone who has experienced cross compiling knows it is an extremely complicated thing to do and not for the weak. Especially when dealing with so many different products which all have to integrate together. It is also amazing that I can use two different operating systems to compile code for a third operating system.

After getting all of those projects compiled, configured and started, I can now say that I'm the first person to successfully perform a Time Machine backup to the Drobo connected to the DroboShare over AFP. This is something that Apple has put a lot of effort into making very difficult to perform because they are trying to sell more of their proprietary and expensive TimeCapsule devices and now I've beaten them at their game. =)

Time to put this project down and next weekend I'll package things up and make it available to other owners of DroboShares.

Update: I've created a Google Code project called DroboCapsule and put up a binary package for you to download and install. Enjoy.

28 comments:

Brendan said...

Holy crap, nowai! I must have this.

U rulz!

Jon Scott Stevens said...

http://code.google.com/p/drobocapsule/

The Red Cowboy said...

Jon -

I left this comment over on GCode:

Thank you SO much for coding this... and I'm having a problem - DroboCapsule? is NOT showing up as a shared volume after installation. droboshare shows up, per usual, and I'm certain that I've dl'd and installed netatalk per your instructions (I think...) Any ideas? My Drobo is configured as three volumes, if that matters, and I've assigned a manual address to DroboShare?. Thank you in advance.

Hans

Jon Scott Stevens said...

Tell me a bit about your network topology. Is your DroboShare plugged into the same hub as your computer? Remember, Bonjour doesn't span routers, so you won't see it on the network if you are routed.

Also, if you ssh into your DroboShare, let me know if you can see avahi in the 'ps w' list.

The Red Cowboy said...

Jon -

Thank you for your reply - sorry I have ben delayed responding. So... Topology. My Drobo is plugged into my DroboShare, which is plugged into an Ethernet switch, which is plugged into a router. My iMac is plugged into the same switch as the DroboShare, so I don't think it's a spanning issue.

I'm not a Drobo/DroboShare expert by any means, but, using DropBear, I do not see 'avahi' in the ps w list:

PID Uid VmSize Stat Command
1 root 328 S init
2 root SWN [ksoftirqd/0]
3 root SW< [events/0]
4 root SW< [khelper]
5 root SW< [kthread]
11 root SW< [kblockd/0]
14 root SW [khubd]
49 root SW [pdflush]
50 root SW [pdflush]
52 root SW< [aio/0]
51 root SW [kswapd0]
165 root SW [scsi_eh_0]
166 root SW [scsi_eh_1]
172 root SW [mtdblockd]
189 root SW [scsi_eh_2]
190 root SW [usb-storage]
195 root SWN [jffs2_gcd_mtd1]
197 root 812 S /bin/sh
281 root 532 S < udevd --verbose
396 root 704 S /bin/ntpclient -l -h pool.ntp.org
397 root 308 S /bin/sh /usr/local/lock_rtc_to_wall_clock.sh
419 root 172 S sleep 3600
420 root 1996 S /usr/sbin/sledd
430 root 312 S /bin/sh /usr/bin/sleddLogRotate
436 root 2064 S usr/sbin/smbd -s /etc/smb.conf
438 root 2064 S usr/sbin/smbd -s /etc/smb.conf
439 root 1552 S usr/sbin/nmbd -s /etc/smb.conf
556 root 1088 S damd
621 root 3216 S /mnt/DroboShares/Drobo1/DroboApps/fuppes/fuppes --config-dir /mnt/DroboShares/Drobo1/DroboApps/fuppes --co
641 root 652 S /usr/bin/dropbear
667 root 924 S /mnt/DroboShares/Drobo1/DroboApps/mt-daapd/mt-daapd -c /mnt/DroboShares/Drobo1/DroboApps/mt-daapd/mt-daapd
668 root 1868 S /mnt/DroboShares/Drobo1/DroboApps/mt-daapd/mt-daapd -c /mnt/DroboShares/Drobo1/DroboApps/mt-daapd/mt-daapd
672 root 1188 S /mnt/DroboShares/Drobo1/DroboApps/netatalk/sbin/afpd -P /mnt/DroboShares/Drobo1/DroboApps/netatalk/var/run
676 root 3636 S /mnt/DroboShares/Drobo1/DroboApps/droboadmin/bin/httpd -d /mnt/DroboShares/Drobo1/DroboApps/droboadmin/bin
702 root 4172 S /mnt/DroboShares/Drobo1/DroboApps/droboadmin/bin/httpd -d /mnt/DroboShares/Drobo1/DroboApps/droboadmin/bin
715 root 4076 S /mnt/DroboShares/Drobo1/DroboApps/droboadmin/bin/httpd -d /mnt/DroboShares/Drobo1/DroboApps/droboadmin/bin
737 root 1072 R /usr/bin/dropbear
743 root 824 S -sh
748 root 172 S sleep 30
749 root 780 R ps w

Any ideas?

Jon Scott Stevens said...

Are you using v3 of the netatalk.tgz? I fixed an issue that would prevent avahi from starting. Install that and reboot.

The Red Cowboy said...

Jon -

I think I am...I downloaded it off the GCode page, after the revision was posted... What's the correct mod date on the file? I'll try again right now.

Hans

The Red Cowboy said...

Jon -

No joy. I deleted the old netatalk folder (over SSH), dl'd the netatalk.tgz file, copied it into the DroboApps folder on Drobo1 (where my other DroboApps reside), restarted, ps w'd, no avahi.

Thoughts?

Hans

Jon Scott Stevens said...

Try starting avahi/dbus manually. You can see how all of this is done by looking at the netatalk-start.sh script.

/mnt/DroboShares/Drobo/DroboApps/netatalk/bin/dbus-daemon --system
ps w (confirm it is running)

then:

/mnt/DroboShares/Drobo/DroboApps/netatalk/sbin/avahi-daemon

it should start up and print information to stdout. you can ctrl-c to stop it. if it prints and error, let me know what the error is.

The Red Cowboy said...

Jon -

Please let me apologize again for not being a UNIX stud - I think I'm doing things right, but I'm not certain. That said - I opened up the netatalk-start.sh file and cut/pasted the appropriate commands into my Terminal session connected to DroboShare. All that seemed to go fine - HOWEVER, I note that the startup file refers to /Drobo/ a number of times - and my Drobo, where the DroboApps folder in question is located, is /Drobo1/. So, I substituted. Wrong? Nothing failed as I went through the script.

Then, I cut/pasted your command "/mnt/DroboShares/Drobo/DroboApps/netatalk/bin/dbus-daemon --system". Boom. Terminal spits this back at me: "/mnt/DroboShares/Drobo1/DroboApps/netatalk $ /mnt/DroboShares/Drobo/DroboApps/netatalk/bin/dbus-daemon --system
-sh: /mnt/DroboShares/Drobo/DroboApps/netatalk/bin/dbus-daemon: not found
/mnt/DroboShares/Drobo1/DroboApps/netatalk $ /mnt/DroboShares/Drobo1/DroboApps/netatalk/bin/dbus-daemon --system
Failed to start message bus: Failed to open "/mnt/DroboShares/Drobo/DroboApps/netatalk/etc/dbus-1/system.conf": No such file or directory". NOTE that I tried the command with /Drobo1/ as well. I'm guessing that, somewhere in a conf file, /Drobo/ needs to be /Drobo1/. Or not. Please advise...

Hans

Jon Scott Stevens said...

Ah ha! The Drobo1 is the problem. The paths are pretty much hard coded as just Drobo.

So, you have three options:

#1. Make an alias of Drobo1 to Drobo: cd /mnt/DroboShares/; ln -s Drobo1 Drobo

#2. Rename Drobo1 to Drobo.

#3. Edit the f*ck out of the configuration files for all the various daemons and shell scripts in netatalk and replace all instances of Drobo in a path with Drobo1.

I'd go for #2. =)

The Red Cowboy said...

Jon -

Thank you for your help. I'm not gonna try editing all the conf files - that's just a recipe for disaster. I did try the ln -s gag, but it didn't work. So, tomorrow, I'll rename (I have to attach Drobo directly to my Mac to do that, apparently). I will report...

Hans

The Red Cowboy said...

Jon -

Never say die. I have the ln -s gag working (I must have done something wrong before). So - now I see both daemons running in ps w:

674 avahi 1628 S avahi-daemon: running [DroboShare.local]

673 messageb 1140 S /mnt/DroboShares/Drobo1/DroboApps/netatalk/bin/dbus-daemon --system

So that's all good. Problem now is that I'm still not seeing DroboCapsule under Shared in the Finder.

Sorry to be so lame.

Hans

Jon Scott Stevens said...

Just to confirm, the DroboShare and the Mac are plugged into the same hub and it still isn't showing up?

If that is the case and things still aren't working, then I'd rename back to just Drobo and see what happens then.

The Red Cowboy said...

Jon -

Hmmm...curious. I have just discovered that "DroboCapsule" _is_ showing up under "Shared" in the Finder - but NOT on my laptop connect via WiFi - it is showing up on my iMac connected via Ethernet.

Am I doing something wrong on the laptop?

Hans

Jon Scott Stevens said...

Now we need an introduction to how networks work. =) Basically, you need to locate your DroboShare on the same side of the network as your wifi network in order to see it from your laptop or you need to bridge the Bonjour networking. I'm sorry, but things are working now for you from my perspective and this is where my free tech support stops.

Simon said...

I also have the issue of seeing neither Drobo nor DroboCapsule in the SHARED Finder sidebar.

Mounting via Command-K works fine though.

So I sshed into the Drobo and tried to start the dbus-daemon and avahi-daemon manually, which yielded the following errors:

/mnt/DroboShares/Drobo/DroboApps/netatalk/sbin/avahi-daemon
/mnt/DroboShares/Drobo/DroboApps/netatalk/sbin/avahi-daemon: error while loading shared libraries: libavahi-common.so.3: cannot open shared object file: No such file or directory

/mnt/DroboShares/Drobo/DroboApps/netatalk/bin/dbus-daemon --system

/mnt/DroboShares/Drobo/DroboApps/netatalk/bin/dbus-daemon: error while loading shared libraries: libexpat.so.1: cannot open shared object file: No such file or directory

I am using v3 of the script and have only one partition on the drobo called "Drobo", totally standard.

I am no expert in Linux stuff, so every help is highly appreciated. Thanks!

Jon Scott Stevens said...

Confirm that avahi isn't running first. You use 'ps w' to check for it running. If it is running, then it is a networking issue (ie: you aren't bridging the bonjour protocol) and there is nothing more for me to do.

If it isn't running...

Try running the start script. It sets up the LD_LIBRARY_PATH for you.

/mnt/DroboShares/Drobo/DroboApps/netatalk/netatalk-start.sh

If it *still* isn't running... write this:

export LD_LIBRARY_PATH=/mnt/DroboShares/Drobo/DroboApps/netatalk/lib:/mnt/DroboShares/Drobo/DroboApps/netatalk/etc/uams

/mnt/DroboShares/Drobo/DroboApps/netatalk/bin/dbus-daemon --system

check to see if dbus is running (with the 'ps w' command)

If it is running, then try to start up avahi (dbus has to be running before avahi will start):

/mnt/DroboShares/Drobo/DroboApps/netatalk/sbin/avahi-daemon

If at any point along the way, it complains about a pid file, you will have to remove that file. You can do that by running the netatalk-stop.sh script and trying to start up dbus and avahi.

If either dbus or avahi give any other errors. let me know.

Jon Scott Stevens said...

I just noticed that the export line above got chopped off. If you copy/paste the entire block (or view source on the page), it will show up.

It should really look something like this (but all one line)

export
LD_LIBRARY_PATH=
/mnt/DroboShares/Drobo/DroboApps/netatalk/lib
:/mnt/DroboShares/Drobo/DroboApps/netatalk/etc/uams

Simon said...

Jon,

Thanks for the extensive help.

I tried the restart script "netatalk-start.sh", but both daemons were not running when I checked ps w.

Then I tried to set the path and start dbus directly but still got the error:
/mnt/DroboShares/Drobo/DroboApps/netatalk/bin/dbus-daemon: error while loading shared libraries: libexpat.so.1: cannot open shared object file: No such file or directory

Any ideas?

Thanks a lot.

Jon Scott Stevens said...

Oh right, I just remembered someone else's posting in the drobo developer forums that could explain the problem you are seeing: Let me guess, you are using NTFS or FAT32 as your filesystem for the Drobo.

If that is the case, then the problem is that it doesn't support symbolic links (ie: aliases to files). You will need to use HFS+ or ext3. I'll update the DroboCapsule page to note this.

There is a way to make it work with NTFS/FAT32, which is to make multiple copies of all the shared libs instead of trying to alias them, but that would be a real waste of space and there is no good reason to use NTFS/FAT32 with a Drobo/DroboShare combination.

Simon said...

Yes, this is exactly the case!

OK, that explains it. Thanks a lot for your help. I was thinking of reformatting the Drobo anyway. So I will do it …

drtidmore said...

Yours and a few others posting allowed me to create a roadmap to get a full APF based TM backup to my DLINK DNS323. Everything is working except I note that in my TM logs that my version of netatalk shows to NOT support TM Lock Stealing and Server Reply Cache. I did see where you indicated that the version of netatalk that you cross compiled supported the latest TM extension to AFP, SO I was wondering if the version you found DOES in fact support these and if so where did you find it. I would like to get these last two pieces in place.

Thanks
David

Jon Scott Stevens said...

Follow the netatalk mailing list. 2.0.5 will support that. It is in a release candidate mode right now, but should be final soon.

Daniel M. Clark said...

Works great! I had the "Drobo1" issue that a previous commenter had, and created a symbolic link to take care of it, as you suggested. Time Machine is backing up as I type this. Thanks a lot!

One question though... how can I know that the backup size won't get out of hand? I know that I told it 500g when I ran the script, but there's no indication that it will max out at that when I look at the volume in Time Machine or Finder. Is that expected?

Jon Scott Stevens said...

The size of the volume (sparsebundle) you are backing up to is fixed. It can't grow any larger than that bundle size.

Daniel M. Clark said...

Great, great. Chalk up my concern to a lack of understanding about sparsebundles. Thanks again!

Jon Scott Stevens said...

http://en.wikipedia.org/wiki/Sparse_image

The only difference is that the 'Create backup volume' script that I use creates 64M bands instead of the default 8M. This works far better for TM backups because it doesn't have to transfer the data about a bazillion tiny files (bands) over the network. Makes backups much faster.