[arch-general] adb backup broken named pipe
hello,
I'm looking for a solution for the problem below : all leads are welcome !
the adb tool is able to make a backup of a smartphone in a file: adb backup -all -f mybackup.ab
however, the file mybackup.ab has a particular format that prevents it from being used "simply". to convert it into a tgz archive, it is possible to do this: ( printf "\x1f\x8b\x08\x08\x00\x00\x00\x00\x00\x00\x00"; tail -c +25 mybackup.ab ) > mybackup.tgz
but in this case, I can easily end up with two files of 16Gb or more... to avoid this volume problem, I tried to do this: mkfifo backup.fifo adb backup -all -f backup.fifo & ( printf "\x1f\x8b\x08\x08\x00\x00\x00\x00\x00\x00" ; tail -c +25 backup.fifo) > mybackup.tgz
but adb systematically breaks my named pipe and my code doesn't work as expected. how to prevent adb from breaking the named pipe ? I'm considering using fuse but there may be a simpler way...
regards, lacsaP.
when I looked a little closer at what adb backup was doing, I realized that it was communicating with an adb server running in parallel and, by mimicking the conversation with python, I managed to get what I wanted. *(note that a small error was present in the question : the string to use with printf is "\x1f\x8b\x08\x00\x00\x00\x00\x00").*
I still have a slight problem because the tar command ends with an error : *(the same error is also present with adb, printf and tail).*
$ adb devices List of devices attached CB5A2B0M9R device $ python3 adbackup.py > /tmp/mybackup.tgz $ tar tvzf /tmp/mybackup.tgz ... all my files are here :-) gzip: stdin: unexpected end of file tar: Child returned status 1 tar: Error is not recoverable: exiting now
adbackup.py :
#!/usr/bin/python3
# strace adb backup -all -f /tmp/backup.ab # ... # unlink("/tmp/backup.ab") = 0 # creat("/tmp/backup.ab", 0640) = 3 # ... # socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 4 # bind(4, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 # connect(4, {sa_family=AF_INET, sin_port=htons(5037), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 # setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0 # write(4, "0012host:transport-any", 22) = 22 # read(4, "OKAY", 4) = 4 # write(4, "000ebackup: '-all'", 18) = 18 # read(4, "OKAY", 4) = 4 # fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x1), ...}) = 0 # write(1, "Now unlock your device and confi"..., 59) = 59 # read(4, "ANDROID BACKUP\n4\n1\nnone\n", 32768) = 24 # write(3, "ANDROID BACKUP\n4\n1\nnone\n", 24) = 24 # read(4, "x\332", 32768) = 2 # write(3, "x\332", 2) = 2 # read(4, "\344VY\256\35E\f}\337o\25\254 \361\U\253A\256\301(\37$Q\22\366\317\251\233\274(\240"..., 32768) = 967
import socket, sys s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(("localhost", 5037)) s.sendall(b"0012host:transport-any") data = s.recv(4) # b"OKAY" s.sendall(b"000ebackup: '-all'") data = s.recv(4) # b"OKAY" data = s.recv(32768) # b"ANDROID BACKUP\n4\n1\nnone\n" sys.stdout.buffer.write(b"\x1f\x8b\x08\x00\x00\x00\x00\x00") data = s.recv(32768) while data: sys.stdout.buffer.write(data) data = s.recv(32768)
regards, lacsaP.
Le jeu. 16 mai 2019 à 10:57, Pascal patatetom@gmail.com a écrit :
hello,
I'm looking for a solution for the problem below : all leads are welcome !
the adb tool is able to make a backup of a smartphone in a file: adb backup -all -f mybackup.ab
however, the file mybackup.ab has a particular format that prevents it from being used "simply". to convert it into a tgz archive, it is possible to do this: ( printf "\x1f\x8b\x08\x08\x00\x00\x00\x00\x00\x00\x00"; tail -c +25 mybackup.ab ) > mybackup.tgz
but in this case, I can easily end up with two files of 16Gb or more... to avoid this volume problem, I tried to do this: mkfifo backup.fifo adb backup -all -f backup.fifo & ( printf "\x1f\x8b\x08\x08\x00\x00\x00\x00\x00\x00" ; tail -c +25 backup.fifo) > mybackup.tgz
but adb systematically breaks my named pipe and my code doesn't work as expected. how to prevent adb from breaking the named pipe ? I'm considering using fuse but there may be a simpler way...
regards, lacsaP.
does anyone have any idea what should be added/removed at the end of the data to properly complete the tgz archive ?
regards, lacsaP.
Le jeu. 16 mai 2019 à 16:00, Pascal patatetom@gmail.com a écrit :
when I looked a little closer at what adb backup was doing, I realized that it was communicating with an adb server running in parallel and, by mimicking the conversation with python, I managed to get what I wanted. *(note that a small error was present in the question : the string to use with printf is "\x1f\x8b\x08\x00\x00\x00\x00\x00").*
I still have a slight problem because the tar command ends with an error : *(the same error is also present with adb, printf and tail).*
$ adb devices List of devices attached CB5A2B0M9R device $ python3 adbackup.py > /tmp/mybackup.tgz $ tar tvzf /tmp/mybackup.tgz ... all my files are here :-) gzip: stdin: unexpected end of file tar: Child returned status 1 tar: Error is not recoverable: exiting now
adbackup.py :
#!/usr/bin/python3
# strace adb backup -all -f /tmp/backup.ab # ... # unlink("/tmp/backup.ab") = 0 # creat("/tmp/backup.ab", 0640) = 3 # ... # socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 4 # bind(4, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 # connect(4, {sa_family=AF_INET, sin_port=htons(5037), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 # setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0 # write(4, "0012host:transport-any", 22) = 22 # read(4, "OKAY", 4) = 4 # write(4, "000ebackup: '-all'", 18) = 18 # read(4, "OKAY", 4) = 4 # fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x1), ...}) = 0 # write(1, "Now unlock your device and confi"..., 59) = 59 # read(4, "ANDROID BACKUP\n4\n1\nnone\n", 32768) = 24 # write(3, "ANDROID BACKUP\n4\n1\nnone\n", 24) = 24 # read(4, "x\332", 32768) = 2 # write(3, "x\332", 2) = 2 # read(4, "\344VY\256\35E\f}\337o\25\254 \361\U\253A\256\301(\37$Q\22\366\317\251\233\274(\240"..., 32768) = 967
import socket, sys s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(("localhost", 5037)) s.sendall(b"0012host:transport-any") data = s.recv(4) # b"OKAY" s.sendall(b"000ebackup: '-all'") data = s.recv(4) # b"OKAY" data = s.recv(32768) # b"ANDROID BACKUP\n4\n1\nnone\n" sys.stdout.buffer.write(b"\x1f\x8b\x08\x00\x00\x00\x00\x00") data = s.recv(32768) while data: sys.stdout.buffer.write(data) data = s.recv(32768)
regards, lacsaP.
Le jeu. 16 mai 2019 à 10:57, Pascal patatetom@gmail.com a écrit :
hello,
I'm looking for a solution for the problem below : all leads are welcome !
the adb tool is able to make a backup of a smartphone in a file: adb backup -all -f mybackup.ab
however, the file mybackup.ab has a particular format that prevents it from being used "simply". to convert it into a tgz archive, it is possible to do this: ( printf "\x1f\x8b\x08\x08\x00\x00\x00\x00\x00\x00\x00"; tail -c +25 mybackup.ab ) > mybackup.tgz
but in this case, I can easily end up with two files of 16Gb or more... to avoid this volume problem, I tried to do this: mkfifo backup.fifo adb backup -all -f backup.fifo & ( printf "\x1f\x8b\x08\x08\x00\x00\x00\x00\x00\x00" ; tail -c +25 backup.fifo) > mybackup.tgz
but adb systematically breaks my named pipe and my code doesn't work as expected. how to prevent adb from breaking the named pipe ? I'm considering using fuse but there may be a simpler way...
regards, lacsaP.
Hi Pascal,
I still have a slight problem because the tar command ends with an error : *(the same error is also present with adb, printf and tail).*
..
$ python3 adbackup.py > /tmp/mybackup.tgz $ tar tvzf /tmp/mybackup.tgz ... all my files are here :-) gzip: stdin: unexpected end of file tar: Child returned status 1 tar: Error is not recoverable: exiting now
Why not keep the non-tar file long term and convert it to gzip'd tar format when required. That way you'll be able to improve your conversion over time, and not break if adb changes its format. That breakage could go unnoticed until it's too late to re-backup the data.
(printf ... && cat foo.ab) | tar ...
Indeed, but it has to face two constraints, the second being directly related to the first : limited storage space and access to the data backed up through archivemount which does not support the android backup format.
Le ven. 17 mai 2019 à 10:32, Ralph Corderoy ralph@inputplus.co.uk a écrit :
Hi Pascal,
I still have a slight problem because the tar command ends with an error
:
*(the same error is also present with adb, printf and tail).*
..
$ python3 adbackup.py > /tmp/mybackup.tgz $ tar tvzf /tmp/mybackup.tgz ... all my files are here :-) gzip: stdin: unexpected end of file tar: Child returned status 1 tar: Error is not recoverable: exiting now
Why not keep the non-tar file long term and convert it to gzip'd tar format when required. That way you'll be able to improve your conversion over time, and not break if adb changes its format. That breakage could go unnoticed until it's too late to re-backup the data.
(printf ... && cat foo.ab) | tar ...-- Cheers, Ralph.
*(sorry, bad Ctrl-C Ctrl-V)* yes, but I have two constraints, the second being directly related to the first : limited storage space and access to data backed up through archivemount that does not support the android backup format.
Le ven. 17 mai 2019 à 11:49, Pascal patatetom@gmail.com a écrit :
Indeed, but it has to face two constraints, the second being directly related to the first : limited storage space and access to the data backed up through archivemount which does not support the android backup format.
Le ven. 17 mai 2019 à 10:32, Ralph Corderoy ralph@inputplus.co.uk a écrit :
Hi Pascal,
I still have a slight problem because the tar command ends with an
error :
*(the same error is also present with adb, printf and tail).*
..
$ python3 adbackup.py > /tmp/mybackup.tgz $ tar tvzf /tmp/mybackup.tgz ... all my files are here :-) gzip: stdin: unexpected end of file tar: Child returned status 1 tar: Error is not recoverable: exiting now
Why not keep the non-tar file long term and convert it to gzip'd tar format when required. That way you'll be able to improve your conversion over time, and not break if adb changes its format. That breakage could go unnoticed until it's too late to re-backup the data.
(printf ... && cat foo.ab) | tar ...-- Cheers, Ralph.
participants (2)
-
Pascal
-
Ralph Corderoy